The SAS® 9.4 JMS Broker services fail to start


When you start the JMS Broker services, the services fail and the errors that follow below are generated in the ActiveMQ logs. These errors occur when the KahaDB database (which ActiveMQ uses to persist messages) becomes corrupted.

2013-12-26 16:46:00,948 | INFO | Apache ActiveMQ 5.7.0 (localhost,

ID:usfkl21as213v-56651-1388094342845-0:1) started | org.apache.activemq.broker.BrokerService |
main 2013-12-27 09:43:59,060 | INFO | Apache ActiveMQ 5.7.0 (localhost,

ID:usfkl21as214v-45739-1382728234428-0:1) uptime 62 days 19 hours |

org.apache.activemq.broker.BrokerService | ActiveMQ ShutdownHook 2013-12-27 09:44:28,822 |
ERROR | Failed to start Apache ActiveMQ (localhost, null).

Reason: java.io.IOException: Detected missing/corrupt journal files. 27 messages affected. |

org.apache.activemq.broker.BrokerService | main java.io.IOException: Detected missing/corrupt
journal files. 27 messages affected.
    at org.apache.activemq.store.kahadb.MessageDatabase.recoverIndex
       (MessageDatabase.java:695)
    at org.apache.activemq.store.kahadb.MessageDatabase$6.execute
       (MessageDatabase.java:505)
    at org.apache.kahadb.page.Transaction.execute(Transaction.java:769)
    at org.apache.activemq.store.kahadb.MessageDatabase.recover(MessageDatabase.java:503)
    at org.apache.activemq.store.kahadb.MessageDatabase.open(MessageDatabase.java:366)
    at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:384)
    at org.apache.activemq.store.kahadb.MessageDatabase.doStart(MessageDatabase.java:239)
    at org.apache.activemq.store.kahadb.KahaDBStore.doStart(KahaDBStore.java:180)
    at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
    at org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.doStart

         (KahaDBPersistenceAdapter.java:195)
    at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
    at org.apache.activemq.broker.BrokerService.doStartPersistenceAdapter
       (BrokerService.java:623)
    at org.apache.activemq.broker.BrokerService.startPersistenceAdapter
       (BrokerService.java:612)
    at org.apache.activemq.broker.BrokerService.start(BrokerService.java:577)
    at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet
       (XBeanBrokerService.java:58)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke
        (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.

        invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1546)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.

        invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1487)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.

        initializeBean(AbstractAutowireCapableBeanFactory.java:1419)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.

        doCreateBean(AbstractAutowireCapableBeanFactory.java:518)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.

        createBean(AbstractAutowireCapableBeanFactory.java:455)
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject

        (AbstractBeanFactory.java:293)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.
        getSingleton(DefaultSingletonBeanRegistry.java:222)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean

        (AbstractBeanFactory.java:290)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean

        (AbstractBeanFactory.java:192)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.
         preInstantiateSingletons(DefaultListableBeanFactory.java:585)
     at org.springframework.context.support.AbstractApplicationContext.
        finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
    at org.springframework.context.support.AbstractApplicationContext.refresh

        (AbstractApplicationContext.java:425)
    at org.apache.xbean.spring.context.ResourceXmlApplicationContext.

        (ResourceXmlApplicationContext.java:64)
    at org.apache.xbean.spring.context.ResourceXmlApplicationContext.

        (ResourceXmlApplicationContext.java:52)
    at org.apache.activemq.xbean.XBeanBrokerFactory$1.
        (XBeanBrokerFactory.java:102)
    at org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext

       (XBeanBrokerFactory.java:102)
    at org.apache.activemq.xbean.XBeanBrokerFactory.createBroker
       (XBeanBrokerFactory.java:66)
    at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)
    at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54)
    at org.apache.activemq.console.command.StartCommand.startBroker
        (StartCommand.java:115)
    at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:74)
    at org.apache.activemq.console.command.AbstractCommand.execute
        (AbstractCommand.java:57)
    at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:148)
    at org.apache.activemq.console.command.AbstractCommand.execute
        (AbstractCommand.java:57)
    at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:90)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke
        (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.apache.activemq.console.Main.runTaskClass(Main.java:262)
    at org.apache.activemq.console.Main.main(Main.java:115) 2013-12-27 09:44:28,829 | INFO |
Apache ActiveMQ 5.7.0 (localhost, null) is shutting down |

org.apache.activemq.broker.BrokerService | main

To solve this problem, complete these steps:

    1. Stop the SAS 9.4 JMS Broker service on the middle-tier machine.
    2. Navigate to the SAS-configration-directory/Lev1/Web/activemq/data/kahadb folder. Rename the db.data file to old.db.data.
    3. Restart the SAS 9.4 JMS Broker service on the middle-tier machine. Wait a few minutes for the ActiveMQ KahaDB index to rebuild.
    4. Check the status of SAS Services to see whether JMS Broker services start successfully with other services. If the broker services start successfully, you should see the status as shown below:
bash-4.1$ ./sas.servers status SAS servers status: SAS JMS Broker is UP SAS Cache Locator Service ins_41415 is UP SAS Web App Server SASServer1_1 is UP SAS Web Server is UP SAS Environment Manager is UP