In certain cases, the number of journal logs that reside in the SAS-configuration-file\Lev1\Web\activemq\data\kahadb directory grows over time, causing disk space to fill up.
A file listing in this directory appears similar to the following:
| month | day | time | dbxxlog |
|---|
| Nov | 5 | 01:39 | db-11.log |
| Nov | 6 | 07:56 | db-12.log |
| Nov | 7 | 13:29 | db-13.log |
| Nov | 8 | 19:02 | db-14.log |
. . .more data lines. . .
This problem occurs because the broadcastTopic messages in ActiveMQ do not expire. As a result, the dead-letter queue (DLQ) fills up, leading to filled kahadb db_*.log files. This broadcast topic is used by the distributed in-process (DIP) scheduler, the DIP job runner, and the job execution services (JES) job runner.
To work around this issue, you have to purge the DLQ.
Purging the Dead-Letter Queue
This workaround cleans up the disk space by enabling ActiveMQ to clean up the kahadb log files that are referenced. The manual purge has no effect on newly created kahadb log files.
To purge the DLQ, complete these steps:
- Use the sas.servers script to stop all of the middle-tier servers. Caution: You must stop the middle-tier servers before you delete the kahadb/*.log files. Deleting the log files from an actively running instance of ActiveMQ can lead to an unrecoverable situation.
- Remove the contents of the SAS-configuration-directory/Levn/Web/activemq/data/ directory by submitting the following command:
rm –r SAS-configuration-directory/Levn/Web/activemq/data/*
Notes:
- Make sure that you remove only the contents of the SAS-configuration-directory/Levn/Web/activemq/data/ directory and not the directory itself.
- Using this workaround effectively cleans up the existing log files, but you might lose any Java Message Service (JMS) messages that are currently active. Unfortunately, those messages might be required by the target application. For example, the deletion of workflow-related messages can cause synchronization issues with the corresponding SAS® solution.
- If your SAS solution uses SAS® Workflow Services, contact SAS Technical Support to remediate any synchronization issues that you encounter after deleting the ActiveMQ data.
- It is possible to purge the DLQ in an active system by using the ActiveMQ administrator console. However, you can do this only if the ActiveMQ administrator console is configured and is running. (By default, the console is not configured to run.) If the console is configured and running, you can use it to purge the DLQ by releasing the active locks in the kahadb log files. Releasing the locks results in the removal of the log files.