After upgrading SAS Viya, some RabbitMQ queues might become orphaned. An orphaned queue means that no application is consuming messages from the queue. The queue will not be deleted, causing RabbitMQ pods to consume more memory as messages are added to the queue.

The RabbitMQ pods might be OOMKilled by Kubernetes continuously, causing the RabbitMQ container to restart. This issue can impact the stability of the SAS Viya system, including errors when starting new jobs or compute sessions, along with general slowness when accessing the SAS Viya applications.

This issue has been reported in LTS 2024.03 and Stable 2024.05-Stable 2024.08.

To identify queues without consumers, run the following command:

kubectl -n <NS> exec sas-rabbitmq-server-0 -it -- bash -c 'source /rabbitmq/data/.bashrc;/opt/sas/viya/home/lib/rabbitmq-server/sbin/rabbitmqctl list_queues name messages_ready consumers | grep 0$'

In the output, look for queues with a large number of messages_ready but zero consumers that is similar to the example below:

sas.application.activities.DeleteConsumer 332871 0

The second column is the number of messages, and the third column is the number of consumers.

To remove these orphaned queues, complete the following steps:

1. Use the list of orphaned queues identified above.
2. Use the following command to delete the queues:

kubectl -n <NS> exec sas-rabbitmq-server-0 -it -- bash -c ' \
QUEUE_TO_DELETE=<queue name with non-zero messages and 0 consumers>; \
source /rabbitmq/data/.bashrc; \
/opt/sas/viya/home/lib/rabbitmq-server/sbin/rabbitmqctl delete_queue $QUEUE_TO_DELETE --if-unused'

A patch to prevent queues from becoming orphaned is available for this issue. Run the Update Checker Report and apply updates to the sas-arke or sas-rabbitmq containers.