Orphaned PersistentVolumeClaims (PVCs) fill up disk space in a SAS® Viya® environment


This SAS KB article provides steps for identifying and deleting any orphaned PVCs that might linger and take up disk space in your SAS Viya environment. Too many orphaned PVCs can lead to a disk-full event and unexpected issues. This problem with PVCs can happen when, for example, you are using a Network File System (NFS) provisioner​​​​​​ and all your PVCs are sharing the same underlying storage, rather than a separate storage for each PVC.

Run the command below to quickly identify such PVCs. Replace Viya-namespace with the namespace where your SAS Viya environment is deployed:

kubectl -n Viya-namespace describe pvc | grep -E "^Name:.*$|^Namespace:.*$|^Used By:.*$" | grep -B 2 "<none>" | grep -E "^Name:.*$| Namespace:.*$"

If the command returns any orphaned PVCs, before deleting them, it is important that you back up your environment by following the instructions from Backup and Restore: Perform an Ad Hoc Backup. Make sure that you follow the correct version of the documentation that matches the cadence version of your SAS Viya software. To delete the PVCs, run the command below:

kubectl –n Viya-namespace delete pvc PVC-name

A common scenario that can create orphaned PVCs is when you have an internal instance of PostgreSQL and you update your SAS Viya software. When the SAS Viya software is updated, the PostgreSQL cluster is terminated and re-created, but the PVCs for the previous PostgreSQL nodes are not deleted. The steps to identify such internal PostgreSQL PVCs and delete them are documented in Unused Persistent Volume Claims (PVC) for PostgreSQL Nodes Are Not Deleted. Make sure that you follow the correct version of the documentation that matches the cadence version of your SAS Viya software. However, the commands in this SAS KB article should work for PostgreSQL PVCs as well.  

If your Consul and RabbitMQ pods fail to start after a disk-full event, refer to these SAS Notes for recovery steps: