A CAS backup that is created with the SAS® Viya® Backup and Restore functionality fails with errors


When you use the SAS Viya Backup and Restore functionality to run a backup, the CAS backup might fail with either an Insufficient authorization error or a java.nio.file.AccessDeniedException error. These types of failures can occur when permissions and group membership requirements are not set correctly. As a result, the CAS backup action might fail or failures might occur in copying the CAS backup from the local-vault location to the shared vault (set by the sharedVault property).

Explanation of How the CAS Backup Works

When you create a backup in this way, the backup for each source type (CAS, PostgreSQL, RabbitMQ, and Consul) is first copied to the local vault (/opt/sas/viya/config/backup). Then, the backup is copied to your shared-vault location. A distinction between the backup of the other source types and the backup of the CAS permstore directory is who manages the backup processes. For source types other than the CAS type, the sas user manages the backup processes. For the CAS permstore directory, the cas user account manages the backup processes.

When the backup runs, first the sas user creates directories with 770 permissions up to this level: /opt/sas/viya/config/backup/backup-ID/__default__/. Here is an example of the permissions that are assigned to __default__ directory:

/opt/sas/viya/config/backup/backup-ID:
total 0
drwxrwx--- 6 sas sas 78 Apr 16 09:35 __default__

Next, the /opt/sas/viya/config/backup/backup-ID/__default__/ path is supplied to the CAS backup action. Then, as the cas user, the CAS backup action creates a directory named cas-shared-default and subdirectories. The user ownership is cas and the group ownership is the cas user's primary group. Here is an example of user and group permissions that are assigned to the cas-shared-default directory and to subdirectories:

/opt/sas/viya/config/backup/backup-ID/__default__:
total 0
drwxrwx--- 3 cas sas 49 Apr 16 09:35 cas-shared-default

/opt/sas/viya/config/backup/backup-ID/__default__/cas-shared-default:
total 4
drwxrwx--- 2 cas sas 4096 Apr 16 09:35 host-name

An Insufficient-Authorization Error Occurs in the CAS Backup 

Failures can occur when the CAS backup action creates /opt/sas/viya/config/backup/backup-ID/__default__/cas-shared-default and subdirectories. These failures occur when the cas user is not in the sas group. This issue occurs because /opt/sas/viya/config/backup/backup-ID/__default__ is owned by the sas group, with 770 permissions set.

The following example errors illustrate those that are displayed in the cas_yyyy-mm-dd_host-name_pid.log CAS logs, which reside in the /opt/sas/viya/config/var/log/cas/default directory. These CAS logs are on the CAS controller in a single-tenant environment.

2018-03-12T01:22:10,923 INFO  [00000007] cas local 88445 sas.deploymentBackup 6309 [tkcasaimp.c:4228] - ++ action accessControl.createBackup / path='/opt/sas/viya/config/backup/2018-03-12T01_22_00_410-0400/__default__/cas-shared-default';

2018-03-12T01:22:10,923 INFO  [00089276] sas.deploymentBackup local MAIN NoUser  [tkperm.c:27371] - A backup is being started.

2018-03-12T01:22:10,923 ERROR [00089276] sas.deploymentBackup local MAIN NoUser  [tkperm.c:27456] - Insufficient authorization to access %s.

2018-03-12T01:22:10,923 ERROR [00089276] sas.deploymentBackup local MAIN NoUser  [tkperm.c:27462] - Failed to create the backup directory (80BFE803): /opt/sas/viya/config/backup/2018-03-12T01_22_00_410-0400/__default__/cas-shared-default/.

2018-03-12T01:22:10,923 ERROR [00000007] cas local 88445 sas.deploymentBackup 6309 [tkperm.c:1336] - Failed to create the backup directory (887FC27C): /opt/sas/viya/config/backup/2018-03-12T01_22_00_410-0400/__default__/cas-shared-default.

2018-03-12T01:22:10,924 INFO  [00000007] cas local 88445 sas.deploymentBackup 6309 [tkcasaimp.c:5963] -  -- 'accessControl.createBackup' FAILED.  0x887fc27c

The following errors are examples of those that are displayed in the cas_yyyy-mm-dd_host-name_pid.log CAS logs that reside in /opt/sas/viya/config/var/log/cas/default. These CAS logs are on the CAS controller in a multi-tenant environment for the provider tenant.

2021-07-26T09:07:30,733 INFO  [00000007] 156663 sas.deploymentBackup 111 [tkcasaimp.c:4504] - ++ action accessControl.createBackup / path='/opt/sas/viya/config/backup/2021-07-26T09_07_17_487-0400/provider/cas-shared-default';

2021-07-26T09:07:30,734 INFO  [00001028] MAIN sas.deploymentBackup 111 [tkperm.c:29054] - A backup is being started.

2021-07-26T09:07:30,736 ERROR [00001028] MAIN sas.deploymentBackup 111 [tkperm.c:29169] - Insufficient authorization to access %s.

2021-07-26T09:07:30,736 ERROR [00001028] MAIN sas.deploymentBackup 111 [tkperm.c:29175] - Failed to create the backup directory (80BFE803): /opt/sas/viya/config/backup/2021-07-26T09_07_17_487-0400/provider/cas-shared-default/.

2021-07-26T09:07:30,737 ERROR [00000007] 156663 sas.deploymentBackup 111 [tkperm.c:1499] - Failed to create the backup directory (887FC27C): /opt/sas/viya/config/backup/2021-07-26T09_07_17_487-0400/provider/cas-shared-default.

2021-07-26T09:07:30,739 INFO  [00000007] 156663 sas.deploymentBackup 111 [tkcasaimp.c:6460] -  -- 'accessControl.createBackup' FAILED.  0x887fc27c

An Access-Denied Exception Occurs When CAS Backup Files Are Copied from the Local Vault to the Shared Vault​

In addition, when the primary group of the cas user is not the sas group, failures can occur when /opt/sas/viya/config/backup/backup-ID/__default__/cas-shared-default is copied to your shared-vault location (set by the sharedVault property). These failures occur because when CAS backup action creates the cas-shared-default directory, group ownership is determined by the primary group that you assigned to the cas user account. When the primary group of the cas user is not the sas group, the sas user does not have permission to copy cas-shared-default from your local vault to your shared vault. In multi-tenancy environments, this issue can occur also when a CAS controller runs with a user account that is not part of sas group. The following examples show various errors that can appear in sas-deploymentBackup logs when these issues occur. The logs that are named sas-deploymentBackup_yyyy-mm-dd_hh-mm-ss.log reside in /opt/sas/viya/config/var/log/deploymentBackup/default.

Example Error 1: 

Error calculating size of CAS backup
java.nio.file.AccessDeniedException: /opt/sas/viya/config/backup/2018-08-13T07_43_28_735-0400/__default__/cas-shared-default

Example Error 2:

The size of the CAS backup file could not be calculated. java.nio.file.AccessDeniedException:/opt/sas/viya/config/backup/2021-08-30T14_25_39_589-0400/__default__/cas-shared-default

Example Error 3:

The files could not be transferred because the specified location of the shared vault or local vault was invalid. java.nio.file.AccessDeniedException:/opt/sas/viya/config/backup/2021-08-30T14_28_17_763-0400/__default__/cas-shared-default

Example Error 4 (multi-tenant environment for a tenant named cas-tenant1-default):

The files could not be transferred because the specified location of the shared vault or local vault was invalid. java.nio.file.AccessDeniedException:/opt/sas/viya/config/backup/2020-10-30T09_08_04_042-0400/tn1/cas-tenant1-default

Resolving the Issues

To resolve these errors, on both the CAS controller (and backup CAS controller, if it exists) ensure that the cas user is in the sas group and ensure that the primary group for the cas user is the sas group. Alternatively, if a CAS controller is running with a user account that is not part of the sas group (such as with multi-tenancy environments), perform the steps to set appropriate permissions using access control lists (ACLs) on the local vault for a successful backup-and-restore process. The steps are available in the section ​​​​​Backup and Restore: Initial Tasks in SAS Viya 3.5 Administration.

Note: If you report a problem to SAS Technical Support, always provide the following logs that are from the same time period as the problem:

After you perform the steps, you might need to restart sas-viya-deploymentBackup-default, sas-viya-backup-agent-default (on CAS controller), and sas-viya-cascontroller-default microservices. Restart these services by submitting the commands below:

systemctl restart sas-viya-backup-agent-default
systemctl restart sas-viya-deploymentBackup-default
systemctl restart sas-viya-cascontroller-default