This SAS KB article describes techniques for troubleshooting and resolving SAS Viya 3.5 service start-up problems on Linux. This information expands upon information from General Servers and Services: Operate (Linux) in the SAS® Viya® 3.5 Administration Guide. As noted in that documentation, the sas-viya-all-services script does not span multiple machines. In a multi-machine environment, always start the core services that are outlined here before running /etc/init.d/sas-viya-all-services.
There are two key services that must be healthy before other SAS Viya services are started. The services and the order in which they need to be started are as follows:
1. sas-viya-consul-default: This service is deployed on all machines in the SAS Viya deployment. Machines assigned to the consul host group in inventory.ini during SAS Viya deployment run Consul in server mode, while other machines run as Consul clients.
2. sas-viya-vault-default: This service is deployed on all machines assigned to the Consul host group in inventory.ini during SAS Viya deployment, so it is co-located with machines that act as Consul servers.
After those two services are started and healthy on all nodes on which they are deployed, the following services should be started. The start-up order for these services is not important, but they must be started after Consul and Vault and before the rest of the SAS Viya services.
sas-viya-sasdatasvrc-* (multiple services for SAS® Infrastructure Data Server/PostgreSQL)
sas-viya-rabbitmq-server-default
sas-viya-httpproxy-default (this starts and dynamically reloads crucial configuration for HTTPD on RHEL or apache2 on SUSE)
After all of these services are started and healthy, the remaining SAS Viya services can be started.
If there is a problem with any of the services noted above, you should stop all SAS Viya services, verify that all SAS Viya processes have been stopped, and then start and test each of the services mentioned above before proceeding to start the next service.
Steps for resolving start-up problems are shown below. If you experience an error with any of these steps, you should not proceed further until that problem is resolved. Review the logs noted below and contact SAS Technical Support if further assistance is needed.
The Ansible command examples below should all be executed in the sas_viya_playbook directory, so that Ansible can find the correct config and inventory information via settings in sas_viya_playbook/ansible.cfg and sas_viya_playbook/inventory.ini. If you are not familiar with Ansible, do not have access to the sas_viya_playbook directory, or encounter problems with Ansible commands, you can use the manual commands instead.
1. Stop all SAS Viya services by running the following command on each SAS Viya machine:
Or you can run the optional Viya-ARK viya-services-stop.yml playbook.
2. Verify that all SAS Viya processes are stopped by running the following command on each SAS Viya machine:
Or you can use Ansible:
If any SAS Viya processes are returned by this command, kill them.
3. Ensure the systemctl status for all services is stopped by running the following on all SAS Viya machines:
Or you can use Ansible:
This might not be necessary in all cases, but it ensures that the systemctl status for all SAS Viya services is in the stopped state and prevents problems that might occur when abnormal process termination results in a mismatch between the reported systemctl status and the true status of a service.
4. On all Viya machines in the deployment, start Consul:
Or you can use Ansible:
5. Verify that Consul is healthy by running the following on all SAS Viya machines:
Or you can use Ansible:
The response should be one of the IP addresses for the machines in the Consul host group from inventory.ini, such as: "10.104.85.157:8300"
You can also check the newest /opt/sas/viya/config/var/log/consul/default/sas-consul_*.log file for messages such as the following:
There are often WARN and ERR messages in the Consul log at start-up that can be ignored. The key messages to look for are that all nodes find each other and a leader is elected.
6. On all SAS Viya machines in the Consul host group in sas_viya_playbook/inventory.ini, start Vault:
If you do not have access to sas_viya_playbook/inventory.ini for reference, the command can be executed on all machines that have the file /etc/init.d/sas-viya-vault-default.
Or you can use Ansible:
7. Verify that Vault is healthy by running the following on all machines assigned to the consul host group in inventory.ini:
Or you can use Ansible:
Good output looks similar to the following:
You can also check the newest /opt/sas/viya/config/var/log/vault/default/sas-vault_<date>_<time>.log file (not sas-vault-audit_<date>_<time>.log). The log should end with output similar to this:
8. Start the template services for sasdatasvrc by running the following on all machines in the pgpoolc and sasdatasvrc host groups in inventory.ini:
If you do not have access to sas_viya_playbook/inventory.ini for reference, the command can be executed on all machines that have /etc/init.d/sas-viya-sasdatasvrc-*template* files.
Or you can use Ansible:
9. Start the remaining sasdatasvrc services by running the following on the machines assigned to the sasdatasvrc host group in inventory.ini:
Or you can use Ansible:
If you have a HA deployment of Postgres and therefore more nodes than just node0, you can start the other nodes by adjusting the command for node1, node2, and so on.
10. Verify that the sasdatasvrc services are up by running the following:
Or you can use Ansible:
The output should indicate a status of up.
If you have a HA deployment of Postgres and therefore more pgpool instances than just pgpool0, you can get the status of the other nodes by adjusting the command for pgpool1, pgpool2, and so on.
If there are any problems with starting sasdatasvrc or getting a status of "up", review the logs in /opt/sas/viya/config/var/log/sasdatasvrc/postgres.
11. On all SAS Viya machines in the RabbitMQ host group in inventory.ini, start RabbitMQ via this command:
If you do not have access to sas_viya_playbook/inventory.ini for reference, the command can be executed on all machines that have the file /etc/init.d/sas-viya-rabbitmq-server-default.
Or you can use Ansible:
12. Verify that RabbitMQ is healthy by running the following on all machines assigned to the RabbitMQ host group in inventory.ini:
Or you can use Ansible:
The command output should list all the expected nodes in the RabbitMQ cluster and no errors.
If there are any problems with starting RabbitMQ or getting good status output, review the logs in /opt/sas/viya/config/var/log/rabbitmq-server/default.
13. On all SAS Viya machines in the HTTPproxy host group in inventory.ini, start the HTTPproxy via this command
If you do not have access to sas_viya_playbook/inventory.ini for reference, the command can be executed on all machines that have the file /etc/init.d/sas-viya-httpproxy-default.
Or you can use Ansible:
14. Verify that the sas-viya-httpproxy-default service started the HTTPD or Apache2 service using curl on the host(s) assigned to the HTTPproxy group in inventory.ini:
Or you can use Ansible:
The output should be an HTTP 302 redirect similar to this:
In case of problems, review the output of "sudo systemctl status httpd" and logs in /var/log/httpd (for RHEL) or "sudo systemctl status apache2" and logs in /var/log/apache2 (for SUSE).
If all of these services start without problems, you can start the remaining services using "sudo /etc/init.d/sas-viya-all-services start" or the Viya-ARK viya-services-start.yml playbook. The /etc/init.d/sas-viya-all-services script starts only services on the local machine, so the script needs to be executed manually on each SAS Viya machine if you use this method in a multi-machine deployment.
The same strategy applies to SAS Viya 3.4, except there are different services for SAS Infrastructure Data Server (PostgreSQL) in that version. In SAS Viya 3.4, the sas-viya-sasdatasvrc-postgres service can be used to start and obtain status for the sasdatasvrc service.
Here is an example: