Diagnostic steps for when you encounter the error, "No qualified launcher servers are available to complete request"


You might see the following error in an application or log:

Unable to create compute server session. No qualified launcher servers are available to complete request. 

The text of the message is accurate, and there are generally three reasons that it occurs:

To determine which reason you are receiving this error, complete the troubleshooting steps below:

  1. Check that the launcher server is up and running.
  1. Check the health of the launcher server by running the following:
     $ curl \
     > http://127.0.0.1:8500/v1/health/checks/launcher-server
For SAS® Viya® 3.4 and later:
$ curl \
     > https://127.0.0.1:8501/v1/health/checks/launcher-server
  1. If the launcher server is up, running, and in a healthy state, look at the Consul log, which is located in the /opt/sas/viya/config/var/log/consul/default directory. Search the log for any WARNS or ERRORS. You might see messages similar to the following:

 "[WARN] agent: Check "memory-check" is now warning". This error message indicates that there might be an issue with high memory 
     consumption. 

 "[WARN] agent: Check 'disk-check' is now warning". This WARN message indicates that there might be an issue with disk utilization. 
  1. Use the following command to check the health of the node. If the node is in an unhealthy state, all services will also be in an unhealthy state:
      $ curl \
      http://127.0.0.1:8500/v1/health/node/my-node
For SAS Viya 3.4 and later:
     $ curl \
     https://127.0.0.1:8501/v1/health/node/my-node
  1. It is possible that a property similar to the following was set in Consul to allow the use of an unhealthy launcher server: config/launcher/sas.launcher.server.locator.consul.only-passing=false.

    Here are some sas-bootstrap-config commands to check the value and change the value:

Note: You will have to source the client token first before running any of the below commands:

source /opt/sas/viya/config/consul.conf

export CONSUL_HTTP_TOKEN=$(sudo cat /opt/sas/viya/config/etc/SASSecurityCertificateFramework/tokens/consul/default/client.token)

# Read the value of a key that does not exist

[root@domain bin]# /opt/sas/viya/home/bin/sas-bootstrap-config kv read config/launcher/sas.launcher.server.locator.consul.only-passing

 

# Write the value of a key that does not exist

[root@domain bin]# /opt/sas/viya/home/bin/sas-bootstrap-config kv write --key config/launcher/sas.launcher.server.locator.consul.only-passing --value false

 

# Read the value of a key and returns the value as you see in italic below

[root@domain bin]# /opt/sas/viya/home/bin/sas-bootstrap-config kv read config/launcher/sas.launcher.server.locator.consul.only-passing

false

 

# Only update the value of a key if it does not exist. In this case, since the property does
# exist and the --force flag was not used, the key’s value will not be updated and you will see the
# error message in italic below

[root@domain bin]# /opt/sas/viya/home/bin/sas-bootstrap-config kv write --key config/launcher/sas.launcher.server.locator.consul.only-passing --value true

level=error msg="Key already exists. Write failed." key=config/launcher/sas.launcher.server.locator.consul.only-passing

 

# Forcibly overwrite the value of a key (for example, changing from “false” to “true”)

# Note that the --force flag must be used to overwrite the value of a key

[root@domain bin]# /opt/sas/viya/home/bin/sas-bootstrap-config kv write --force --key config/launcher/sas.launcher.server.locator.consul.only-passing --value true

 

# Re-read the value of the key noting that the value is now “true,” as you see in italic below

[root@domain bin]# /opt/sas/viya/home/bin/sas-bootstrap-config kv read config/launcher/sas.launcher.server.locator.consul.only-passing

true

 

# Delete the key

[root@domain bin]# /opt/sas/viya/home/bin/sas-bootstrap-config kv delete config/launcher/sas.launcher.server.locator.consul.only-passing

 

# Confirm that the key does not exist and return the message in italic below if it does not exist

[root@domain bin]# /opt/sas/viya/home/bin/sas-bootstrap-config kv exists config/launcher/sas.launcher.server.locator.consul.only-passing

The key config/launcher/sas.launcher.server.locator.consul.only-passing was not found.

[root@domain bin]# echo $?

64

 

# Re-read the key noting that it is no longer defined

[root@domain bin]# /opt/sas/viya/home/bin/sas-bootstrap-config kv read config/launcher/sas.launcher.server.locator.consul.only-passing

  1. If you view the sas-launcher log and you see a debug message similar to the following, you can safely ignore it:
  failed to qualify launcher server due to port mismatch: port: Port{port=36735} : required port:
  Port{port=32770} 

These messages are expected when a launcher server is restarted between the time that a process is launched and when that process reaches a terminal state. 

  1. Check for multiple launcher servers that did not successfully de-register:

Run 

curl https://127.0.0.1:8501/v1/health/checks/launcher-server --cacert /opt/sas/viya/config/etc/SASSecurityCertificateFramework/cacerts/trustedcerts.pem
If more than one server appears in the output similar to the following, then you need to de-register the critical launcher servers:

[root@STG-SAS-CS sas_viya_playbook]# curl https://127.0.0.1:8501/v1/health/checks/launcher-server --cacert /opt/sas/viya/config/etc/SASSecurityCertificateFramework/cacerts/trustedcerts.pem

[{"Node":"STG-SAS-CS","CheckID":"service:db2f077f-3ff5-41bd-8835-f092e2ca9b4d","Name":"Service 'launcher-server' check","Status":"critical","Notes":"","Output":"dial tcp 10.1.87.20:14555: getsockopt: connection refused","ServiceID":"db2f077f-3ff5-41bd-8835-f092e2ca9b4d","ServiceName":"launcher-server","ServiceTags":["tls","linux"],"Definition":{},"CreateIndex":3016120,"ModifyIndex":3016909},{"Node":"STG-SAS-CS","CheckID":"service:e8df8649-c9b1-4ef6-a6ca-9e4b6c0f1262","Name":"Service 'launcher-server' check","Status":"critical","Notes":"","Output":"dial tcp 10.1.87.20:29913: getsockopt: connection refused","ServiceID":"e8df8649-c9b1-4ef6-a6ca-9e4b6c0f1262","ServiceName":"launcher-server","ServiceTags":["tls","linux"],"Definition":{},"CreateIndex":3016120,"ModifyIndex":3016896},{"Node":"STG-SAS-CS","CheckID":"service:fd505d89-c1bb-4875-8702-3a3906f51174","Name":"Service 'launcher-server' check","Status":"passing","Notes":"","Output":"TCP connect STG-SAS-CS:22623: Success","ServiceID":"fd505d89-c1bb-4875-8702-3a3906f51174","ServiceName":"launcher-server","ServiceTags":["tls","linux"],"Definition":{},"CreateIndex":3016890,"ModifyIndex":3016892}][root@STG-SAS-CS sas_viya_playbook]#

 

De-register the critical servers from above:

source /opt/sas/viya/config/consul.conf

export CONSUL_HTTP_TOKEN=$(sudo cat /opt/sas/viya/config/etc/SASSecurityCertificateFramework/tokens/consul/default/client.token)

cd /opt/sas/viya/home/bin/

./sas-bootstrap-config agent check deregister --id service:db2f077f-3ff5-41bd-8835-f092e2ca9b4d

./sas-bootstrap-config agent check deregister --id service:e8df8649-c9b1-4ef6-a6ca-9e4b6c0f1262

./sas-bootstrap-config agent service list | grep -i launcher-server  (this is optional!!!)

./sas-bootstrap-config agent service deregister db2f077f-3ff5-41bd-8835-f092e2ca9b4d

./sas-bootstrap-config agent service deregister e8df8649-c9b1-4ef6-a6ca-9e4b6c0f1262

In a few cases, the underlying issue has been high-memory consumption or disk utilization. You can check the SAS® Environment Manager dashboard to see whether there is an issue with resources. Restarting the node might eliminate the error, in some cases. But the underlying issue will probably reappear if it occurs due to a lack of resources. 

See List Checks for Service for more information about health-checking options using the Hashicorp Consul HTTP API.