SAS Technical Support has received reports about previously working SAS Stored Process Servers becoming unresponsive over time for unknown reasons. Unresponsive means that the SAS Stored Process Servers are running, but no requests from client applications are getting through to the server. These servers might also be referred to as hung or orphaned SAS processes.
You might have encountered this problem in one or more ways. You might be the end user working with one of the SAS® Business Intelligence client applications, such as SAS® Enterprise Guide®, SAS® Web Report Studio, SAS® Add-in for Microsoft Office, SAS® Information Delivery Portal, SAS® Stored Processes Web Application, or another web-based application. In this scenario, you click a button, expecting a report to be returned, but instead you receive a generic error or a Java dump. Or, you might be the systems administrator who gets a call from the end user. Then, you determine that there is no SAS Stored Process Server responding, or at least discover that the problem involves a SAS server rather than a client.
What do you do?
1. Initially. with all customers, troubleshooting starts as a "Put out the Fire" situation, in which Technical Support offers suggestions to confirm that the servers are down or unresponsive, and to clean up and recover from the problem immediately. This document provides tips to evaluate and restore your SAS Stored Process Servers.
2. After the "fire" is out, a long-term strategy is needed to gather information and determine why the problem occurs. See SAS Note 60306, "Tips for addressing unresponsive SAS® 9.4 Stored Process Servers, Part 2" for a suggested approach.
Use the following five-step approach to evaluate the status of and restore your SAS Stored Process Servers:
2. Check the Status of the Stored Process Server Ports by Running the NETSTAT System Command
3. Evaluate SAS Processes after Stopping the SAS Object Spawner
4. Review Applicable Log Files
5. Terminate Orphaned SAS Stored Process Servers and Restart the SAS Object Spawner
Step 1: Test the Connection to the SAS Stored Process Server from the SAS® Management Console
A basic test of stored process server functionality is available in the SAS® Management Console. To conduct this test, complete the following steps:
A pop-up message appears with the message Test Connection Successful! if the test was successful.
You usually receive the following error in SAS Management Console when your stored process servers are unresponsive:
The command-line tool NETSTAT (network statistics) displays incoming and outgoing network connections, routing tables, and various network interface statistics. This tool is available on UNIX and Windows operating systems. Use this tool to provide information about the status of the ports on which a particular stored process server runs, which are ports 8611, 8621, and 8631 by default.
Complete the following steps:
See RFC_793 (pages 20 and 21) for details about the progression of states for a TCP/IP connection.
When port 8611 is unresponsive, you see NETSTAT output similar to the following:
By default, the SAS Stored Process Server is configured to execute using the sassrv user account.
If executing the NETSTAT command reveals unresponsive servers, you should stop the object spawner and search for remaining SAS processes that are owned by the sassrv user account. Any SAS processes owned by the sassrv account that persist after the SAS Object Spawner shuts down are likely to be unresponsive SAS Stored Process Servers (although other possible explanations exist).
Complete the following steps:
When stored process servers become unresponsive, error and warning messages might appear in log files for the object spawner, stored process servers, metadata servers, and (for Windows systems only) the Windows Event Viewer.
You should review all the log file types noted below.
SAS Object Spawner Logs
Go to the appropriate folder and view the object spawner log file.
Stored Process Server Logs
Go to the appropriate folder and view the stored process server log file.
Example Windows path: SAS-configuration-directory\Lev1\SASApp\StoredProcessServer\Logs
Windows filename: SASApp_STPServer_yyyy-mm-dd_pid.log
Example UNIX path: SAS-configuration-directory/Lev1/Logs
UNIX filename: SASApp_STPServer_yyyy-mm-dd_pid.log
It is possible that the log from an unresponsive stored process server will contain no errors. In this case, you should note the last step or program that executed successfully and the last entry that was written in the log.
Often the last entry in an unresponsive stored process server log file notes that a request has started executing, similar to the example below:
INFO [00000417] 4:sasdemo - STP: 1: Executing c:\mycode stp_report.sas
If servers are unresponsive and you restart the object spawner, log entries similar to the following will appear in the stored process server log file when the stored process server attempts to restart:
ERROR [00000007] :sassrv - The TCP/IP tcpSockBind support routine failed with error 10048
(The specified address is already in use.).
ERROR [00000007] :sassrv - Bridge Protocol Engine Socket Access Method was unable to bind
the listen socket to port 8611.
ERROR [00000007] :sassrv - The Bridge Protocol Engine Socket Access Method listen thread
failed during the definition of the server listen.
INFO [00000007] :sassrv - Bridge protocol engine is unloading.
Note: These errors occur because the orphaned stored process server is still bound to the designated port for the server that is trying to start. If the stored process server log contains the statements above, you must terminate the existing stored process servers before you restart the object spawner.
For instructions, see the Step 5: Terminate Orphaned Stored Process Servers and Restart the Object Spawner section.
Metadata Server Log
Go to the appropriate folder and view the metadata server log file.
Example Windows path: SAS-configuration-directory\Lev1\SASMeta\MetadataServer\Logs
Windows filename: SASMeta_MetadataServer_<yyyy-mm-dd_pid>.log
Example UNIX path: SAS-configuration-directory/Lev1/Logs
UNIX filename: SASMeta_MetadataServer_yyyy-mm-dd_pid.log
Note: In most cases, no errors are logged by the metadata server when stored process servers become unresponsive. However, you should check this log as a precautionary measure.
Windows Event Viewer (Windows operating system only)
The Windows Event Viewer is a Windows System Tool found in the Microsoft Management Console. Complete these tasks to check the Windows Event Viewer for errors:
Step 5: Terminate Orphaned SAS Stored Process Servers and Restart the SAS Object Spawner
Use Task Manager or the kill command to terminate the unresponsive SAS Stored Process Servers.
Use the kill command. See the example below:
Start the Object Spawner service through the Windows Services manager.
Locate your ObjectSpawner.sh file in your SAS configuration directory. Here is an example:
From a system prompt, change directories to where the ObjectSpawner.sh script is located and submit the following:
4. Using SAS Management Console, validate the SAS Stored Process Server to ensure that the servers were successfully restored. See Step 1 for the procedure.
5. Clean up leftover WORK library files. These files accumulate as a result of abnormal termination of the stored process server sessions.
For SAS® 9.4M0 (TS1M0) and SAS® 9.4M1 (TS1M1) releases, use the cleanwork command.
For SAS® 9.4M2 (TS1M20 and later releases, another option is to use the Cleanwork Utility.
Use the cleanwork command.
If the above steps do not restore the SAS Stored Process Servers to normal functionality, compile results from the tests and the log files referenced in these steps and contact SAS Technical Support for further assistance.
If you are experiencing unresponsive stored process servers using SAS® 9.3, refer to the following note:
If you are experiencing unresponsive stored process servers using SAS® 9.2, refer to the following notes:
If you are experiencing unresponsive stored process servers using SAS® 9.1.3, refer to the following notes: