When you use SAS^® compliance solutions (SAS^®Anti-Money Laundering, SAS^® Enterprise Case Management, or both), you might encounter a performance issue where a specific task or tasks take longer to complete than expected. The following sections in this note serve as a starting point for diagnosing and resolving such performance issues in SAS 9.3 and 9.4.

• Initial Diagnostics
• Additional Diagnostics

Notes:

• Not every task that fails to perform as expected is a performance issue. Unreasonable expectations or failure to follow best practices might lead to scenarios that are incorrectly perceived to be performance issues.
• The information in this SAS KB article is intended to assist with the diagnosis of actual performance issues. Before you begin, consider whether using a best practice might be a more appropriate solution.

---

Initial Diagnostics

The following diagnostic steps are necessary to begin investigating a performance issue. In some cases, the results of these initial steps lead to immediate resolution by SAS Technical Support. In other cases, additional diagnostics and iterations might be required before resolution is reached.

1. Describe the behavior and environment.
   • Describe the observed behavior and how it differs from expected behavior.
   • Answer the following questions:
     • When was this behavior observed? When did it start, when did it end, and so on?
     • Who observed the problem? (for example, one user, some users, or all users?)
     • Can you replicate the behavior? If so, provide detailed steps for doing so.
     • What is the system architecture? How many metadata, middle-tier, and application servers do you have, and what is the underlying database.
     • Does the problem occur when you test other SAS^® web applications (for example, like SAS^® Web Administration Console, SAS^® Studio, or SAS^® Stored Processes)?
     • Which hot fixes have been applied? See: SAS KB0036131, "Using the ViewRegistry Report and other methods to determine the SAS^® 9.2 and later software releases and hot fixes that are installed."
     • Has anything on the system changed? If so, what?
2. Replicate the issue and provide the following logs and configuration files.

   It is helpful if you activate the debug and trace level logging and re-create the issue prior to log collection. If this is not possible, then provide the logs listed below. You can use the Problem Determination Analysis tool (PDATool) to collect all logs from your environment or you can manually gather and provide the following files logs from all servers for the time period during which the performance issue occurred:

---

Additional Diagnostics

• Collect log files for the following:
  • • metadata servers: (If the metadata tier is clustered, provide logs for all servers in the cluster.): MetadataServer and MetadataServer_console logs from SAS-configuration-directory/Lev1/SASMeta/MetadataServer/Logs
  
  
  • middle-tier servers (If the middle tier is clustered, provide logs for all servers in the cluster.):
    • all .log files from SAS-configuration-directory/Lev1/Web/WebServer/logs
    • content.xml and server.xml files from SAS-configuration-directory/Lev1/Web/WebAppServer/SASServer1_x/conf
    • content.xml and server.xml files from SAS-configuration-directory/Lev1/Web/WebAppServer/SASServer8_x/conf
    • the server.log and all other .log files, localhost_access_log..yyyy-mm-dd.txt, and catalina.out files from these directories:
      • SAS-configuration-directory/Lev1/Web/WebAppServer/SASServer1_x/logs
      • SAS-configuration-directory/Lev1/Web/WebAppServer/SASServer8_x/logs
    • all .log files from these directories:
      • SAS-configuration-directory/Lev1/Web/Logs/SASServer1_x
      • SAS-configuration-directory/Lev1/Web/Logs/SASServer8_x
  • SAS application-tier (compute-tier) servers (If the application tiers are clustered, provide logs for all servers in the cluster.):
    • SAS-configuration-directory/Lev1/ObjectSpawner/Logs/ObjectSpawner_yyyy-mm-dd_server_PID.log
    • SAS-configuration-directory[/Lev1/WebInfrastructurePlatformDataServer/Logs/webinfdsvrc-yyyy-mm-dd_PID.log
    • SAS-configuration-directory/Lev1/WebInfrastructurePlatformDataServer/Logs/webinfdsvrc_console.log
    • SAS-configuration-directory/Lev1//SASApp/StoredProcessServer/Logs/SASApp_STPServer_yyyy-mm-dd_server_PID.log (You should find multiple log files for any given date.)
  • any load balancers that you are using.
• Set trace and debug logging levels:
  • Thread and time precision

    When you research performance issues, it really helps with the diagnosis if the thread and time precision are added to the SASEntCaseManagement6.x.log file. Doing so enables you to see which threads take more time to run than anticipated and to tie together the various logs so you can drill down into the issue. For details about how to enable this logging, see SAS KB0036294, "How to add thread and time precision to enhance debugging for SAS^® Enterprise Case Management."

  • Enable Trace logging on the SAS^® Object Spawner and SAS^® Stored Process Servers.

    Note: You can use the SAS BI Color Coding and Reporting Tools in SAS^® Integration Technologies to analyze the workload and performance of the SAS Stored Process servers. To use these tools, you must enable debugging for the object spawner and the stored-process server, as explained in the previous two list items.

    • Enable debug logging for the stored-process server by following the instructions in SAS Note 34114, "Creating a detailed SAS^® Stored Process Server log by default."
    • Enable tracing for the SAS Object Spawner by following the instructions in SAS Note 34893, "Turning on additional logging for the SAS Object Spawner."

After the initial diagnostics are collected and analyzed, you might need to collect additional diagnostic information in subsequent iterations. Here is a list of some additional diagnostic techniques that SAS Technical Support might use while diagnosing and resolving performance issues:

• SAS KB0036262, "Tool to test I/O throughput on UNIX platforms"
• SAS KB0039569, "Testing Throughput for your SAS^® 9 File Systems: Microsoft^® Windows platforms"
• SAS Note 45561, "How to use the nmon command to monitor the performance of SAS^® on an AIX system"
• SAS KB0039726, "How to use the nmon command to monitor the performance of SAS^® on a Linux system"
• SAS KB0039801, "Using the gather_info.sh script to gather performance-related information about RHEL 6 and RHEL 7 environments"
• SAS Note 60306, "Tips for addressing unresponsive SAS^® 9.4 Stored Process Servers, Part 2"
• SAS KB0036430, "Using the web browser developer tools to investigate issues in SAS^® Web Applications"
• SAS KB0036301, "Enabling garbage-collection logging for SAS^® 9.4 Environments"