PoolDiagnostics.jsp is a tool that is available with a hot fix for both SAS® Decision Services 5.5 M1 and 6.1. (See SAS Note 51957 "SAS Real-Time Decision Manager does not provide tuning information for pooled resources without a hot fix.") These releases are contained in SAS Real-Time Decision Manager 5.41 M1 and 5.6, respectively. This tool is also available without a hot fix in subsequent releases of SAS Decision Services (6.3 and 6.4) and SAS Real-Time Decision Manager (6.3 and 6.4). Note that due to security concerns, beginning with SAS Decision Services 6.4 M1 (SAS Real-Time Decision Manager 6.5), PoolDiagnostics.jsp is only available from SAS Professional Services.
Contents
Background
SAS Decision Services uses pools of resources such as threads and database connections. The size and persistence parameters for these pools need to be configured in order to provide the best throughput. PoolDiagnostics.jsp is a web page on the SAS Decision Services Engine web application that displays the status of the primary thread pool and the JDBC Connection pools for that engine. Based on the data that is displayed, you can adjust the pool configuration for the best performance. Information on how to adjust the pool configuration is available in the SAS Decision Services Administrators Guide. This information is also available in the most recent version of the SAS Real-Time Decision Manager Administrators Guide.
There are two types of pools that the PoolDiagnostics.jsp page monitors.
- Activity and Subflow Thread Pool: The SAS Decision Service Engine is a web application that executes decision flows (campaigns) that call out to other subflows and activities. Each initial incoming request (event) is executed in the application server's servlet thread pool. However, the calls to sub flows and activities are executed on threads borrowed from the primary thread pool. If there are not enough threads available in this pool, there will be contention for threads across requests. When this contention occurs, the overall system throughput will fall. If the top–level flows calls out to subflows and activities concurrently, then you might need to increase the size of this thread pool to accommodate the number of concurrent requests.
- JDBC Connection Pools: SAS Decision Services uses Apache DBCP connection pools to connect to resources that use JDBC. Currently, the resources that use JDBC are the SAS® Federation Server and the databases that provide data for the flows. SAS Federation Server is used by SAS® activities (SAS process nodes) and scoring nodes to execute DS2 code. Database data marts are used by the GeneralIO activity (data process nodes).
For the SAS Federation Server, every connection from the engine creates a service thread that loads and compiles any DS2 packages that are called for execution on that thread. Once a DS2 package is loaded on a thread, it remains in that thread's memory until the connection is discarded. Loading and compiling DS2 packages is expensive because of the time to create each thread and compile code, as well as the memory that the thread and its compiled DS2 packages occupies. Initializing the thread pool to be large enough to accommodate this demand will increase performance for flows that use the SAS Federation Server. However, making the thread pool too large uses resources unnecessarily and, in some environments, can actually decrease performance or use memory unnecessarily.
For database connections, the size of the JDBC connection pool should be sized for the capacity and throughput of the database server that it is referencing. If there is insufficient capacity on the database server to accommodate more connections, increasing the connection pool size merely moves any I/O bottleneck to the database server. Although the increase in available connections moves the bottleneck from the engine, it does not improve the throughput unless the database's availability for connections is also increased.
PoolDiagnostics.jsp
Usage
The PoolDiagnostics.jsp page can be displayed using any browser that has access to the SAS Decision Services Engine web application. To see the PoolDiagnostics.jsp page, enter the following URL:
http://ip-address-of-engine-server:port-of-engine-web-application-server/RTDM/PoolDiagnostics.jsp
The page automatically refreshes every 5 minutes. If you prefer a different refresh frequency, use an appropriate value for the query parameter refreshSeconds. The following URL refreshes the page every two seconds:
http://ip-address-of-engine-server:port-of-engine-web-application-server/RTDM/PoolDiagnostics.jsp?refreshSeconds=2
Items Displayed
Activity and Subflow Thread Pool
The following are displayed:
- Bar chart - represents the primary thread pool of the engine in question.
- Active threads - the number of active threads in the thread pool that are currently executing activities or sub flows. This number is represented by a dark green bar in the above chart. If the number of active threads is more than 90% of the Max pool size, the color is set to red.
- Pool size - the size of the pool that includes both active and idle threads. Idle threads are shown as a medium green bar not as dark as active and not as light as the largest pool size. If the size of the pool is more than 90% of the Max pool size, the color is set to yellow.
- Largest pool size - the largest value of the pool size that has been achieved since the most recent restart of the Engine. This value is represented as a light green bar in the graph. This is the high water mark that indicates whether the present pool size is enough to deal with transient load peaks. If the size of the pool is more than 90% of the Max pool size, the color is set to yellow.
- Core pool size - the configuration property that creates a set of threads in the pool when the engine is started. If this value is too low, the engine will spend time initially to create the threads needed to support the initial load and might not be able respond to initial spikes.
- Max pool size - the maximum configured size of the thread pool.
JDBC Connection Pools
For each JDBC connection pool, the following information is displayed:
- Resource name - name of the JDBC connection resource
- URL - all the server URL values for the named resource. SAS Decision Services supports multiple server URLs for a JDBC resource and distributes the load across them using a uniform random distribution. Every URL is associated with its own connection pool. For every URL, the following are displayed:
- Display bar - represents the status of the connection pool.
- Created - time and date when this pool was created. Pools might be invalidated and re-created if there are errors connecting to the server. The value that is displayed for Created can be used to detect these server connection problems.
- Active - the number of connections active in the connection pool. This number is represented by a dark green bar. If this value is more than 90% of the maximum size of the pool's Max active setting, then the color is set to red.
- High - the maximum number of connections that were in use since the pool was created. This number is represented by a medium green bar as the Pool Size in the Thread pool. If this value is greater than 90% of the pool's Max active setting, the color is set to yellow. The value indicates any spikes in load that have been encountered.
- Max active - the maximum number of connections that the pool is configured to hold.
- Idle - the number of connections that are idle in the pool. This value is represented as a light green bar. The existence of idle connections implies that the connection-specific objects have already been created in the database or SAS Federation Server and are ready for use.
- Max idle - the configured maximum number of idle connections that the pool can have before the connections are reaped and closed by the idle connection closing process.
- Min idle - the number of idle connections that the pool maintains and that the idle connection closing process will not close.
Example Screenshot