When troubleshooting performance problems, be aware that SAS processes place different demands on file systems and I/O than traditional databases or simple query processes. For that reason, you need to be able to determine the throughput rates of any file system that SAS uses. This KB article documents a shell script that system administrators and IT teams can use to help make that determination.

The iotest.sh script uses UNIX/Linux dd commands to measure the I/O behavior of the system under defined loads. The script is easy to use and can be used to launch individual or multiple concurrent I/O tests to flood the file system and determine its raw performance.

Note: If you have installed SAS Software on Red Hat Enterprise Linux (RHEL), use the rhel_iotest.sh script documented in SAS KB0039548.

The iotest.sh script creates files and writes them to the file system being tested. It then reads them back, to test both Write and Read performance of the file system. The primary output from this tool is the megabytes written per second (Write MB/sec) and megabytes read per second (Read MB/sec). These metrics can help you to analyze and tune I/O for file systems that support SAS.

You can compare those results to specific recommendations given in any SAS Enterprise Excellence Center System sizing, or general per-file-system recommendations in SAS papers referenced below.

There are several papers available that will help you understand how SAS performs I/O, as well as minimum I/O recommendations for SAS file systems. These papers outline the recommended I/O metrics for file systems that support SAS deployments, and they identify guidelines that can help with provisioning and tuning I/O characteristics for optimal SAS performance. You can access these papers via SAS KB0036235, "Tuning guidelines and best practices for your hardware infrastructure (systems, storage, virtualization, file systems, and operating systems)."

• Best Practices for Configuring your IO Subsystem for SAS 9 Applications
• Frequently Asked Questions Regarding Storage Configurations
• How to Maintain Happy SAS 9 Users

Additionally, SAS KB0036235 contains papers related to set-up and tuning of specific vendor operating systems and storage arrays.

Before beginning I/O throughput tests, it is important to consider the following concepts:

• SAS performs all Write and Read operations via the host system file cache. File sizes used for the write and read tests should be larger than your host system file cache. (It is safe to make them greater than the RAM in your host system.) For example, if you run your test with a 1GB file size, and your system has 5GB of cache, the tests will exercise limited disk I/O on the read test. The file that is written will remain in the system file cache, and the subsequent read operation will read it from cache instead of from disk. If the read tests are satisfied via cache instead of coming from disks, the read performance metrics will be misleading, and you will not be able to generate consistent, repeatable results.
  
  
• An invocation of a single instance test of either of these tools (one write/read test) will show how a single SAS job might fare at that point in time on that file system. If your system is already very busy with other users' workloads, this may be adequate.
  
  
• A multiple instance invocation of the tests will better show what a concurrent SAS working load might experience, and is recommended for systems experiencing no other current jobs or a very light workload.
  
  
• It is wise to conduct several of these test sets at various times of the day, and over several busy days, to get a comprehensive profile of I/O characteristics on systems that run a variety of workloads.
  
  
• The timeframe in which you run your tests can make a difference in the results. It is preferable to place a targeted test during the same time of day and file system(s) as you are currently experiencing under-performing SAS jobs.
  
  
• Run the tests against the file system you want to target. Direct the results output to a file system that is different from the one under test by executing the iotest.sh program in a different file system, and/or directing the results to a different file system using the UNIX pipe command, if running in batch.
  
  
• The test output shows throughput results in MB/sec. Note that a minimum throughput rate of 100MB/sec per core should be provided for any SAS file system.
  
  
• If you are testing your file system throughput to solve an existing performance problem, refer to the following papers on performance problem resolution using SAS logs in combination with host monitors:
  • Solving SAS Performance Problems: Employing Host Based Tools
  • A Practical Approach to Solving Performance Problems with SAS
  • SAS Performance Monitoring - A Deeper Discussion

The iotest.sh script

On UNIX and Linux host platforms, you can use the UNIX dd command to perform I/O tests. SAS Technical Support has created a script named iotest.sh that executes dd commands. The script accepts command-line parameters and formats the output for concurrent Write and Read tests. The utility simulates SAS Read and Write activity, although it is a stand-alone program (which means you can run this program on systems where SAS software is not yet installed).

For each run, the program writes an output file that captures the elapsed real time and the I/O rate, expressed as megabytes per second (MB/sec). The script creates a test file whose size is a multiple of the number of blocks * block size (file system transfer size), and it allows you to select how many simultaneous iterations to run as expressed by the –i option. The script automatically launches these iterations and collects their individual results into a test average in a single final report.

Utility Deployment from the SAS Technical Support Sample Tools (SASTSST) Repository

To deploy the utility from the SASTSST repository, complete the following steps when you are logged on to your problematic server using the SAS installer user ID:

1. Click SASTSST_UNIX_installation.zip to download the SASTSST_UNIX_installation.sh utility.
2. Execute the utility: bash SASTSST_UNIX_installation.sh
3. Review and accept the license terms and conditions.
4. When prompted, provide an absolute path to a deployment directory (such as /usr/local/etc/SAS/) to define the location for your SAS Technical Support Sample Tools.
5. Select the Testing Throughput for your SAS 9 File Systems: UNIX and non-RHEL Linux platforms option. After the utility is deployed, you see the following content in your deployment directory:

      6. Make the iotest.sh file executable (chmod 0555 iotest.sh). The script is now ready to use.

Usage

Run the test script from a file system separate from the target file system that you are testing. Ensure that the test file(s) you create are individually larger than your host RAM. The shell script is parameter-driven, and an example of the syntax of the command follows:

# bash iotest.sh –i 5 –t /saswork –b 8404992 –s 64

Here is an explanation of the parameters:
-i 5 executes 5 concurrent Write, then Read, iterations of the test.
-t /saswork specifies writes to /saswork (the file system being tested for throughput).
-b (number of blocks) is set to 8404992, which when multiplied by the
-s (block size) of 64K will result in a 513 GB file size, which is larger than the 512 GB of RAM on the machine used in this example.

The command above launches 5 simultaneous Write tests first. Each Write test creates a unique 513 GB file, which will each be stored on the target file system /saswork. After all 5 Write tests complete, 5 simultaneous Read tests will be launched to read those files, then subsequently delete them when they are finished. The script then collects the Read and Write times, as well as the throughput times in MB/sec for each run, then averages the Writes together, and the Reads together to obtain the final statistics presented in the example results below.

The script produces output to the screen as well as to a single output file named iotest.sh.results.<n>, where <n> is the number of iterations that you put in the –i parameter of the run. The results file contains the Read and Write test results averaged across all of the iterations: in this example, across 5 iterations. Below is an example of the output/results from the example iotest.sh command:

iotest.sh-readtest.out.1:Real 7200.45
iotest.sh-readtest.out.2:Real 7190.58
iotest.sh-readtest.out.3:Real 7335.65
iotest.sh-readtest.out.4:Real 7538.80
iotest.sh-readtest.out.5:Real 7343.72
iotest.sh-writetest.out.1:Real 3788.11
iotest.sh-writetest.out.2:Real 3786.86
iotest.sh-writetest.out.3:Real 3889.87
iotest.sh-writetest.out.4:Real 4156.95
iotest.sh-writetest.out.5:Real 3896.14

RESULTS
-------
INVOCATION : iotest.sh -i 5 -t /saswork -b 8404992 -s 64

TARGET DETAILS
directory : /saswork
df : /dev/mapper/mpathdvp1 6142314488 2648206708 3182096108 46% /fusion/raid0
mount point : /dev/mapper/mpathdvp1 on /fusion/raid0 type ext4 (rw,noatime,nodiratime)
filesize : 550829555712 bytes or 525312.00 megabytes

STATISTICS
average read time in seconds : 7321.84
average read throughput rate : 71.74 megabytes per second
average write time in seconds : 3903.58
average write throughput rate : 134.57 megabytes per second

The output shows the launch of each of the Write test iterations. When the Writes have completed, the Read tests begin. The invocation of the command is shown for purposes of clarity. The target file system tested is shown, with results from the UNIX df command showing the file system's address and how full it is at the end of the Write tests.

The mount-point is reported, along with file system type and mount options. The size of the test file is computed and given in megabytes. Following this are the average read time and read throughput statistics for the average of all runs, and then the same for the write tests.

In the example above, you can see that the average Read throughput is about 72 MB per second per core.
The average Write throughput for these 5 simultaneous tests is about 135 MB per second per core.

What Should You Expect to See?

For nominally good SAS I/O performance, you should see job steps in your SAS logs with combined CPU + User CPU time within 10% of the Real Time. If this is not the case, there is generally some benefit to tuning or better provisioning I/O from your SAS file systems.

A good performance target for a typical system is for throughput from the file system to match at least 75 MB/sec per processor core for small to average systems, and upwards of 100 for larger, more heavily used systems. A minimum throughput rate of 100 MB/sec should be provided for any SAS file system.

Work with your Storage Administrator if you believe you are not getting the performance that you need. Share with them the various operating system, I/O, and storage array tuning papers that can be found in SAS KB0036235.