Three Python programs can help you determine an optimal Elasticsearch indexer configuration, and you can also check the indexer logging for information about health. The key and the cert file must be generated prior to use of the script. These commands show how to do that:

Notes:

• This SAS KB article applies to SAS^® Visual Investigator 10.6, SAS^® Visual Investigator 10.7, SAS^® Visual Investigator  and 10.8.
• Replace host-name:port-number with the correct values for your server. You can use "https://localhost:8200" if using default port number and running on the same server as the vault micro service. Otherwise, specify the host name and port number.
• Confirm the path and file names for all files used above, such as for cacert, key, cert, and vault-token.

This first Python program, elasticsearch-diagnostic.py, collects diagnostic data. Provide the host and port information for your environment (if a port number is not provided, it defaults to 9200):

The following command runs the program:

The --ephemeral option results in the information being displayed on your screen. If you do not use the --ephemeral option, all the collected information is put in a compressed ZIP file. Here are the optional parameters that you can use with the command:

-h, --help                show this help message and exit
--host HOST              the host name of Elasticsearch (default: localhost)
--port PORT             the port of Elasticsearch (default: 9200)
--key KEY                 the key file (default: /opt/sas/viya/config/var/tmp/sgadminkey.pem)
--cert CERT             the cert file (default: /opt/sas/viya/config/var/tmp/sgadmincert.pem)
--cacert CACERT       the cacert file (default: /opt/sas/viya/config/etc/SASSecurityCertificateFramework/cacerts/trustedcerts.pem)
--insecure            disable certificate validation; --cacert is not used when this option is used
--timeout TIMEOUT   the timeout for requests in seconds (default: 30)
--ephemeral           write to stdout instead of tarball
--verbose, -v         increase output verbosity

The second Python program, cat_indices_analysis.py, uses indices content that is created from the elasticsearch-diagnostic.py program. If you ran the first script to stdout (the screen), you can copy and paste the indices information into a text file and use that file as input to this script. 

However, if you ran the first script without using the --ephemeral parameter, you need to extract the indices output file from the compressed ZIP file. The extracted indices file is in the form cat_indices_yyyy-mm-*.txt. 

This script analyzes output from the Elasticsearch /_cat/indices endpoint and warns you if it detects potential problems. The script will give both suggestions for remediation and an explanation of each problem. The script looks for the following:

• Duplicate indices
• Indices that utilize a suboptimal number of primary shards (or indices that use too many shards)
• A large document_link index that could benefit from assigning custom/dedicated indices

The following example shows the command that runs the cat_indices_analysis.py program. Replace "path" with the correct information for your environment:

The third Python program, indexer_log_analysis.py, uses the indexer logging as input. The script analyzes Indexer logs and reports common problems. It will provide both an analysis of the problem and give suggested next steps to resolve (or help mitigate) the issue.

The script will look for the following:

• Tolerance exceeded DB reading warnings
• Garbage collection warnings​​
• Elasticsearch is down warnings
• Lost connections to the distributed cache

The following example shows the command that runs the program. Replace path with the correct information for your environment:

The source files for these programs are provided in the link below.

https://gitlab.sas.com/fsitactical/diagnostic-tools/-/tree/main?ref_type=heads