Leveraging the Elasticsearch Diagnostics Utility

The first thing we always examine when troubleshooting any issue is the log files and the same is true with an Elasticsearch cluster. However, Elasticsearch also provides an excellent robust set of Rest APIs as well. These APIs are extremely helpful in understanding details about the cluster that are not present in the logs, such as number of shards, size of indexes, unassigned shards and more.

Elasticsearch administration is not my specialty, but I have learned to depend on a number of the CAT APIs that I usually curl one-by-one when troubleshooting a cluster. However, would it not be great to automatically execute a fuller set of the APIs all at once when looking into an issue and save them to files? I think so, especially when referring back and forth from the results of the different APIs.

Well, I recently ran across the following article, https://www.elastic.co/blog/why-does-elastic-support-keep-asking-for-diagnostic-files on Elastics’s Support Diagnostic Utility. This tool does exactly that, it automates the execution of a complete list of APIs in order to help Elastic's Support troubleshoot and advise on cluster issues. The results are stored in json and text files and then nicely zipped up at the end of the run. You can read the article for more details about the tool, but I wanted to demonstrate how easy it is to setup and use and more specifically in Liferay’s PaaS environment.

 

Setup & Execution

1. In the Liferay Cloud Console, navigate to the shell of a Liferay service container.

2. From the Liferay Home directory( /opt/liferay), make a directory for the diagnostics tool and diagnostic files, e.g. mkdir es-diagnostics

3. Next, from the tool’s github repository, find and copy the link to the most recent release, https://github.com/elastic/support-diagnostics

4. Use the curl command to download the diagnostics zip file using the copied link.

   1. curl -LJO https://github.com/elastic/support-diagnostics/releases/download/v8.5.0/diagnostics-8.5.0-dist.zip

5. Unzip the zip file and change directory inside the resulting folder (e.g. diagnostics-8.5.0)

6. Lastly, execute the tool. There are several parameters that can be passed on the command line or it can be ran interactively. More details can be found in the project’s README, but it should be possible to execute the tool with the following basic parameters.

   1. ./diagnostics.sh --type api -h search

 

 

 

After execution a new date stamped archive will be created in the current directory, e.g. api-diagnostics-20240327-191750.zip. This archive will contain several json and text files with the results of the Elasticsearch APIs that were invoked.

Downloading the Results

To download results there basically two options. The first is to move the archive to the /opt/liferay/data directory, which is persisted and gets backed up by the Backup Service. This would allow the archive files to be downloaded as part of a backup. This might be helpful if you were running the tool multiple times, creating several diagnostic archives. However, the simplest is to move it to the tomcat webapps folder.

1. Create a directory in the Tomcat webapps directory to contain the diagnostic archives, e.g. mkdir tomcat/webapps/esdiag/

2. Move the archive, e.g. mv api-diagnostics-20240327-191750.zip /opt/liferay/tomcat/webapps/esdiag/

3. Lastly, open a browser tab and navigate to the Liferay site, adding a /esdiag/ARCHIVE_NAME on the end of the url. For example, https://webserver-lfrtest-dev.lfr.cloud/esdiag/api-diagnostics-20240327-191750.zip

After the file is downloaded, the diagnostic results can be viewed more easily in an IDE or tool of your choice or even shared with other team mates.

 

 

One final note, this installs the diagnostic tool in the Liferay container’s ephemeral storage. Therefore, the installation steps would need to be repeated if the container is deleted, which is in a way self cleanup.

[doria avallone]

The CAT APIs, short for Compact and Aligned Text APIs, are designed to provide a concise and human-readable overview of various cluster, node, and index metrics. Unlike raw JSON responses, the CAT APIs present data in a column-aligned format, making it easy to read and ideal for quick checks and troubleshooting. These APIs are intended for human consumption using tools like the Kibana console or the command line, rather than for application use.