Troubleshooting Elasticsearch-related issues

Recovering from Elasticsearch read-only indices

You may encounter issues with Elasticsearch (ES) indices becoming locked in read-only mode. ES requires free disk space available and implements a safety mechanism to prevent the disk from being flooded with index data that:

• For non-DCE – locks all indices in read-only mode when the 95% used disk usage watermark is reached.
• For DCE – locks all or some indices in read-only mode when one or more node reaches the 95% used disk usage watermark.

ES shows warnings in the logs as soon as disk usage reaches 85% and 90%. At 95% usage and above, indices turning read-only causes errors in the web and compute engine.

Freeing disk space will not automatically make the indices return to read-write. To make indices read-write, you also need to:

• For non-DCE – restart SonarQube Server.
• For DCE – restart ALL application nodes (the first application node restarted after all have been stopped will make the indices read-write).

SonarQube Server's built-in resilience mechanism allows SonarQube Server to eventually recover from the indices being behind data in the DB (this process can take a while).

If you still have inconsistencies, you'll need to rebuild the indices (this operation can take a long time depending on the number of issues and components):

non-DCE:

1. Stop SonarQube Server.
2. Delete the data/es8 directory.
3. Restart SonarQube Server.

DCE:

1. Stop the whole cluster (ES and application nodes).
2. Delete the data/es8 directory on each ES node.
3. Restart the whole cluster.

Failed background tasks during reindexing

During Elasticsearch reindexing, you may have failed tasks in your branches or pull requests:

• If you only have a few failed tasks, you can reanalyze your branch or pull request. You may want to use web services to remove branches and pull requests that can't be reanalyzed because they have been removed from version control. 
• If you have many failed tasks, you may want to delete your Elasticsearch directory and reindex again. To do so, see Forcing an Elasticsearch reindex on the Reindexing page.

If background tasks of type Project Data Reload fail for a particular project, see Reindexing a single project on the Reindexing page.

Exception java.lang.RuntimeException: cannot run elasticsearch as root

SonarQube Server starts an Elasticsearch process, and the same account that is running SonarQube Server itself will be used for the Elasticsearch process. Since Elasticsearch cannot be run as root, that means SonarQube Server can't be either. You must choose some other, non-root account with which to run SonarQube Server, preferably an account dedicated to the purpose.

Related pages

• Checking the server logs
• Performance issues
• Database-related issues
• Other issues