Configure and operate a cluster

High availability and cluster scalability are features of the Data Center Edition.

Once the SonarQube cluster is installed, you have a high availability configuration that allows your SonarQube instance to stay up and running even if there is a crash or failure in one of the cluster's nodes. Your SonarQube cluster is also scalable, and you can add application nodes to increase your computing capabilities.

Start, stop, or upgrade the cluster

Start the Cluster

To start a cluster, you need to follow these steps in order:

1. Start the search nodes
2. Start the application nodes

Stop the Cluster

To stop a cluster, you need to follow these steps in order:

1. Stop the application nodes
2. Stop the search nodes

Upgrade SonarQube

1. Stop the cluster.
2. Upgrade SonarQube on all nodes (application part, plugins, JDBC driver if required) following the usual upgrade procedure but without triggering the setup phase.
3. Once all nodes have the same binaries: restart the cluster.
4. At this point, only one of the application nodes is up. Try to access node_ip:port/setup on each application node, and trigger the setup operation on the one that responds.

Start or stop a node

You can start or stop a single node in the same way as starting and stopping an instance using a single server. By default, it's a graceful shutdown where no new analysis report processing can start, but the tasks in progress are allowed to finish.

Install or upgrade a plugin

1. Stop the application nodes.
2. Install or upgrade the plugin on the application nodes.
   • If upgrading, remove the old version.
   • You don't need to install plugins on search nodes.
3. Restart the application nodes.

Scalability

You have the option of adding application nodes (up to 10 total application nodes) to your cluster to increase computing capabilities.

Scaling in a Traditional Environment

Adding an Application Node

To add an application node: 

1. Configure your new application node in sonar.properties. The following is an example of the configuration to be added to sonar.properties for a sixth application node (server6, ip6) in a cluster with the default five servers:
   
   Server6:

...
sonar.cluster.enabled=true
sonar.cluster.node.type=application
sonar.cluster.node.host=ip6 
sonar.cluster.node.port=9003
sonar.cluster.hosts=ip1,ip2,ip6
sonar.cluster.search.hosts=ip3:9001,ip4:9001,ip5:9001
sonar.auth.jwtBase64Hs256Secret=YOURGENERATEDSECRET
...

2. Update the configuration of the preexisting nodes to include your new node. While you don't need to restart the cluster after adding a node, you should ensure the configuration is up to date on all of your nodes to avoid issues when you eventually do need to restart.

Removing an Application Node

When you remove an application node, make sure to update the configuration of the remaining nodes. Much like adding a node, while you don't need to restart the cluster after removing a node, you should ensure the configuration is up to date on all of your nodes to avoid issues when you eventually do need to restart.

Scaling in a Docker Environment

Adding Application Nodes

If you're using docker-compose, you can scale the application nodes using the following command:

docker-compose up -d --scale sonarqube=3

Removing Application Nodes

You can reduce the number of application nodes with the same command used to add application nodes by lowering the number.

Monitoring

CPU and RAM usage on each node have to be monitored separately with an APM.

In addition, we provide a Web API api/system/health you can use to validate that all of the nodes in your cluster are operational.

• GREEN: SonarQube is fully operational
• YELLOW: SonarQube is usable, but it needs attention in order to be fully operational
• RED: SonarQube is not operational

To call it from a monitoring system without having to give admin credentials, it is possible to setup a system passcode. You can configure this through the sonar.web.systemPasscode property in <SONARQUBE_HOME>/conf/sonar.properties if you're using a traditional environment or through the corresponding environment variable if you're using a Docker environment.

Cluster Status

On the System Info page at Administration > System, you can check whether your cluster is running safely (green) or has some nodes with problems (orange or red).

Maximum Pending Time for Tasks

On the global Background Tasks page at Administration > Projects > Background Tasks, you can see the number of pending tasks as well as the maximum pending time for the tasks in the queue. This shows the pending time of the oldest background task waiting to be processed. You can use this to evaluate if it might be worth configuring additional Compute Engine workers (Enterprise Edition) or additional nodes (Data Center Edition) to improve SonarQube performance.

Compute engine workers

If you change the number of compute engine workers in the UI, you must restart each application node to have the change take effect.

Project move

When the Project move feature is used in a DC installation:

• Projects are exported on only one of the application nodes
• The archive of the exported projects must be copied to all the applications nodes in the target server

Configuration details

There are three TCP networks to configure:

• the network of application nodes that relies on Hazelcast.
• the network used for Elasticsearch internal communication between search nodes (es properties).
• the network between application nodes and search nodes (search properties).

Hazelcast is used to manage the communication between the cluster's application nodes. You don't need to install it yourself, it's provided out of the box.

Docker environment configuration

In a Docker environment, your properties are configured using environment variables.

Traditional environment configuration

The following properties may be defined in the <SONARQUBE_HOME>/conf/sonar.properties file of each node in a cluster. When defining a property that contains a list of hosts (*.hosts) the port is not required if the default port was not overridden in the configuration.

All nodes

Application nodes

Search nodes

Elasticsearch authentication

For Elasticsearch authentication, the following properties need to be configured on specific nodes:

Application nodes

Search nodes

When you're using the SonarSource Docker images, the truststore/keystore should be provided as volumes. On Kubernetes, you need to create a new Secret from the truststore/keystore and provide the name to the Helm chart.

Secure your network

To further lock down the communication in between the nodes in your SonarQube Cluster, you can define the following network rules:

You can further segment your network configuration if you specify a frontend, a backend and a search network.

Limitations

• Cluster downtime is required for SonarQube upgrades or plugin installations.
• All application nodes must be stopped when installing, uninstalling, or upgrading a plugin.
• Plugins are not shared, meaning if you install/uninstall/upgrade a given plugin on one application node, you need to perform the same actions on the other application node.
• There is no way to perform actions on the cluster from a central app - all operations must be done manually on each node of the cluster.

Frequently asked questions

Does Elasticsearch discover automatically other ES nodes?

No. Multicast is disabled. All hosts (IP+port) must be listed.

How do I trigger a full Elasticsearch reindex?

You can trigger a full Elasticsearch reindex. During the reindex, SonarQube will detect out-of-sync indices and correct them.

1. Stop the cluster as follows: stop first the application nodes and then the search nodes.
2. On each search node, remove the contents of the <sonarqubeHome>/data/es8 directory where <sonarqubeHome>:
   • For a ZIP installation: is the location where the SonarQube distribution has been unzipped.
   • For a Docker installation: is the installation directory of SonarQube within your container. This path is stored in the SONARQUBE_HOME environment variable.  
3. Start your cluster as follows: start first the search nodes and then the application nodes.

Can different nodes run on the same machine?

Yes, but it's best to have one machine for each node to be resilient to failures. To maintain an even higher level of availability, each of your three search nodes can be located in a separate availability zone within the same region.

Can the members of a cluster be discovered automatically?

No, all nodes must be configured in $SQ_HOME/conf/sonar.properties.

My keystore/truststore cannot be read by SonarQube

Make sure that the keystore/truststore in question was generated with an algorithm that is known to Java 11. See JDK-8267599 for reference