Step 4: Test failover scenarios

This page describes the procedure for testing the failover of the Data Center Edition (DCE) deployed on Kubernetes. The following disaster scenarios are considered with our disaster recovery architecture example based on Azure resources:

• A regional failure of Azure Kubernetes Service.
• A regional failure of Azure Database for PostgreSQL flexible server.

Regional failure of AKS

1. Stop the Master AKS cluster and go to Azure portal's Home > Kubernetes services. 

2. Select the master cluster and select Stop.

3. Wait until the cluster's Power state changes to Stopped and the Cluster operation status changes to Succeeded.

4. Power on the Replica cluster and wait for the cluster's Power state to change to Started and Cluster operation status to Succeeded.

5. Once the cluster starts, perform a forced Elasticsearch reindexing.

6. Using the Azure CLI, make sure the correct cluster credentials are set to your Helm and kubectl commands context with the command below.

az aks get-credentials --resource-group <YOUR_RESOURCE_GROUP> --name <YOUR_REPLICA_CLUSTER_NAME>

7. Run the following command to reduce the replica count of the search nodes from 3 to 0. The monitoring passcode is required for the helm upgrade operation.

helm upgrade sonarqube-dce sonarqube/sonarqube-dce  --set searchNodes.replicaCount=0,monitoringPasscode="mypassword" -n sonarqube-dce

8. If you are using any PVC with SonarQube deployment, delete the PVC, which should delete any PV, assuming the reclaim policy is Delete. Otherwise, manually delete/remove any PV. Typically, there is one PVC for each search node. Repeat this step for all the PVCs associated with the search nodes.

kubectl delete pvc 

sonarqube-dce-sonarqube-dce-sonarqube-dce-sonarqube-dce-search-0 -n sonarqube-dce

9. Run the following commands to bring the replica count back to 3 for the search nodes.

export JWT_SECRET=$(echo -n "your_secret" | openssl dgst -sha256 -hmac "your_key" -binary | base64)

helm upgrade sonarqube-dce sonarqube/sonarqube-dce  --set searchNodes.replicaCount=3,monitoringPasscode="mypassword",applicationNodes.jwtSecret=$JWT_SECRET -n sonarqube-dce

10. Login to your SonarQube Server instance using the Azure FrontDoor endpoint to confirm the failover was successful.

Regional failure of Azure Database for PostgreSQL flexible server

1. On the Azure portal home page, go to Azure Database for PostgreSQL flexible servers.
2. Select the primary SonarQube database from the list.
3. On the database home page, go to Settings > High availability
4. Select Planned failover or Forced failover. For less downtime, select Planned failover. 

5. Once the failover is complete, open your SonarQube server instance and check the integrity of your data. 

Related pages

• Disaster recovery architecture example with Azure resources
• Step 1: Deploy the primary and replica databases
• Step 2: Set up the primary and replica clusters
• Step 3: Configure the Azure Front Door