Step 4: Test failover scenarios

How to test the failover of the Data Center Edition (DCE) deployed on Kubernetes.

The disaster scenarios described below are based on a Disaster recovery architecture example with Azure resources.

Regional failure of AKS

1. Stop the Primary AKS cluster and go to Azure portal's Home > Kubernetes services.
2. Select the primary cluster and select Stop.
3. Wait until the cluster's Power state changes to Stopped and the Cluster operation status changes to Succeeded.

1. Power on the Replica cluster and wait for the cluster's Power state to change to Started and Cluster operation status to Succeeded.
2. Once the cluster starts, perform a forced Elasticsearch reindexing. The Reindexing page has a special article about Forcing an Elasticsearch reindex.
3. Using the Azure CLI, make sure the correct cluster credentials are set to your Helm and kubectl commands context with the command below.

az aks get-credentials --resource-group <YOUR_RESOURCE_GROUP> --name <YOUR_REPLICA_CLUSTER_NAME>

1. Run the following command to reduce the replica count of the search nodes from 3 to 0. The monitoring passcode is required for the helm upgrade operation.

helm upgrade sonarqube-dce sonarqube/sonarqube-dce  --set searchNodes.replicaCount=0,monitoringPasscode="mypassword" -n sonarqube-dce

1. If you are using any PVC with SonarQube deployment, delete the PVC, which should delete any PV, assuming the reclaim policy is Delete. Otherwise, manually delete/remove any PV. Typically, there is one PVC for each search node. Repeat this step for all the PVCs associated with the search nodes.

kubectl delete pvc 
sonarqube-dce-sonarqube-dce-sonarqube-dce-sonarqube-dce-search-0 -n sonarqube-dce

1. Run the following commands to bring the replica count back to 3 for the search nodes.

export JWT_SECRET=$(echo -n "your_secret" | openssl dgst -sha256 -hmac "your_key" -binary | base64)
helm upgrade sonarqube-dce sonarqube/sonarqube-dce  --set searchNodes.replicaCount=3,monitoringPasscode="mypassword",applicationNodes.jwtSecret=$JWT_SECRET -n sonarqube-dce

1. Login to your SonarQube Server instance using the Azure FrontDoor endpoint to confirm the failover was successful.

Regional failure of Azure Database for PostgreSQL flexible server

1. On the Azure portal home page, go to Azure Database for PostgreSQL flexible servers.
2. Select the primary SonarQube database from the list.
3. On the database home page, go to Settings > High availability.
4. Select Planned failover or Forced failover. For less downtime, select Planned failover.

1. Once the failover is complete, open your SonarQube server instance and check the integrity of your data.

Related pages

• Disaster recovery architecture example with Azure resources

• Step 1: Deploy the primary and replica databases

• Step 2: Set up the primary and replica clusters on AKS

• Step 3: Configure the Azure Front Door