Log in to the UI on one of the healthy master nodes. Select the System Resources tab, select Nodes.

Power off the old master node that is being removed. In the UI, make sure the status for this node changes to Inactive. Verify that the serial number matches that of node that needs to be removed.

Power on the new node and complete bootstrap. Use the same parameters that were used to set up the old node (including name and network information). Note down the serial number of this node.

In the UI, select the checkbox next to the inactive master node. Click on Actions and select Replace. When prompted, enter the new node’s serial number under New Serial Number and proceed.

In the UI, you should see the serial number get updated. The status will change to Active once the master has successfully joined the cluster.

Power down the two master nodes that have failed.

Power on and bootstrap the two new nodes with the same parameters that were used to bootstrap the old nodes.

Log in to the healthy master (cli) and run the acidiag recover save command.

[rescue-user@node1 ~]$ acidiag recover save
Warning: Cluster recovery can be a disruptive operation and should only
be performed as last resort option to recover cluster from disasters
where two master nodes have lost their state due to hardware faults. Proceed? (y/n): y

cluster snapshot '/tmp/cluster_snapshot.tar.gz' generated successfully.
Copy to other devices as '/tmp/cluster_snapshot.tar.gz' before performing restore.

Copy the .tar file that was generated in previous step to the two new nodes as /tmp/cluster_snapshot.tar.gz and perform the acidiag recover restore command on all the nodes. The nodes will reboot.

[rescue-user@node1 ~]$ acidiag recover restore
Warning: This command will restart this device to perform recovery.
Make sure, you have copied cluster snapshot to other devices
if you are recovering the cluster from this device. Proceed? (y/n): y
Connection to 172.20.6.119 closed.

Wait for all nodes to form the cluster and show their status as healthy. Use the acidiag health command to verify the total cluster health.

Log in to the GUI of one of the healthy master nodes. Select the System Resources < Nodes.

Power off the old worker node that is being removed. In the GUI, make sure the status for this node changes to Inactive. Verify that the serial number matches that of node needs to be replaced.

In the GUI, select the checkbox next to the worker node that needs to be removed. Click Actions < Delete. The entry for this node is removed from the Nodes page.

# acidiag touch clean

# acidiag reboot

Power on the new worker node and complete bootstrap. Use the same parameters that were used to set up the old worker node. Note down the serial number of this node.

In the GUI, a worker node entry should show up as Unregistered. Verify that the serial number matches that of the new node.

In the GUI, select the checkbox next to this worker node. Click on Actions and select Register. Verify the details in the next screen and select Save.

In the GUI, you should see the node’s status change to Discovering, and then it changes to Active.