SvKMS Failure Scenarios and Expected Outcomes – StorMagic

This article is intended for administrators wishing to understand and test the expected outcomes for an SvKMS Cluster when there are different environmental failures.

Note: Scope of this document is to show the expected behaviour of a SvKMS Cluster in various failure scenarios.

Table of Contents

2-Node VIP Cluster configuration scenarios
1. ‘Lights on’ failure scenarios starting from a Healthy 2-node VIP Cluster
2. ‘Lights off’ failure scenarios starting from a Healthy 2-node VIP Cluster
Standard 2n+1 Cluster configuration scenarios
1. ‘Lights on’ failure scenarios starting from a Standard 2n+1 Healthy Cluster
2. ‘Lights off’ failure scenarios starting from a Standard 2n+1 Healthy Cluster

Resolution/Information

Note: Each scenario listed below outlines the procedure that can be taken to simulate the expected outcome of the given scenario.

2-Node VIP Cluster configuration scenarios:

This is example image set of a healthy VIP SvKMS Cluster for a reference point to the below scenarios.

SvKMS example image for healthy VIP Activator page.

SvKMS example image for healthy VIP Cluster Status page.

‘Lights on’ failure scenarios Starting from a Healthy 2-node VIP Cluster
Scenario	Description	Procedure	Expected Outcome
V01 - SvKMS Master Node offline	Due to maintenance or failure of the storage or host the SvKMS master node runs on.	Power off Master SvKMS VM.	BACKUP will be promoted to MASTER and will show display API (OK). Node that has failed will show Internal Server Error: EHOSTUNREACH. Cluster VIP available throughout and able to process KMIP requests and API calls etc. No impact or sign of Outage to vCenter or SvSAN VSAs. Example result output in images below
V02 - SvKMS Backup Node offline	Due to maintenance or failure of the storage or host the SvKMS backup node runs on.	Power off Backup SvKMS VM	Reset on the Cluster VIP login. Master will continue to display API (OK). Node that has failed will show Internal Server Error: EHOSTUNREACH. Cluster VIP available throughout and able to process KMIP requests and API calls etc. No impact or sign of Outage to vCenter or SvSAN VSAs. Example result output in images below
V03 - Both Nodes become network Isolated	Network communications between nodes is lost. Whereby both VM's remain up and running but cannot talk to each other (stretched Cluster for instance).	Remove network connectivity between the two nodes.	Reset on the Cluster VIP login. Each node will determine itself to be Master and will display API (OK). Remote node will display Internal Server Error: EHOSTUNREACH. Cluster VIP available throughout and able to process KMIP requests and API calls. No impact or sign of Outage to vCenter or SvSAN VSAs that can connect to one of the nodes. Encrypted VSA's or vCenter will continue to connect to local SvKMS node to operate. SvKMS's will synch up when network restored Example output will be same as shown for previous scenarios.

SvKMS Activator Page output for Scenario V01

SvKMS Cluster status output for Scenario V01

SvKMS Activator Page output for Scenario V02

SvKMS Cluster status output for Scenario V02

‘Lights off’ failure scenarios Starting from a Healthy 2-node VIP Cluster
Scenario	Description	Procedure	Expected Outcome
V04 - SvKMS both nodes offline	Due to maintenance or hardware failure of the host or environment the SvKMS runs on.	Power off each SvKMS VM.	No access to SvKMS cluster Cluster VIP unavailable throughout and unable to process KMIP requests and API calls etc. Encrypted vCenter and SvSAN VSAs will continue to operate as normal providing they remain online as the key is stored in memory. If rebooted, will be locked. Unable to encrypt or decrypt new targets or VM while down. No example image output as page not navigable
V05 - Both Nodes lose network connection.	Due to maintenance or network issues, both hosts lose network connection.	Remove network interface from each SvKMS VM.	Each node will fail to load IP address and SvKMS cluster will become unavailable. Encrypted vCenter and SvSAN VSAs will continue to operate as normal providing they remain online as the key is stored in memory. If rebooted, will be locked. Unable to encrypt or decrypt new targets or VM while down. Example image output from console of SvKMS below.

SvKMS Console example output for Scenario V05

Standard 2n+1 Cluster configuration scenarios:

This is example image set of a healthy 2n+1 SvKMS Cluster for a reference point to the below scenarios.

SvKMS example image for healthy 2n+1 Activator page.

SvKMS example image for healthy 2n+1 Cluster Status page.

‘Lights on’ failure scenarios Starting from a Standard 2n+1 Healthy Cluster
Scenario	Description	Procedure	Expected Outcome
201 - Single SvKMS node offline.	Due to maintenance or failure of the storage of the host the SvKMS node runs on.	Power off single SvKMS VM.	Remaining nodes will show display API (OK). Node that has failed will show Internal Server Error: EHOSTUNREACH. Cluster remains available throughout via and able to process KMIP requests and API calls etc. No impact or sign of Outage to vCenter or SvSAN VSAs. Example result output in images below
202 - Single SvKMS node network failure.	Single SvKMS node loses all network connections but remains running.	Remove network interface from SvKMS VM	Remaining nodes will show display API (OK). Node that has failed will show Internal Server Error: EHOSTUNREACH. Cluster remains available throughout via and able to process KMIP requests and API calls etc. No impact or sign of Outage to vCenter or SvSAN VSAs. Example result output in images below same as above scenario.

SvKMS Activator Page output for Scenario 201 and 202

SvKMS Cluster status output for Scenario 201 and 202

‘Lights off’ failure scenarios Starting from a Standard 2n+1 Healthy Cluster
Scenario	Description	Procedure	Expected Outcome
203 - Majority of SvKMS nodes offline	Due to maintenance or hardware failure of the host or environment the SvKMS runs on.	Power off majority of SvKMS VM's.	There will be no access to the cluster via the Remaining node(s). Cluster will reject any login attempts with authentication failure. Activator page will not load, will show error Checking node state failed: Request Timeout. Unable to process KMIP requests and API calls etc. Encrypted vCenter and SvSAN VSAs will continue to operate as normal providing they remain online as the key is stored in memory. If rebooted, will be locked. Unable to encrypt or decrypt new targets or VM while down. Example result output in images below
204 - All SvKMS nodes offline.	Due to maintenance or hardware failure of the host or environment the SvKMS runs on.	Power off all SvKMS VM's.	No access to SvKMS cluster Cluster unavailable throughout and unable to process KMIP requests and API calls etc. Encrypted vCenter and SvSAN VSAs will continue to operate as normal providing they remain online as the key is stored in memory. If rebooted, will be locked. Unable to encrypt or decrypt new targets or VM while down. No example image output as page not navigable
205 - Majority of nodes become Isolated	Network communications between majority of SvKMS nodes are lost simultaneously. Whereby each VM remains up and running but cannot talk to each other.	Remove network connectivity between the nodes.	There will be no access to the cluster via the remaining node(s). Cluster will reject any login attempts with authentication failure. Activator page will not load on remaining node(s), will show error Checking node state failed: Request Timeout. Unable to process KMIP requests and API calls etc. Encrypted vCenter and SvSAN VSAs will continue to operate as normal providing they remain online as the key is stored in memory. If rebooted, will be locked. Unable to encrypt or decrypt new targets or VM while down. Example image output in images below.