25 Mar Recover From Loss of Majority (Search Head Clustering Service Not Ready Error)

A Splunk cluster generally utilizes dynamic captain that keeps shifting over time. The periodic elections choose the dynamic captain. In the dynamic captain, a majority of all cluster members should agree on the chosen captain.    A captain can’t be elected if a cluster suffers the loss of most of its members, and cannot continue to perform wrongly. By reconfiguring the cluster, this can be worked around to use a static captain in place of a dynamic captain.    A static captain, unlike dynamic captain, usually does not alter over time. The cluster won’t carry out an election to choose the static captain. Rather, a member is designated as a static captain. That member lasts as the captain till another member is designated as captain. 

To make a member a Static Captain 

1. Run the below mentioned CLI command on the member, you would like to nominate as captain.  
splunk edit shcluster-config -mode captain -captain_uri : – false election

2. On every member that is not a captain, run the following CLI command: 
Splunk edit shcluster-config -mode member -captain_uri : – false election 
• -mode -> this parameter specifies if the sample should perform as captain or only as member. • -captain_uri -> this parameter defines the management port and URI of the captain sample. • -election -> by placing election as “false”, it is conveyed that the cluster makes use of a static captain. Retrograde to Dynamic Captain While the provoking circumstance has resolved, retrograde the cluster to steer using one dynamic captain. To divert to the dynamic captain, reconfigure every member that you configured for static captain earlier.   

Note: Also clean every Instance KV Store and also sync. 

1. As soon as the cluster retrieves its majority, shift all members once again to dynamic captain using the command mentioned below. Convert the current, static captain last. 
Splunk edit shcluster-config -election true -mgmt_uri : 

2. Bootstrap one of the many members. This member is selected as the first dynamic captain. It is adviced to bootstrap the member which was the static captain earlier. 
Splunk bootstrap shcluster-captain -servers_list “:,: 
 
Don’t forget to follow us on 👍 Social networks.