Solving Splunk KV Store Errors

Splunk environment is strained by errors connected to the KV store. Along with such errors, several warnings associated with the KV store and Buckets arise. It affects our clustering and makes it unsteady.    Note: Here we are facing issues with our search head cluster
The snip below shows an error message: 

Steps to Solve KV Store Error: 

 1. Firstly, to solve this we need to sync the KV store on all the available members of the Search-Head cluster.   

Resynchronize the KV Store: 

It might be stale if a KV store member flunks to alter its data from the write operations. To solve this matter, you need to resync the member. 

 Identify the Stale Member 

Log in to all the members from Putty and run the following command: ./splunk show kvstore-status. This would return the KV store members’ summary, and also information related to all the members in the cluster of KV store. Let’s look at the replication status area and find out if any member has neither “KV store captain” nor “Non-captain KV store member” as values. So take it as worn out that needs to Re-Sync. 

Splunk KV

                  (Ensure that all the members are showing the right info.)

The below steps are to resynchronize all the members.  

  • Identify which node acts as search head cluster captain currently, by running the following command in any of the Sh’s. 
  • /opt/splunk/bin/splunk show shcluster-status. 
  • Log in to SH Cluster captain and run ‘Splunk resync KV store’. 
  • Employ the ‘splunk show kvstore-status’ command to check if the cluster is resynchronized. 
  • On cluster members, execute the following steps individually. 
  • Terminate the Splunk on search head of each member. 
  • Execute the command ‘splunk clean kvstore –local’. 
  • Reinitiate the search head. It activates the initial synchronization from other members of the KV store. 
  • Execute the command ‘splunk show kvstore-status’ to confirm synchronization. 

    2. Verify and modify the permission of the splunk.key file if needed on each occasion. Go to the path /opt/splunk/var/lib/splunk/kvstore/mongo/ and verify the permission of the file by command ll or ls –lrth, on each incidence having the error. 

Splunk KV

 Alter the permission of the file to read-only.  By command: – chmod 400 splunk.key    

Splunk KV

Restart Splunk each time to view the changes.