Search
  • Todd Waller

Splunk - Fixing - Bucket Already Added as Clustered Attempted to Add Again as Standalone

Hello Everyone


Welcome back to Old Logs New Tricks!


Todays post we are going to talk about an issue I ran into recently and thought it would be a good idea to post about.


Here's the message I got from an indexer after we had some architecture issues with the underlying hardware Spunk was running on(all images and messages are sanitized for security reasons....of course):

Failed to register with cluster master reason: failed method=POST path=/services/cluster/master/peers/?output_mode=json master=xxx.xxx.xxx:8089 rv=0 gotConnectionError=0 gotUnexpectedStatusCode=1 actual_response_code=500 expected_response_code=2xx status_line=“Internal Server Error” socket_error=“No error” remote_error=Cannot add peer=xxx.xxx.xxx.xxx mgmtport=8089 (reason: bucket already added as clustered, peer attempted to add again as standalone. guid=12345A67-BC8D-9E0F-G1H2-34I56J7K8L9M bid=windows_perfmon~571~12345A67-BC8D-9E0F-G1H2-34I56J7K8L9M).


So this bucket was preventing the indexer from re-joining the indexer cluster.


So what I had to do is go into the, in my case custom data library path, and find the bucket in question:




Based on the warning message I know the index to look in and the name of the bucket so I can grep that index and find the bucket in question:

[splunk@hostname splunkdata]$ grep -r '_571 ' windows_perfmon/db/



This found the bucket I was looking for:


To fix this I tacked the GUID of the cluster master to the end of the bucket to force it to send it to the cluster master to replicate/fix. TO do this I had to find the GUID of the cluster master. You can do this by checking the file located at $SPLUNK_HOME/etc/instance.cfg:



To add the GUID I simply used 'mv' to move the bucket to the new bucket name: [splunk@hostname db]$ mv db_1627425087_1627424688_571 db_1627425087_1627424688_571_12345D67-CE8C-9F0A-A1D2-34E56B7A8E9F


And this is the result:


After is this was completed, I simply restarted Splunk on the indexer and it send the bucket to the cluster master and re-joined the cluster



I hope this helps in the event some of you may run into this issue. I definitely was tricky to figure out but once I did fixing it was easy.


Have a great day everyone and talk to you soon!


Todd

76 views0 comments

Recent Posts

See All

So I've been thinking today about how useful Splunk is on the "blue" side of the house. What about the Purple and Red side? What features would Splunk need to gain to add value to the other 2/3rds of