Search
  • Todd Waller

Splunk - Fixing - Bucket Already Added as Clustered Attempted to Add Again as Standalone

Hello Everyone


Welcome back to Old Logs New Tricks!


Todays post we are going to talk about an issue I ran into recently and thought it would be a good idea to post about.


Here's the message I got from an indexer after we had some architecture issues with the underlying hardware Spunk was running on(all images and messages are sanitized for security reasons....of course):

Failed to register with cluster master reason: failed method=POST path=/services/cluster/master/peers/?output_mode=json master=xxx.xxx.xxx:8089 rv=0 gotConnectionError=0 gotUnexpectedStatusCode=1 actual_response_code=500 expected_response_code=2xx status_line=“Internal Server Error” socket_error=“No error” remote_error=Cannot add peer=xxx.xxx.xxx.xxx mgmtport=8089 (reason: bucket already added as clustered, peer attempted to add again as standalone. guid=12345A67-BC8D-9E0F-G1H2-34I56J7K8L9M bid=windows_perfmon~571~12345A67-BC8D-9E0F-G1H2-34I56J7K8L9M).


So this bucket was preventing the indexer from re-joining the indexer cluster.


So what I had to do is go into the, in my case custom data library path, and find the bucket in question:




Based on the warning message I know the index to look in and the name of the bucket so I can grep that index and find the bucket in question:

[splunk@hostname splunkdata]$ grep -r '_571 ' windows_perfmon/db/



This found the bucket I was looking for:


To fix this I tacked the GUID of the cluster master to the end of the bucket to force it to send it to the cluster master to replicate/fix. TO do this I had to find the GUID of the cluster master. You can do this by checking the file located at $SPLUNK_HOME/etc/instance.cfg:



To add the GUID I simply used 'mv' to move the bucket to the new bucket name: [splunk@hostname db]$ mv db_1627425087_1627424688_571 db_1627425087_1627424688_571_12345D67-CE8C-9F0A-A1D2-34E56B7A8E9F


And this is the result:


After is this was completed, I simply restarted Splunk on the indexer and it send the bucket to the cluster master and re-joined the cluster



I hope this helps in the event some of you may run into this issue. I definitely was tricky to figure out but once I did fixing it was easy.


Have a great day everyone and talk to you soon!


Todd

42 views0 comments

Recent Posts

See All

Do you need to dedup when using stats?

I had to do some casual counting of sourcetypes today. In the process I was trying to decide if I needed to dedup before going to stats. It seemed to me a dedup would, in theory, pass less data to sta

How do I learn Splunk administration?

Had an old coworker hit me up a week ago. He took a job as a SOC analyst where part of his job is going to be supporting Splunk. He's a smart guy but Splunk is more complex than it looks. Given I've a