Hello again Everyone
In our last post we talked about migrating you Indexer Cluster to new hosts. Today I am going to go to the next step in rebuilding creating an Indexer Cluster and expand on how to migrate your Indexer Cluster Master.
its not really a complicated process. I did it this way to minimize downtime, its is possible to do it in a slightly different order.
I am migrating to a new freshly installed and configured machine.
1. Setup the new Cluster Master - prior to stopping the current, just dont restart yet
A. Copy files to the new Cluster Master
scp -r /opt/splunk/etc/master-apps/ splunk@<newcm>.domain.com:/opt/splunk/etc/
B. Copy server.conf to new Cluster Master
scp -r /opt/splunk/etc/system/local/server.conf splunk@<newcm>.domain.com:/opt/splunk/etc/system/local/
C. On the new Cluster Master change encrypted passwords to clear text, identify all with:
1. find /opt/splunk/etc -name '*.conf' -exec grep -inH '\$[0-9]\$' {} \;
2. /opt/splunk/bin/splunk show-decrypted --value '$encryptedpassword'
D. Make sure the above decrypted the main 2 passwords below and replace them in the copied server.conf, in clear text, on the new Cluster Master until it is restarted when it will then encrypt.
[general] sslPassword=
[clustering]
pass4SymmKey=
2. Backup the etc directory on the old Cluster Master
tar -czvf splunkEtcBackup<date>.tgz /opt/splunk/etc
3. Stop splunk on the old Cluster Master
/opt/splunk/bin/splunk stop
4. Copy the old Cluster Masters remote bundle to the new Cluster Master
scp -r /opt/splunk/var/run/splunk/cluster/remote-bundle splunk@<newcm>.domain.com:/opt/splunk/var/run/splunk/cluster/remote-bundle
5. Start/Restart the new Cluster Master
/opt/splunk/bin/splunk restart
6. Point indexers to the new Cluster Master
/opt/splunk/bin/splunk edit cluster-config -mode peer -manager_uri https://<newcm>.domain.com:8089 -replication_port 9887 -secret <secret>
7. Point the search heads to the new Cluster Master
/opt/splunk/bin/splunk edit cluster-config -mode searchhead -manager_uri https://<newcm>.domain.com:8089 -secret <your_key_decypted>
And there you have it! Nothing major, check for errors and make sure everything is functioning as intended.
Thanks again everyone!
Have a great day!