Recently we ran into an issue where our existing Deployment Server was becoming overburdened with too many enpoints checking in too frequently causing the UI to become slow and tedious to use. We decided to split the load across a total of 3 servers, based on our endpoint count. Assuming we could put a VIP and a load balancer in front of the cluster.
Honestly this was all hypothetical as this functionality is only eluded to in the serverclass confgiuration file.
We built and tested out this functionality, that is really only eluded to in configuration files.
There are a few "gotchas" that we found but its pretty ok otherwise.
The setup is pretty simple. VIP set to a URL with a load balancer with 3 servers.
So get the VIP created and the DNS configured(Will elaborate on this later). For our example the VIP is splunkdeploy.domain.com:8089. So when the endpoints/UFs connect to that VIP, they get load balanced across our 3 Deployment Servers: 192.3.3.1, 192.3.3.2, 192.3.3.3
Once you verify connections and DNS things are working as expected we can move on to adding the actual configs.
Additionally, we had a deployment server we were already using that we had existing apps and cofngiurations for, as well as serverclass configs that we wanted to make sure stayed.
To start 192.3.3.1 was our original server. We added 2 new ones, 192.3.3.2 and 192.3.3.3 and left Splunk OFF.
192.3.3.1 is going to be our Parent Deployment Server and 192.3.3.2 and 192.3.3.3 will be our Child Deployment Servers.
We copied the existing serverclass.conf from 192.3.3.1 to 192.3.3.2 and 192.3.3.3 in $SPLUNK_HOME/etc/system/local/
on the Parent Deployment Server, in the serverclass.conf file we added this to the top:
[global]
crossServerChecksum = true
*THESE SHOULD NOT BE ADDED TO THE CHILDREN*
[serverClass:DSChild_AppReplication]
restartSplunkd = false
stateOnClient = noop
whitelist.0 = 192.3.3.*
[serverClass:DSChild_AppReplication:app:*]
restartSplunkd = false
stateOnClient = noop
On the Child Deployment Servers we only added this in the serverclass.conf file: [global]
crossServerChecksum = true
We also created a deploymentclient.conf to point the Child Deployment Servers back to the Parent:
[deployment-client]
repositoryLocation = /opt/splunk/etc/deployment-apps
serverRepositoryLocationPolicy = rejectAlways
phoneHomeIntervalInSecs = 30
[target-broker:deploymentServer]
# Change the targetUri
targetUri = 192.3.3.1:8089
Next we started Splunk back up on the new servers and reloaded all serverclasses on the Parent.
All apps are now in the deployment-apps directory of the Children and, throetically, all deployment app changes made in the Parent will replicate to the Children.
Lessons Learned!!!!
serverclass.conf does not replicate, so when changes are made to serevrclass.conf(i.e new app added to serverclass, hosts in serverclass are updated etc.) the file must scp'd to the Children and added to $SPLUNK_HOME/etc/system/local/
Replication of apps, at times, seems slow or unreliable
Editing serverclass.conf is easier on the command line as opposed to working in the UI