Hello Everyone!
So I have been going down the rabbit hole of performance testing and trying to find the "needle in the haystack". So I thought it might be helpful for other people to see what I did.
There are a couple notes before starting. Splunk must be stopped before running the test. For the Bonnie++ Analysis for Splunk app you need an index for the bonnie++ data unless you use the app with the provided indexes.conf file. I use a indexes.conf managed by an app in my indexer cluster so I didn't want the file in the app. I tested for each storage mount point.
So.....here we go!
The first thing I did was download the Bonnie++ Analysis for Splunk app from splunkbase.
Next I used the Bonnie++ Resources view to create the corresponding bonnie++ commands to run. Here's an example:
My commands we similar to that, one for each storage path:
echo -e "start=`date`" > /splunk//`hostname`.log && /opt/splunk/bonnie/bonnie++ -d /splunk/ -s 1031624 -u splunk:splunk -fq -n 5\npath="/splunk/" >> /splunk//`hostname`.log && echo -e "finish=`date`\n--- EOF ---" >> /splunk//`hostname`.log
echo -e "start=`date`" > /opt/splunk//`hostname`.log && /opt/splunk/bonnie/bonnie++ -d /opt/splunk/ -s 1031624 -u splunk:splunk -fq -n 5\npath="/opt/splunk/" >> /opt/splunk//`hostname`.log && echo -e "finish=`date`\n--- EOF ---" >> /opt/splunk//`hostname`.log
echo -e "start=`date`" > /splunk_cold//`hostname`.log && /opt/splunk/bonnie/bonnie++ -d /splunk_cold/ -s 1031624 -u splunk:splunk -fq -n 5\npath="/splunk_cold/" >> /splunk_cold//`hostname`.log && echo -e "finish=`date`\n--- EOF ---" >> /splunk_cold//`hostname`.log
The result is a file in the output path with the contents similar to this (sanitized for security):
Once you have this data you can one shot it into an index to use it the Bonnie++ Analysis for Splunk app. One command for each file.
/opt/splunk/bin/splunk add oneshot /splunk/`hostname`.log -sourcetype bonnie -index bonnie
/opt/splunk/bin/splunk add oneshot /opt/splunk/`hostname`.log -sourcetype bonnie -index bonnie
/opt/splunk/bin/splunk add oneshot /splunk_cold/`hostname`.log -sourcetype bonnie -index bonnie
Now you can take a look at your Bonnie++ Analysis for Splunk app and investigate results. Something like this:
Additionally you can use the built in export csv to html python command, boncsv2html.py, in bonnie++. This is handy because you can then send that to n html file that you can open in a web browser. You can do that like this:
Loaded into Chrome for example it looks like this:
Now you can take the results and act on them if you need to.
It was really helpful for me and some thing I've been working on.
Thanks and have a great week and weekend!
Cheers!
Todd