Search
  • Todd Waller

Splunk and Bonnie++ Testing

Hello Everyone!


So I have been going down the rabbit hole of performance testing and trying to find the "needle in the haystack". So I thought it might be helpful for other people to see what I did.

There are a couple notes before starting. Splunk must be stopped before running the test. For the Bonnie++ Analysis for Splunk app you need an index for the bonnie++ data unless you use the app with the provided indexes.conf file. I use a indexes.conf managed by an app in my indexer cluster so I didn't want the file in the app. I tested for each storage mount point.


So.....here we go!


The first thing I did was download the Bonnie++ Analysis for Splunk app from splunkbase.

Next I used the Bonnie++ Resources view to create the corresponding bonnie++ commands to run. Here's an example:




My commands we similar to that, one for each storage path:


echo -e "start=`date`" > /splunk//`hostname`.log && /opt/splunk/bonnie/bonnie++ -d /splunk/ -s 1031624 -u splunk:splunk -fq -n 5\npath="/splunk/" >> /splunk//`hostname`.log && echo -e "finish=`date`\n--- EOF ---" >> /splunk//`hostname`.log


  echo -e "start=`date`" > /opt/splunk//`hostname`.log && /opt/splunk/bonnie/bonnie++ -d /opt/splunk/ -s 1031624 -u splunk:splunk -fq -n 5\npath="/opt/splunk/" >> /opt/splunk//`hostname`.log && echo -e "finish=`date`\n--- EOF ---" >> /opt/splunk//`hostname`.log


  echo -e "start=`date`" > /splunk_cold//`hostname`.log && /opt/splunk/bonnie/bonnie++ -d /splunk_cold/ -s 1031624 -u splunk:splunk -fq -n 5\npath="/splunk_cold/" >> /splunk_cold//`hostname`.log && echo -e "finish=`date`\n--- EOF ---" >> /splunk_cold//`hostname`.log


The result is a file in the output path with the contents similar to this (sanitized for security):



Once you have this data you can one shot it into an index to use it the Bonnie++ Analysis for Splunk app. One command for each file.


  /opt/splunk/bin/splunk add oneshot /splunk/`hostname`.log -sourcetype bonnie -index bonnie


  /opt/splunk/bin/splunk add oneshot /opt/splunk/`hostname`.log -sourcetype bonnie -index bonnie


  /opt/splunk/bin/splunk add oneshot /splunk_cold/`hostname`.log -sourcetype bonnie -index bonnie

 

Now you can take a look at your Bonnie++ Analysis for Splunk app and investigate results. Something like this:




Additionally you can use the built in export csv to html python command, boncsv2html.py, in bonnie++. This is handy because you can then send that to n html file that you can open in a web browser. You can do that like this:



Loaded into Chrome for example it looks like this:


Now you can take the results and act on them if you need to.


It was really helpful for me and some thing I've been working on.


Thanks and have a great week and weekend!


Cheers!


Todd

293 views0 comments

Recent Posts

See All

So I've been thinking today about how useful Splunk is on the "blue" side of the house. What about the Purple and Red side? What features would Splunk need to gain to add value to the other 2/3rds of