• Todd Waller

Splunk and Bonnie++ Testing

Hello Everyone!

So I have been going down the rabbit hole of performance testing and trying to find the "needle in the haystack". So I thought it might be helpful for other people to see what I did.

There are a couple notes before starting. Splunk must be stopped before running the test. For the Bonnie++ Analysis for Splunk app you need an index for the bonnie++ data unless you use the app with the provided indexes.conf file. I use a indexes.conf managed by an app in my indexer cluster so I didn't want the file in the app. I tested for each storage mount point. we go!

The first thing I did was download the Bonnie++ Analysis for Splunk app from splunkbase.

Next I used the Bonnie++ Resources view to create the corresponding bonnie++ commands to run. Here's an example:

My commands we similar to that, one for each storage path:

echo -e "start=`date`" > /splunk//`hostname`.log && /opt/splunk/bonnie/bonnie++ -d /splunk/ -s 1031624 -u splunk:splunk -fq -n 5\npath="/splunk/" >> /splunk//`hostname`.log && echo -e "finish=`date`\n--- EOF ---" >> /splunk//`hostname`.log

  echo -e "start=`date`" > /opt/splunk//`hostname`.log && /opt/splunk/bonnie/bonnie++ -d /opt/splunk/ -s 1031624 -u splunk:splunk -fq -n 5\npath="/opt/splunk/" >> /opt/splunk//`hostname`.log && echo -e "finish=`date`\n--- EOF ---" >> /opt/splunk//`hostname`.log

  echo -e "start=`date`" > /splunk_cold//`hostname`.log && /opt/splunk/bonnie/bonnie++ -d /splunk_cold/ -s 1031624 -u splunk:splunk -fq -n 5\npath="/splunk_cold/" >> /splunk_cold//`hostname`.log && echo -e "finish=`date`\n--- EOF ---" >> /splunk_cold//`hostname`.log

The result is a file in the output path with the contents similar to this (sanitized for security):

Once you have this data you can one shot it into an index to use it the Bonnie++ Analysis for Splunk app. One command for each file.

  /opt/splunk/bin/splunk add oneshot /splunk/`hostname`.log -sourcetype bonnie -index bonnie

  /opt/splunk/bin/splunk add oneshot /opt/splunk/`hostname`.log -sourcetype bonnie -index bonnie

  /opt/splunk/bin/splunk add oneshot /splunk_cold/`hostname`.log -sourcetype bonnie -index bonnie


Now you can take a look at your Bonnie++ Analysis for Splunk app and investigate results. Something like this:

Additionally you can use the built in export csv to html python command,, in bonnie++. This is handy because you can then send that to n html file that you can open in a web browser. You can do that like this:

Loaded into Chrome for example it looks like this:

Now you can take the results and act on them if you need to.

It was really helpful for me and some thing I've been working on.

Thanks and have a great week and weekend!



345 views0 comments

Recent Posts

See All