Search
  • Todd Waller

Splunk and Bonnie++ Testing

Hello Everyone!


So I have been going down the rabbit hole of performance testing and trying to find the "needle in the haystack". So I thought it might be helpful for other people to see what I did.

There are a couple notes before starting. Splunk must be stopped before running the test. For the Bonnie++ Analysis for Splunk app you need an index for the bonnie++ data unless you use the app with the provided indexes.conf file. I use a indexes.conf managed by an app in my indexer cluster so I didn't want the file in the app. I tested for each storage mount point.


So.....here we go!


The first thing I did was download the Bonnie++ Analysis for Splunk app from splunkbase.

Next I used the Bonnie++ Resources view to create the corresponding bonnie++ commands to run. Here's an example:




My commands we similar to that, one for each storage path:


echo -e "start=`date`" > /splunk//`hostname`.log && /opt/splunk/bonnie/bonnie++ -d /splunk/ -s 1031624 -u splunk:splunk -fq -n 5\npath="/splunk/" >> /splunk//`hostname`.log && echo -e "finish=`date`\n--- EOF ---" >> /splunk//`hostname`.log


  echo -e "start=`date`" > /opt/splunk//`hostname`.log && /opt/splunk/bonnie/bonnie++ -d /opt/splunk/ -s 1031624 -u splunk:splunk -fq -n 5\npath="/opt/splunk/" >> /opt/splunk//`hostname`.log && echo -e "finish=`date`\n--- EOF ---" >> /opt/splunk//`hostname`.log


  echo -e "start=`date`" > /splunk_cold//`hostname`.log && /opt/splunk/bonnie/bonnie++ -d /splunk_cold/ -s 1031624 -u splunk:splunk -fq -n 5\npath="/splunk_cold/" >> /splunk_cold//`hostname`.log && echo -e "finish=`date`\n--- EOF ---" >> /splunk_cold//`hostname`.log


The result is a file in the output path with the contents similar to this (sanitized for security):



Once you have this data you can one shot it into an index to use it the Bonnie++ Analysis for Splunk app. One command for each file.


  /opt/splunk/bin/splunk add oneshot /splunk/`hostname`.log -sourcetype bonnie -index bonnie


  /opt/splunk/bin/splunk add oneshot /opt/splunk/`hostname`.log -sourcetype bonnie -index bonnie


  /opt/splunk/bin/splunk add oneshot /splunk_cold/`hostname`.log -sourcetype bonnie -index bonnie

 

Now you can take a look at your Bonnie++ Analysis for Splunk app and investigate results. Something like this:




Additionally you can use the built in export csv to html python command, boncsv2html.py, in bonnie++. This is handy because you can then send that to n html file that you can open in a web browser. You can do that like this:



Loaded into Chrome for example it looks like this:


Now you can take the results and act on them if you need to.


It was really helpful for me and some thing I've been working on.


Thanks and have a great week and weekend!


Cheers!


Todd

212 views0 comments

Recent Posts

See All

Do you need to dedup when using stats?

I had to do some casual counting of sourcetypes today. In the process I was trying to decide if I needed to dedup before going to stats. It seemed to me a dedup would, in theory, pass less data to sta

How do I learn Splunk administration?

Had an old coworker hit me up a week ago. He took a job as a SOC analyst where part of his job is going to be supporting Splunk. He's a smart guy but Splunk is more complex than it looks. Given I've a