Search
  • Todd Waller

Route ugly data to Splunk nullQueue and save some cash

Updated: Oct 30, 2018

Welcome to another installment of Old Logs New Tricks. Here we are going to look at routing some of the uglier non-useful data from the logs, pre-index time, to save on licensing as well as make the logs prettier. So say we have a log like this(edited for security reasons!):

So, for example, let's say we don't want debug logs. So what we can do is go into Splunk and on the Heavy Forwarder tier(some dont have this type of deployment, in that instance it would be added to the indexer) and pipe those logs to the "nullQueue", with props.conf and transforms.conf: props.conf [new_sourcetype]

TRANSFORMS-bye_debug=kill_debug transforms.conf

[kill_debug]

REGEX=\"level\"\:\"debug\"

DEST_KEY=queue

FORMAT=nullQueue Always test your regex:

Now in Splunk the debug logs will be gone laving only the informational messages

Doing this will save you time as well as money since sending un-necessary data to the nullQueue at index time keeps that data from being indexed and using the license that you pay for. Thanks for following and have a great day!

187 views0 comments

Recent Posts

See All

Do you need to dedup when using stats?

I had to do some casual counting of sourcetypes today. In the process I was trying to decide if I needed to dedup before going to stats. It seemed to me a dedup would, in theory, pass less data to sta

How do I learn Splunk administration?

Had an old coworker hit me up a week ago. He took a job as a SOC analyst where part of his job is going to be supporting Splunk. He's a smart guy but Splunk is more complex than it looks. Given I've a