Search
  • Todd Waller

Creating, testing and optimizing Splunk index time extraction

Updated: Oct 16, 2018

Hello everyone, welcome back to Old Logs New Tricks. If you are like us you are always looking to improve your Splunk indexing or search performance. Some things you can do to improve search performance is to move search time extractions to index time. While this is not recommended it is acceptable for a few use cases. As an example we have host names that look something like: nsab000css005.domain.com In this example we call the portion "css" role. Every host has this role and extracting it at search time causes really poor performance so moving it to index time on our Intermediate forwarders, where much of our pre-processing happens, is an acceptable change. To do this I created a file called fields.conf in the $SPLUNK_HOME/etc/apps/<app_name>/local on my Intermediate tier. Its a very simple file and looks something like this:

[example_fieldname]

INDEXED=true


Next I created a transforms.conf in $SPLUNK_HOME/etc/apps/<app_name>/local for the new field like:

[extract_role]

REGEX=\D{3}\D\d{1,4}(?<example_fieldname>...)\d{1,5}

FORMAT = example_fieldname::$1

SOURCE_KEY = MetaData:Host

WRITE_META = true



Then I created a props.conf in $SPLUNK_HOME/etc/apps/<app_name>/local to apply to all hosts. Here's a new "trick." Splunk doesn't work with wildcards (*) in stanzas, so "[host::*]" won't work. TO WORK AROUND THIS you can do this:

[(?::){0}host::*]

TRANSFORMS-roleextract = extract_role Next I went to $SPLUNK_HOME/etc/apps/<app_name>/metadata and created a local.meta file to set permissions to global for the search heads. (As we will copy this app to them in the next steps.)

[example_fieldname]

export = system Lastly, you will want these configurations on the search heads as well. For us in a clustered environment we copied the app to the deployer and then deployed it to the cluster, otherwise just add it to your search head(s). Verify in search that the data works as expected: host=* | dedup example_fieldname| table example_fieldname


example_fieldname css

hdn

dst

ifd

chb

dtx

cob

hnn

hen

usp

api

itl

jes



Now that it verified and you're done! Hope that helps you save some time and improve performance, it helped us. Have a great day!



24 views0 comments

Recent Posts

See All

How do I learn Splunk administration?

Had an old coworker hit me up a week ago. He took a job as a SOC analyst where part of his job is going to be supporting Splunk. He's a smart guy but Splunk is more complex than it looks. Given I've a