top of page
  • Writer's pictureTodd Waller

Creating, testing and optimizing Splunk index time extraction

Updated: Oct 16, 2018

Hello everyone, welcome back to Old Logs New Tricks. If you are like us you are always looking to improve your Splunk indexing or search performance. Some things you can do to improve search performance is to move search time extractions to index time. While this is not recommended it is acceptable for a few use cases. As an example we have host names that look something like: In this example we call the portion "css" role. Every host has this role and extracting it at search time causes really poor performance so moving it to index time on our Intermediate forwarders, where much of our pre-processing happens, is an acceptable change. To do this I created a file called fields.conf in the $SPLUNK_HOME/etc/apps/<app_name>/local on my Intermediate tier. Its a very simple file and looks something like this:



Next I created a transforms.conf in $SPLUNK_HOME/etc/apps/<app_name>/local for the new field like:



FORMAT = example_fieldname::$1

SOURCE_KEY = MetaData:Host


Then I created a props.conf in $SPLUNK_HOME/etc/apps/<app_name>/local to apply to all hosts. Here's a new "trick." Splunk doesn't work with wildcards (*) in stanzas, so "[host::*]" won't work. TO WORK AROUND THIS you can do this:


TRANSFORMS-roleextract = extract_role Next I went to $SPLUNK_HOME/etc/apps/<app_name>/metadata and created a local.meta file to set permissions to global for the search heads. (As we will copy this app to them in the next steps.)


export = system Lastly, you will want these configurations on the search heads as well. For us in a clustered environment we copied the app to the deployer and then deployed it to the cluster, otherwise just add it to your search head(s). Verify in search that the data works as expected: host=* | dedup example_fieldname| table example_fieldname

example_fieldname css













Now that it verified and you're done! Hope that helps you save some time and improve performance, it helped us. Have a great day!

155 views0 comments

Recent Posts

See All
Post: Blog2_Post
bottom of page