Splunk and Regex Fun

Hello Everyone!

Todays post is a fun one!

I was tasked with onboarding some data. In this instance the logs file was a text file that was created once per session so its not constantly written to. Most of the file is syslog formatted logs BUT the beginning log the file has a paragraph of useless information I don't need followed by the data I want to index.

So I wanted to just use sed to drop that data.

Here's an example file(with clean logs for security)

Last login: Wed Sep  4 16:42:58 EDT 2019 from 111.22.33.444 on pts/0
Video Communication Server X1.1.1
SW Release date: 2017-01-11 14:22, build
OK
?
Supported commands:
  about            - Displays the system version information
  bye              - Exits the shell
  configlog        - Displays lines from log files
  eventlog         - Displays lines from log files
  exit             - Exits the shell
  help             - Displays help for the top-level commands
  license          - Lists and displays third party software licenses
  networklog       - Displays lines from log files
  quit             - Exits the shell
  relkey           - Gets and sets the system release key
  xcommand         - <type "xcommand help" for more details>
  xconfiguration   - <type "xconfiguration help" for more details>
  xfeedback        - Displays change notifications on this console
  xgetxml          - Displays an XML description of some configuration
  xhistory         - <type "xhistory help" for more details>
  xstatus          - <type "xstatus help" for more details>
OK
configlog all
2019-05-17T00:23:19.244-04:00 infoserv UTCTime="2019-05-17 04:23:19,244" Event="System Configuration Changed" Node="cluster@192.0.0.1" PID="<0.111.0>" Detail="xconfiguration licensePoolLimitsStatus uuid 123aa444-7d94-40c6-1ec2-bd100222ed11 license_type: nontraversal timestamp - changed from: 1558063399 to: 1558066999"
2019-05-17T00:23:19.249-04:00 infoserv UTCTime="2019-05-17 04:23:19,249" Event="System Configuration Changed" Node="cluster@192.0.0.1" PID="<0.111.0>" Detail="xconfiguration licensePoolLimitsStatus uuid 1234abc56-abcd-1234b-b332-122ef3ce1234 license_type: traversal timestamp - changed from: 1558063399 to: 1558066999"
2019-05-17T00:23:19.253-04:00 infoserv UTCTime="2019-05-17 04:23:19,253" Event="System Configuration Changed" Node="cluster@192.0.0.1" PID="<0.111.0>" Detail="xconfiguration licensePoolLimitsStatus uuid 123aa444-7d94-40c6-1ec2-bd100222ed11 license_type: turnrelay timestamp - changed from: 1558063399 to: 1558066999"
2019-05-17T00:23:19.257-04:00 infoserv UTCTime="2019-05-17 04:23:19,257" Event="System Configuration Changed" Node="cluster@192.0.0.1" PID="<0.111.0>" Detail="xconfiguration licensePoolLimitsStatus uuid 1234abc56-abcd-1234b-b332-122ef3ce1234 license_type: user_registration timestamp - changed from: 1558063399 to: 1558066999"

So to removed this I put a sedcmd command in my props.conf for this sourcetype like this:

SEDCMD-removepara = s/^Last(?s)(.*)all//g

This basically starts at the beginning of the string matching "Last" all the way until "all" is reached, then it replaces that with nothing(deleted).

Then I turned on the UF that sends this data and it removed the data I didnt want and indexed the rest of the logs:

Maybe you can find this useful when you get stuck in a similar situation.

Have a great day!

-Cheers Todd