Search
  • Todd Waller

Splunk and Regex Fun

Hello Everyone!


Todays post is a fun one!


I was tasked with onboarding some data. In this instance the logs file was a text file that was created once per session so its not constantly written to. Most of the file is syslog formatted logs BUT the beginning log the file has a paragraph of useless information I don't need followed by the data I want to index.


So I wanted to just use sed to drop that data.


Here's an example file(with clean logs for security)

Last login: Wed Sep  4 16:42:58 EDT 2019 from 111.22.33.444 on pts/0
Video Communication Server X1.1.1
SW Release date: 2017-01-11 14:22, build
OK
?
Supported commands:
  about            - Displays the system version information
  bye              - Exits the shell
  configlog        - Displays lines from log files
  eventlog         - Displays lines from log files
  exit             - Exits the shell
  help             - Displays help for the top-level commands
  license          - Lists and displays third party software licenses
  networklog       - Displays lines from log files
  quit             - Exits the shell
  relkey           - Gets and sets the system release key
  xcommand         - <type "xcommand help" for more details>
  xconfiguration   - <type "xconfiguration help" for more details>
  xfeedback        - Displays change notifications on this console
  xgetxml          - Displays an XML description of some configuration
  xhistory         - <type "xhistory help" for more details>
  xstatus          - <type "xstatus help" for more details>
OK
configlog all
2019-05-17T00:23:19.244-04:00 infoserv UTCTime="2019-05-17 04:23:19,244" Event="System Configuration Changed" Node="cluster@192.0.0.1" PID="<0.111.0>" Detail="xconfiguration licensePoolLimitsStatus uuid 123aa444-7d94-40c6-1ec2-bd100222ed11 license_type: nontraversal timestamp - changed from: 1558063399 to: 1558066999"
2019-05-17T00:23:19.249-04:00 infoserv UTCTime="2019-05-17 04:23:19,249" Event="System Configuration Changed" Node="cluster@192.0.0.1" PID="<0.111.0>" Detail="xconfiguration licensePoolLimitsStatus uuid 1234abc56-abcd-1234b-b332-122ef3ce1234 license_type: traversal timestamp - changed from: 1558063399 to: 1558066999"
2019-05-17T00:23:19.253-04:00 infoserv UTCTime="2019-05-17 04:23:19,253" Event="System Configuration Changed" Node="cluster@192.0.0.1" PID="<0.111.0>" Detail="xconfiguration licensePoolLimitsStatus uuid 123aa444-7d94-40c6-1ec2-bd100222ed11 license_type: turnrelay timestamp - changed from: 1558063399 to: 1558066999"
2019-05-17T00:23:19.257-04:00 infoserv UTCTime="2019-05-17 04:23:19,257" Event="System Configuration Changed" Node="cluster@192.0.0.1" PID="<0.111.0>" Detail="xconfiguration licensePoolLimitsStatus uuid 1234abc56-abcd-1234b-b332-122ef3ce1234 license_type: user_registration timestamp - changed from: 1558063399 to: 1558066999"

So to removed this I put a sedcmd command in my props.conf for this sourcetype like this:


SEDCMD-removepara = s/^Last(?s)(.*)all//g

This basically starts at the beginning of the string matching "Last" all the way until "all" is reached, then it replaces that with nothing(deleted).


Then I turned on the UF that sends this data and it removed the data I didnt want and indexed the rest of the logs:

Maybe you can find this useful when you get stuck in a similar situation.


Have a great day!


-Cheers Todd


78 views0 comments

Recent Posts

See All

So I've been thinking today about how useful Splunk is on the "blue" side of the house. What about the Purple and Red side? What features would Splunk need to gain to add value to the other 2/3rds of