So hopefully you read my Splunking the Linux Endpoint Part 1 blog ( https://www.oldlogsnewtricks.com/home/splunking-the-linux-endpoint-part-1 ) . And you've solved system logging, Config_file monitoring, performance data, patching data and some simple File Integrity Management (FIM) problems.
I think it's fair to say you've noticed I danced around two thing 1) fschange is limited and unsupported 2) Auditd logs suck
FIM is a common regulatory requirement and also a critical component to monitoring the integrity of you systems both against the bad guys and your own system administrators.
So you're probably thinking, I'll use this fancy fschange thing Daniel mentioned - well. Yeah. I do! It's awesome. And maybe some very simple use cases you should. It requires no extra software, the logs pop out in a quality way, they are even tagged for premium tools like Splunk ES and ITSI.
But fschange isn't really supported, eventually it will just stop working in Splunk. So you're thinking I know what I'll do I'll just take a day and really tweak up Auditd. And this works. Of course the issue with auditd is that it's logs are not well through out. Where a single event and can be on multiple line.
So the answer? Splunk biggest competitor Elastic has an open source solution that can pop right into your environment called "Auditbeat".
A "beat" is basically a log shipper like a Universal Forwarder from Splunk. But they are generally single tasked and don't have a lot of the flexibility of the App/TA model Splunk uses.
In this case Elastic has solved the problem of multi line events by merging them into one event that is packaged neatly into a JSON document. While the resulting ouput isn't CIM it's pretty slick. I tossed a TA on Splunkbase to get you started with CIMing the data if you're in a ITSI/PCI/ES envionment. Otherwise you're good to go out of the box really. https://splunkbase.splunk.com/app/4436/
Auditbeat offers three features in it's current interation to solve compliance and security problems. 1) System Info Auditbeat periodically logs out the system profile info. Host name, IP, etc
2) Audit Replacement Auditbeat's audit subsystem hooks the Linux kernel and generates beautiful JSON documents. The Config file (auditbeat.yml) takes standard auditd configurations. So as your testing out your audit configuration you can borrow directly from existing recipes.
3) File Integrity Monitoring My personal favorite use case if FIM. Auditbeat hooks the OS kernel and keep running track of change types. These changes are merged into the Audit documents making tracing back the changes easy easy.
I don't normally like to blog recipes and how-to's, because they become out of date over night. So let me point you to a few as I close out the day with people way better at keeping up their technical details than I am.
Auditd Recipe to get you going with PCI annotation right in the logs! https://github.com/linux-audit/audit-userspace/blob/master/rules/30-pci-dss-v31.rules
Install Auditbeat https://www.elastic.co/guide/en/beats/auditbeat/current/auditbeat-installation.html
Auditbeat TA for Splunk https://splunkbase.splunk.com/app/4436/
SANS Paper on FIM https://www.sans.org/reading-room/whitepapers/detection/paper/35327
Getting Beats Data Into Splunk https://www.mos-eisley.dk/display/it/Beats+for+splunk
So if you've been following along with the series so far we've only setup two tools so far and we're already gaining keep insights into the Linux endpoint.
Splunk_TA_Nix 1) OS performance data 2) System logging/Syslog/Update logs 3) Config management monitoring
TA-Auditbeat 1) System Profiling 2) Audit Logs with Compliance annotataion 3) File Integrity Management