I am HUGE fan of the Splunk Conf presentations on Splunking the Endpoint done by James Brodsky but they are missing something! Linux! With so many endpoints like Linux jump server, desktop, development workstations, laptop and Cloud instances Linux is a very valid endpoint conversation to have.
Sure, maybe Linux isn't getting popped as much as Windows but it's definitely happening. This isn't even counting the risk and exposure you have from your compliance and auditing teams.
So how do you Splunk your Linux endpoint for security? I can't claim to be the world leader in Linux endpoint security but I wanted to take some time to cover the four systems I have in place to gain operational visibility.
Splunk_TA_nix
Splunk_TA_nix - Critical System files with tags Auditd logs, patching logs (yum), syslog system messages, CRON and the half-a-dozen facilities logging to /var Do you remember the difference between Debian, macOS and CentOS authentication logs? Me either. However, I can remember tag=authentication. While Splunk_TA_nix brings in the original files on all these platforms it includes DOZENS complex eventtypes and tags that make navigating a breeze. Toss these values in a data model you can search YEARs of data in seconds. Result of top and netstat is critical to tracing down RCA and forensics.
Splunk_TA_nix - Config_File monitoring Sure, Splunk isn't going to replace Github anytime soon, nor should it. Did the sysadmin remember to check his work in? (Do you check your work in?) Did a tool change that file? Was that a patch that altered grub or the bad guy? Even with the best GIT and change control process it can be complex and sometimes impossible to connect your audit logs back to actual config file changes. Splunk_TA_nix collects your default .conf files in near real time with little to no I/O cost and stores them for as long as you want. This allows you run diffs and correlate changes back to their source without leaving the Splunk console. And these are config_file folks we're talking BYTES of data.
Splunk_TA_nix - Shhh... fschange So fschange was officially removed from Splunk but if you're running Splunk forwarders 7.3 or lower you have a custom input called "fschange". What this does is scan your favorite files periodically and stores the change information locally. If the monitored file changes, it logs out all the critical details of the file integrity change. While there are better options than fschange out there, I find it drives quick value for specific monitoring jobs. I have fschange cross monitor my FIM for changes. Yes, I am scanning my FIM with a FIM to make sure they FIM.
Sure Splunk and Splunk_TA_nix let you search some logs but when you've tuned up your Splunk_TA_nix config you start to gain real operational visibility. System performance, config_file monitoring, critical system logging, event tagging and file integrity management. As I close out I'll mention that Splunk_TA_NIX comes with some interesting normalization techniques for the auditd log on Linux. It's true to Splunk approach of protecting the original data but I am not fan. It's hard to read and not well normalized. I've got a better option for you I'll blog about later.