I’ve been doing a lot of work recently setting up auditing of Splunk. One of the components of this is ingesting bash history, as most Splunk config changes are made from the command line. I found this great blog post by Duanne Waddle that explains how to enhance and capture bash history, which saved me a lot of work! I did need to make a few tweaks to his script for my environment. I suggest reading Duanne’s post first, as he does an excellent job of explaining the script and the enhancements it makes to the bash history.
Modifying the script
The first thing I did differently was to drop bash-history.sh into /etc/profile.d, which eliminated the need to source the script in each user’s .profile script. Any scripts in /etc/profile.d are loaded via /etc/profile, so it’s the perfect place to put scripts that you want to run for all users.
Secondly, the script seems to assume that Splunk is running as root. As I’m not running Splunk as root, I needed to make a few changes to how the permissions are applied.
When I created the folder /var/log/bashhist, I ran a few extra commands to apply permissions for the splunk group:
sudo mkdir -m 1777 /var/log/bashhist
chown -hR root:splunk /var/log/bashhist
chmod g+s /var/log/bashhist
The chown command gives the splunk group access to the bashhist folder. The chmod command sets the group ID, which means that all new files and subdirectories inherit the group ID of the directory, rather than the primary group ID of the user who created the file.
I also altered the permissions on the line in the script that creates the bashhist subfolders. I changed the mkdir permissions from 700 to 750. 700 means the owner can read, write and execute. 750 also allows the owner to read, write and execute, and additionally allows the group to read and execute. This allows the splunk group to traverse folders(execute) and read the log files.
mkdir -m 750 $HISTBASEDIR/$EFFNAME >/dev/null 2>&1
Finally, I added a touch line to the script to ensure that new log files are created with the right permissions:
I was able to follow Duanne’s blog for the most part in this section. I have a distributed environment, so I split out props.conf into Heavyweight Forwarder and Search Head components. I also added the SEDCMD line in props.conf to obscure the password in any events containing -auth.
Heavyweight Forwarder props.conf:
TIME_PREFIX = ^#
TIME_FORMAT = %s
SEDCMD-password = s/(-auth[^:]+:)[^\s]+/\1xxxxx/g
EXTRACT-userids = ^\/var\/log\/bashhist\/(?<effective_user>[^\/]+)/history-(?<real_user>.*)$ in source
EXTRACT-command = \d+\n(?P<command>[^\n]+)
A restart of Splunk was required to apply the inputs.conf configuration.
Viewing the results
Now that the bash history is being ingested into Splunk, all that remains is to search for the events in Splunk.
Run the following search to confirm bash history is onboarded for all Splunk servers:
sourcetype=bash:history | dedup host | table host
This search generates a report of bash history, excluding read actions:
sourcetype=bash:history NOT (command=more* OR command=ll* OR command=cat* OR command="splunk btool*" OR command=find* OR command=ls* OR command=pwd* OR command=history OR command=exit)
| table _time host real_user effective_user command | sort host _time