Deploying Splunk Universal Forwarder on Macs/OSX

[ndeal]

Anyone had to do this before? If so, mind sharing how you did it? When using the basic instructions from Splunk to install, a plist file is put (by the splunk service when calling an enable-boot command) into /Library/LaunchAgents which isn't going to work well for gathering logs from system.log when logging in as a standard user account, and in fact it fails to launch at all. Per some internet searches, I placed the plist in LaunchDeamons which seems to work but is causing some concerning login behaviors (delay, screen blanks out for 10 seconds, etc). Wanted to see if anyone had done this before and I'm just missing something about the "proper" way to do this. Before you ask why we'd want to install the forwarder on individual endpoints, let me just clarify that this is not my idea.

[gachowski]

@ndeal

I did try this a few year ago but kinda gave up on the "Splunk Universal Forwarder" after realized that you can have the logs forwarded straight from the OS with adding a "special" app.

That said that was before apple change to the new universal logging so I have to read up on that, but here is the old code that worked for me to send to a Splunk test server... we could see in real time when a usb drive was plugged in. In the end we didn't' use it..

#!/bin/sh
# Stopping syslog to edit
#sudo launchctl unload /System/Library/LaunchDaemons/com.apple.syslogd.plist

#Adding sshd module to syslog need for full CIS syslog fowarding
syslog -module com.openssh.sshd enable 1

# Restarting syslog
sudo launchctl load /System/Library/LaunchDaemons/com.apple.syslogd.plist

and much later in my config script I used this

# Appending the following line to syslog.conf for syslog fowarding
echo "*.*   @XX.XX.X.XXX:9997" >> /etc/syslog.conf

# Starting and stoping syslog.plist for syslog fowarding
launchctl unload /System/Library/LaunchDaemons/com.apple.syslogd.plist

#sleep 3

launchctl load -w /System/Library/LaunchDaemons/com.apple.syslogd.plist

This was from 3 or 4 years ago so not sure if it's still working with HS and newer version of Splunk

Hope this gets you started in the right direction...

C

[hulsebus]

Haven't done this before, but I'm in the same boat trying to get it to work.

All of the reports I've found of people getting it to to work with 10.12+ (with the unified logging system) required a "glue" script that would dump out logs (or a subset) into a text file that the forwarder is watching. Again, haven't done this myself (yet...) but I'm looking at/testing options.

[merps]

Just had to go through this on Mojave - haven't yet tried it on Catalina but I've seen in the Splunk forums that people are having issues with unsigned binaries there.

Anyway, I was having tons of issues getting a silent install using the .pkg installer provided by Splunk - due to TCC and the inability to whitelist binaries because they aren't signed, etc. Ended up finding an install script for Linux in the Splunk forums and adapted it to work for our needs. There are two components needed in addition to the .tgz download from Splunk.

1. deploymentclient.conf
2. user-seed.conf

deploymentclient.conf looks like this:

[deployment-client]

[target-broker:deploymentServer]
targetUri = ip.addr.to.splunk:8089

user-seed.conf is generated with an already installed splunk binary:

sudo /opt/splunkforwarder/bin/splunk hash-passwd password_here

and the file looks like this:

[user_info]

USERNAME = splunkuser
HASHED_PASSWORD = $9$this_is_a_long_string_provided_by_the_command_above_the_rest_is_random.C.EoL5jgk74jFmPljaidjshadjduejskcKDHiSHiskslclOS.oIHDhkxezKBDLMiahdEdu88dcD.

The 3 files (splunkforwarder*.tgz, deploymentclient.conf, user-seed.conf) are delivered to /var/tmp by the install package.

Here's a screenshot of the installer built with Packages, but Composer would also be fine.

Finally, here is the the postinstall script that's working as of today.

#!/bin/sh

# NOTE: script assumes there is exactly ONE splunkforwarder*.tgz package already present in /tmp/splunkforwarder*.tgz
clear;
set -x;

cd /opt/;export SPLUNK_HOME=/opt/splunkforwarder
tar xvf /var/tmp/splunkforwarder*.tgz
cd ./splunkforwarder/

# kill splunkd if running

PID=$(pgrep splunkd)

if [ ! -z "$PID" ]
    then
        killall splunkd
fi

# Generate Deployment Client app
mkdir -p ${SPLUNK_HOME}/etc/system/local/

cp /var/tmp/deploymentclient.conf "${SPLUNK_HOME}"/etc/system/local/deploymentclient.conf

cp /var/tmp/user-seed.conf "${SPLUNK_HOME}"/etc/system/local/user-seed.conf

# Splunk First-Time-Run (FTR)
${SPLUNK_HOME}/bin/splunk start --accept-license --auto-ports --no-prompt --answer-yes

sleep 3
# NOTE: This part MUST BE RUN AS root (or sudo)!
# NOTE: If running splunk as non-root, add "-user splunk" to the argument list of "enable boot-start"


"${SPLUNK_HOME}"/bin/splunk enable boot-start
launchctl unload /Library/LaunchDaemons/com.splunk.plist

sleep 3

launchctl load /Library/LaunchDaemons/com.splunk.plist

# Ownership probably does not need to be changed because
# probably script is being run as correct user which may be root or splunk or other.
# sudo chown -R splunk: 
${SPLUNK_HOME}/bin/splunk status

exit $?

[MichaelMcG]

Hey, so I know this is quite old, but I have been using a version of your script and it's worked perfectly, but now in Ventura, when the 9.0.3 installer runs from the .tgz my users are being prompted that the app was downloaded from the internet and they need to click install to continue, and man I have been trying for a week or so now to find a solution? Are you having any issues on Ventura? 

[obi-k]

Hi,

Is anyone able to install Splunk Universal Forwarder 8.2 on 10.15 Macs? Here is my package, which puts all the files in /tmp directory. I use the Jamf to run the shell script and to clean up the mess in /tmp after.

Errors in script seem to be opening the .tar file in Jamf:

Details

[STEP 1 of 5]
Executing Policy Splunk 8.2.0 (10.15+)
[STEP 2 of 5]
Mounting xxxxxxxxxxxxxxx
Installing Splunk 8.2.0.pkg...
Successfully installed Splunk 8.2.0.pkg.
[STEP 3 of 5]
Running script Install and Remove Splunk files...
Script exit code: 0
Script result:

##################################

Welcome to the Splunk 8.2 Universal

Forwarder auto-installer

for Mac.

Last updated 06/04/2021.

##################################

/private/tmp/Splunk/splunkUF_bash_install_script-mac.sh: line 22: /opt/splunkforwarder/bin/splunk: No such file or directory
Creating Splunk Linux OS account
/private/tmp/Splunk/splunkUF_bash_install_script-mac.sh: line 25: useradd: command not found
cat: ospwdfile: No such file or directory
passwd: illegal option -- -
usage: passwd [-i infosystem] -l location]] [-u authname] [name] infosystem: file NIS OpenDirectory PAM location (for infosystem): file location is path to file (default is /etc/master.passwd) NIS location is NIS domain name OpenDirectory location is directory node name PAM location is not used

Installing Splunk Universal Forwarder
chmod: splunkforwarder-8.2.0-e053ef3c985f-darwin-64.tgz: No such file or directory
tar: Error opening archive: Failed to open 'splunkforwarder-8.2.0-e053ef3c985f-darwin-64.tgz'
mkdir: /opt/splunkforwarder/etc/apps: No such file or directory

/private/tmp/Splunk/splunkUF_bash_install_script-mac.sh: line 45: runuser: command not found
chown: splunk: illegal group name
/private/tmp/Splunk/splunkUF_bash_install_script-mac.sh: line 49: /opt/splunkforwarder/bin/splunk: No such file or directory
cp: splunkpwdfile: No such file or directory
chown: splunk: illegal group name

chown: splunk: illegal group name
chmod: /opt/splunkforwarder/etc/splunk-launch.conf: No such file or directory

/private/tmp/Splunk/splunkUF_bash_install_script-mac.sh: line 59: runuser: command not found

Splunk test start and stop complete. Enabled Splunk to start at boot. Also, adjusted splunk-launch.conf to mitigate privilege escalation attack.

mv: rename Splunk_ACLs to /etc/logrotate.d/: No such file or directory
chown: root: illegal group name
Setting *nix log files facl to allow Splunk to read
/private/tmp/Splunk/splunkUF_bash_install_script-mac.sh: line 74: /usr/bin/setfacl: No such file or directory
/private/tmp/Splunk/splunkUF_bash_install_script-mac.sh: line 75: /usr/bin/setfacl: No such file or directory
/private/tmp/Splunk/splunkUF_bash_install_script-mac.sh: line 76: /usr/bin/setfacl: No such file or directory
/private/tmp/Splunk/splunkUF_bash_install_script-mac.sh: line 77: /usr/bin/setfacl: No such file or directory
/private/tmp/Splunk/splunkUF_bash_install_script-mac.sh: line 78: /usr/bin/setfacl: No such file or directory
/private/tmp/Splunk/splunkUF_bash_install_script-mac.sh: line 79: /usr/bin/setfacl: No such file or directory
/private/tmp/Splunk/splunkUF_bash_install_script-mac.sh: line 80: /usr/bin/setfacl: No such file or directory
/private/tmp/Splunk/splunkUF_bash_install_script-mac.sh: line 81: /usr/bin/setfacl: No such file or directory
/private/tmp/Splunk/splunkUF_bash_install_script-mac.sh: line 82: /usr/bin/setfacl: No such file or directory
/private/tmp/Splunk/splunkUF_bash_install_script-mac.sh: line 83: /usr/bin/setfacl: No such file or directory
Splunk Universal Forwarder installation has FAILED install!
[STEP 4 of 5]
[STEP 5 of 5]
Unmounting file server...