Restarting the Jamf binary when Macs stopping checking in

[jamesandre]

Howdy... I’ve detected an issue where the Jamf binary stops checking in, this can be for hours, days, weeks or even months. This is evident when Macs have run their Inventory Update for "x days". It would appear that the Jamf binary begins its check-in process but never completes, this stops any further check-in attempts as the process is still running and wont attempt to check-in until the original process has completed. If you attempt to manually check-in you will get a similar error to:

This policy trigger is already being run: root 88591 0.0 0.0 34245156 1048 ?? Ss 21Jul22 0:03.85 /usr/local/jamf/bin/jamf policy -stopConsoleLogs -randomDelaySeconds 300

I suspect this issue is caused by a network interuption when a policy is running or a script within a policy that cannot complete (the softwareupdated process has been hanging on some versions of macOS Big Sur). There does not appear to be a time-out for the Jamf binary.

CasperCheck or the new Jamf-Management-Framework-Redeploy API function do not resolve ths issue. Killing the Jamf binary resolves the issue and the Mac can check-in with the jamf server again. Since getting access to a Mac that isn't checking-in is can be somewhat difficult, I have made something I am calling "Jamf Restart" which lifts ideas and code from CasperCheck and AppProcessKiller.

Jamf Restart consists of:

• A Launch Daemon ( /Library/LaunchDaemons/com.expample.jamfRestart.plist) that will run the script once a day.
• A script (/Library/Scripts/jamfRestart.sh) that checks if the process has been running for 1 day or more.
• A log file (/var/log/jamfRestart.log) that captures that output of the script.
• An Extension Attribute (Jamf Restart) that reads the log and displays if the Jamf binary has been killed.

The LaunchDaemon and Script can be packaged and deployed via Jamf (a lot easier to do when all Macs are checking-in). You can launch the LaunchDaemon with the command

/bin/launchctl load /Library/LaunchDaemons/com.example.jamfRestart.plist

Once Jamf Restart has been deployed to a Mac, it will check if the process for the Jamf binary has been running for more than 1 day, if the Jamf binary has been running for more than a day, it will kill the process. After the Jamf binary has been killed, the next scheduled check-in will run correctly. Any policy run from Jamf can reasonably be expected to complete within 1 day, so killing the process when in it has been running for so long will not stop any policy with any chance of success from completing.

Jamf Restart will not fix the "Device Signature Error” which stops the Jamf binary from running, I have been testing the Jamf-Management-Framework-Redeploy API function for that.

Jamf Restart should ensure Macs keep checking into Jamf, and will allow you to identify which Macs have had issues checking in, so they can be investigated further.

The LaunchDaemon:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>Label</key>
	<string>com.example.jamfrestart</string>
	<key>ProgramArguments</key>
	<array>
		<string>sh</string>
		<string>/Library/Scripts/jamfRestart.sh</string>
	</array>
	<key>RunAtLoad</key>
	<false/>
	<key>StartInterval</key>
	<integer>86400</integer>
</dict>
</plist>

The Script:

#!/bin/bash

processRuntime=$(ps -ax -o user,pid,etime,args | grep "/usr/local/[j]amf/bin/jamf policy -stopConsoleLogs -randomDelaySeconds 300" | awk '{ print $3; }' | grep -o '.*[-]' | awk -F\- '{print $1}')
processCheck=$(ps -ax -o user,pid,etime,args | grep "/usr/local/[j]amf/bin/jamf policy -stopConsoleLogs -randomDelaySeconds 300" | awk '{ print $2; }')

logLocation="/var/log/jamfRestart.log"
scriptLogging(){

    DATE=`date +%Y-%m-%d\ %H:%M:%S`
    LOG="$logLocation"
    
    echo "$DATE" " $1" >> $LOG
}

	if [ "${processRuntime}" = "" ]; then
        scriptLogging "JamfBinary has not run for more than 1 day"
    else
        scriptLogging "JamfBinary has run for ${processRuntime} days"
        scriptLogging "JamfBinary Process ID: ${processCheck}"
        scriptLogging "Quitting JamfBinary..."
        sudo kill -9 ${processCheck}
    fi
    
exit 0

The Extension Attribute:

#!/bin/bash

jamfRestart=`/usr/bin/tail -10 /var/log/jamfRestart.log | grep "JamfBinary has run for"`

if [ "$jamfRestart" != "" ]; then
   echo "<result>$jamfRestart</result>"
else
   echo "<result>No Restarts</result>"
fi

Hopefully this is of use to someone.

[joser3]

For the "Load failed: 5: Input/output error" I found out the issue.
There's a typo on the LauncDaemon where:
"<key>Label</key>
<string>com.example.jamfrestart</string>"

needs to be:
"<string>com.example.jamfRestart</string>"
The "r" is lowercase and needs to be the uppercase "R".

[kacey3]

Change that final line to:

launchctl bootstrap system "/Library/LaunchDaemons/com.example.jamfRestart.plist"

That should fix it.

[itinspectorio]

 dont know if I fixed the issue, but I dont have any errors any more. Can someone tell me if this is a good result of policy? 

Executing Policy Jamf Binary Restart

Running script Jamf binary Restart...

Script exit code: 0

Script result:

[itinspectorio]

@jamesandre Could you please help validate the latest process? As I understand there is issue with new MacOS to execute sudo /bin/launchctl load /Library/LaunchDaemons/com.example.jamfRestart.plist

Did you try on Sonama and Ventura OS? 

[jamesandre]

Yes, this is working on Sonoma, you can check the jamfRestart.log in Console. You will get the Load failed: 5: Input/output error if the Launch Daemon has already been loaded. 

[kacey3]

For anyone still looking to deploy this, I took what @MCfreiz put together and added some variables to make it a bit more adaptable. There are also some additional operations to do prep and post work. Other changes were made to bring the code up to compliance with current changes to MacOS.

#!/bin/bash

##########################################################################################
#
# Written by KClose
# Based on original scripts by "JamesAndre" and "MCfreiz"
#
# This script simply writes out the required script and launchdaemon 
#
##########################################################################################

### VARIABLES ###
scriptDir="/Library/Scripts/JamfAT"
scriptFile="$scriptDir/jamfRestart.sh"
daemonFile="/Library/LaunchDaemons/com.unt.at.jamfrestart.plist"

### MAIN SCRIPT ###

# Check for Script Directory.
if [[ ! -d "$scriptDir" ]]; then
	mkdir -p "$scriptDir"
fi

# Cleanup old files for recreation.
if [[ -e "$scriptFile" ]]; then
	rm -f $scriptFile
fi

if [[ -e "$daemonFile" ]]; then
	launchctl bootout system $daemonFile
	rm -f $daemonFile
fi

# Write out the jamfRestart script.
cat << 'EOF' > $scriptFile
#!/bin/bash

### VARIABLES ###

processRuntime=$(ps -ax -o user,pid,etime,args | grep "/usr/local/[j]amf/bin/jamf policy" | awk '{ print $3; }' | grep -o '.*[-]' | awk -F\- '{print $1}')
processCheck=$(ps -ax -o user,pid,etime,args | grep "/usr/local/[j]amf/bin/jamf policy" | awk '{ print $2; }')
logLocation="/var/log/jamfRestart.log"

### FUNCTIONS ###

scriptLogging(){
    
    DATE=`date +%Y-%m-%d\ %H:%M:%S`
    LOG="$logLocation"
    
    echo "$DATE" " $1" >> $LOG
}

### MAIN SCRIPT ###

if [ "${processRuntime}" = "" ]; then
    scriptLogging "JamfBinary has not run for more than 1 day"
else
    scriptLogging "JamfBinary has run for ${processRuntime} days"
    scriptLogging "JamfBinary Process ID: ${processCheck}"
    scriptLogging "Quitting JamfBinary..."
    sudo kill -9 ${processCheck}
fi

exit 0
EOF

# Set owenership and permissions on jamfRestart script.
chmod 644 $scriptFile
chown root:wheel $scriptFile

# Write out LaunchDaemon.
cat << EOF > $daemonFile
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>Label</key>
	<string>$(basename $daemonFile | sed 's/.plist//')</string>
	<key>ProgramArguments</key>
	<array>
		<string>sh</string>
		<string>$scriptFile</string>
	</array>
	<key>RunAtLoad</key>
	<false/>
	<key>StartInterval</key>
	<integer>86400</integer>
</dict>
</plist>
EOF

# Set ownership and permissions on LaunchDaemon.
chmod 644 $daemonFile
chown root:wheel $daemonFile

# Load LaunchDaemon
launchctl bootstrap system "$daemonFile"

[JevermannNG]

@kacey3 what are the macOS Version requirements? Thank you!

[kacey3]

The main thing I added was "launchctl bootout system" to unload any previously existing Launch Daemons of the same name. This particular command line was first introduced in MacOS 10.11. Ideally, this script should be compatible with all current versions of MacOS (10.11 and beyond).

Another simple change I made was I made the grep command a bit more "flexible" as it seems like there are a couple of different outputs when jamf policy gets stuck. I also have the script check for existing scripts and launch daemons created by previous policies and delete them rather than overwriting them.

Lastly, I moved some of the script variables to the top, allowing admins to more easily customize the name and location of the script as well as the name of the Launch Daemon.