Machines not checking in with JSS

[JasonAdams]

I called JAMF Support yesterday and still working with them, but wanted to throw out a lifeline so to speak if anyone else has the same issue.
We have about 75 MacMini's running a mix of 10.7.5 and 10.8.4, they are all used as Digital Signage. So the machines never move, and are always connected to the LAN, never shut down, etc, etc.
However they have been losing connection with the JSS. So yesterday I went thru and connected via SSH and ran sudo jamf recon on the machines that hadnt checked in greater than 1 day.
So the recon runs successfully, however I cant get the JSS inventory to update. I have stopped and restarted Apache and MySql, but it still shows the old inventory date. So how do I know they have checked in? I have it set to show the Antivirus date, so the machines are reporting current AV defs in the JSS, but not current dates of last report in/checked in times.
Has anyone else ran into this?
JSS is 8.64 running on a OS X 10.7.5 server

[mm2270]

Can you monitor these systems to see if the jamf binary is getting hung up during the regularly scheduled recons? I know you said you can run a manual recon, but you may want to look at the /var/log/jamf.log file on these Macs before doing anything to force them to check in to see what's actually going on and what the last process run on them was.

Why am I saying this? Because we've seen this happen as well and tracked it down to cases of the inventory collection getting stuck and seemingly never completing, or else timing out. At the time we were running JSS 8.63, but recently upgraded to 8.71 which so far seems to have resolved this for us. I don't know that 8.64 would have still had the same issue, but its possible.

[jhalvorson]

Any chance the time on the Digital Signage computers are not close to the time on the JSS? Usually the recon will fail if you have set it to enforce the maximum clock skew and the client times are drifting. (The time drift issue happens on our Windows devices that are 24X7X365, but not so much on our Macs.) You mentioned the recon was successful, so that is a bit strange the report time is not updated.

On one of the failing devices, try ssh'ing to it and then using the following command: sudo jamf enroll

[Chris_Hafner]

Just a quick check. Run the following

sudo jamf manage

just to see what comes up.

[donmontalvo]

When JAMF went to "enrollment" on their QuickAdd, it caused this problem for many of the Macs we manage...in several environments. The only way we were able to get the Macs re-enrolled was to run sudo jamf enroll -prompt and authenticate with the appropriate account. Me thinks the enrollment process is inherently fragile, I really miss the old way (even if it meant having the management account name and password hash in the QuickAdd). :)

[JasonAdams]

Thanks for the responses... to reply on the time, I dont have time skew enforced on the JSS, however all our Macs in our environment use the Windows DC that is the PDC emulator. Why? Because that time server is the same time server for Windows devices as well so we ensure all devices OS neutral have the same time.
For sudo jamf manage, when I run that from one of the Mac Minis, I see the response "The management framework will be enforced as soon as all policies are done executing." However when I refresh the JSS the machine check in date doesnt change, but something strange is when I look at the Computer Information its showing /usr/sbin/jamf version 8.62 even though the JSS is on 8.64.
And for Don's post, when I run "sudo jamf enroll -prompt" I use the creds of the service acct we have on all our JAMF connected Macs thats only used from JAMF and recon. I see the response back the computer was successfully enrolled to the JSS with the following device certificate... When I wait 5 + minutes and go back to the JSS, the record still hasnt updated, still showing Last Contact Time of 03/18/2013. However Report Date is showing 07/30/2013.
This is under Details, Computer Information.
Just to note back to Don, I have run sudo jamf enroll on several of the impacted Mac Minis, they will successfully re-enroll, but never update/show a correct checked in date in the JSS. If that info helps as well.

[donmontalvo]

@JasonAdams A couple things I'd try. First, I'd delete one of the problem Macs from JSS, then run sudo jamf removeFramework, then install QuickAdd again and see if the Mac comes in with correct Last Contact and Last Report. After the above is done, maybe trigger a policy to make sure you're able to push to it. We've had problems with a few (not many) Macs that were resolved this way.

[Joel_Peasley]

Jason,

There is a difference between Last Contact Time and Last Report Date. Last Contact Time is updated anytime a computer checks in on its regular check in frequency. To simulate this you can run the command "sudo jamf log" and this should update Last Contact Time. Last Report Date is referencing the last time that the computer did a full inventory update. This occurs when a computer is enrolled either by command line or the Recon Application. It can also occur when you have the check box for update inventory checked in a policy. Hopefully this helps clear up any questions you have regarding those two items.

Thanks,
Joel

[donmontalvo]

sudo jamf policy works too.

[Chris_Hafner]

Have you checked to see if there's an inventory report getting stuck on those units? At some point last year I ended up having to shut off the "gather mobile devices" checkbox in "Inventory Collection preferences" for a while for a similar reason. One of the mobile apps in iTunes was causing the inventory to hang. Can't quite remember what I did to sort it out. I'll happily dig through my notes if you end up looking in that direction.

[acostj]

What version of JSS are you running? We had this issue with 8.63 and so far moving to latest has helped but 8.64 made a note that Application collection was a problem and we saw that it would hang systems and they could go on for days with it hung. Once we moved to 8.71 we have seen weird issues with reports and also the hanging die down.

Ditto on the report getting hung also. Good Luck!

[bsuggett]

We are experiencing similar issues here @ Sydney Uni.

Inventory collection isn't working too well when collecting directory bindings. The status shows "Not Bound" even though machines are bound, network accounts are available and working. It takes at least 2 days for it to correct itself in inventory.

Caspersuite 8.71

[JPDyson]

Run ```
tail -50 /var/logs/jamf.log
``` and investigate... something is likely hung up, and the log may tell you what.

[RobertHammen]

What happens if you ssh into the client, and type:

sudo jamf recon -verbose

[JasonAdams]

Thanks everyone for the responses, I have been working with Mitch @ JAMF Support, and have a couple things to report.
I have updated the JSS to 8.71. Has it made a difference? Unfortunately no.
But here is what Mitch had me try running on the machines that no longer check in. So I SSH into them and run the below commands:
Ps auxx | grep jamf
Sudo kill “processID” (locate the process ID for the policy –action every 15, it will be the first number after root)
Sudo launchctl unload /Library/LaunchDaemons/com.jamfsoftware.task.3.plist
Sudo jamf manage –verbose
Sudo jamf policy
Sudo jamf recon

After running them, they do check in and the report and last contact dates show current.
However a day later the machines have fallen back off the cliff so to speak. If you reboot the machine it may check back in for that point in time and then stop again.
I havent made any changes to the JSS from an inventory standpoint, nor adding any extension attributes etc.
I also have never had the "Gather mobile devices" option checked. (Settings, Inventory Options, Inventory Collection Preferences).
And I have never had anything checked under the Software tab for Inventory Collection.

What is strange is if I am on one of the impacted machines and I run "ps auxx | grep jamf" I dont see the every15 process running.
If I manually run it "sudo jamf policy -trigger every15" it does run. It just doesnt stay running in memory/running process.
And looking in /var/log/jamf.log doesnt show anything of value in regards to errors, warnings, stop messages, etc.

Forgot to add, even restarting the machines doesnt kick back off the every15 trigger.
Am I crazy, but this should always be running when you type "ps auxx | grep jamf" right?

[acdesigntech]

if you manually load the jamfsoftware daemon, what happens?

sudo launchctl load -w /Library/LaunchDaemons/com.jamfsoftware.jamf.daemon.plist

[JasonAdams]

@acdesigntech....
If I run via SSH, sudo launchctl load -w /Library/LaunchDaemons/com.jamfsoftware.jamf.daemon.plist
I get the warning/error/message, that "com.jamfsoftware.jamf.daemon: Already loaded"

If I then run "sudo launchctl load -w /Library/LaunchDaemons/com.jamfsoftware.task.3.plist"
It runs, so I then type "ps auxx | grep jamf" but do not see the every15 running.

[bbass]

I'm curious what's returned if you do this:

ls -a /Library/LaunchDaemons/ | grep jamf

We had a similar situation that was caused by a malfunctioning LaunchDaemon:

com.jamfsoftware.task.checkForTasks.plist

If that's listed I can probably offer some help.

[JasonAdams]

@bbass...
When I connect to one of the F'd machines, and run "ls -a /Library/LaunchDaemons/ | grep jamf"
this is returned:
com.jamfsoftware.jamf.daemon.plist
com.jamfsoftware.startupItem.plist
com.jamfsoftware.task.3.plist

[bbass]

How does that list compare to machines that seem to be working OK?

I ask because we don't have com.jamfsoftware.jamf.daemon.plist in our /Library/LaunchDaemons folders. We are running v8.64.

[acdesigntech]

Hmmm, can you post the results of a ```
sudo launchctl unload -w /Library/LaunchDaemons/com.jamfsoftware.jamf.daemon.plist
```?

Then reload it and post what happens (also results of the ps command. Just for giggles, also try a ```
ps -ax | grep jamf
``` to see if you get the every15 task as a result.

Can you also post the contents of your com.jamfsoftware.startupitem.plist? For example, mine is

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.jamfsoftware.startupItem</string>
    <key>LaunchOnlyOnce</key>
    <true/>
    <key>RunAtLoad</key>
    <true/>
    <key>ProgramArguments</key>
    <array>
        <string>/Library/Application Support/JAMF/ManagementFrameworkScripts/StartupScript.sh</string>
    </array>
</dict>
</plist>

And because even the simplest things can trip up the most experienced sys admin - have you tried resetting PRAM on the affected Macs? ALso, have you tried unplugging one of the Macs, letting it sit for about 5 minutes, then plugging back in and starting up? It sounds dumb, I know, but might be worth a try to reset the PMU.

[RobertHammen]

Also "sudo jamf checkJSSConnection" to make sure you clients can connect to the JSS.

[flyboy]

I've taken over for @JasonAdams][/url on this one, and want to get things restarted on this particular topic. All the Mac Minis in question are running a mixture of 10.7.5 and 10.8.x. Current JSS version is 9.2. The upgrade to 9.22 is in the works, just not completed yet. Below is some data specific to an example machine that has the problem. I have a work-around, but would of course like a permanent fix. For any machine that has stopped communicating, I can kill the "jamf policy -randomDelaySeconds..." process. A second one is immediately spawned, kill that one too. Running "sudo jamf policy" then completes successfully. However, it's a gamble as to when the machine will stop checking in.

uptime:
11:52 up 53 days, 4:32, 2 users, load averages: 1.77 1.50 1.21

Running JAMF processes:
root 90479 0.0 0.1 730184 1444 ?? Ss 21Dec13 6:51.33 /usr/sbin/jamf policy -randomDelaySeconds 300
root 11331 0.0 0.1 653572 1956 ?? SN 17Dec13 366:04.32 /usr/sbin/jamfAgent
root 11320 0.0 0.0 654360 888 ?? SNs 17Dec13 2:23.37 /usr/sbin/jamf launchDaemon -monitorUsage -enforceRestrictions -monitorNetworkStateChanges

JAMF binary version: 9.2

OS Info:
ProductName: Mac OS X
ProductVersion: 10.7.5
BuildVersion: 11G63b

Hardware Overview:

Model Name: Mac mini Model Identifier: Macmini5,1 Processor Name: Intel Core i5 Processor Speed: 2.3 GHz Number of Processors: 1 Total Number of Cores: 2 L2 Cache (per Core): 256 KB L3 Cache: 3 MB Memory: 2 GB Boot ROM Version: MM51.0077.B10 SMC Version (system): 1.76f0

jamf.log excerpt:
Sat Dec 21 19:11:27 C07J707XDJD0 jamf[87184]: Installing all available Software Updates...
Sat Dec 21 19:25:20 C07J707XDJD0 jamf[88046]: Checking for policies triggered by "recurring check-in"...
Sat Dec 21 19:25:21 C07J707XDJD0 jamf[88046]: Executing Policy My Company Today Policy - Weekly Patch Policy for My Company Today - every15 ongoing - NO Reboot...
Sat Dec 21 19:25:21 C07J707XDJD0 jamf[88046]: Setting Software Update Server to http://chapdt3mac01.ad.mycompany.com:8088/index.sucatalog for all accounts...
Sat Dec 21 19:25:21 C07J707XDJD0 jamf[88046]: Installing all available Software Updates...
Sat Dec 21 19:43:37 C07J707XDJD0 jamf[88919]: Checking for policies triggered by "recurring check-in"...
Sat Dec 21 19:43:39 C07J707XDJD0 jamf[88919]: Executing Policy My Company Today Policy - Weekly Patch Policy for My Company Today - every15 ongoing - NO Reboot...
Sat Dec 21 19:43:39 C07J707XDJD0 jamf[88919]: Setting Software Update Server to http://chapdt3mac01.ad.mycompany.com:8088/index.sucatalog for all accounts...
Sat Dec 21 19:43:39 C07J707XDJD0 jamf[88919]: Installing all available Software Updates...
Sat Dec 21 19:45:04 C07J707XDJD0 jamf[88919]: Executing Policy My Company Today Policy - Weekly Restart for Apple Remote Desktop - NO Reboot...
Sat Dec 21 19:45:04 C07J707XDJD0 jamf[88919]: Setting Software Update Server to http://chapdt3mac01.ad.mycompany.com:8088/index.sucatalog for all accounts...
Sat Dec 21 19:45:04 C07J707XDJD0 jamf[88919]: Installing all available Software Updates...
Sat Dec 21 19:56:50 C07J707XDJD0 jamf[90479]: Checking for policies triggered by "recurring check-in"...
Sat Dec 21 19:56:52 C07J707XDJD0 jamf[90479]: Executing Policy My Company Today Policy - Weekly Patch Policy for My Company Today - every15 ongoing - NO Reboot...
Sat Dec 21 19:56:52 C07J707XDJD0 jamf[90479]: Setting Software Update Server to http://chapdt3mac01.ad.mycompany.com:8088/index.sucatalog for all accounts...
Sat Dec 21 19:56:52 C07J707XDJD0 jamf[90479]: Installing all available Software Updates...

[bsuggett]

We've encountered
On OSX10.8.x built from scratch then enrolled, the machines check in fine up until they go to sleep @ which point checkins stop occurring until woken up either by physically going up to the machine or by using ARD, Casper remote or SSH to the machines.

To resolve this we've disabled sleep all together to get around this issue.

[bsuggett]

We've encountered
On OSX10.8.x built from scratch then enrolled, the machines check in fine up until they go to sleep @ which point checkins stop occurring until woken up either by physically going up to the machine or by using ARD, Casper remote or SSH to the machines.

To resolve this we've disabled sleep all together to get around this issue.

[bsuggett]

We've encountered
On OSX10.8.x built from scratch then enrolled, the machines check in fine up until they go to sleep @ which point checkins stop occurring until woken up either by physically going up to the machine or by using ARD, Casper remote or SSH to the machines.

To resolve this we've disabled sleep all together to get around this issue.

[flyboy]

If it was only that easy. These are true machines, they never sleep.

[NightFlight]

I manage /Library/LaunchDaemons with my own custom script distribution repository tool. Long story short, I accidentally wiped jamf services.

For the fix:

root@itch (0):~# jamf version
version=9.7
root@itch (0):~# ls /Library/LaunchDaemons|grep jamfsoftware
root@itch (1):~# jamf manage
Getting management framework from the JSS...
Enforcing management framework...
Checking availability of https://casper.xxxxxxx.com:8443/...
The JSS is available.
Enforcing scheduled tasks...
Creating launch daemon...
Creating launch agent...
Checking availability of https://casper.xxxxxx.com:8443/...
The JSS is available.
root@itch (0):~# ls /Library/LaunchDaemons|grep jamfsoftware
-rw-r--r--   1 root  wheel   777B 24 Apr 16:06 com.jamfsoftware.jamf.daemon.plist
-rwxr--r--   1 root  wheel   474B 24 Apr 16:06 com.jamfsoftware.startupItem.plist
-rw-r--r--   1 root  admin   528B 24 Apr 16:06 com.jamfsoftware.task.1.plist
root@itch (0):~# ps x|grep jamf
96960   ??  SNs    0:00.27 /usr/sbin/jamf launchDaemon -monitorUsage -enforceRestrictions -monitorNetworkStateChanges
97007   ??  SN     0:00.09 /usr/sbin/jamfAgent
97088 s001  S+     0:00.00 grep jamf
root@itch (0):~#

[user-lFdRjzkONM]

Jumping on this thread...

Random endpoints managed by our Jamf Pro instance are not checking in/performing an inventory update for even more than 10 days.

Our ad-hoc solution is to request the user to run a policy from the Self Service app which forces Inventory Update (Policies -> Maintenance -> Update Inventory)

Does anyone have an idea regarding what causes it and how do we eliminate the issue for the long term?