The every15 trigger not triggering

[llitz123]

I have a few policies using every15 trigger... Here is one example with 25 computers remaining:
external image link

Here is another example with 39 computers remaining:
external image link

It's now 12:07pm with no activity. I restarted the Casper server among other things earlier, yet it didn't seem to help.
Can I please get some help troubleshooting?
Thanks.

[justinrummel]

On one of your test clients you can run
[code]sudo jamf policy -trigger every15 -verbose[/code]

It may give reasons why the client is not triggering.

[mm2270]

You may also want to check on one of the clients that isn't running the policies to see if the actual LaunchDaemon is running. We have seen odd issues where the LaunchDaemon stops running and needs to be kicked in the pants to get it going. I'm not sure what causes that, but again, have seen it very sporadically. We may have some more cases of this we just discovered and will troubleshoot.

sudo launchctl list | grep jamf

You should see a launchd in the policy named something like: com.jamfsoftware.task.Every 15 Minutes

If you don't see that, the daemon probably isn't active and that's your issue. I hope its not though since this is a tricky one to fix remotely. If thy aren't checking in, there's no easy way to drop them into a Smart Group with, say, an Extension Attribute.

[llitz123]

sudo jamf policy -trigger every15 -verbose on one test machine that isnt working returned:

verbose: Checking for an existing instance of this application...
This policy trigger is already being run: root      
1393   0.0  1.0   749992  41248   ??  Ss    8:24AM   0:10.94 /usr/sbin/jamf policy -action every15 -randomDelaySeconds 300

I tried sudo launchctl list | grep jamf on 2 test machines returned:

1792    -   com.jamfsoftware.task.Every 15 Minutes
55  -   com.jamfsoftware.jamf.daemon

It looks like most if not all of my active test machines ran the policies, yet production machines didn't.
Any other ideas?
Thanks for your help and any further assistance.

[llitz123]

For kicks, I checked the time on the Casper server and client and the times are fine.
I had one production machine log out and then log in as a new user and the every15 policies ran...
I guess I can run a reboot command tonight and see if that works, yet it would be nice to know what the problem is.
Thanks.

[mm2270]

Look at the jamf log in Console (shows up under /var/log) You can also just tail the log file.

See if those Macs are in the process of running a long succession of policies based on the every15 trigger. It might have gotten stuck on something, like a recon stage. Really only one policy run can happen at any given time, so if they haven't finished their current policies already in play, any other ones will need to wait until its done.

[llitz123]

I did run a software update policy last night and it's very possible the machines in question didnt restart/logout for some reason and are hung so that makes sense. I'll try and reboot all the machine tonight and see how it goes.
Thanks.

[llitz123]

It does look like the software update policy from last night may be the culprit. I tailed the log file:

Tue Mar  5 18:48:54 Client1214 jamf[1477]: Checking for policies triggered by "every15"...
Tue Mar  5 19:04:28 Client1214 jamf[1538]: Checking for policies triggered by "every15"...
Tue Mar  5 19:19:58 Client1214 jamf[1601]: Checking for policies triggered by "every15"...
Tue Mar  5 19:31:06 Client1214 jamf[1665]: Checking for policies triggered by "every15"...
Tue Mar  5 19:46:41 Client1214 jamf[1729]: Checking for policies triggered by "every15"...
Tue Mar  5 20:01:15 Client1214 jamf[1792]: Checking for policies triggered by "every15"...
Tue Mar  5 20:01:15 Client1214 jamf[1792]: Executing Policy Install Apple Software Updates...
Tue Mar  5 20:01:15 Client1214 jamf[1792]:  Installing all available Software Updates...
Wed Mar  6 08:45:25 Client1214 jamf[2655]: Checking for policies triggered by "login" for user "staffuser"...
Wed Mar  6 08:45:25 Client1214 jamf[2655]: Executing Policy Get Casper Usernames...

every15 trigger looks like it stopped after the software update, yet the login trigger seems to work... I checked the logs from the "Install Apple Software Updates" policy and Client1214 isn't even listed in the "Install Apple Software Updates" policy logs from Tue Mar 5 20:01:15.
Thanks for any further assistance.

[llitz123]

So it looks like as people are logging out for the day, the policies are running.
Is there any way I can troubleshoot back to exactly what happened?
Thanks again!

[mm2270]

Again, the /var/log/jamf.log on the Macs may reveal something. Since no logs were being uploaded to the JSS while the issue was happening, your only real hope of backtracking will be on the clients themselves.

Another thought, do you have different subnets or locations you're talking about here? If so, is it possible one location or subnet just wasn't properly seeing the JSS for a period of time?

[llitz123]

I appreciate the idea yet we are all on the same subnet. I'll see what I can find from the client end. If anything shows up I'll post back here. Thank everyone for your help.

[mm2270]

Sure. I was just wondering out loud. Looks like you can essentially rule out a network connectivity issue then. I would capture some of those logs as soon as you can to see what turns up.

Good luck and keep us posted.

[Chris_Hafner]

How have you assigned the policy to said computers? Not that I doubt something is getting hung on you in certain circumstances. However, it looks like you've got some computers properly running the every15 trigger and not finding any policies to execute. Well, at least it looks that way from what's posted.

[llitz123]

2 policies I'm having issues with are for Apple Java update and Flash update running on 10.6.8 machines. The clients which are not running these new policies are all 2010 MacBooks on the same subnet as the JSS server.

I assigned Apple Java policy using Smart Computer Group.
For the Java update I used Available SWUs has JavaForMacOSX10.6-14.0.
I assigned Flash policy using an EA and I specified specific departments (all except IT and Servers basically):
EA "Flash Version" is not 11.6.602.171
Both policies have now run on most machines as users have logged in and logged out, yet the policies were set for every15. During troubleshooting I changed the Java policy to "any" trigger and it didnt seem to help.

For reference, this is the EA I stole from the forums:

#!/bin/sh
#
############################################################################
#
# Extension Attribute checks to display Adobe Flash Player Version number.
#
#
############################################################################
FlashPluginVersion=`/usr/bin/defaults read /Library/Internet Plug-Ins/Flash Player.plugin/Contents/Info CFBundleVersion`
echo "<result> $FlashPluginVersion </result>"

exit 0

[llitz123]

I should mention I asked 3 users to reboot their machines to see what happened. Two ran the policies without issue after restart. One client rebooted and didn't run the policies.
I ran the sudo launchctl list | grep jamf on the client which didn't run the policies and this is what came up after reboot:

Client1218 (192.168.111.98)
213 -   0x10041dc00.anonymous.jamf
1111    -   com.jamfsoftware.task.Every 15 Minutes
59  -   com.jamfsoftware.startupItem
60  -   com.jamfsoftware.jamf.daemon

[Chris_Hafner]

Can you verify that the "Flash Version" in the computer records is NOT listed as 11.6.602.171 (for the units still not grabbing the policy). It's just that this 'feels' like a weird recon issue. I only say this because pretty much everything seems to be setup and running properly (as per the logs posted here). Obviously there's still a problem with those units.

[llitz123]

"Flash Version" was not listed as 11.6.602.171 in any of the problem computers.
Update: I was able to get a hold of one of the problem MacBooks. Symptom was it was hanging on shutdown. I ran all sorts of utilities (AppleJack/TTP/Apple HW/Disk Warrior) and found no serious errors and tried a variety of other common troubleshooting techniques.
I booted into safe mode and noticed that the Activity Montor was lit red for both processors. The main process was a child process softwareupdate with parent jamf. So I let is sit and watched in console as both policies ran without issue. I rebooted a few times and all seems fine. I have one more critical client machine that isnt running the policies. If I can get it, I'll boot in safe mode and see how it goes. I'll report back here if anything new comes up.
Thanks.

[CAG_1337]

I am having a similar issue with all my older Macs (over 300 of them and running 10.6.8). It seems like my software update policy (triggered by every15 after midnight) is hanging. No other every15 policies run after that, of course. And when I try to manually run software updates on such a machine from the command line, I get this:

Updates are being installed by uid 0

And and when I do a sudo jamf policy -trigger every15 -verbose, I get this:

This policy trigger is already being run: root 355 0.0 0.1 105848 4664 ?? Ss 3:06AM 0:00.25 /usr/sbin/jamf policy -action every15 -randomDelaySeconds 300

So the machines are basically like this until I reboot the computer. The next time my software update policy triggers sometime after midnight, same thing happens, night after night. I don't think it is a problem with the Apple Updates per se, because after I reboot I the machines, I can manually run software updates from the command line and they work beautifully. It is just that when Casper tries to run software updates, it hangs and hangs forever. Here is an example from a jamf log:

Sat Mar 9 03:09:22 XXXXXX jamf[355]: Checking for policies triggered by "every15"...
Sat Mar 9 03:09:22 XXXXXX jamf[355]: Executing Policy Install All Apple SW Updates...
Sat Mar 9 03:09:22 XXXXXX jamf[355]: Installing all available Software Updates...

And that's where the log ends...

Doing manual intervention on them isn't really an option. Like I said, there are over 300 of them and they are scattered across two campuses in two cities, Minneapolis and St. Paul.

[llitz123]

Well that is definitely a symptom of what is happening with me with the software updates not completing and freezing the every15 trigger.
Another update. I was able to access the last critical computer that was having this issue. I safe booted and Casper ran the stuck every15 update with no intervention from me. I may need to open a support ticket with JAMF to troubleshoot this out as I still have machines having issues. I'll post back here if I find anything helpful.
Thanks.

[mm2270]

Just curious, since this was never asked before on the thread, but what version of the Casper Suite is your JSS running?

[llitz123]

v8.62

[mm2270]

OK, I ask because we just recently updated our Casper Suite server to 8.63 and for some odd reason it seems to have addressed some sporadic issues we've noticed with the jamf binary getting hung up on some of the longer processes. I can't really be sure its related of course, and the 8.63 release notes only mention the fix for Restricted Software and non ascii characters.

You could try updating your JSS to see if it helps.

[llitz123]

Yeah I may try that. I have never done a JSS version update.
I'm a Casper noob struggling through all setups. When I get it working it works great, yet I don't know all the ins and outs...
For example, my issue could be the way I'm running items. All the clients having issues are mobile clients which are in and out of the office and on and off various networks all day long.
My Java update package is a .pkg from the Apple Support or Oracle. And the Force Distribution Points to use AFP/SMB instead of HTTP is unchecked as this needs to run on remote mobile clients outside the corporate subnet.
My Flash update is a ,dmg which I created using guidance from the manual and these forums. The Force Distribution Points to use AFP/SMB instead of HTTP is unchecked as this policy also needs to run on remote mobile clients outside the corporate subnet.
These policies do work on some remote client machines yet others haven't been updating. It could be because of issues outside my control such as unmanaged remote networks...
Would it be better to cache updates somehow or are there some tricks of the trade I'm missing?
Thanks for any assistance.

[CAG_1337]

It seemed to start when I upgraded to 8.62 a couple of months ago. I upgraded to 8.63 last week to see if it would help, and it didn't. Definitely seems like a bug with Casper to me. I can consistently reproduce it. I can even fire up Casper Remote and kick off a software update and get the same hang as when the policy triggers after midnight. Running software updates manually from the command line works fine though.

Fortunately for me, all these old 10.6 machines are leased and going away in the summer, so it is hard to be too worried, but still...

[acostj]

Not sure if anyone has a fix for this but found a way to kill the every15 trigger if it has been running for more than 1 day. Built a policy that runs on "any trigger", "once a day" and runs the command via advanced tab.

It returns the process id and then kills it via one line. If the every15 is running normally it will not kill the process.

ps -eo pid,etime,args | grep -i jamf | grep every15 | awk '$2~/-/ {if ($2>0) print $1}' | sudo xargs kill