enrollmentComplete issues with network state change

[ftiff]

I'm having issues with enrollmentComplete trigger since 9.97:

enrollmentComplete policies fail to execute (as if they were not scoped)

A blackout period was introduced (PI-003515) to ensure that a policy would not execute twice. In practice, if a policy fails, you won't be able to run it again until the blackout period expires. A good way to test is to launch a policy from the self service, cancel it, and try to run it again.
Workaround: Edit and Save policies to reset the counter.
From 9.100.0 you can reach support and ask them to change the blackout period.

NetworkStateChange makes enrolment freeze

In jamf.log it looks like this:

Fri Aug 25 00:42:06 FLevaux’s MacBook jamf[923]: Checking for policies triggered by "enrollmentComplete" for user "flevaux"... Fri Aug 25 00:42:11 FLevaux’s MacBook jamf[904]: Informing the JSS about login for user flevaux Fri Aug 25 00:42:12 FLevaux’s MacBook jamf[904]: Network state changed, checking for policies... Fri Aug 25 09:42:15 FLevaux’s MacBook jamf[1109]: Checking for policies triggered by "networkStateChange" for user "flevaux"... Fri Aug 25 09:58:24 FLevaux’s MacBook jamf[1323]: Checking for policies triggered by "recurring check-in" for user "flevaux"...

in the process list:

FLevauxs-MacBook:~ flevaux$ ps aux|grep jamf root 1049 0.0 0.2 2496900 18500 ?? SN 9:42AM 0:00.10 /usr/local/jamf/bin/jamfAgent root 923 0.0 0.5 2508988 44320 ?? S 9:42AM 0:00.65 /usr/local/jamf/bin/jamf policy -action enrollmentComplete root 904 0.0 0.4 2503368 35720 ?? SNs 9:42AM 0:00.19 /usr/local/jamf/bin/jamf launchDaemon -monitorUsage -monitorNetworkStateChanges root 477 0.0 0.5 2505716 45620 ?? S 9:41AM 0:01.03 /usr/local/jamf/bin/jamf enroll -invitation 84381567390768067806887290803661559097 root 444 0.0 0.0 2452852 2420 ?? Ss 9:41AM 0:00.01 /bin/bash /usr/local/jamf/bin/enroll flevaux 1285 0.0 0.0 2432804 1408 s001 R+ 9:43AM 0:00.00 grep jamf flevaux 1271 0.0 0.0 2432796 1752 s000 S+ 9:42AM 0:00.00 tail -f /var/log/jamf.log FLevauxs-MacBook:~ flevaux$

/usr/local/jamf/bin/jamf policy -action enrollmentComplete will never finish. But other triggers will execute, such as Recurring Check-in.

It always stops at:

Thread 0x5a3c DispatchQueue 1 1000 samples (1-1000) priority 20 (base 20) 1000 start 1 (libdyld.dylib 21045) [0x7fffaeb7d235] 1000 main 591 (jamf 936799) [0x10737eb5f] 1000 -[JBCommand validateAndExecute] 135 (jamf 33415) [0x1072a2287] 1000 -[PolicyCommand execute] 54 (jamf 2211926) [0x1074b6056] 1000 -[PolicyController execute] 162 (jamf 2732114) [0x107535052] 1000 -[PolicyController executePolicies] 355 (jamf 2732723) [0x1075352b3] 1000 -[PolicyController executePolicy:] 58 (jamf 2733546) [0x1075355ea] 1000 -[Policy execute] 479 (jamf 387967) [0x1072f8b7f] 1000 -[Policy doPolicyStartUserInteractionAndAskUserAboutDefer] 785 (jamf 397377) [0x1072fb041] 1000 -[PolicyUserInteraction displayInformationalMessage] 187 (jamf 2496011) [0x1074fb60b] 1000 -[MessageDisplayService showInformationMessage:] 131 (jamf 2506579) [0x1074fdf53] 1000 -[MessageDisplayService showInformationMessageWithTitle:andMessage:] 236 (jamf 2505804) [0x1074fdc4c] 1000 -[NotificationCenterDisplayStrategy showInformationMessageWithTitle:andMessage:] 494 (jamf 2508382) [0x1074fe65e] 1000 -[JAMFTask execute:asUser:error:] 206 (jamf 3015864) [0x10757a4b8] 1000 -[JAMFTask execute:andAnswerPromptWith:error:] 74 (jamf 3015977) [0x10757a529] 1000 -[JAMFTask execute:andAnswerPromptWith:error:ignoreStderr:writeToConsole:] 1501 (jamf 3017518) [0x10757ab2e] 1000 -[NSConcreteFileHandle readDataOfLength:] 498 (Foundation 804604) [0x7fff9aeb26fc] 1000 read 10 (libsystem_kernel.dylib 111174) [0x7fffaecad246] 1000 hndl_unix_scall64 22 (kernel 634262) [0xffffff800029ad96] 1000 unix_syscall64 597 (kernel 6439157) [0xffffff80008240f5] 1000 read_nocancel 115 (kernel 5878083) [0xffffff800079b143] 1000 ??? (kernel 5878746) [0xffffff800079b3da] 1000 ??? (kernel 5898680) [0xffffff80007a01b8] 1000 ??? (kernel 5794979) [0xffffff8000786ca3] 1000 lck_mtx_sleep 132 (kernel 1015028) [0xffffff80002f7cf4] 1000 thread_block_reason 222 (kernel 1057470) [0xffffff80003022be] 1000 ??? (kernel 1062043) [0xffffff800030349b] 1000 machine_switch_context 206 (kernel 2070942) [0xffffff80003f999e] Thread 0x5ac1 Thread name "com.apple.NSURLConnectionLoader" 1000 samples (1-1000) priority 20 (base 20) 1000 thread_start 13 (libsystem_pthread.dylib 12429) [0x7fffaed9608d] 1000 pthread_start 286 (libsystem_pthread.dylib 14471) [0x7fffaed96887] 1000 _pthread_body 180 (libsystem_pthread.dylib 14651) [0x7fffaed9693b] 1000 NSThread__start 1243 (Foundation 207021) [0x7fff9ae208ad] 1000 [NSURLConnection(Loader) _resourceLoadLoop:] 313 (CFNetwork 26420) [0x7fff9853a734] 1000 CFRunLoopRunSpecific 420 (CoreFoundation 553236) [0x7fff993fd114] 1000 __CFRunLoopRun 1361 (CoreFoundation 555201) [0x7fff993fd8c1] 1000 __CFRunLoopServiceMachPort 212 (CoreFoundation 558132) [0x7fff993fe434] 1000 mach_msg_trap 10 (libsystem_kernel.dylib 74570) [0x7fffaeca434a] 1000 ipc_mqueue_receive_continue 0 (kernel + 850128) [0xffffff80002cf8d0]

Only way to recover is to manually:

sudo killall jamf
sudo /usr/local/jamf/bin/jamf policy -action enrollmentComplete

(the same with the first issue above would not do anything).

In our environment, the computers are setup with DEP and initial setup is done over wifi.

[Chris]

+1 for enrollmentComplete not running/getting killed by networkStateChange.
>> PI-004144

[jmariani]

We also intermittently see this issue in our environment. No resolution from jamf, they weren't sure what the cause was.

[ftiff]

Thank you!

Jamf confirmed it was indeed the PI @Chris mentioned.

It seem that it happens at every enrolment on 10.13. Is this something you can reproduce?

[cybertunnel]

We too have the issue in our environment. During DEP enrollment, even if I flush the policies, and try again, it still only runs the policies originally ran on the system and not the others. This usually consists of ~ 10 policies before the enrollmentComplete trigger dies/freezes.

[bpavlov]

We are also effected by PI-004144 and it really is frustrated. Now, we've had to rely on running our first boot script on a USB drive because of this bug. I swear, every time we try to come up with something to make Jamf Pro usable, we run into a bug that just makes life harder for us. It's quite frustrating and disappointing. We were temporarily using the network state change trigger but have found that this was not working well for us either.

I don't understand why they would have made such a change with this "black out policy time period" without mitigating against this particular issue with triggers and how they're used.

[nvandam]

I have the same issue. If the enrollment is done over ethernet, everything works great, but on wifi is like throwing darts in the dark. Are deployment strategy relies on DEP enrollment working reliably. We can't guarantee the end user will be on ethernet and need to know that enrolling on wifi is just as reliable.

[rblaas]

We have this issue as well. (also on 10.11 / 10.12) I have create a workaround where I don't have enrollment policies But I manualy cal a policy which in turn calls all the enrollment scripts.. (yes it is not neat... )

[franton]

Yes, there is zero resiliency on this trigger where there desperately needs some. Anything that interrupts the wifi causes the entire trigger to fail ... and it wasn't exactly reliable with four or more policies attached to it.

The policy timeout needs to go, as that is particularly infuriating during testing.

An example. You first boot a Mac, it comes up with the wifi prompt at setup assistant. You attach wifi and because of cert issues, it asks you to trust the certificate chain. Ok, you go through it picks up DEP and goes ahead. You get to the point of user first logon .. macOS drops the wifi and the user account logs in.

At this point bad things happen. In the background, the jamf dep enrolment process is running but in the user foreground, the user is again prompted to trust the same set of wifi certificates because they're stored per user not per system. The issue is until the certs are manually trusted again, there is no network connectivity but the background enrolment process continues on and just fails. There's no sign in the jamf logs of the binary retrying, it just fails. You end up with a device that has a jamf binary, eventually gets check in policies but anything scoped to enrolment complete fails hard.

And that's my deployment process broken because of wifi certs I can't get fixed at my end, and an enrolment policy that doesn't retry over and over until it gets a valid server connection.

Please fix!

[tig]

franton exactly describes the things happening to me. Like this, DEP enrolment over WiFi is practically unusable for me.

[sepiemoini]

Seeing this as well but it appears that folks have/@ftiff has identified this as PI-004144. I'd like to throw my log file into the mix as well.

Fri Nov 03 09:32:25 compName jamf[1389]: Checking for policies triggered by "enrollmentComplete" for user "administrator"...
Fri Nov 03 09:32:26 compName jamf[1389]: Executing Policy 01 Enrollment: Computer Naming
Fri Nov 03 09:32:32 compName jamf[1377]: Network state changed, checking for policies...
Fri Nov 03 09:32:32 compName jamf[1377]: Network state changed, checking for policies...
Fri Nov 03 09:32:33 compName jamf[1389]: Executing Policy 02 Enrollment: Splash Buddy

[MLBZ521]

I have seen and have had Site Admins report symptoms that were exactly like the behavior reported in this discussion for months now.

I hadn't looked into it much until recently while testing DEP workflows more in depth and I see this is the same issue.

I've seen the -trigger networkStateChange axe a currently running -trigger enrollmentComplete process.

And also when a Network Configuration Profile is provided and resets the current connection during a DEP workflow, the enrollmentComplete process stops and never completes.

[Cem]

PI-003515 vote-up here!

[jmariani]

Throwing my hat back in the ring for this issue. We are also hit with this "networkstatechange" issue - PI-004144. I don't have an exact number but I would bet it's close 50% of our DEP enrolments failing due to the PI above.

We just had our staging Jamf Pro server upgraded to 10.2 and the first 3 DEP enrolments I tested all failed with the same issue. Checking for policies triggered by "networkStateChange" for user "first_lastname"...

According to Jamf this is going to require an "enrolment complete trigger" code re-write. No ETA on this yet.

Jamf please help!!