Macs not fully enrolling / self service not installing / management framework missing

[sram]

Right. Where do I begin.... I am going to try and be as constructive as possible rather than just complaining. 

We've been running Jamf Pro (currently 11.4.2) for almost two years. Since day one we had issues with out of the box or EACAS Macs fully completing enrolment. The usual state they would end up in was for the profiles to install fine, VPP apps to install, etc, but for no policies to run nor for Self Service to install. When I looked on Jamf Nation and other sources I saw that this can sometimes happen and so because I'm a scripting god I came up with an amazingly complex script that is installed as part of an PreStage Profile's enrolment package workflow. The script does the following:

/usr/local/bin/jamf policy

This immediately caused enrolment to complete, the Self Service app appeared in /Applications and everything was once more right in the world. A reboot would achieve the same, but we didn't want to randomly reboot the Mac halfway through enrolment or ask the user to do so. This simply shouldn't be necessary!

We accepted this workaround for about a year because it Just Worked and I didn't need to involve Jamf Support. Until now...

(no, it's not a firewall issue and it's not DNS (or is it?) - the symptom is the same whether the devices are on the corporate network, tethered to an iPhone or sitting here at home with me on a bog-standard consumer broadband connection)

With one of the recent-ish OS updates, we saw a race condition whereby the enrolment package's attempt to call /usr/local/bin/jamf failed because the jamf binary had not actually been installed at the time the enrolment package's postinstall script was called. Rather than try to work around this with tests in the script to wait for the presence of the binary, I thought I'd try my first attempt at reaching around to Jamf Support. I logged a ticket during the first week of March and the issue is still unresolved. You would have thought that I've found a brand new issue that has completely baffled the most intelligent minds in Minneapolis.

<grumble>

Expand

I've been back and forth with the support rep, trying all kinds of wacky things that made me question their thinking. I know from looking through the archives on here that others have been offered suggestions similar to mine such as changing the port within the SMTP server settings.... Needless to say this made no difference. I have repeatedly asked to be called on the phone rather than trying to do everything via the unfriendly Service Now portal, but these requests just get passed over in replies. I then thought I'd be proactive and attempt to call Jamf UK's support line. Perhaps I missed something obvious but when I look on https://www.jamf.com/contact/ there is no UK support number listed. I scrolled down to office locations thinking that I'd just call a general UK number and speak to a human being to be pointed in the correct location. When I call the number listed at:

London
45 Mortimer Street
London W1W 8HJ
United Kingdom
Phone +44 (0) 208 164 4053

It is answered "Welcome to Wandera"... I pressed the keypad number for support but after ringing a few times I am told that there is nobody available.

</grumble>

Now, to be fair, in my time dealing with Jamf Pro I have rarely come across an issue that wasn't able to be resolved by good old fashioned logical troubleshooting or with the aid of the very helpful and generous community here. But symptoms identical or similar to mine keep coming up in search results with no reliable fix.

I spent some more time on the issue today and have discovered what I think is the cause of the stalled enrolment: After the Mac loads the Finder for the first time and drops you at the desktop, a LaunchDaemon named com.jamf.management.enroll.plist is present with the following contents:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>KeepAlive</key>
<dict>
<key>SuccessfulExit</key>
<false/>
</dict>
<key>Label</key>
<string>com.jamf.management.enroll</string>
<key>ProgramArguments</key>
<array>
<string>/usr/local/jamf/bin/jamf</string>
<string>postMdmEnrollment</string>
<string>-server_url</string>
<string>https://blahblah.jamfcloud.com</string>
<string>-invitation</string>
<string>215477778593833692970966644604081438781</string>
</array>
<key>RunAtLoad</key>
<true/>
<key>UserName</key>
<string>root</string>
</dict>
</plist>

The LaunchDaemon is being bootstrapped by whatever process is responsible for that, because it's listed in launchctl print (or lauchctl list for saner humans). `jamf postMdmEnrollment ...` is even listed as a process in ps, so it's been fired but isn't actually completing what it's supposed to do. The jamf log is of no help to me nor seemingly jamf support:

Wed May 01 14:45:07 MacBook Air jamf[984]: Removing existing launchd task /Library/Application Support/JAMF/tmp/com.jamfsoftware.task.startssh.plist...
Wed May 01 14:45:07 MacBook Air jamf[1012]: The SSL Certificate for https://blahblah.jamfcloud.com must be trusted for the jamf binary to connect to it.
Wed May 01 14:45:07 MacBook Air jamf[1012]: Enrolling computer...
Wed May 01 14:45:09 MacBook Air jamf[1012]: JMFCommons.JamfKeychain.JamfKeychainSecurityError.failedToReadJmfKeychainPassword
Wed May 01 14:45:09 MacBook Air jamf[1012]: JMFCommons.JamfKeychain.JamfKeychainSecurityError.failedToReadJmfKeychainPassword
Wed May 01 14:45:12 MacBook Air jamf[1012]: The device certificate was created successfully.

If the com.jamf.management.enroll.plist LD is manually bootout-ed and then bootstrapped into place then the expected stuff in /Library/Application Support/JAMF and Self Service appear as expected and enrolment is completed as expected. 

launchctl bootout system /Library/LaunchDaemons/com.jamf.management.enroll.plist 
launchctl bootstrap system /Library/LaunchDaemons/com.jamf.management.enroll.plist 

Post bootstrap log:

Wed May 01 14:45:07 MacBook Air jamf[984]: Removing existing launchd task /Library/Application Support/JAMF/tmp/com.jamfsoftware.task.startssh.plist...
Wed May 01 14:45:07 MacBook Air jamf[1012]: The SSL Certificate for https://blahblah.jamfcloud.com must be trusted for the jamf binary to connect to it.
Wed May 01 14:45:07 MacBook Air jamf[1012]: Enrolling computer...
Wed May 01 14:45:09 MacBook Air jamf[1012]: JMFCommons.JamfKeychain.JamfKeychainSecurityError.failedToReadJmfKeychainPassword
Wed May 01 14:45:09 MacBook Air jamf[1012]: JMFCommons.JamfKeychain.JamfKeychainSecurityError.failedToReadJmfKeychainPassword
Wed May 01 14:45:12 MacBook Air jamf[1012]: The device certificate was created successfully.
Wed May 01 23:02:17 Ben’s MacBook Air jamf[5080]: The SSL Certificate for https://blahblah.jamfcloud.com must be trusted for the jamf binary to connect to it.
Wed May 01 23:02:17 Ben’s MacBook Air jamf[5080]: Enrolling computer...
Wed May 01 23:02:23 Ben’s MacBook Air jamf[5080]: Creating user jadmin...
Wed May 01 23:02:28 Ben’s MacBook Air jamf[5080]: The device certificate was created successfully.
Wed May 01 23:02:40 Ben’s MacBook Air jamf[5080]: Error Domain=NSCocoaErrorDomain Code=4099 "The connection to service named com.jamf.management.daemon.binary was invalidated from this process." UserInfo={NSDebugDescription=The connection to service named com.jamf.management.daemon.binary was invalidated from this process.}
Wed May 01 23:02:40 Ben’s MacBook Air jamf[5080]: Removing existing launchd task /Library/LaunchDaemons/com.jamfsoftware.task.bgrecon.plist...
Wed May 01 23:02:40 Ben’s MacBook Air jamf[5080]: Downloading the Jamf Bundle...
Wed May 01 23:02:43 Ben’s MacBook Air jamf[5080]: Enforcing management framework...
Wed May 01 23:02:43 Ben’s MacBook Air jamf[5080]: Enforcing scheduled tasks...
Wed May 01 23:02:43 Ben’s MacBook Air jamf[5080]: Adding launchd task com.jamfsoftware.task.1...
Wed May 01 23:02:44 Ben’s MacBook Air jamf[5080]: Updating daemon settings
Wed May 01 23:02:44 Ben’s MacBook Air jamf[5080]: Flushing the /Library/Application Support/JAMF/tmp directory was successful
Wed May 01 23:02:44 Ben’s MacBook Air jamf[5080]: Upgrading jamfHelper.app...
Wed May 01 23:02:45 Ben’s MacBook Air jamf[5080]: Upgrading JAMF notification service...
Wed May 01 23:02:45 Ben’s MacBook Air jamf[5080]: Upgrading Self Service.app...
Wed May 01 23:03:03 Ben’s MacBook Air jamf[5080]: Setting Self Service icon to custom
Wed May 01 23:03:03 Ben’s MacBook Air jamf[5080]: Removing existing launchd task /Library/Application Support/JAMF/tmp/com.jamfsoftware.task.policy.plist...
Wed May 01 23:03:03 Ben’s MacBook Air jamf[5080]: Enroll return code: 0
Wed May 01 23:03:03 Ben’s MacBook Air jamf[5080]: Enrollment Complete. Exiting.
Wed May 01 23:03:03 Ben’s MacBook Air jamf[5855]: Checking for policies triggered by "enrollmentComplete" for user "ben"...

(the sudden jump in time is the timezone changing from Cupertino to GMT)

So..... do I keep digging in an attempt to find out why the jamf postMdmEnrollment command stalls or do I just knock up a quick enrolment package script to bootout and bootstrap the LaunchDaemon (perhaps testing/waiting until its presence but running the risk of unloading it in the middle of the original jamf postMdmEnrollment completing successfully on its own - it hasn't happened yet but a random Jamf Pro or OS update could fix things). Given that I'm like a dog with a bone I'd like to resolve the problem 'properly' without having to resort to a workaround, but I appreciate that nothing in life is perfect!

Thanks for reading and apologies for the length of post.

[jamf-42]

seems you are not alone... is this on prem or cloud?  Maybe see if you get the same errors with your sandbox jamfcloud instance.. 

https://community.jamf.com/t5/jamf-pro/jmfcommons-jamfkeychain-jamfkeychainsecurityerror/m-p/310086

[jtrant]

I have been battling this for months with no clear pattern. Sometimes it will fail at creating the mangement account, other times that will succeed but Self Service will fail to download.

Very strange but I will for sure try your workaround!

[jtrant]

This has resolved any issues with enrollments failing, thanks again!

[sram]

That's great to hear, but what did you do exactly? 😁

[nicolaswojczak]

hoping to see your updates on this.

[MatG]

We are also getting numerous Macs that have been working an enrolld for weeks, months, years have these errors but tis seems to be on Jamf upgrade.

We have to manually kill the binary on the Mac, however this may not fix as after a hour, day or few days we see the same devices so the same.

So far no fix from Jamf

Fri May 17 13:43:21 C02FD2JNXXXX jamf[51423]: Error Domain=NSCocoaErrorDomain Code=4099 "The connection to service named com.jamf.management.daemon.binary was invalidated from this process." UserInfo={NSDebugDescription=The connection to service named com.jamf.management.daemon.binary was invalidated from this process.}
Fri May 17 14:01:54 C02FD2JNXXXX jamf[58397]: Error Domain=NSCocoaErrorDomain Code=4099 "The connection to service named com.jamf.management.daemon.binary was invalidated from this process." UserInfo={NSDebugDescription=The connection to service named com.jamf.management.daemon.binary was invalidated from this process.}
Fri May 17 14:20:37 C02FD2JNXXXX jamf[65359]: Error Domain=NSCocoaErrorDomain Code=4099 "The connection to service named com.jamf.management.daemon.binary was invalidated from this process." UserInfo={NSDebugDescription=The connection to service named com.jamf.management.daemon.binary was invalidated from this process.}

[JPost]

I am experiencing this too however running the above terminal commands did not help. I do see the policies applied on the Mac but something else I have noticed is that the policy's are showing as notI have re-enrolled twice. Still not getting the Self Service to install either. When I go into policies, and look at a log that has All computers assigned and all users assigned, the device I enrolled does not even show as applicable. I am seeing that my profiles are showing as Unverified so I will look at my cert to see if something is happening there.

Device - Sonoma
Version of Jamf Connect - 2.29
Using SYM for Prestage Script

[FutureFacinLuke]

Seeing this on some re-purposed Macs, "Device Signature Error" if you run jamf policy. Turns out they were Mx Macs which had been 'Erased' in Recovery rather than via MDM Command, Settings or Firmware recovery.

Last time I saw similar was when there was an On-Prem-Cloud migration between rebuilds.

I've recently done a CA renewal so I'm guessing that an Erase in Recovery doesn't fully clear out the secure enclave or trigger re-activation like the other methods would and there's some old junk left in there that breaks re-enrollment.