Unable to communicate with iOS 11.3

[wpzr]

Hi,

We have JSS version 9.101.0-t1504998263 . This morning after updating from 11.2.6 to 11.3 . I can no longer communicate with iPads that are on newer version. Is anyone else having a similar issue?

[RobbieG]

Happened to us too - we were running 10.2.2 in the cloud and the issue persists after they upgraded us to 10.3.0-t1521825567. Apparently the JSS is still trying to send the update but the iPads won't take it because they don't need it.

[miregan]

The JSS is trying to send which update?

[RobbieG]

iOS 11.2.2 to 11.3 - It seems that in some cases it starts the update but communication is severed prior to completion - it can no longer communicate with the ipad so it doesn't "know" that it doesn't need so it keeps trying and everything else just lines up behind it. We are on Jamf cloud so we are borked until they fix it in Jamfland.

[wpzr]

Here is an example for me. It started an update, and just haven't checked back in. It doesn't know that update is finished.
We do have Single app mode profiles and etc that were turned off prior to doing an update. Now those profiles wouldn't reinstall back since there is no connection

[miregan]

Ah okay. You are talking about updating the iOS of the devices through the JSS. I thought you were referencing configurations profiles etc or commands in general.

[snourse]

We are seeing the same thing. Wiping and re-enrolling the device causes it to work fine afterwards. Existing devices that update to 11.3 have mostly stopped communicating--- we have 6 still checking in and around 6500 not checking in. We just had the server updated to 11.3 from 9.101 Thursday.

[jcline]

We have seen it too and the issue has been reported to apple. "From what i hear apple messed up the code and is working on a fix"

[miregan]

Do the devices work consistently after a wipe and re-enroll or is it a persistent issue?

[SBergman]

There was some discussion on this last night with @bradleychambers on Twitter. I confirmed it with a test device today also.

https://twitter.com/bradleychambers/status/980908183613997058

[michael_devins]

Jamf is investigating an issue related to the MDM Software Update workflow on iOS 11.3. While we don't have all of the details just yet, I want to share what we know for transparency:

MDM Software Updates are composes of three distinct steps:
1. Send an 'AvailableOSUpdates' command to determine what is available for a device
2. Send a 'ScheduleOSUpdate' command to tell the device what to install
3. Send an 'OSUpdateStatus' command to monitor download/installation status and completion

The response for 'OSUpdateStatus' includes a key for 'IsDownloaded', which should return a boolean True/False. We are investigating an issue where the 'OSUpdateStatus' is returning a '0' instead of a boolean True/False. This is causing a failure and affecting subsequent MDM communication. We have filed Radar 39116010 with Apple regarding this response change and our teams are investigating ways that we can handle these scenarios if the response does not conform to the documented standards.

[hyprmatt]

Does this affect JSS 10.3 or just prior versions? Also is there a way to figure out which iOS devices are affected by this?

[RobbieG]

@miregan They seem fine after wiping and re enrolling - the bug seems to basically just block the queue so that no new commands after the OSUpdate are sent.

[RobbieG]

@hyprmatt We were on Jamf 10.2.2 and another poster was on 9.101 - all our iPads were on 11.2.6

[nstrauss]

"We filed a radar."

What I want to see is, "We will meet to improve our QA process to avoid scenarios like this in the future."

This is a major bug. Yes, due to a change (maybe undocumented) Apple made. Jamf still did not do their due diligence in testing. Hire me to do QA. I'll press all the buttons before shipping a release.

[hyprmatt]

@RobbieG Maybe 10.3 is safe then. I haven't seen any issues yet but most of our devices are still on 11.2.6.

[RobbieG]

@hyprmatt The issue happens when you push the 11.3 iOS update from Jamf - the update completes on the device but the JSS thinks it is still in process so all new management commands get stuck behind whatever the JSS is still chewing on. As long as you don't push the iOS update until they get the issue sorted you should be fine no matter what version of Jamf you are on.

[hyprmatt]

I see. What if I cancel all pending/failed commands. Does that fix it or is there something under the hood I can't correct for without Jamf providing the fix?

[CasperSally5432]

I get mistakes happen, whether it's jamf or apple, but I don't understand why Jamf would not send an email by now to all customers saying to hold off on 11.3 to lessen the potential blow (not that customers shouldn't test pushing the update in dev, and/or in test group in production, too).

[RobbieG]

@hyprmatt Canceling the pending commands does not fix it - so far the only thing that does is wiping and re-enrolling the iPads

[michael_devins]

Hey everyone. Thank you so much for your rapid feedback on this issue. I've posted an update about an upcoming maintenance release of Jamf Pro that will address this problem here.

[jcline]

This isn't just on 10.3 we are still on 9.101 and it happens to us. I've heard it's an iOS issue

[FastGM3]

We're on Jamf Pro 10.2.2 and seeing this issue occur on devices all the way down to iOS 9.3.5 but not all of the devices. The reset on each iPad re-enrolls the device and gets the 3rd Party apps back. I have a support call in about 10 minutes hopefully we'll learn a little more.

[miregan]

Im seeing far more issues on 10.3 than just ios update commands.

[michael_devins]

@FastGM3 The issue exists for devices that get upgraded to iOS 11.3, regardless of Jamf Pro version or previous iOS version. Once the iOS device is upgraded, Jamf Pro attempts to validate that the upgrade is complete. This is the step that is failing.

[FastGM3]

Hi Michael,

Does the iPad actually get the iOS update and then the MDM problem appears? Because our iPads that lost enrollment still have iOS versions older than 11.3. 9.35 to 11.3

Thanks,
Chuck

[miregan]

I havent issued the iOS update commands from the JSS. All of my ipads are on 11.2.5 and commands either take forever to actually apply on the ipad, (Erase device has taken upwards of 45 minutes after consistently sending blank pushes) or they dont take at all. As far as i know this has only started happening since yesterday when our Jamf Cloud instance was upgraded to 10.3. I have checked with out networking team as well and have looked through the firewall logs and nothing is being blocked or has changed.

[michael_devins]

@FastGM3 When you send the MDM Software Update command to upgrade to iOS 11.3, the device should go through the upgrade properly and the device should be usable by end users. The 'OSUpdateStatus' command that I outlined above is then designed to verify the upgrade is complete. Because that command fails on iOS 11.3, it prevents any subsequent commands from completing successfully. That includes update inventory, which would update your reported iOS version. That's why no devices are reporting their successful upgrade to iOS 11.3. With the upcoming maintenance release, that failed response will be accepted and devices will resume normal behavior as managed devices.

[miregan]

I havent issued any iOS update commands though

[michael_devins]

@miregan I would recommend contacting Jamf Support so we can help determine what might be causing the performance issues you're describing.

[miregan]

Done already. It literally started happening after the 10.3 upgrade yesterday

[michael_devins]

Today, we released Jamf Pro 10.3.1. See discussion and release notes for full details.

[hendersong]

In the same boat...... Doing support with Jamf, they recommended upgrading to JSS 10.5: issue persist. Can't even do an OTA update on the iPad itself, it fails. Never even downloads the iOS, message shows on the iPads the update request failed. I've had to connect iPads to a computer and do the update to iOS 11.4. That seems to fix the issue and all works as expected after the iPads are updated to 11.4. Now I have to locate and do this for my 1600 out of 2200 devices having issues........