iOS Mass Action Update Failures

[cstout]

Happy Saturday, JN.

I've been working on and off with support regarding this for several months now and still don't have a solid answer so I thought I'd throw it out here to see what is being experienced out in your environments:

My questions to JN are:
1. Are any of you seeing the same behavior?
2. What have you found to be the most reliable procedure to follow regarding mass iOS updates?

Scenario: Apple releases a new iOS version.
My basic process: When I have a smart group or a regular inventory search and choose to perform an action on the listed iOS devices, I choose Action->Send Remote Commands->Update OS Versions on supervised devices->Latest version & Download and install the update chosen->Next (Finish)

Pretty straightforward, right? Well, for the last two iOS releases I've had about a 10% success rate with this process.

My test iPads, which have no passcode set, simply log the following and never actually process my update request. I've had only marginally better results by specifying the iOS version instead of choosing latest available, but even then have seen the same results.

Here is what I see on the management command history:

OSUpdateStatus  2 minutes ago   
OSUpdateStatus  17 minutes ago  
OSUpdateStatus  32 minutes ago  
OSUpdateStatus  47 minutes ago  
OSUpdateStatus  About an hour ago   
OSUpdateStatus  Today at 8:14 AM    
OSUpdateStatus  Today at 7:59 AM    
OSUpdateStatus  Today at 7:44 AM    
OSUpdateStatus  Today at 7:29 AM    
OSUpdateStatus  Today at 7:14 AM    
OSUpdateStatus  Today at 6:59 AM    
OSUpdateStatus  Today at 6:44 AM    
OSUpdateStatus  Today at 6:29 AM    
OSUpdateStatus  Today at 6:14 AM    
OSUpdateStatus  Today at 5:59 AM    
OSUpdateStatus  Today at 5:44 AM    
OSUpdateStatus  Today at 5:29 AM    
OSUpdateStatus  Today at 5:14 AM    
OSUpdateStatus  Today at 4:59 AM    
OSUpdateStatus  Today at 4:44 AM    
OSUpdateStatus  Today at 4:29 AM    
OSUpdateStatus  Today at 4:14 AM    
OSUpdateStatus  Today at 3:59 AM    
OSUpdateStatus  Today at 3:44 AM    
OSUpdateStatus  Today at 3:29 AM    
OSUpdateStatus  Today at 3:14 AM    
OSUpdateStatus  Today at 2:59 AM    
OSUpdateStatus  Today at 2:44 AM    
OSUpdateStatus  Today at 2:29 AM    
OSUpdateStatus  Today at 2:14 AM    
OSUpdateStatus  Today at 1:59 AM    
OSUpdateStatus  Today at 1:44 AM    
OSUpdateStatus  Today at 1:29 AM    
OSUpdateStatus  Today at 1:14 AM    
OSUpdateStatus  Today at 12:59 AM   
OSUpdateStatus  Today at 12:44 AM   
OSUpdateStatus  Today at 12:29 AM   
OSUpdateStatus  Today at 12:14 AM   
OSUpdateStatus  Yesterday at 11:59 PM   
OSUpdateStatus  Yesterday at 11:44 PM   
OSUpdateStatus  Yesterday at 11:29 PM   
OSUpdateStatus  Yesterday at 11:14 PM   
OSUpdateStatus  Yesterday at 10:59 PM   
OSUpdateStatus  Yesterday at 10:44 PM   
OSUpdateStatus  Yesterday at 10:29 PM   
OSUpdateStatus  Yesterday at 10:14 PM   
OSUpdateStatus  Yesterday at 9:59 PM    
OSUpdateStatus  Yesterday at 9:44 PM    
OSUpdateStatus  Yesterday at 9:29 PM    
OSUpdateStatus  Yesterday at 9:14 PM    
OSUpdateStatus  Yesterday at 8:59 PM    
Schedule OS Update - iOSUpdate16E227    Yesterday at 8:48 PM    
AvailableOSUpdates  Yesterday at 8:48 PM    
Update Inventory    Yesterday at 3:39 PM

[cstout]

@nsdjoe I always use latest based on eligibility. I've only ever used "specific version" when troubleshooting issues like this.

[CorpIT_eB]

@cstout This may not just be limited to iOS devices as I tried to do this for macOS patch 10.14.5 via JAMF's Actions and it did absolutely nothing had to create a policy to update them.

So we will see what JAMF Pro 10.13 brings.

More importantly how is everyone seeing the results or logs of those mass actions?

[nsdjoe]

I wish I could say my results were like that of @cstout , but unfortunately I'm seeing the same thing as @ejculpepper .

We updated to JAMF Pro 10.13.0 this morning. I cleared all pending and failed commands. Selected a device that was on iOS 12.2 and sent the iOS update to "Latest version based on device eligibility." It's been almost 2 hours and I'm still seeing the "OSUpdateStatus" command completing every 15 minutes.

JAMF says it's an Apple issue, so I encourage JAMF Pro users who are experiencing this issue to call AppleCare Enterprise support at 1-866-752-7753 to report it.

[ayork]

Has anyone found a resolution yet?

[ejculpepper]

After working with Apple Enterprise Support, the final answer was that our smart covers were causing the issue. Apparently, an assertion is preventing the update from being applied and restarting the device when the smart cover is closed. It seems to be sporadic which devices are affected though, as we have the same covers on all of our iPads connected in carts and several are updating successfully.

According to Apple, improvements were made in iOS 13 that allow for updates to be installed with the cases closed. At this point, we are holding off on updating our iPads until iOS 13 is released. Sounds like we'll be visiting each cart to update the iPads using AC2.

[nsdjoe]

@ejculpepper - Oh no... not AC2!!! :)

We don't have covers on our cases, so it must be something else for us. I'm going to follow up with our Apple SE today to see if I can get a status update on the issue.

~Joe

[ayork]

Covers.....really? I guess I will have to check that out even though I really don't think so.

[ejculpepper]

@ayork Yeah, I have my doubts as well, especially since some of ours are updating fine even with the smart covers closed on them. That's what was provided to me from Apple based on several sysdiagnose logs I submitted from affected devices though.

Upgrading from 12.x to 12.3.1 isn't a pressing matter for us right now though, so we'll wait for iOS 13 and see what happens.

[krispayne]

We do not have any covers, smart or otherwise, on our iPads, so I feel that's a dubious cause to the issue.

[ayork]

As of yesterday, 7/9/19, the Apple tech I had on my case said that their engineers are aware of this issue and that the problem should be resolved with iOS 12.3 so that means we have to physically touch any iPad that will not auto update. I have a call with JAMF today to review some logs just be to sure.

[rebelati]

Manually updating to 12.3 was the method that we used as well. Physically touching 2500 iPads was not fun so we are really hoping this is the last time we have to update in this way.

[krispayne]

Weird, I had the issue when upgrading to 12.3.1 from 12.3.

[ayork]

I finally have an answer for my situation!

What is happening is when the iPads are set to automatically do software updates, they get stuck when our command is pushed out. If you were to turn off those automatic updates and wait the 15 minutes for the OSScheduledUpdate command to push out again the iPad will update. At this point that is the workaround or wait for the device to update at 2am.

Jamf Case: JAMF-0688047
Apple Radar Ticket: 20000054935452
Jamf Product Issue: PI-007214

[krispayne]

@ayork I'm not certain, but I don't think I have automatic OS updates enabled on any iPad. Where can I go to confirm this?

[ayork]

@krispayne We have a restrictions config profile so under the payload configuration, it's under the iOS and tvOS, Functionality. I changed ours to defer the software update for 60 days.

[dubprocess]

Still getting this issue when trying to update tvOS 13 today. We are currently on Jamf 10.15. We do have a configuration profile to defer software updates for 90days, but was under the impression if you force a "remote command>Update OS Versions on supervised devices>etc" to update over the air it would still continue to update regardless of the software update deferral profile we have set already.

Well to test this I went ahead and disabled the "software defer for 90 days" and pushed it out to the test AppleTvs but I am still getting the OSUpdateStatus hanging.

[Adam_h]

Hi

jamf version:10.15.0-t1567105782

Also can confirm @dubprocess ipad ios update have the same issue here with software deferal updates and command up update the ipad on 13.0 error message Complete The remote command failed to send.

pushing 13.1 has same output.

tests were on ipad MD785X/B air and MTXN2X/A pro

[lee_smith]

I have opened a support case with JAMF regarding this same issue. We are attempting to go to 12.14.1. We are receiving the same error message:

"The remote command failed to send"

The tech I spoke with indicated there is a known issue with 12.14.

When we attempt to go to 13.1.1 we had no issue. I have sent JAMF the debug logs and a JAMF Summary.

I am currently waiting to hear from them.

[CanmerLagerhaus]

Have the same issue now, we have a retail with several iPads in different stores.
Some stores updates iPads without any concerns but some stores you need to manage for a couple of days before it updates.

It seems like Jamf need to release a best practice how to handle the updates.
One thing that would ease up the update handling is to have some information in Jamf on the device that the update has been downloaded and ready to install.

I´ve worked with other MDM-tool which had the feature to display if an update was available which was great and I didn´t see this type of behavior on that system either.

I use to do like this:
1. A couple of days before the update I download the update without installation
2. When the day for update come, I´ll press on the button to initiate the update
3. Waiting game begins...

This works sometimes, but not always which is weird!

I have three iPads now which is not updated... And it seems not to update either... What am I doing wrong or is it just a bug in Jamf? It should be instant as long the update is downloaded.

[mhegge]

This issue has been an eternal frustration. Currently requires too much hands-on or user interaction when it should be just downloading, installing, and restarting. Could be because I am running v10.12 currently...

[RobbieG]

Still does it on v10.16 iOS versions 12 & 13. Cases prevent updates, looking at the update screen on the iPad prevents updates, Updates being turned on prevents updates, having a passcode prevents updates, when by some miracle updates do try to run Apple throttles the connection so it takes 2 days to download it.

[RobbieG]

Out of about 60 iPads eligible for iOS 13 roughly 40 never updated to 13 at all after pushing 13.0 - 13.2.3 - I walked around to each one and 20 of those are stuck at Update Requested / Estimating Time Remaining for non-current OS versions. By stuck I mean no change after 24 hours - persists through restarts and resets.

[Tolandese]

Can confirm that we are seeing this same issue. I just reached out to support to get some answers as even one off downloads offsite over the weekend seem to take an extremely long time. Device restart appears to cancel the operation but the limited network usage makes me think it never really was running.

[rstasel]

Anyone had any luck with this issue? It's been several months since last post so wondering if it's solved itself, or if we're still in a "this sucks and I don't want to deal with it" state. =/

[mooler]

@rstasel It's been a major improvement since then, in my view. The things that have worked best, and without being able to confirm if they are related are:
1. Turn off automatic update of all devices.
2. As we use a lot of shared iPads, make sure to push and push the info to remember to logout. As we still do not have a way to mass logout users from JAMF. Remind the teachers to use classroom and logout students. As you figured this is in a school enviroment.
3. The command do not succed 100% when pushed out. You have to send remote command, wait. After 1-2 days, cancel command (and failed commands) and resend the remote command. This will somehow help with the bottleneck of deploying iOS (ipadOS) updates.

This is totally unsupported by any logs or whatever, it just is what works for us at the moment.

[rstasel]

Thanks! These aren't shared ipads I'm dealing with, but that's a good tidbit.

We're testing with turning off automatic updates.

Will also try removing the "failed" (it's not really failed, it's just sitting there in pending) command. Also going to see if 10.21 fixes the issue since it has some improvements to app catalog updates (we're seeing what looks like an app update is "in front of" the OS update, but stalled.