iOS Mass Action Update Failures

[cstout]

Happy Saturday, JN.

I've been working on and off with support regarding this for several months now and still don't have a solid answer so I thought I'd throw it out here to see what is being experienced out in your environments:

My questions to JN are:
1. Are any of you seeing the same behavior?
2. What have you found to be the most reliable procedure to follow regarding mass iOS updates?

Scenario: Apple releases a new iOS version.
My basic process: When I have a smart group or a regular inventory search and choose to perform an action on the listed iOS devices, I choose Action->Send Remote Commands->Update OS Versions on supervised devices->Latest version & Download and install the update chosen->Next (Finish)

Pretty straightforward, right? Well, for the last two iOS releases I've had about a 10% success rate with this process.

My test iPads, which have no passcode set, simply log the following and never actually process my update request. I've had only marginally better results by specifying the iOS version instead of choosing latest available, but even then have seen the same results.

Here is what I see on the management command history:

OSUpdateStatus  2 minutes ago   
OSUpdateStatus  17 minutes ago  
OSUpdateStatus  32 minutes ago  
OSUpdateStatus  47 minutes ago  
OSUpdateStatus  About an hour ago   
OSUpdateStatus  Today at 8:14 AM    
OSUpdateStatus  Today at 7:59 AM    
OSUpdateStatus  Today at 7:44 AM    
OSUpdateStatus  Today at 7:29 AM    
OSUpdateStatus  Today at 7:14 AM    
OSUpdateStatus  Today at 6:59 AM    
OSUpdateStatus  Today at 6:44 AM    
OSUpdateStatus  Today at 6:29 AM    
OSUpdateStatus  Today at 6:14 AM    
OSUpdateStatus  Today at 5:59 AM    
OSUpdateStatus  Today at 5:44 AM    
OSUpdateStatus  Today at 5:29 AM    
OSUpdateStatus  Today at 5:14 AM    
OSUpdateStatus  Today at 4:59 AM    
OSUpdateStatus  Today at 4:44 AM    
OSUpdateStatus  Today at 4:29 AM    
OSUpdateStatus  Today at 4:14 AM    
OSUpdateStatus  Today at 3:59 AM    
OSUpdateStatus  Today at 3:44 AM    
OSUpdateStatus  Today at 3:29 AM    
OSUpdateStatus  Today at 3:14 AM    
OSUpdateStatus  Today at 2:59 AM    
OSUpdateStatus  Today at 2:44 AM    
OSUpdateStatus  Today at 2:29 AM    
OSUpdateStatus  Today at 2:14 AM    
OSUpdateStatus  Today at 1:59 AM    
OSUpdateStatus  Today at 1:44 AM    
OSUpdateStatus  Today at 1:29 AM    
OSUpdateStatus  Today at 1:14 AM    
OSUpdateStatus  Today at 12:59 AM   
OSUpdateStatus  Today at 12:44 AM   
OSUpdateStatus  Today at 12:29 AM   
OSUpdateStatus  Today at 12:14 AM   
OSUpdateStatus  Yesterday at 11:59 PM   
OSUpdateStatus  Yesterday at 11:44 PM   
OSUpdateStatus  Yesterday at 11:29 PM   
OSUpdateStatus  Yesterday at 11:14 PM   
OSUpdateStatus  Yesterday at 10:59 PM   
OSUpdateStatus  Yesterday at 10:44 PM   
OSUpdateStatus  Yesterday at 10:29 PM   
OSUpdateStatus  Yesterday at 10:14 PM   
OSUpdateStatus  Yesterday at 9:59 PM    
OSUpdateStatus  Yesterday at 9:44 PM    
OSUpdateStatus  Yesterday at 9:29 PM    
OSUpdateStatus  Yesterday at 9:14 PM    
OSUpdateStatus  Yesterday at 8:59 PM    
Schedule OS Update - iOSUpdate16E227    Yesterday at 8:48 PM    
AvailableOSUpdates  Yesterday at 8:48 PM    
Update Inventory    Yesterday at 3:39 PM

[thejenbot]

Yep, seeing the same thing. The OSUpdateStatus running every 15 minutes is normal/expected behavior.

If I have everything straight, what is required for this to work properly is that the device be at least at iOS 12 - there were software issues with 11.3 and 11.4. Also, the iPad should be plugged in, and it seems to need to be plugged in when you issue the command so that when the download is done it can complete the process; if the download completes and it is not plugged in, even if there is 50%+ battery, it will not automatically start to install. Users can manually go in and click Install Now if they have sufficient battery, but the mass action stops dead at that point if not plugged in. Additionally, if you have a case that locks the screen, the lid should be open. This was the consensus after having one case open with Apple and one with Jamf.

Which all told really sucks; we have stm DUX cases (recommended by our Apple rep) that never gave us an issue with updates before iOS 11 came out. Now the magnet that keeps the lid closed (and locks/unlocks the screen) screws everything up. If people are using charging carts or crates or practice good cord management, this is now much harder and requires user intervention if everything isn't just so. The changes Apple made on the software side don't translate well for enterprise, this seems to be much more for individual/consumer.

[cstout]

So, @thejenbot, even in your best case scenario, about how long does it take for you to actually see success from a mass action such as this?

My three test iPads in this scenario are on iOS 12.1.4, charged to 100% (edit: and plugged in), no passcode set, and not in any case that would activate the display lock function.

I cleared all pending commands (there were no failed commands) and reissued the mass action 35 minutes ago and have this so far in the device's log:

OSUpdateStatus  8 minutes ago   
Update Inventory    14 minutes ago  
OSUpdateStatus  23 minutes ago  
Schedule OS Update - iOSUpdate16E227    35 minutes ago  
AvailableOSUpdates  35 minutes ago

[thejenbot]

@cstout...I'm not sure on the timing part of it. What I was doing was sending commands out at the end of the day and checking on them as I got in the next morning. I would have devices at 100% battery and it still wouldn't work if it wasn't plugged in. Apple pointed me to an article - https://support.apple.com/en-us/HT204204 - that calls out the need to plug in right at the top...super lame.

[kdavis]

We're a school system with iPads in carts. The carts are plugged into power and the iPads have no passcode. We're experiencing the same issues as you described. iPads constantly report command OSUpdateStatus about every 15 minutes, but the update never completes and there are no failed commands. We tried updating the OS to a "version" and the OSUpdateStatus command doesn't repeat, we see the Schedule OS Update command, however the iPads never finish the update. We're on Spring Break this week and were hoping to complete updates, but it's been a failure so far.

[cstout]

@kdavis Thank you for sharing that. I’m sorry to hear you’re having the same issue but I’m honestly relieved it’s not just me. Exactly as you described, both deployment methods (latest and specified) are giving me the same results as you wrote.

Most recently in my testing I left the iPads plugged in, screens up, on WiFi, no passcode, and then cleared all pending commands. Then I sent an update inventory, restart, and issued a fresh update iOS mass action. One for specified 12.2 and two for latest. The one that was specified recorded my action in history and did not repeat any status messages but more importantly did not update at all. The other two are still repeating the OSUpdateStatus message. It’s been about 36 hours in that state and the iPads have not been touched during that time.

[mmcallister]

I have updated some iPads remotely that showed the repeated OSUpdateStatus commands. The iPad had updated, but was waiting for user interaction on the device to step through a few setup screens.

[krispayne]

I'm having the same issue. All of mine are in single app mode, whenever I take them out of single app mode, go into settings and check, the update says "Update requested" then quickly shows 3 hours, 2 hours, etc until it's now waiting for install. I typically have cancelled the update command at this point because of needing to use the ipads (conference room controllers), but then the next available time, I can quickly run the update in about 15 minutes. I was upgrading from 12.1.4 to 12.2 today and noticed this on 4 iPads, the rest of my iPads worked fine (even same generation iPads.) I was able to run updates on 7 iPads yesterday pretty quickly without needing to go around to each iPad. Today is a completely different story, only 1/5 was able to update without me touching it.

Hopefully it's just a bug, because I live in a different state than the iPads, so remotely updating is pretty important :-/

[RobbieG]

I have seen the same thing in our environment. One thing that helped in the past was to send remove passcode - even though none of the iPads being updated actually had one but that trick seems to have stopped helping since 12.1.4. I have seen some iPads display download failed on the software update screen when I walk around to check but they seem to be in the minority (and a very few show the upgraded as failed in Jamf and allow us to clear and retry.)

This behavior has persisted for several months - we are on Jamf Cloud so I am unsure about which Jamf Version it started on - but at least the past 5 iOS versions.

[isheldon91]

Not a solution, but I have seen the exact same behavior as of late. Currently pulling my hair out over it as we are on Spring break this week and I planned to update all 500ish iPads. Same sort of situation...classroom carts, iPads all plugged in, and in cases that have auto sleep/wake magnet feature. (Not sure if that really matters for this as @cstout tested on fully awake iPads with no case and saw same behavior..will test this later today and report back.) If I send the command as latest, I see a string of OSUpdateStatus, and if I send specific, I see one instance of "ScheduleOSUpdate - " but nothing completes, other than a few rare cases where the student has already opened the Software Update pane and the download has been requested and completed (those devices seem to apply the update and then subsequently restart).

[isheldon91]

This definitely seems to be tied to mainly the download portion of the Download, Install and Restart devices command. I discovered this today by opening the software update pane in settings on an entire group of iPads. Opening the pane in most cases automatically started an iOS download (I manually forced it to download on the devices that did not). I then let the iPads sit and complete the download. I returned later and all these devices had a message that the update would be installed automatically overnight (thanks to iOS 12s Automatic Updates feature). Once they were all in this state I was able to send yet another Download, Install, and Restart devices command from JSS, and this time it went through without a hitch. The same workflow worked for another group of iPads in which I turned automatic updates off when I visited the software updates pane. Once the download is completed, automatic updates being on or off doesn't seem to matter.

However, I am curious for when the next round of updates is released if the group with automatic updates off will function how it should when sent the initial Download, Install, and Restart devices command. Perhaps this new feature is blocking the initial download request from the JSS. Time will tell on that one.

[MarshallW]

Just adding my case to this issue, I've been struggling with this for a few weeks now but all I have for my efforts are strings of "OSUpdateStatus". They've all been wiped and set up again but no luck. I've tried all of the tips in this discussion but nothing has improved the situation.

[ibrahimk]

I had the same issue and below tip from Jamf support fixed this for me.

Go to Mobile Devices -> Advanced Search -> perform blank search -> Action -> Cancel Management Commands -> Cancel All Pending and Failed Commands and let's retry to schedule OS update again.

[_andrea]

Same issue here and wiping devices or deleting mobile commands didn't sort the issue.

[cstout]

@ibrahimk For whatever reason...that worked for me. My three test iPads that have been sitting here for weeks on "OSUpdateStatus" all just rebooted and started the update process. I cleared all pending and failed commands via actions (instead of individually, by pressing the "cancel all" button - which is what I've been doing) and then I issued a new mass action to update and install the latest available update. This is how it should work...but what was hung up and preventing it that was not visible in the UI and why?

Update Inventory    4 minutes ago   
Schedule OS Update - iOSUpdate16E227    9 minutes ago   
AvailableOSUpdates  9 minutes ago

[cstout]

I had a new iPad that I enrolled two days ago and decided to see if it would update after the last successful batch of updates. I searched for the device, chose latest available and install/restart, and it didn't do anything snappy. Instead of being impatient and clicking around I let it sit to see what'd happen today. To my surprise, only 30 minutes later, it ran the update successfully.

Schedule OS Update - iOSUpdate16E227    5 minutes ago   
OSUpdateStatus  5 minutes ago   
OSUpdateStatus  17 minutes ago  
Update Inventory    20 minutes ago  
Schedule OS Update - iOSUpdate16E227    28 minutes ago  
AvailableOSUpdates  28 minutes ago

[rzoppi]

Background to our issue:
A recent windows update was the first domino to fall for us. At the time we were on JamfPro 10.9.0. After a windows update, JamfPro would consistently crash when updating CFG profiles to ~600 ipads. During this time, only the current iOS version would show up when trying to update iPads en masse to a specific version (trying to upgrade to 12.1.4 and only 12.2 would show). We were advised to upgrade from 10.9.0 to 10.11. After upgrade, all versions would appear but we ended up having to roll back the windows updates to prevent Jamf from crashing. We then sent out an update command to which only ~10% of ipads were successful.

Currently we are having the same issue as many stated by others: When updating to a non current iOS version, MDM shows "Schedule OS Update -" and doesn't repeat. When updating to current version, MDM command "Schedule OS Update - iOSUpdate16D57" appears and repeats every 15 minutes, but device never completes upgrade.

Devices are plugged in, cleared of pending/failed commands, cleared of passcode, have no restrictions applied, and sent update iOS to latest available en masse.

[justin_wiltjer]

Glad to know I'm not alone in this. Our iPads are on iOS 12.1.4 and I left them over the weekend to update but came back to the same behavior noted here.

The iPads I currently manage function as a kiosk and are in running in Single User Mode. Would this stop an iPad from updating if the Update came from the Send Remote Commands feature?

Schedule OS Update - iOSUpdate16E227 5 minutes ago
AvailableOSUpdates 5 minutes ago
OSUpdateStatus 8 minutes ago
OSUpdateStatus 8 minutes ago
OSUpdateStatus 23 minutes ago
OSUpdateStatus 23 minutes ago

[pcacs_jamf]

We are new too JAMF.
We have 96 iPads. 32 of them are in "Shared" mode on iOS 12.1
Obviously in Shared mode we can't get to the iOS updates section so we need to update iOS via the MDM.
I have 3 iPads in front of me, 2 in shared mode (Students) and 1 as a managed supervised "Teacher" iPad. All 3 iPads FAIL to update via the Force update iOS via Actions.

Now, I don't care too much about the Teacher iPads, as updates can be done via the UI, but we are screwed with the Shared iPads.

In preparation for the iOS update, the shared iPads have had the user logged out, restarted, plugged into power with battery at 100%, have their magnetic cases open (and an attempt to clear passcode, though this causes an error on Shared devices, so this isn't an option, the failed command request was also purged.)

I don't get an error in the management queue, it just sits there as pending after about 15minutes, the command queue is clear. No errors.

The only environment variable I can think of that might be screwing up JAMF is that we do have a Mac mini Server doing update caching, could this pose an issue?

Kind Regards,
Aaron

[krispayne]

I'm still having trouble with this.

Until I go in, remove the single app mode profile, go into settings, and tap on software update, it doesn't move.

I opened a support case, too, not sure why I put it off?

[cstout]

Another update releases and more of the same. Endless OSUpdateStatus across all iPads. Tried the tricks from last time with no success.

Side note: I've never had this issue with any Apple TV's. They run the command first time, every time. This is only iOS updates that are not processing at all for me.

[krispayne]

@cstout what version of JSS are you on?

[cstout]

@krispayne 10.11.1-t1553545638 on prem

[cstout]

Just upgraded to 10.12 and cancelled all commands. I'll reissue the mass update command when I get back to the office today and see if anything's improved.

[krispayne]

@cstout I am on 10.12 and tried updating an iPad to 12.3 this morning and had the same problem as updating to 12.2. I still have my support case open regarding it, so hopefully we can figure out what's wrong.

The workaround stills works (cancel all pending and failed commands, re-issue command)

[RobbieG]

My test batch from 12.2 to 12.3 went 0 for 8 sending it out with no frills then 0 for 8 sending it out after cancelling all pending and failed then 0 for 8 sending out the remove passcode, verifying that that command pushed, canceling all pending and failed, the resending.

I had half the batch manually on and unlocked and half in closed cases - but none budged.

[rebelati]

Seeing this here also. We sent out the first update command 7 days ago and so far only 42 out of 2300 iPads have updated. 500 of these are K2 iPads that are constantly plugged in to carts with no passcodes. Jamf recommended opening a ticket with Apple, which I have done.

We have discovered that iPads on 12.3 will process the remote update command and update to 12.3.1 and iPads that manage to download the update will install it if we send the command again. Other than that, the issue is frustratingly inconsistent. Sometimes two iPads will be sitting next to each other, both plugged in, both on the same iOS version, both with the case open, and one will process the update command while the other will not.

[lee_smith]

We are K-12 School as well. We are seeing the same issue as you all mentioned above.

We have iPad Carts plugged in and communicating. We can send out the "Update Inventory" command and they check right in. I send out the Update OS command and they sit there for days.

As with many of you, this is frustrating but like others have said, I am glad I am not the only one.

[ChrisRupert]

I'm seeing the same as the rest of you. I created a smart group to show me any student iPad that's not on 12.3.1. When I ran that I had 839 devices not on the most current iOS. I've run the mass update for over a week now and my numbers are only down to 782. Very frustrating when there's no rhyme or reason. All the iPads don't leave the school and end up in cart. I've had no issue with over the air updates before iOS 12, and now it's a crap shoot when and if they will update.

[nsdjoe]

I'm seeing the exact same issue. I've tried wiping all pending and failed commands and sending out a new update command, but that didn't fix it. Most of our iPads are on 12.1.4. Hoping this gets resolved soon!

[ayork]

We are a K-12 school with about 1000 ipads and having this issue too. Why can't we get this resolved????

[jmendoza]

We are a K-12 school with about 1000 devices as well, and this issue is becoming very frustrating.

[ayork]

Anyone have this error in their logs?
An error occurred while consuming the Apple Software Lookup service

[marktaylor]

+1 for this issue too Jamf 10.12
Mixture of iOS

[RainerNRW]

+1 leider tritt der Fehler seid 10.9. auf und es wird immer gesagt „mit der nacharten Version klappt es dann“

[nsdjoe]

JAMF Support suggested I contact AppleCare support and report this issue. So I did. You can too... call AppleCare Enterprise support at 1-866-752-7753. Then let your Apple SE know the ticket number so that they can follow up with the engineering team regarding this issue. Hoping this issue gets resolved soon!!!

[rzoppi]

Does anyone have a test environment to see if an upgrade to 10.13 fixes our issue?

Jamf Pro Release Notes
Version 10.13.0
[PI-006714] Fixed an issue that prevented Jamf Pro from sending the Update iOS Version remote command when sending it as a mass action to mobile devices with iOS 10.2 or earlier.
[PI-006745] Fixed an issue that sometimes prevented Jamf Pro from sending the Update iOS version remote command.

[cstout]

Well, @rzoppi, we'll see what others say but as soon as my test environment booted up on 10.13.0 as soon as I sent the mass update action...they all did exactly that. All of my devices immediately started installing the update. They've just completed successfully. Pretty sweet.

[ejculpepper]

I'm still having mixed results even after updating our environment to 10.13.0. I used one of our campuses as a test, canceled all current pending commands, then deployed the iOS update to around 200 devices. It's been right at 6 hours now, and 150 devices are still showing the "OSUpdateStatus" command completing every 15 minutes.

I currently have a ticket opened with Jamf and Apple for this issue, hoping to hear something back from Apple soon regarding information I sent in to them.

[nsdjoe]

Hey @cstout , for Target Version did you choose a "Specific version" or "Latest version based on device eligibility"?