Pending - Application is not available to install

[JKoopman]

We use VPP-managed device-based app assignments for our iPads. Starting about 3-4 days ago, suddenly all our app installs have become stuck at "Pending - Application is not available to install". The strange thing is, this is only happening on newer iPads, as our old Air 1's are able to enroll and receive their apps fine.

I've tried letting some of them sit for upwards of 24 hours in case it was just slowness, but they never progress beyond there. We've tried full restarts of the iPads, and clearing all pending commands and issuing another inventory update / push, none of which do anything.

The only temporary workaround is to go into our Volume Purchasing settings in JAMF Pro and manually refresh each individual app's licenses. This generally will cause the app installs to proceed for iPads that are currently stuck, but not in 100% of cases, and any iPads set up or apps requested after the refresh will continue to be stuck until the licenses are refreshed yet again.

Anyone else seeing or seen something similar happen and know of a solution? I have a support request open, but we're deploying thousands of iPads next week and students are going to need to start installing apps soon.

[nsbickhart]

We have over 10,000 iOS/iPadOS devices and are seeing the same thing for about a year and it's getting worse. Are you noticing the issue more when distributing apps automatically vs. from self service?

The issue seems to only be affecting our auto install apps. I was able to reproduce all of the same issues on our test server running 10.23 while our prod is on 10.18. I ran wireshark and found a couple things and sent to our network team for further analysis. I also ran conosole on an iPad and grabbed some logs. Every time that I reproduce the issue, the first error I see is "unrecognized token from <private>". I'm not sure if that's of any significance, but I see it on both test and prod while reproducing the issue.

[JKoopman]

I tested a few manual installs from Self Service and saw the same issue, so I don't believe it's limited to our automatic installs.

[RLR]

I'm seeing the same issue. I have a call open with Jamf. It was working fine Thursday but the next day it stopped working. Licenses are just not being applied for some reason. They are aware it's a PI and are working on a fix for me at the moment. I've said I can't wait for a Jamf update and this needs doing as soon as possible as we've just rolled out 100s of iPads for students and teachers which they can't currently use fully. Manually refreshing licenses fixes it but this is very time consuming as you can only do one at a time: Global Management > Volume Purchasing > VPP Account > Content > IOS App > Refresh the app you want.

I'm just focusing on core ones at the moment. Although we're still setting iPads up so I have to keep going back and doing the same ones. Any new iPads that get setup will not get their licenses. This is not just auto install apps. It's any apps that have been assigned after the break happened.

[mbonifacio]

We are having the same issue.

[RLR]

Possibly an Apple Issue then as this only started happening on Friday for us?

[palmna]

We're in the same boat only ours was working fine until this morning (Wed). In our case it doesn't matter if the app is a required install or availble in self-service it just seems like the licenses are broken. I've gone through about 30 of the (essential) apps in VPP and performed a manual refresh but as @RLR alluded, that could take days depending on your deployment size.

If anyone hears concrete positive news from Jamf please share.

[fsjjeff]

Adding us to the "me too" list - we're frantically trying to setup 1500 more iPads and have ground to a halt here because of this issue.... I think it's happening on our Macs as well but haven't dug into that yet.

When I look in my JSS logs I'm seeing tonnes of error messages about managing licenses, something like the following, only repeated thousands of times:

2020-09-09 13:01:38,125 [ERROR] [eralPool-15] [VppCommService           ] - Error managing licenses. ErrorNumber: 0 , ErrorMessage:null
2020-09-09 13:01:38,161 [ERROR] [eralPool-16] [VppCommService           ] - Error managing licenses. ErrorNumber: 0 , ErrorMessage:null
2020-09-09 13:01:38,281 [ERROR] [eralPool-19] [VppCommService           ] - Error managing licenses. ErrorNumber: 0 , ErrorMessage:null
2020-09-09 13:01:38,385 [ERROR] [eralPool-16] [VppCommService           ] - Error managing licenses. ErrorNumber: 0 , ErrorMessage:null
2020-09-09 13:01:38,386 [ERROR] [neralPool-8] [VppCommService           ] - Error managing licenses. ErrorNumber: 0 , ErrorMessage:null
2020-09-09 13:01:38,500 [ERROR] [eralPool-13] [VppCommService           ] - Error managing licenses. ErrorNumber: 9603 , ErrorMessage:Internal error

[russeller]

@fsjjeff I had a similar issue, here is what support asked me about it:

Have you noticed any VPP issues? With this error, we would need to enable HTTP debug logging to get more information on the root cause. Do you know if a VPP token might be in use on another server by chance?

In our case one of our VPP accounts was being using in Apple Configurator 2, and we had to reclaim the token.

[fsjjeff]

@ssrussell I've got all the VPP issues ;)... As in nothing is installing right now and I'm just getting all those error messages :(. I don't think we have any of our VPP accounts logged into any Apple Configurator instances, but trying to check as I type.

[russeller]

@fsjjeff If you click on each one of your VPP accounts in Settings in your jamfPro instance you should see a banner at the top of the VPP account page if the token is in use. Then you can click "Reclaim token".

[fsjjeff]

@ssrussell Interesting, didn't even know that was a feature. Just checked and no banner to reclaim a token, but that doesn't surprise me too much as I was pretty certain we weren't signed in with anything else.

[mahughe]

Apple has been having some issues the last couple of weeks I've seen. Apps/books not available in ASM, ASM errors, iPads
timing out to activate to name a few. That message means that Apple hasn't issued a license to that device for that app, and with them having issues as they have been I'm not surprised.

[russeller]

Based on the 9603 error in the log from @fsjjeff I found this from here: https://developer.apple.com/documentation/devicemanagement/app_and_book_management/interpreting_erro...

Receiving a 9603 Internal Error response typically indicates the VPP server couldnʼt provide timely processing. Nothing is necessarily wrong with the request. When the MDM server receives this response, it should send the current request again. If it continues to receive 9603 errors after more than five attempts, it may mean that the VPP service is unexpectedly down and further retries should be scheduled for minutes later, instead of seconds.

[nsbickhart]

I've been comparing jss logs from our prod and test server. Reproducing on the test server, I see [ERROR] [eralPool-10] [VppCommService ] - Error managing licenses. ErrorNumber: 0 , ErrorMessage:null

I see this on both prod and test and was able to grab a fresh log while reproducing the error. I sent the info to support and am waiting to hear back. The initial error is pending - application is not available to install, but eventually it will install.

[fsjjeff]

@ssrussell That's definitely sounding like it's more of an Apple thing than a JAMF thing :(.

[paulbenham]

I ran into this issue today. In my case it was because I had the app scoped to a static group that had more members then we had licenses for in Apple School manager. It was a free app, so I added a bunch more licenses, then the app started installing.

Hope that helps somebody!?

[sdecook]

We are starting to see this as well on newly registered iPads. Typically we would see the message for around 30 minutes as the licenses were getting assigned. Now with new ones we are seeing it for days without it progressing. We have plenty of licenses available.

[sbistrup]

I have the exact same issue with auto-install apps and apps from Self Service....in the managment pane they sit there and say: Pending - Application is not available to install

[gcarmichael]

I opened a ticket on this today. Our student testing window starts Monday and only 1/3rd or so of folks actually prepped for it.

[JKoopman]

I got confirmation from higher level support that it does appear to be an Apple issue. I've been told some changes have been pushed out which will hopefully fix it, but that app installs impacted by it may still need to be refreshed at least once more. The full relevant text is below.

We've been seeing this issue in a variety of environments, and we've opened communication with Apple regarding this behavior and it looks like multiple MDM's are affected. There have been some changes pushed, so we should see some improvement soon. However, we may need to refresh each app one last time. The following is a brief overview of what's happening on the backend, feel free to skip over this if not curious!

This issue at its root has two causes that compound on each other: 1: When Jamf Pro attempts to assign a license to a device, Apple's VPP server does assign the license, but returns an error code (9603) to Jamf Pro. 2: Jamf Pro is unaware that a license was assigned, so it attempts to assign the license again. Apple returns another error code (9616), indicating that the license in question is already assigned. Jamf Pro interprets this as a failure and continues trying.

From here: We have preliminary reports that Apple has pushed a fix to prevent their servers from returning the initial error code (9603), so new license assignments should not run into this issue. However, the fix that was implemented will not fix license assignments that are impacted by the second of the two issues, the 9616 error code. To fully clear out this issue, any applications that are affected may need to be refreshed again in Jamf Pro > Settings > Volume Purchasing Accounts > Click on the token in question > Content > Refresh next to the app in question. If you do run into this issue on any apps after refreshing their licenses, let me know. We're still in the process of verifying the fix from Apple across all environments, so we'll want to catch this as soon as possible if the issue pops up again.

[nsbickhart]

Hope this works

[RLR]

Still not working for me today. Just setup a new iPad and still getting the same error message. Jamf support said they applied a fix last night for us but this hasn't changed anything either.

[nsbickhart]

I been on our test server and after a reset, I still see the "Pending-Application is not available to install", however, it is pushing the apps in a timely fashion. I am noticing a new error in the jss logs while triggering an enrollment.

2020-09-11 08:52:34,570 [ERROR] [duledPool-0] [iceEnrollmentSyncRunnable] -

com.jamfsoftware.jss.exceptions.communication.InvalidRequestDataException: java.net.SocketTimeoutException: Read timed out

Renewed the VPP token on prod and still seeing. constant error in the jss logs.
[VppCommService ] - Error managing licenses. ErrorNumber: 0 , ErrorMessage:null

[khinkelman]

Following thread.

We have been having issues since August with a handful of our devices. Over the weekend we updated our 2,000 iPads to the latest iOS. We noticed Google Classroom was having a lot of glitches, pushed an app update. Now, all of our iPads have Google Classroom greyed our "waiting". I also have a support ticket with JAMF.

[cboatwright]

+1 I have been fighting this for over a week as well, and it is an AWFUL issue when every student and teacher are working remotely and need their apps! Sounds like some of you have been getting better advice, I had to discover the refreshing content licenses myself and a Jamf agent was telling me its likely I just doubled-up scoping and applied multiple licenses to the same devices... right...
Have not gotten anywhere on the Apple-end of this problem either! Just gets pushed higher up each day.

[JKoopman]

@RLR I can confirm we are still seeing the issue as well this morning, including on apps whose licenses we have refreshed multiple times since being told a fix had been pushed out.

@cboatwright We were initially told the same, but after checking that the scoping wasn't improper for any of the apps they were seeing errors for, we were ultimately kicked up to a higher level of support.

[palmna]

FWIW, I've been going back and forth with Jamf's escalation team the past few days and while I'm still gettings thousands of the errors @nsbickhart mentioned in the server logs our apps our starting to install.

The most recent response I got from Jamf mentioned that they would parse the logs I sent and send API commands to apple to disassociate the licenses. In addition, I uploaded a fresh server token from ASM to our Jamf Pro Server (as a shot in the dark). Something seems to be working because today the problematic apps (that I hadn't manually refreshed) are now installing or just "pending" instead of saying "pending - application is not available to install".

We're still sorting through today's logs to determine why Apple/Jamf is dropping the licenses (and what we can do about it) but there seems to be light at the end of the tunnel for us...hopefully, others will start seeing forward movement as well.

[Emmert]

Unfortunately, it looks like this is either model or iOS version specific, as Air 1 iPads seem to be (mostly) working for us today, but 7th gen iPads are getting absolutely nothing.

edit: who knows what's going on. I'm told we were setting up 7th gen iPads perfectly fine last weekend. Hopefully Apple can get this figured out quickly for us.

[gcarmichael]

Yesterday at least all my apps had licenses, today they are all 0d out. 290 ios apps, and 85 Mac apps. Roughly 11k devies in total now SOL and we have testing windows starting Monday. I have not heard anything back from Apple or Jamf on this from either side after opening tickets.

[fsjjeff]

I'm on with Apple Support right now and they don't seem to have any awareness of this as an issue. Any chance someone else has a case # I can reference?

[gfarabaugh]

@gcarmichael Seemed to just happen to me at 12:54PM EST. All my app licenses 0'd out. I can't create a new token either in VPP at apples site

[fsjjeff]

Yeah, all my apps just disappeared from School Manager as well. They were there 20 minutes ago but now nothing :(

[rice-mccoy]

Same here, Apple Status page silent about this issue although they list a School Manager problem that was resolved earlier today.

[eizaguirre]

Same here! Been having this problem for a few weeks but we've been able to send blank pushes or inventory updates to force. Now we're seeing licenses 0'd out. All app pushes are halted or stuck. Will try to renew token and see if that does anything.

[gcarmichael]

@fsjjeff Apple Care Ticket number or Jamf Ticket number? I have opened a enterprise ticket, and have an open Jamf ticket.

[adam_richardson]

According to Jamf support yesterday they didn't have any fix or options for the "Pending - Application is not available to install" outside of refreshing the licensing for each individual app. Jamf support did state that they had an enterprise support ticket open with Apple (#101145962620) and suggested I contact Apple and have them add our school district to the ongoing ticket.

[hphan]

It seems the 0 app issue is fixed for us and a neighbor school district.

[gcarmichael]

Got this 10 mins ago from my Apple care enterprise ticket - Can you refresh your licenses in Jamf? In the last hour or so, we fixed some stuff on our end (which hopefully resolved the issue for you).

[tusdafif]

We've experienced this to our instance as well! The delay of apps being pushed out, and under VPP content on the instance we were 0'd out. Not sure what the issue is in particular if it's Apple or Jamf. If anyone knows please let me know!