We use VPP-managed device-based app assignments for our iPads. Starting about 3-4 days ago, suddenly all our app installs have become stuck at "Pending - Application is not available to install". The strange thing is, this is only happening on newer iPads, as our old Air 1's are able to enroll and receive their apps fine.
I've tried letting some of them sit for upwards of 24 hours in case it was just slowness, but they never progress beyond there. We've tried full restarts of the iPads, and clearing all pending commands and issuing another inventory update / push, none of which do anything.
The only temporary workaround is to go into our Volume Purchasing settings in JAMF Pro and manually refresh each individual app's licenses. This generally will cause the app installs to proceed for iPads that are currently stuck, but not in 100% of cases, and any iPads set up or apps requested after the refresh will continue to be stuck until the licenses are refreshed yet again.
Anyone else seeing or seen something similar happen and know of a solution? I have a support request open, but we're deploying thousands of iPads next week and students are going to need to start installing apps soon.
We have over 10,000 iOS/iPadOS devices and are seeing the same thing for about a year and it's getting worse. Are you noticing the issue more when distributing apps automatically vs. from self service?
The issue seems to only be affecting our auto install apps. I was able to reproduce all of the same issues on our test server running 10.23 while our prod is on 10.18. I ran wireshark and found a couple things and sent to our network team for further analysis. I also ran conosole on an iPad and grabbed some logs. Every time that I reproduce the issue, the first error I see is "unrecognized token from <private>". I'm not sure if that's of any significance, but I see it on both test and prod while reproducing the issue.
I'm seeing the same issue. I have a call open with Jamf. It was working fine Thursday but the next day it stopped working. Licenses are just not being applied for some reason. They are aware it's a PI and are working on a fix for me at the moment. I've said I can't wait for a Jamf update and this needs doing as soon as possible as we've just rolled out 100s of iPads for students and teachers which they can't currently use fully. Manually refreshing licenses fixes it but this is very time consuming as you can only do one at a time: Global Management > Volume Purchasing > VPP Account > Content > IOS App > Refresh the app you want.
I'm just focusing on core ones at the moment. Although we're still setting iPads up so I have to keep going back and doing the same ones. Any new iPads that get setup will not get their licenses. This is not just auto install apps. It's any apps that have been assigned after the break happened.
We're in the same boat only ours was working fine until this morning (Wed). In our case it doesn't matter if the app is a required install or availble in self-service it just seems like the licenses are broken. I've gone through about 30 of the (essential) apps in VPP and performed a manual refresh but as @RLR alluded, that could take days depending on your deployment size.
If anyone hears concrete positive news from Jamf please share.
Adding us to the "me too" list - we're frantically trying to setup 1500 more iPads and have ground to a halt here because of this issue.... I think it's happening on our Macs as well but haven't dug into that yet.
When I look in my JSS logs I'm seeing tonnes of error messages about managing licenses, something like the following, only repeated thousands of times:
2020-09-09 13:01:38,125 [ERROR] [eralPool-15] [VppCommService ] - Error managing licenses. ErrorNumber: 0 , ErrorMessage:null 2020-09-09 13:01:38,161 [ERROR] [eralPool-16] [VppCommService ] - Error managing licenses. ErrorNumber: 0 , ErrorMessage:null 2020-09-09 13:01:38,281 [ERROR] [eralPool-19] [VppCommService ] - Error managing licenses. ErrorNumber: 0 , ErrorMessage:null 2020-09-09 13:01:38,385 [ERROR] [eralPool-16] [VppCommService ] - Error managing licenses. ErrorNumber: 0 , ErrorMessage:null 2020-09-09 13:01:38,386 [ERROR] [neralPool-8] [VppCommService ] - Error managing licenses. ErrorNumber: 0 , ErrorMessage:null 2020-09-09 13:01:38,500 [ERROR] [eralPool-13] [VppCommService ] - Error managing licenses. ErrorNumber: 9603 , ErrorMessage:Internal error
@fsjjeff I had a similar issue, here is what support asked me about it:
Have you noticed any VPP issues? With this error, we would need to enable HTTP debug logging to get more information on the root cause. Do you know if a VPP token might be in use on another server by chance?
In our case one of our VPP accounts was being using in Apple Configurator 2, and we had to reclaim the token.
Apple has been having some issues the last couple of weeks I've seen. Apps/books not available in ASM, ASM errors, iPads
timing out to activate to name a few. That message means that Apple hasn't issued a license to that device for that app, and with them having issues as they have been I'm not surprised.
Based on the 9603 error in the log from @fsjjeff I found this from here: https://developer.apple.com/documentation/devicemanagement/app_and_book_management/interpreting_erro...
Receiving a 9603 Internal Error response typically indicates the VPP server couldnʼt provide timely processing. Nothing is necessarily wrong with the request. When the MDM server receives this response, it should send the current request again. If it continues to receive 9603 errors after more than five attempts, it may mean that the VPP service is unexpectedly down and further retries should be scheduled for minutes later, instead of seconds.
I've been comparing jss logs from our prod and test server. Reproducing on the test server, I see [ERROR] [eralPool-10] [VppCommService ] - Error managing licenses. ErrorNumber: 0 , ErrorMessage:null
I see this on both prod and test and was able to grab a fresh log while reproducing the error. I sent the info to support and am waiting to hear back. The initial error is pending - application is not available to install, but eventually it will install.
I got confirmation from higher level support that it does appear to be an Apple issue. I've been told some changes have been pushed out which will hopefully fix it, but that app installs impacted by it may still need to be refreshed at least once more. The full relevant text is below.
We've been seeing this issue in a variety of environments, and we've opened communication with Apple regarding this behavior and it looks like multiple MDM's are affected. There have been some changes pushed, so we should see some improvement soon. However, we may need to refresh each app one last time. The following is a brief overview of what's happening on the backend, feel free to skip over this if not curious! This issue at its root has two causes that compound on each other: 1: When Jamf Pro attempts to assign a license to a device, Apple's VPP server does assign the license, but returns an error code (9603) to Jamf Pro. 2: Jamf Pro is unaware that a license was assigned, so it attempts to assign the license again. Apple returns another error code (9616), indicating that the license in question is already assigned. Jamf Pro interprets this as a failure and continues trying. From here: We have preliminary reports that Apple has pushed a fix to prevent their servers from returning the initial error code (9603), so new license assignments should not run into this issue. However, the fix that was implemented will not fix license assignments that are impacted by the second of the two issues, the 9616 error code. To fully clear out this issue, any applications that are affected may need to be refreshed again in Jamf Pro > Settings > Volume Purchasing Accounts > Click on the token in question > Content > Refresh next to the app in question. If you do run into this issue on any apps after refreshing their licenses, let me know. We're still in the process of verifying the fix from Apple across all environments, so we'll want to catch this as soon as possible if the issue pops up again.
I been on our test server and after a reset, I still see the "Pending-Application is not available to install", however, it is pushing the apps in a timely fashion. I am noticing a new error in the jss logs while triggering an enrollment.
2020-09-11 08:52:34,570 [ERROR] [duledPool-0] [iceEnrollmentSyncRunnable] -
com.jamfsoftware.jss.exceptions.communication.InvalidRequestDataException: java.net.SocketTimeoutException: Read timed out
Renewed the VPP token on prod and still seeing. constant error in the jss logs.
[VppCommService ] - Error managing licenses. ErrorNumber: 0 , ErrorMessage:null
We have been having issues since August with a handful of our devices. Over the weekend we updated our 2,000 iPads to the latest iOS. We noticed Google Classroom was having a lot of glitches, pushed an app update. Now, all of our iPads have Google Classroom greyed our "waiting". I also have a support ticket with JAMF.
+1 I have been fighting this for over a week as well, and it is an AWFUL issue when every student and teacher are working remotely and need their apps! Sounds like some of you have been getting better advice, I had to discover the refreshing content licenses myself and a Jamf agent was telling me its likely I just doubled-up scoping and applied multiple licenses to the same devices... right...
Have not gotten anywhere on the Apple-end of this problem either! Just gets pushed higher up each day.
@RLR I can confirm we are still seeing the issue as well this morning, including on apps whose licenses we have refreshed multiple times since being told a fix had been pushed out.
@cboatwright We were initially told the same, but after checking that the scoping wasn't improper for any of the apps they were seeing errors for, we were ultimately kicked up to a higher level of support.
FWIW, I've been going back and forth with Jamf's escalation team the past few days and while I'm still gettings thousands of the errors @nsbickhart mentioned in the server logs our apps our starting to install.
The most recent response I got from Jamf mentioned that they would parse the logs I sent and send API commands to apple to disassociate the licenses. In addition, I uploaded a fresh server token from ASM to our Jamf Pro Server (as a shot in the dark). Something seems to be working because today the problematic apps (that I hadn't manually refreshed) are now installing or just "pending" instead of saying "pending - application is not available to install".
We're still sorting through today's logs to determine why Apple/Jamf is dropping the licenses (and what we can do about it) but there seems to be light at the end of the tunnel for us...hopefully, others will start seeing forward movement as well.
Unfortunately, it looks like this is either model or iOS version specific, as Air 1 iPads seem to be (mostly) working for us today, but 7th gen iPads are getting absolutely nothing.
edit: who knows what's going on. I'm told we were setting up 7th gen iPads perfectly fine last weekend. Hopefully Apple can get this figured out quickly for us.
Yesterday at least all my apps had licenses, today they are all 0d out. 290 ios apps, and 85 Mac apps. Roughly 11k devies in total now SOL and we have testing windows starting Monday. I have not heard anything back from Apple or Jamf on this from either side after opening tickets.
According to Jamf support yesterday they didn't have any fix or options for the "Pending - Application is not available to install" outside of refreshing the licensing for each individual app. Jamf support did state that they had an enterprise support ticket open with Apple (#101145962620) and suggested I contact Apple and have them add our school district to the ongoing ticket.