Hey!

just leaving a "Me too!" here to follow poosible news and solutions :)
And we got this Problem for Mac App Installation, too.

Greetings!

Dealing with this problem as well. Hosted on-prem for 8500 devices. Refreshing 500 individual apps is not a pleasant experience.

I feel this isn't getting prioritised as a fix? This seems to be affecting a lot of people and is making any newly enrolled devices pointless. Some information from Jamf would help. Even an explanation of what is happening and whether it's an apple issue or not.

I've seen this on and off over the past yeae, but usually clears itself out. This pass week has been a bit notorious, yet still the apps have installed properly after several hours.

I have several reports of parents not able to install Apps for learners at home for over a week. Staff is at a standstill with setting up iPads for new students. The problem is so widespread and inconsistent that troubleshooting is impossible. Refreshing individual Apps in VPP content has maybe helped but it also may be a placebo effect.

I feel this isn't getting prioritised as a fix? This seems to be affecting a lot of people and is making any newly enrolled devices pointless. Some information from Jamf would help. Even an explanation of what is happening and whether it's an apple issue or not.

This is what I got from JAMF support:

Regarding the app deployment issue, I'm afraid we're seeing this behavior in a number of environments, and it's even been reported in other MDMs. Apple has been adjusting a few things on their side to try and resolve this issue, but once it's fixed we'll need to refresh the app licenses one last time for any app affected. They've asked us to recommend refreshing app licenses, and closely monitor to see if the apps continue to fail after being refreshed. Please refresh those apps in Jamf Pro > Settings > Global Management > Volume Purchasing > Account in question > Content > Refresh next to any app that is affected. After refreshing, please let us know if the apps still fail to deploy after 30 minutes. I'll wait to hear from you.

I asked when this is going to be fixed because we have over 20K iPads and we can't keep doing this workaround. No response yet.

We are in the same boat. Teachers need apps for tomorrow and they are stuck as not available to install. I have an open case with Jamf but it doesnt look promising reading these replies.

Same problem here with ABM. I opened a ticket. Now we can't adopt Macs into Jamf properly, OR install apps either.

I enrolled a device this morning and the apps installed straight away so it looks like it might be fixed. Although, all the previous devices that where enrolled are still in the pending state as I guess they will all need refreshing to get the licenses. I'm going to have to ask Jamf to do this as I'm not refreshing 1000 apps manually.

Edit: Looks like this isn't working. I just realised that the iPad I setup this morning was a students iPad that we erased all content and settings on and set it up again. This meant the licenses for the iPad where still in jamf which is why it worked this morning. We didn't delete the iPad out of jamf before we re-set the ipad back up.

Having exact issues here as well. Intermittent issues last week, Friday was a day of 0 licenses Apple acknowledging the issue was also ABM (even though only School Manager was marked with the issue on the Status page) -- real rollercoaster of a few days here. I feel like Apple made an error code change and JAMF (and other MDMs) cannot or may not be able to interpret what to do with it so it just errors out itself.

Working with JAMF support, but at this point I think it's a combination Apple issue (making a change) and JAMF now needing to react to that change. So far there is no "easy button" fix, and I have to have our support team every hour or so go in and hit the Refresh button on all of our apps (I certainly ain't gonna do it, but they might claim workman's comp soon for their wrists and clicking 100 times each time they need to do it).

We're seeing some success this morning with students not only being able to download apps on newly enrolled iPads, but being able to download apps from self service when they were part of the group that had lost that functionality altogether. I'm not sure yet if it's Apple, JAMF, or both making them, but whatever changes are being made on the back end are having a significant, positive impact on our end. 

I'm hope we get a detailed incident report that lists the root cause and remediation of these issues once the dust settles. I'm thankful people at both Apple and JAMF have been working together to resolve the this, but these issues have made for a rough start to the year for many students and teachers in our district. Any lessons that can be learned to prevent issues like it in the future would benefit all of us if shared openly.

We are coming up on a full week of not being able to deploy devices. We've got tickets open with Jamf and Apple, but neither have given much of an update. Which sucks because there's no indication on how long this could drag out. Thankfully for us, we were in the middle of a year long migration from Airwatch/WorkspaceONE and have been flipping DEP devices back to that server to get them up and running.

We worked with Jamf support and added some debugging on our jss server today. Worked with Jamf support to generate some more info and they're taking it to engineering. I'll post if we hear anything that may help.

After days of sending debug logs and loads of guided (and unguided) troubleshooting the problem resists all of our troubleshooting. I did get a response yesterday (Tue) from Jamf that mentioned 2 key points.
1. They (Jamf) are working closely with Apple to find a "complete solution"
2. Jamf is currently testing a workaround that would reset the license connection between Apple's servers and Jamf Pro but this may not be viable until they can test its impact.

Regarding point #2, there was some allusion in their response that the temporary workaround could adversely affect the performance of the Jamf Pro server depending on the number of licenses that are being managed.

I think my next step is to open a case with Apple in the hopes that the more we pile on the more pressure it will apply to properly resolve this once and for all.

I'm having exactly the same issue on newer iPads as well. We have around 200 iPads 7th Gen, and some randomly have this issue.
I'm posting this alternative workaround that currently fixed the issue for me, and I found it easier / faster than refresh each single app.

Generally our app are scoped to the department the iPad is part of.
I therefore unscope the iPads from their department, re-scope them and send a blank push. I see then the apps reinstalling without problems.

Knowing that this is very likely not a permanent solution, I hope nevertheless that it helps you as well.

It's frustrating that we all seem to be dealing with different people at jamf support and uploading logs. It sounds like we're all having the same issue so it would make sense to try centralise everyone's case. This would allow other jamf support people to get on with something else instead of spending time on an issue that seems to be already known.

I have had this issue in the past, and just had it happen again. So far my experience is concerned, it has usually been a lack of amount of licenses to cover the total devices scoped. You can't scope an App to more 'possible' devices than you have licenses to cover. In that case the solution is usually to either reduce your scope to be within the number of licenses you have available, or go to ASM and purchase/transfer enough licenses to cover the scope you are targeting.

In this current case mine was in the Mac App Store and being distributed via Self Service.
I upped my licenses and deleted the App from the the JSS, and rebuilt it.
That only resulted in a "Pending - Application is not available to install" error.
So I then built yet another App distribution, but this time I set it to 'Install Automatically' the App and this time it went through.
So, make sure you have the numbers to cover everything first.

I think something funky is going on with the App Store since they issued the new licensing guidelines so make sure you log into ASM and confirm the new licensing guidelines too. That could be contributing.

"On September 16, 2020 Apple will post updated versions of operating system software license agreements to Apple School Manager. Once posted, your organization won’t be able to enroll devices or deploy new apps until an administrator has signed in to Apple School Manager and accepted the new terms."

We are seeing the light. Worked with Jamf and gathered tons of logs. Ran a command from the mysql command line tool and things are finally installing without a hitch within 2 minutes of enrollment. We have some more testing to do, but it's looking like things are fixed for us.

@nsbickhart Do you have a case # that I can refer my JAMF support ticket to so hopefully I can get those mysql commands as well?

@nsbickhart What command did they have you run?

OK, my rep just got ahold of me with some info on this - currently doing a database backup in preparation before trying. Will update...

I'd rather not post the command because it may have been specific to our environment and I'm honestly not 100% sure what it did lol, but it worked. Case info is below

Case #: JAMF-1491612 (Incident Ticket)

OK, so I want to do more testing to make sure, but things do seem to be working a lot better after applying the 'fix' (I almost hesitate to call it that as it was basically removing 1 corrupted record from the vpp_licenses table in the database)...

Before executing the fix, I was seeing tonnes of errors in my VPP.log file, kind of like:

2020-09-11 13:48:03,627 [DEBUG] [eralPool-14] [viceAssignResponseHandler] - VPP License: 1725275771 already assigned for app: 378813687. Updating...
2020-09-11 13:48:03,628 [DEBUG] [eralPool-14] [viceAssignResponseHandler] - VPP License: 1725274570 already assigned for app: 378813687. Updating...
2020-09-11 13:48:03,419 [DEBUG] [eralPool-11] [viceAssignResponseHandler] - ManagedLicensesResponse(status: -1 Adam ID: 873198123 Association[ Client User ID Str: null Serial Number: XXXXXXX License ID Str: null Error Message: License already assigned Error Number: 9616 License Already Assigned[ License ID Str: 1727878071 Client User ID Str: null Serial Number: XXXXXXXXX]]  Association[ Client User ID Str: null Serial Number: XXXX License ID Str: null Error Message: License already assigned Error Number: 9616 License Already Assigned[ License ID Str: 1727878125 Client User ID Str: null Serial Number: XXXX]]  Association[ Client User ID Str: null Serial Number: XXXXX License ID Str: null Error Message: License already assigned Error Number: 9616 License Already Assigned[ License ID Str: 1727878055 Client User ID Str: null Serial Number: XXXXX]]  Association[ Client User ID Str: null Serial Number: XXXXX License ID Str: null Error Message: License already assigned Error Number: 9616 License Already Assigned[ License ID Str: 1727878054 Client User ID Str: null Serial Number: XXXXXX]]  Association[ Client User ID Str: null Serial Number: XXXXXX License ID Str: null Error Message: License already assigned Error Number: 9616 License Already Assigned[ License ID Str: 1727877980 Client User ID Str: null Serial Number: XXXXX]]  Association[ Client User ID Str: null Serial Number: XXXXX License ID Str: null Error Message: License already assigned Error Number: 9616 License Already Assigned[ License ID Str: 1727878171 Client User ID Str: null Serial Number: XXXXX]]  Association[ Client User ID Str: null Serial Number: XXXXXX License ID Str: null Error Message: License already assigned Error Number: 9616 License Already Assigned[ License ID Str: 1727878131 Client User ID Str: null Serial Number: XXXXXX]]  Association[ Client User ID Str: null Serial Number: XXXXXXX License ID Str: null Error Message: License already assigned Error Number: 9616 License Already Assigned[ License ID Str: 1727878167 Client User ID Str: null Serial Number: XXXXX]]  Association[ Client User ID Str: null Serial Number: XXXXX License ID Str: null Error Message: License already assigned Error Number: 9616 License Already Assigned[ License ID Str: 1727878066 Client User ID Str: null Serial Number: XXXXX]]  Association[ Client User ID Str: null Serial Number: XXXXXX License ID Str: null Error Message: License already assigned Error Number: 9616 License Already Assigned[ License ID Str: 1727878035 Client User ID Str: null Serial Number: XXXXX]] )
2020-09-11 13:48:03,419 [ERROR] [eralPool-11] [VppCommService           ] - Error managing licenses. ErrorNumber: 0 , ErrorMessage:null
2020-09-11 13:48:03,468 [DEBUG] [neralPool-7] [viceAssignResponseHandler] - VPP License: 1727891965 already assigned for app: 611452042. Updating...
2020-09-11 13:48:03,471 [DEBUG] [neralPool-7] [viceAssignResponseHandler] - VPP License: 1727891935 already assigned for app: 611452042. Updating...
2020-09-11 13:48:03,472 [DEBUG] [neralPool-7] [viceAssignResponseHandler] - ManagedLicensesResponse(status: -1 Adam ID: 611452042 Association[ Client User ID Str: null Serial Number: XXXXXXXXXX License ID Str: null Error Message: License already assigned Error Number: 9616 License Already Assigned[ License ID Str: 1727891965 Client User ID Str: null Serial Number: XXXXXXX]]  Association[ Client User ID Str: null Serial Number: XXXXXXXXXX License ID Str: null Error Message: License already assigned Error Number: 9616 License Already Assigned[ License ID Str: 1727891935 Client User ID Str: null Serial Number: F9FD84K9LMPD]] )

Afterwords I'm seeing much more reassuring messages, more like this, without all the errors:

2020-09-17 14:22:26,201 [DEBUG] [duledPool-4] [VppCommService           ] - Performing license request using the previous since modified token.
2020-09-17 14:22:26,201 [DEBUG] [duledPool-4] [VppLicenseUpdater        ] - Updating VPP licenses for admin account 4
2020-09-17 14:22:26,474 [DEBUG] [eralPool-16] [ResponseMonitor          ] - Put a response in the out queue.
2020-09-17 14:22:26,474 [DEBUG] [eralPool-16] [ResponseMonitor          ] - 1 of 1 DonePills have been received.
2020-09-17 14:22:26,474 [DEBUG] [eralPool-16] [ResponseMonitor          ] - Finished the ResponseMonitor having seen 1 Responses. The final pill was: com.jamfsoftware.vpp.comm.parallel.Drop$DonePill@428540d9
2020-09-17 14:22:27,936 [DEBUG] [eralPool-68] [AssignmentService        ] - Filtered 10 licenses down to 0 licenses
2020-09-17 14:22:27,949 [DEBUG] [eralPool-74] [AssignmentService        ] - Filtered 528 licenses down to 0 licenses
2020-09-17 14:22:28,986 [DEBUG] [eralPool-68] [AssignmentService        ] - Filtered 148 licenses down to 0 licenses
2020-09-17 14:22:30,167 [DEBUG] [eralPool-68] [AssignmentService        ] - Filtered 7 licenses down to 0 licenses
2020-09-17 14:22:30,703 [DEBUG] [eralPool-74] [AssignmentService        ] - Filtered 184 licenses down to 0 licenses
2020-09-17 14:22:30,741 [DEBUG] [eralPool-68] [AssignmentService        ] - Filtered 214 licenses down to 0 licenses
2020-09-17 14:22:36,775 [DEBUG] [eralPool-68] [AssignmentService        ] - Filtered 1659 licenses down to 0 licenses
2020-09-17 14:22:38,735 [INFO ] [duledPool-4] [VppLicenseMonitor        ] - License monitor completed after 13.6 seconds
2020-09-17 14:23:38,735 [INFO ] [duledPool-3] [VppLicenseMonitor        ] - Running license monitor
2020-09-17 14:23:38,735 [DEBUG] [duledPool-3] [VppLicenseMonitor        ] - Updating VPP services directory

Verify

If you can do a mysql query against your database, you can at least verify if you have the same issue as us by running the following query:

select * from vpp_licenses where apple_license_id=-1;

If you find matching records, wait a minute or two and rerun the query - if you get the same records, from what I was told, you probably have this issue.

I'm hypothesizing that there was some kind of change or hiccup with Apple's servers that resulted in a corrupt record in random JAMF servers, thus triggering this issue for random customers.

Fix

Since this involves modifying your database directly, proceed with extreme caution, and get help if you're unsure what you're doing.

The 'fix' was simply to backup the database, shut down Tomcat, remove that corrupted record and then restart Tomcat, at which point the issue seems to be resolving itself.

DISCLAIMER: The following is what worked for me in my environment. Each environment is unique and will likely require additional or different steps. This was all performed under the direction of a Jamf engineer in realtime. A.K.A. If you're experiencing the issues described in this discussion contact Jamf for guided help.
DO NOT PERFORM THESE STEPS.

TL;DR
1. Stop tomcat on everything
2. Backup the Jamf Database
3. Run this command in the MySQL DB: delete from vpp_licenses where apple_license_id=-1;
4. Restart tomcat on everything
＿ ＿ ＿ ＿ ＿ ＿ ＿ ＿ ＿ ＿ ＿ ＿ ＿ ＿ ＿ ＿ ＿ ＿

Just got off the phone with Jamf support. Earlier today they had me run this in the DB:

select * from vpp_licenses where apple_license_id=-1;

Which confirmed that I had a row in the 'jamfsoftware' database with the license id of "-1".

The fix, as @fsjjeff described was to remove that entry from the database.
After performing a backup up the DB I ran the following in MySQL:

delete from vpp_licenses where apple_license_id=-1;

Once tomcat was running again on the Jamf servers I was able to confirm, for the first time in days, clean server logs. From my initial testing iOS apps are installing without issue. Some mac apps still say they're "Pending - Application is not available to install" but as the license monitor re-runs it's clearing most of them out.

Jamf's wrap-up email referenced PI-008838.

@palmna We received the same instructions and things are working much better as well.