02-23-2022 01:59 PM - edited 02-23-2022 02:10 PM
As this is already working correctly again, it's not really a super urgent issue and I mostly found it a minor annoyance that required a trip across town to my office from the site I was working at. Here's the scenario:
I was working on a couple of iPads that for reasons I won't get into needed a physical touch. After wiping them through Jamf I noticed that Self Service and all the apps scoped to the smart group it was in were taking their sweet time installing. I looked up the device records in Jamf and saw that the apps wouldn't install because there were no licenses. I thought, "that's weird, there should be plenty," and checked in VPP, only to discover that *gasp* all our app licenses were zeroed out. Everything in the Content tab under all categories, just showed "0" for Total, In-Use, and Reported.
Panicking slightly, I headed across town to my office and called Apple Support first while I got logged into ASM. The licenses were all there. So why aren't they being seen in Jamf? I check the server token, Good until 10/22/22. Licenses are still zeroed. Before i click the "Update Purchased Content" button and send my on-prem servers into cardiac arrest while they verify a gazillion app licenses, I hit "Edit" to see if there were any options I could look at.
In the "Edit" mode, all my app licenses are back! What in tarnation?? I click "Cancel", the app licenses are still there! I refresh the page. Yep, things are back to normal. In the mean time I've finally gotten an apple rep on the phone and I just sort of went "uhhhh I guess it's working now...?"
So for about an hour, my VPP token lost its mind I guess, and then magically fixed itself? Has anyone else had this issue???
Solved! Go to Solution.
Posted on 03-17-2022 09:21 AM
Welp, I'm glad some other people had this issue to at least validate my experience. Phew, they do like to make us sweat, don't they.
Posted on 04-18-2022 02:03 PM
Talked to Apple and have a case going there.
They looked at a copy of my VPP token and said there is nothing that should be coming through so that seemed to confirm that jamf isn't at fault and that Apple did something that essentially blanked out the account.
Apple engineering team is investigating and is supposed to get back to me.
Posted on 04-19-2022 04:44 AM
No good! - I did finally get a jamf and apple to open a "co-op" case together around 4:00pm central time. That is the last I have heard any updates on my end.
Posted on 04-19-2022 06:49 AM
Nothing really to add here except that we are seeing the same issue, started on Friday as well. As of today, we are able to get apps installed from the Self Service cart even though Jamf is still reporting all apps at 0. Please keep updating if you receive new information.
Posted on 04-19-2022 07:02 AM
I spoke with a customer success manager at jamf and he said it was helpful to submit as many tickets as possible. Your welcome to reference our cases jamf support case is CS0813412 and apple support case is 101679467651. Ask your jamf contact if your potentially being affected by PI104395 or PI101079. Lots of others having the same issue since last friday.
04-19-2022 07:13 AM - edited 04-19-2022 01:56 PM
We do have a case open as well, ours is CS0813463 if anybody wants to reference ours. Thanks for the Apple support case # and PI #s, good to know.
Edit: Our Apple support case # is 101681429370.
Posted on 04-19-2022 08:01 AM
In our JAMFSoftwareServerVPP.log it looks like the 9603 error is zeroing out the app counts one one of our tokens. That 9603 error is an Apple side error according to their documentation.
2022-04-19T08:13:11,784 [DEBUG] [ttp-epoll-2] [vppClientTraffic ] - [id: 0x8553af59, L:/192.168.0.38:50150 - R:vpp.itunes.apple.com/17.125.249.4:443] READ: 564B HTTP/1.1 200 OK
Date: Tue, 19 Apr 2022 13:13:10 GMT
x-apple-jingle-correlation-key: R6AB7RTGVCLQLPLNW3Z6X6SNR4
x-apple-request-uuid: 8f801fc6-66a8-9705-bd6d-b6f3ebfa4d8f
b3: 8f801fc666a89705bd6db6f3ebfa4d8f-97d9443c7528d8ad
x-b3-traceid: 8f801fc666a89705bd6db6f3ebfa4d8f
x-b3-spanid: 97d9443c7528d8ad
apple-seq: 0
apple-tk: false
apple-originating-system: MZVppMdm
x-responding-instance: MZVppMdm:012385:::
Content-Type: application/json
apple-timing-app: 1516 ms
Content-Length: 65
{"errorNumber":9603,"errorMessage":"Internal error.","status":-1}
Posted on 04-19-2022 09:10 AM
We started seeing this yesterday. Still licenses showing as zeroes. Did all VPP token re upload and server token fine too. All licenses there in ASM. Tried refreshing the apps in content didn't work, still 0's.
Posted on 04-19-2022 10:08 AM
I was told that jamf opened a Apple care enterprise ticket (101682355197) and that I should reach out to my Apple contacts and ask to be associated with it. I dont know if that means it's a ticket just for our instance or if thats like a globabl ticket jamf submitted to apple? Waiting to hear back from our Apple customer success manager currently.
Posted on 04-19-2022 11:19 AM
Figured I'd chime in. Same exact issues for us. Started last Thursday. Nothing good to report today other than Jamf verified it's the same issue for us. The workaround of using a different location/token isn't really viable due to having to transfer the licenses for 1,500 or so apps from the affected location to a new location, then updating the app assignments all manually too.
04-19-2022 03:17 PM - edited 04-19-2022 03:17 PM
I don’t know if it’s actually fixed yet but currently we show licenses in our MDM again and apps are being installed.
🥳
Hopefully this means the issue was found and resolved by Apple?
Anyone else see success currently?
Posted on 04-19-2022 03:19 PM
Yup, can confirm ours are back too!
Posted on 04-19-2022 04:09 PM
Looks like ours are back as well.
Posted on 04-19-2022 04:12 PM
Same, though they were coming in and out of Jamf all weekend. Curious if it *sticks*
Posted on 04-19-2022 04:23 PM
VPP licenses appear to be back for us as well.
Posted on 04-19-2022 05:57 PM
All of my licenses are showing as well - thanks everyone that put in tickets for this!
Posted on 04-20-2022 04:44 AM
Can confirm our licenses are back as of this morning! - I have not had an update from Apple or jamf on the cases mentioning that things are fixed though.
Posted on 04-20-2022 05:29 AM
Same here. Looks good, no confirmations yet from Apple or Jamf.
Posted on 04-20-2022 06:30 AM
Looks good for us as well, no word from Apple or Jamf as to what exactly got resolved?
Posted on 04-20-2022 06:37 AM
Apple did just confirm the issue is resolved but no details. Asked for details and can share if I get an answer.
I told Jamf and closed that case since correcting the issue wasn't on them.
Posted on 04-20-2022 08:22 AM
No details were given by Apple.
Posted on 04-20-2022 08:24 AM
Same here - no details at all.
Posted on 04-21-2022 05:19 AM
Hey Everyone,
I had a ticket open for the exact same problem. My licenses are back as well. Here is part of the email I received from the JAMF support team
@johntgeck wrote:As this is already working correctly again, it's not really a super urgent issue and I mostly found it a minor annoyance that required a trip across town to my office from the site I was working at. Here's the scenario:
I was working on a couple of iPads that for reasons I won't get into needed a physical touch. After wiping them through Jamf I noticed that Self Service and all the apps scoped to the smart group it was in were taking their sweet time installing. I looked up the device records in Jamf and saw that the apps wouldn't install because there were no licenses. I thought, "that's weird, there should be plenty," and checked in VPP, only to discover that *gasp* all our app licenses were zeroed out. Everything in the Content tab under all categories, just showed "0" for Total, In-Use, and Reported.
Panicking slightly, I headed across town to my office and called Apple Support first while I got logged into ASM. The licenses were all there. So why aren't they being seen in Jamf? I check the server token, Good until 10/22/22. Licenses are still zeroed. Before i click the "Update Purchased Content" button and send my on-prem servers into cardiac arrest while they verify a gazillion app licenses, I hit "Edit" to see if there were any options I could look at.
In the "Edit" mode, all my app licenses are back! What in tarnation?? I click "Cancel", the app licenses are still there! I refresh the page. Yep, things are back to normal. In the mean time I've finally gotten an apple rep on the phone and I just sort of went "uhhhh I guess it's working now...?"
So for about an hour, my VPP token lost its mind I guess, and then magically fixed itself? Has anyone else had this issue???
The following message is from Apple: This was a one off issue, and we don't expect the issue to reoccur. One of the load balancing nodes that helps with the VPP service was down, and Apple got it back online yesterday around 5 PM CT.