Jamf Pro gaslighting me? VPP licenses zeroed out, reappear with no changes.

[johntgeck]

all zeroesback to normalAs this is already working correctly again, it's not really a super urgent issue and I mostly found it a minor annoyance that required a trip across town to my office from the site I was working at. Here's the scenario:

I was working on a couple of iPads that for reasons I won't get into needed a physical touch. After wiping them through Jamf I noticed that Self Service and all the apps scoped to the smart group it was in were taking their sweet time installing. I looked up the device records in Jamf and saw that the apps wouldn't install because there were no licenses. I thought, "that's weird, there should be plenty," and checked in VPP, only to discover that *gasp* all our app licenses were zeroed out. Everything in the Content tab under all categories, just showed "0" for Total, In-Use, and Reported.

Panicking slightly, I headed across town to my office and called Apple Support first while I got logged into ASM. The licenses were all there. So why aren't they being seen in Jamf? I check the server token, Good until 10/22/22. Licenses are still zeroed. Before i click the "Update Purchased Content" button and send my on-prem servers into cardiac arrest while they verify a gazillion app licenses, I hit "Edit" to see if there were any options I could look at.

In the "Edit" mode, all my app licenses are back! What in tarnation?? I click "Cancel", the app licenses are still there! I refresh the page. Yep, things are back to normal. In the mean time I've finally gotten an apple rep on the phone and I just sort of went "uhhhh I guess it's working now...?"

So for about an hour, my VPP token lost its mind I guess, and then magically fixed itself? Has anyone else had this issue???

[emily]

I saw something very similar today! We have lots of licenses and saw 0s everywhere. I refreshed one app and then everything reappeared. Smells like an Apps & Books issue to me. I also saw some odd behavior today when I was in our ABM portal trying to add additional licenses for apps and got some weird error messages.

[cbrewer]

Same thing here today. Happened around 1:30p pacific.

[timbyler1890]

That is interesting. We had the same thing happen to us on the 23rd. I had apps that showed zero and also had a Location disappear from ABM. Before I got Appel called, things seemed to go back to normal. At least I know that I wasn't just going crazy.

[johntgeck · Accepted solution]

Welp, I'm glad some other people had this issue to at least validate my experience. Phew, they do like to make us sweat, don't they.

[aarond]

Figured I'd jump in here rather than start a new post since it's the exact same issue and save the explaining.

We're experiencing this right now.

We have had it happen before but it resolved itself after a couple hours.

This time it's been intermittent but mostly showing 0 licenses for 24 hours now.
Has anyone else experienced this again in the past day?

Wondering if it's a widespread VPP issue or something we are alone in.

I saw Apple listed a VPP outage on April 12th for about 2.5 hours - wondering if that is related?

Other thoughts or ideas?

[jr139]

Were experiencing this as well. Zero licenses, started overnight. Renewed the VPP token (which was NOT expired) to see if that would resolve it. Had a support case open for hours, no resolution yet. Called Apple around 10:00am central time and opened a case, they walked through the normal stuff and said stuff looks good on their end. 

Glad to know we are not alone I guess. Best of luck!

[timbyler1890]

I just checked and two out of my three locations are showing 0 apps. I don't know when it started as Jamf hasn't given me an alert as yet. It just seems to be on my main production server, our Mosyle instance and our Jamf test sever are all showing that I have licenses available

So far all my locations are still showing in ABM which is a good thing.

[jr139]

We checked and all of our licenses in ASM and they are all there and fine. So perhaps this is on Apples end if you are having problems in mosyle as well.

[jr139]

If anyone wants to reference them, my jamf support case is CS0813412 and apple support case is 101679467651 for this zeroed out license issue we are seeing today. 

[aarond]

I've had a jamf case opened for 24 hours now and haven't heard a word from them on this case.
Thanks for sharing case numbers. I often share them when jamf tells me they haven't heard of an issue we're looking at but I see others talking about the same issue and have open cases.

Haven't jumped in with Apple yet.
Was thinking about checking with our regional Apple engineer.

[jr139]

I just sent an email to our Apple SE for the region but they are out of the office until Tuesday. I will post back here if I get any traction from either side. Happy Friday!

[aarond]

Interesting that other instances and servers are up.

Thanks for sharing and letting me know I'm not alone. :)

[timbyler1890]

I'm pretty sure that this is an Apple issue as there is some chatter on the Jamf Nation channel on Slack. I'm going to see if I can't ride it out till Monday and see what happens by then.

[tusdafif]

We are experiencing the same thing here at my organization, Apps are not able to be downloaded via Self Service and newly enrolled devices do not catch any of the apps as well. We renewed our tokens and that didn't seem to solve it either. 

What was the solution for everyone?  I don't if I should start contacting Apple - Thanks

[jr139]

No resolution here over the weekend, still showing zero licenses (have tried re-uploading token too). Apple says it's not their issue on my case 101679467651. Contacted jamf this morning giving them an update and asking for next steps. 

[msnowdon]

Same here. Was trying to figure why no apps were coming down and noticed that all my licenses are "0" for one instance. It was fine yesterday. Assuming a problem on Apple's end since I now know Im not the only one.  Maybe I can stop panicking now.

[jefferyhoffman]

So, I just updated Jamf Pro last week, went home for spring break, then came back to this.  After a quick heart attack, I checked here and see other people with the same issue.  Has anyone figured out a solution for this issue, or are you all still running zeroes since Friday?

[timbyler1890]

I'm still down as of this morning. I'm wondering how many of you are on Jamf Cloud or on an on prem system? Is there a difference between the cloud servers and those that are on prem? All of my systems are on prem.

[jefferyhoffman]

on prem as well

[jr139]

Still showing zero licenses this morning. Was told by jamf it looked like a connection issue between jamf server (cloud for us) and ASM based on our VPP logs, which I don't totally buy (see below for reasons why). Trying to get our jamf engineer and Apple engineer on a call together currently as this is what was suggested by our Apple customer success manager. 

Licenses always display correctly for us in ASM, only having issues with them displaying in jamf cloud instance. Re-uploading the VPP token for the site we have all our licenses in did not resolve the issue. We also re-uploaded the jamf pro key into ASM and then the ASM token back into jamf per the reccoemendation of the Apple engineer. None of which resolved the issue. 

We did find that if we uploaded a VPP token for an unsused site, the licenses show up just fine (although it was only 50 licenses total). Not sure why jamf is having issues decifering the main VPP token with all of our licenses. Our license count is over 700,000 on the token having issues. 

[aarond]

On prem also.

[wkelly1]

Just wanted to add my organization to the list - we have ~30 distinct VPP locations and only 2 locations (from 2 different ASM instances) are showing 0 for all license content. For one of the locations, I renewed the ASM Server token and then the VPP token, which didn't have any noticeable effect.

As others have reported, I recall this happening in the past (for sure with 1 of these locations even), but it cleared itself up within a day or so. I noticed this issue late last week and hoped it would clear up over the weekend.

[aarond]

To be clear, you do have multiple installations that are showing license and app installations are possible?

We had this happen previously and like you said, it cleared up after a day or so and I was pretty sure this would clear up by Monday and it hasn't yet.

[wkelly1]

Yes, that is correct - 28 of our 30 VPP locations are showing accurate license counts and are allowing app installations, but 2 locations are reporting 0 licenses and (unsurprisingly) not allowing app installations.

Since I didn't mention this in my previous comment, we are on-prem and on Jamf Pro 10.37.2

[aarond]

Interesting.

Despite multiple people experiencing this, it's not a total blackout like I thought it would be.

On prem and a few version behind. Pro 34.2

[KCOURTEAU]

Same problem here this morning. Guess it was a problem last week too but we had a bunch of weather events so I wasn't even in JAMF to notice. Our primary VPP location with all non-paid apps is reporting 0s across the board. The other 15ish locations are fine.

[jr139]

my suggestion to all would be call jamf and apple and bug them both about it, hoping that strength in numbers makes them both dump some resources at this issue. 

You are welcome to reference our case numbers, jamf support case is CS0813412 and apple support case is 101679467651. I am still working on getting a meeting between both sides engineers scheduled. 

[aarond]

Just got off the phone with jamf. Other than a few fixes that worked for some (refreshing a license, renewing the vpptoken, etc... No real help.

Hopefully you're able to get Apple and jamf on the phone together so they don't just point at each other.

[JKoopman]

We're seeing this at our organization as well. It presumably also happened Friday, but we didn't have school that day so no one noticed until this morning. Already tried refreshing content and loading a new server token, but we're still zeroed out across the board and nothing can obviously install, so we're completely crippled on new device setups.

Also on-prem if that seems to be a common factor. 

[KCOURTEAU]

Whatever it is it's affecting both on-prem and cloud, as we are hosted by JAMF.

[jr139]

Cloud here as well, using ASM. I have been trying to set a meeting with both sides engineers since this morning and havent made it super far. Jamf does seem receptive to it but getting apple aligned with that has been a sturggle (even though apple suggested a meeting between the two). If possible, open a case with jamf so they are aware of the issue. My jamf case is CS0813412. I was just told a bit ago that this is "potentially" related to a jamf product issue and they were working to verify that. 

[aarond]

I was just cited a known issue as a possible cause: PI101079

When I looked it up I didn't find anything, so maybe a new known product issue that hasn't had details released yet?

[jr139]

Thanks for that. I was told it may be a potential PI but they didnt know yet. I asked what the PI number was but havent heard back. 

[jr139]

Looks like PI101079 got logged with our case as well. Syncs from jamf to ASM/VPP failing, in turn causing jamf to report zero licenses. 

For others - If you have submitted a case for this issue and they haven't associated it with PI101079, I would suggest contacting jamf and telling them you believe you might be running into this PI. 

[eizaguirre]

Same here. Been following this thread as well as the slack channel. Multiple people have the same problem. I've also reached out to Jamf and Apple support. Renewing tokens doesn't seem to make a difference. ASM appears to be working fine but things arent syncing back to Jamf. Is everyone affected education sector using ASM? 

Also hosted, not on prem.

[KCOURTEAU]

Yup, another ASM user here.

[jr139]

ASM here as well, I did see atleast one ABM user on slack who reported the issue. It seems like it could be affecting both. 

[aarond]

ASM, but I believe I have seen others with ABM.

[timbyler1890]

I'm strictly ABM, on prem 10.37.2
I don't really think it is related to the version of Jamf. We upgraded the first of last week and I didn't see this problem until Friday.