Skip to main content
Solved

Jamf Pro gaslighting me? VPP licenses zeroed out, reappear with no changes.

  • February 23, 2022
  • 62 replies
  • 401 views

Show first post

62 replies

KCOURTEAU
Forum|alt.badge.img+1
  • New Contributor
  • April 18, 2022

Just wanted to add my organization to the list - we have ~30 distinct VPP locations and only 2 locations (from 2 different ASM instances) are showing 0 for all license content. For one of the locations, I renewed the ASM Server token and then the VPP token, which didn't have any noticeable effect.

 

As others have reported, I recall this happening in the past (for sure with 1 of these locations even), but it cleared itself up within a day or so. I noticed this issue late last week and hoped it would clear up over the weekend.


Same problem here this morning. Guess it was a problem last week too but we had a bunch of weather events so I wasn't even in JAMF to notice. Our primary VPP location with all non-paid apps is reporting 0s across the board. The other 15ish locations are fine.


jr139
Forum|alt.badge.img+6
  • New Contributor
  • April 18, 2022

my suggestion to all would be call jamf and apple and bug them both about it, hoping that strength in numbers makes them both dump some resources at this issue. 

You are welcome to reference our case numbers, jamf support case is CS0813412 and apple support case is 101679467651. I am still working on getting a meeting between both sides engineers scheduled. 


Forum|alt.badge.img+5
  • April 18, 2022

my suggestion to all would be call jamf and apple and bug them both about it, hoping that strength in numbers makes them both dump some resources at this issue. 

You are welcome to reference our case numbers, jamf support case is CS0813412 and apple support case is 101679467651. I am still working on getting a meeting between both sides engineers scheduled. 


Just got off the phone with jamf. Other than a few fixes that worked for some (refreshing a license, renewing the vpptoken, etc... No real help.

Hopefully you're able to get Apple and jamf on the phone together so they don't just point at each other.


Forum|alt.badge.img+9
  • Contributor
  • April 18, 2022

We're seeing this at our organization as well. It presumably also happened Friday, but we didn't have school that day so no one noticed until this morning. Already tried refreshing content and loading a new server token, but we're still zeroed out across the board and nothing can obviously install, so we're completely crippled on new device setups.

Also on-prem if that seems to be a common factor. 


KCOURTEAU
Forum|alt.badge.img+1
  • New Contributor
  • April 18, 2022

We're seeing this at our organization as well. It presumably also happened Friday, but we didn't have school that day so no one noticed until this morning. Already tried refreshing content and loading a new server token, but we're still zeroed out across the board and nothing can obviously install, so we're completely crippled on new device setups.

Also on-prem if that seems to be a common factor. 


Whatever it is it's affecting both on-prem and cloud, as we are hosted by JAMF.


Forum|alt.badge.img+3
  • New Contributor
  • April 18, 2022

Same here. Been following this thread as well as the slack channel. Multiple people have the same problem. I've also reached out to Jamf and Apple support. Renewing tokens doesn't seem to make a difference. ASM appears to be working fine but things arent syncing back to Jamf. Is everyone affected education sector using ASM? 

Also hosted, not on prem.


jr139
Forum|alt.badge.img+6
  • New Contributor
  • April 18, 2022

Whatever it is it's affecting both on-prem and cloud, as we are hosted by JAMF.


Cloud here as well, using ASM. I have been trying to set a meeting with both sides engineers since this morning and havent made it super far. Jamf does seem receptive to it but getting apple aligned with that has been a sturggle (even though apple suggested a meeting between the two). If possible, open a case with jamf so they are aware of the issue. My jamf case is CS0813412. I was just told a bit ago that this is "potentially" related to a jamf product issue and they were working to verify that. 


KCOURTEAU
Forum|alt.badge.img+1
  • New Contributor
  • April 18, 2022

Same here. Been following this thread as well as the slack channel. Multiple people have the same problem. I've also reached out to Jamf and Apple support. Renewing tokens doesn't seem to make a difference. ASM appears to be working fine but things arent syncing back to Jamf. Is everyone affected education sector using ASM? 

Also hosted, not on prem.


Yup, another ASM user here.


jr139
Forum|alt.badge.img+6
  • New Contributor
  • April 18, 2022

Yup, another ASM user here.


ASM here as well, I did see atleast one ABM user on slack who reported the issue. It seems like it could be affecting both. 


Forum|alt.badge.img+5
  • April 18, 2022

Same here. Been following this thread as well as the slack channel. Multiple people have the same problem. I've also reached out to Jamf and Apple support. Renewing tokens doesn't seem to make a difference. ASM appears to be working fine but things arent syncing back to Jamf. Is everyone affected education sector using ASM? 

Also hosted, not on prem.


ASM, but I believe I have seen others with ABM.


Forum|alt.badge.img+5
  • April 18, 2022

Cloud here as well, using ASM. I have been trying to set a meeting with both sides engineers since this morning and havent made it super far. Jamf does seem receptive to it but getting apple aligned with that has been a sturggle (even though apple suggested a meeting between the two). If possible, open a case with jamf so they are aware of the issue. My jamf case is CS0813412. I was just told a bit ago that this is "potentially" related to a jamf product issue and they were working to verify that. 


I was just cited a known issue as a possible cause: PI101079

When I looked it up I didn't find anything, so maybe a new known product issue that hasn't had details released yet?


jr139
Forum|alt.badge.img+6
  • New Contributor
  • April 18, 2022

I was just cited a known issue as a possible cause: PI101079

When I looked it up I didn't find anything, so maybe a new known product issue that hasn't had details released yet?


Thanks for that. I was told it may be a potential PI but they didnt know yet. I asked what the PI number was but havent heard back. 


Forum|alt.badge.img+7
  • Contributor
  • April 18, 2022

I'm strictly ABM, on prem 10.37.2
I don't really think it is related to the version of Jamf. We upgraded the first of last week and I didn't see this problem until Friday.


jr139
Forum|alt.badge.img+6
  • New Contributor
  • April 18, 2022

I was just cited a known issue as a possible cause: PI101079

When I looked it up I didn't find anything, so maybe a new known product issue that hasn't had details released yet?


Looks like PI101079 got logged with our case as well. Syncs from jamf to ASM/VPP failing, in turn causing jamf to report zero licenses. 

 

For others - If you have submitted a case for this issue and they haven't associated it with PI101079, I would suggest contacting jamf and telling them you believe you might be running into this PI. 


Forum|alt.badge.img+5
  • April 18, 2022

Talked to Apple and have a case going there.

They looked at a copy of my VPP token and said there is nothing that should be coming through so that seemed to confirm that jamf isn't at fault and that Apple did something that essentially blanked out the account.

Apple engineering team is investigating and is supposed to get back to me.


jr139
Forum|alt.badge.img+6
  • New Contributor
  • April 19, 2022

Talked to Apple and have a case going there.

They looked at a copy of my VPP token and said there is nothing that should be coming through so that seemed to confirm that jamf isn't at fault and that Apple did something that essentially blanked out the account.

Apple engineering team is investigating and is supposed to get back to me.


No good! - I did finally get a jamf and apple to open a "co-op" case together around 4:00pm central time. That is the last I have heard any updates on my end. 


Forum|alt.badge.img+2
  • New Contributor
  • April 19, 2022

Nothing really to add here except that we are seeing the same issue, started on Friday as well. As of today, we are able to get apps installed from the Self Service cart even though Jamf is still reporting all apps at 0. Please keep updating if you receive new information.


jr139
Forum|alt.badge.img+6
  • New Contributor
  • April 19, 2022

Nothing really to add here except that we are seeing the same issue, started on Friday as well. As of today, we are able to get apps installed from the Self Service cart even though Jamf is still reporting all apps at 0. Please keep updating if you receive new information.


I spoke with a customer success manager at jamf and he said it was helpful to submit as many tickets as possible.  Your welcome to reference our cases jamf support case is CS0813412 and apple support case is 101679467651. Ask your jamf contact if your potentially being affected by PI104395 or PI101079. Lots of others having the same issue since last friday.


Forum|alt.badge.img+2
  • New Contributor
  • April 19, 2022

I spoke with a customer success manager at jamf and he said it was helpful to submit as many tickets as possible.  Your welcome to reference our cases jamf support case is CS0813412 and apple support case is 101679467651. Ask your jamf contact if your potentially being affected by PI104395 or PI101079. Lots of others having the same issue since last friday.


We do have a case open as well, ours is CS0813463 if anybody wants to reference ours. Thanks for the Apple support case # and PI #s, good to know.

Edit: Our Apple support case # is 101681429370.


Forum|alt.badge.img+9
  • Contributor
  • April 19, 2022

In our JAMFSoftwareServerVPP.log it looks like the 9603 error is zeroing out the app counts one one of our tokens. That 9603 error is an Apple side error according to their documentation.

2022-04-19T08:13:11,784 [DEBUG] [ttp-epoll-2] [vppClientTraffic ] - [id: 0x8553af59, L:/192.168.0.38:50150 - R:vpp.itunes.apple.com/17.125.249.4:443] READ: 564B HTTP/1.1 200 OK Date: Tue, 19 Apr 2022 13:13:10 GMT x-apple-jingle-correlation-key: R6AB7RTGVCLQLPLNW3Z6X6SNR4 x-apple-request-uuid: 8f801fc6-66a8-9705-bd6d-b6f3ebfa4d8f b3: 8f801fc666a89705bd6db6f3ebfa4d8f-97d9443c7528d8ad x-b3-traceid: 8f801fc666a89705bd6db6f3ebfa4d8f x-b3-spanid: 97d9443c7528d8ad apple-seq: 0 apple-tk: false apple-originating-system: MZVppMdm x-responding-instance: MZVppMdm:012385::: Content-Type: application/json apple-timing-app: 1516 ms Content-Length: 65 {"errorNumber":9603,"errorMessage":"Internal error.","status":-1}

 

 


Forum|alt.badge.img+3
  • New Contributor
  • April 19, 2022

We started seeing this yesterday. Still licenses showing as zeroes. Did all VPP token re upload and server token fine too. All licenses there in ASM. Tried refreshing the apps in content didn't work, still 0's.


jr139
Forum|alt.badge.img+6
  • New Contributor
  • April 19, 2022

I was told that jamf opened a Apple care enterprise ticket (101682355197) and that I should reach out to my Apple contacts and ask to be associated with it. I dont know if that means it's a ticket just for our instance or if thats like a globabl ticket jamf submitted to apple? Waiting to hear back from our Apple customer success manager currently. 


Forum|alt.badge.img+2
  • New Contributor
  • April 19, 2022

Figured I'd chime in. Same exact issues for us. Started last Thursday. Nothing good to report today other than Jamf verified it's the same issue for us. The workaround of using a different location/token isn't really viable due to having to transfer the licenses for 1,500 or so apps from the affected location to a new location, then updating the app assignments all manually too.


Forum|alt.badge.img+5
  • April 19, 2022

I don’t know if it’s actually fixed yet but currently we show licenses in our MDM again and apps are being installed.
🥳


Hopefully this means the issue was found and resolved by Apple?

Anyone else see success currently?


Forum|alt.badge.img+3
  • New Contributor
  • April 19, 2022

I don’t know if it’s actually fixed yet but currently we show licenses in our MDM again and apps are being installed.
🥳


Hopefully this means the issue was found and resolved by Apple?

Anyone else see success currently?


Yup, can confirm ours are back too!