Version 11 - Availability Pending takes 60 minutes to clear no matter the package size

[scrjeff]

I'm curious if any others using JAMF cloud with the cloud distribution point are seeing issues with uploaded packages, or any size, taking 60 minutes to clear the "availability pending" notice?  This seems to have started with the 11.x updates.  For example, a 200MB chrome package took 60 minutes (almost to the second) as did a 6GB pre-zipped Adobe package.  It's as if the 60 minutes point is an internal timeout where it just gives up and assumes all is well.   The packages work and can be deployed, but I'm also seeing random machines that deploy the package, and others that will return package download errors.   It's as if the package isn't being replicated properly in AWS, and if a client happens to get the wrong backend, it's as if the package doesn't exist. 

I have a support case open but I'm struggling with the responsiveness of the agent. I get one email per day and that's it. 

[jamf-42]

jamf / jcds has been.. wobbly for the last few days.. see: https://status.jamf.com

this maybe the impact.. 

[scrjeff]

I'm aware of the current issues, but this has been ongoing for much longer than the current wobbly. I guess my issue could be the precursor event to what they are dealing with, but the support rep hasn't mentioned it.  

[jamf-42]

there is also the jdcs / IP changes... if your endpoints are on prem.. firewall rules changes required..

[scrjeff]

That's not it. That's all taken care of and I see a mix of on-prem and at-home devices running into the same thing.  I know that JAMF uses cloudfront and I'm wondering if the service issues with their "third party" was them. Packages are fronted by the service, and this distributed issue i'm seeing may be them. 

[jamf-42]

testing MS office 2.42GB once uploaded available within 30 seconds.. 

[New2JAMF]

Yep, somewhere between December  2-3 and 11th something major changed. All packages are failing to install via policy or self service. All of our apps are affected too. Been going back and forth with support this week who said it was OUR network but that is not the case. So far the "fix" is to redo ALL of our packages and upload again. The JAMF distribution point in settings is empty aside from the one package I rebuilt this morning, so it's almost like our repo changed or wiped or something..

[scrjeff]

I have a case open, and support found there to be a lot of JCDS errors in their backend logs, and my agent has escalated further to their product specialist.   For me, it's been inconsistent even with the same package, and "feels" like a replication issue within their CDN (CloudFront?).  Some machines get the packages and others don't. 

This is what the agent had to say:

I pulled logs from our end and I'm seeing a lot of " 2023-12-06T18:55:22,185 [ERROR] [duledPool-3] [LoggingCcmErrorHandler ] - Response code 404 received when sending request to null." errors in the JCDS logs.

[New2JAMF]

Since commenting earlier, I emailed them twice with further findings and the confirmed/admitted about an issue that occurred from the JCDS 1 to 2 migration and asked if I would like to have the existing packages restored. I'm definitely frustrated that they couldn't/haven't identified customers that were/might've been affected. 

Luckily we don't have a ton of users/devices affected, but being a small shop, it would've taken us a while to re-building stuff. While our situations aren't cut and paste, I hope that they're narrowing down your issue!

[scrjeff]

Do you have a case number? I'd like to reference it with my guy.  My case is CS1081379.  And an update five minutes ago mentioned this as the possible ca: PI111180 - package_upload_status table can become bloated. 

[New2JAMF]

CS1086258 is what mine is and I'll update here once I get the confirmation they've finished restoring and whether that was our issue or if its something more complex

[New2JAMF]

I have confirmed several successful deployments now after they emailed stating the recover was complete. 

[dthrall]

Jamf got back to me and gave me a solution that worked:

The very first thing we’d like to do is re-authenticate to Cloud Services Connection as this is what provides connectivity between Jamf Pro and the Cloud Distribution Point. Please navigate to Jamf Pro > Settings > Global > Cloud Services Connection - even if this looks good (showing a green check mark), let’s re-enable this with a Jamf ID associated with your account (the same you’d use to access account.jamf.com).

Can you please now go to Jamf Pro > Settings > Server > Cloud distribution point > click “Edit” - [DO NOT MODIFY ANYTHING] - click “Save”.

This process will reinitiate the authentication of Jamf Pro to the backend of JCDS.As always, if there is anything that we can clarify, or if you hit any roadblocks with these steps, please don’t hesitate to reach out.

[scrjeff]

Already been down that road and it didn't help the issue.  I've been keeping track of my uploads and adding date/times to my ticket so that the backend people can look at the logs.  So far, one the 60 min timeout occurs, the packages do seem OK and appear to distribute.  

[Utilizator]

This appears to have worked for me - thanks for the info... one more glitch patch to add to the ever growing list!

[user-VwlRMRYoow]

This worked for me. Appreciate the clear steps.

[JasonWHand]

It still helping and just fixed it for me too.  Thanks.