Has anyone else encountered failures when uploading large files? I have been attempting to upload a pkg containing the installer for Catalina to our instance and the upload keeps failing.
We have a cloud instance. I have renamed the file multiple times. I have tried in Jamf admin AND directly through the Jamf pro Webb interface. I've tried it remote from my home office (gigabit fiber) both wired/wireless, and from both wired/wireless in office. I have had this happen in the past and it was never resolved. The pkg in question is 7.7gb. It has done the same with 2+gb files as well (office pkgs).
I think these are all symptoms of the same general issue. There seems to be something broken in the communication between the Jamf Cloud web servers and their JCDS buckets in AWS.
The only common thing I have noticed with multiple postings is that everyone who has mentioned it is hosted on US-East.
Ok, I combined a couple of the workarounds posted and it seems to be working for me.
I usually use Safari for everything and I have been using underscores in package names for a decade. I had my first success with a hyphenated name in Chrome. I then tried an underscore name in Chrome and it failed. I then tried a hyphenated name in Safari and it failed.
I have now done half a dozen new uploads using the hyphen/Chrome combination. Quitting Chrome in between just to reset the connection as a test. They were all 100mb or smaller since those were what I had waiting to go.
Can a few others give this combo a try so I know if I just got lucky or if this is really a valid workaround?
For me. I've only ever been using chrome. Hyphen, underscores and spaces whether there or not seem to make no difference. Size makes no difference (had a 4Gb package work earlier), renaming after deleting the failed package makes no difference. No matter the combination used only about 1 in 15 uploads works.
@tomt I tried your suggestion to get the latest TeamViewer installer package (49 MB) uploaded to our JCDS. I named the package TeamViewer-15.7.6.pkg and uploaded with Chrome. It failed. I deleted this package from JamfPro, renamed the installer package to TeamViewer_15.7.6, uploaded with Chrome and it worked. I don't think there is a magic formula, at least I have not found one yet. It is just hit or miss and mostly miss.
Thanks for all the comments and feedback, as the technical product owner working on the JCDS I wanted to reach out with an update after we've had some engineers take a closer look at what is going on and have been able to plan out our next steps.
While we are still working to identify all the underlying causes, we've seen a large increase in IOPs in our AWS infrastructure particularly in the last week. As a result, we are going to need to make some changes that will most likely include a planned outage of the JCDS in the near future. At the moment, we are not anticipating this change will need to be linked or associated with an upgrade or hotfix for Jamf Pro but will let you know if we learn otherwise.
Please know we are investigating these issues, planning our next steps, and have started actively working on the JCDS as this is currently the top priority for the teams working on this.
Once we have more details including a target date range, we’ll reach back out with an update.
My organization has also been impacted on this since late January 2020. Its now late June 2020 - FIVE MONTHS with no fix!
Yesterday I spent several hours trying to upload a simple 50MB package with a simple three character file name (e.g. JC7.pkg)
Safari, Firefox, Chrome, all failed. After about 8 tries I got it to work. It feels like I'm trying to buy tickets to a sellout concert from Ticketmaster when millions of people are trying to access the page at the same time.
I'm very disappointed in Jamf's transparency on this issue. At a minimum, the status page should at least be updated to reflect this. It's clear a lot of customers are having this same issue and it seems like Jamf is trying to hide that fact.
I am having this upload issue, persists regardless of .pkg size. More frustrating, though, is packages are unable to be downloaded to endpoints from our JC Distribution Point, so any package policies fail to run with a 403 when attempting to download. This is true even of packages that were uploaded and functioning fine before this week.
Yes @tomt, I agree this is definitely not all new and we did have work planned but the recent issues have now made it our priority. So to answer your, @hrhnick, and other's questions on timing here are our current plans.
@drhoten Thanks for the update. The "next several weeks" and "considering" part is a bit concerning. At this point your customers need action and not words. You mention that you had some remediation already planned. Since this has been an ongoing issue for at least a year, why was nothing done last year to prevent the current major problem?
To clarify, we are continuing to review and consider possible changes and are implementing what we can now vs. other options that may require longer maintenance windows for the JCDS or a coordinated upgrade of both Jamf Pro and the JCDS at the same time.
After the Jamf Cloud update yesterday, I decided to try and do an initial sync of my on-prem distribution point to the Jamf Cloud Distribution point. It seems faster now, and I’ve gotten further along compared to the first time I tried syncing.
However, it looks like after about 24 hours of syncing I’m seeing constant retries of the same package again. It doesn’t matter if it is a 16 GB pkg file or a 1.5 MB file. I’m going to let it to continue for the time being, but thought I would chime in on my current experience.
So folks are aware, we will be performing urgent maintenance to the JCDS for US Cloud customers overnight starting at 1 AM CST.
Emails with additional details about the changes and affected services have been sent out to all primary technical contacts and decision-makers for each Jamf Cloud customer serviced in our US Region.
To subscribe to status updates, see https://status.jamf.com/incidents/ksf6fsfttbfd.
Can someone please confirm if these issues have been officially been resolved? In the past two weeks I've received a total of 6 emails (although maybe only 3...each one seemed to be duplicated) regarding emergency maintenance to correct these issues. The status page is now showing "clear", but the latest email I got says more maintenance is going to be completed on 7/11. It would be nice to get a confirmation on if this issue has actually been confirmed fixed or if this is still in the monitoring or troubleshooting stage. Thank you.
Over the past few days, I've been trying to get my Jamf Cloud Distribution point sync'd with my on-prem primary distribution point again since I've been experiencing a lot of issues. Things have been faster syncing overall, but any new files it's taking 4 to 5 times to upload to the JCDS – doesn't matter the size. One thing I keep noticing is that I get a warning message in the Casper Admin Sync Log that there is a "connection failure: "The operation couldn't be completed ( error 502.)"":
At the end of sync, I got a list of files that didn't upload - which was 6 this time (a lot better than in the past). I decided to run a sync again to see what would happen. It took about 15 seconds to run and I got a message saying "Replications of the drives complete." Huh? I don't know what to believe on this one. This is the first time I actually got a favorable message after syncing and am a little leery on the integrity of what's uploaded to JCDS.