That's why I'd love someone else to test a few files using the Chrome/hyphen combination. I'm going to try a larger package here in a few.
That's my point I've tried it and it failed at the same rate as any other combination.
@tomt I tried your suggestion to get the latest TeamViewer installer package (49 MB) uploaded to our JCDS. I named the package TeamViewer-15.7.6.pkg and uploaded with Chrome. It failed. I deleted this package from JamfPro, renamed the installer package to TeamViewer_15.7.6, uploaded with Chrome and it worked. I don't think there is a magic formula, at least I have not found one yet. It is just hit or miss and mostly miss.
Hello Everyone,
Thanks for all the comments and feedback, as the technical product owner working on the JCDS I wanted to reach out with an update after we've had some engineers take a closer look at what is going on and have been able to plan out our next steps.
While we are still working to identify all the underlying causes, we've seen a large increase in IOPs in our AWS infrastructure particularly in the last week. As a result, we are going to need to make some changes that will most likely include a planned outage of the JCDS in the near future. At the moment, we are not anticipating this change will need to be linked or associated with an upgrade or hotfix for Jamf Pro but will let you know if we learn otherwise.
Please know we are investigating these issues, planning our next steps, and have started actively working on the JCDS as this is currently the top priority for the teams working on this.
Once we have more details including a target date range, we’ll reach back out with an update.
@drhoten Are we looking at 48 hours, a week, a month? None of the workarounds like zipping or renaming the files are working for me.
@drhoten This is not a new issue, it has been happening sporadically for at least a year. Now that it blew up it's finally moved off the "ignore" list. What kind of a timetable are we looking at?
we have tried all the tricks and still cannot upload our 65mb installer pkg that we made in composer.
My organization has also been impacted on this since late January 2020. Its now late June 2020 - FIVE MONTHS with no fix!
Yesterday I spent several hours trying to upload a simple 50MB package with a simple three character file name (e.g. JC7.pkg)
Safari, Firefox, Chrome, all failed. After about 8 tries I got it to work. It feels like I'm trying to buy tickets to a sellout concert from Ticketmaster when millions of people are trying to access the page at the same time.
I'm very disappointed in Jamf's transparency on this issue. At a minimum, the status page should at least be updated to reflect this. It's clear a lot of customers are having this same issue and it seems like Jamf is trying to hide that fact.
I am having this upload issue, persists regardless of .pkg size. More frustrating, though, is packages are unable to be downloaded to endpoints from our JC Distribution Point, so any package policies fail to run with a 403 when attempting to download. This is true even of packages that were uploaded and functioning fine before this week.
Same issues here. The only workaround I've found to be somewhat reliable is to upload directly through Jamf Pro (not Jamf Admin) on an Ethernet connection, after 9pm EST.
Hello Everyone,
Yes @tomt, I agree this is definitely not all new and we did have work planned but the recent issues have now made it our priority. So to answer your, @hrhnick, and other's questions on timing here are our current plans.
- Overnight, we're planning on performing server maintenance during non-peak hours.
- From there we'll continue to monitor the performance and error rates, and make additional changes if needed.
- Over the next several weeks, we'll be considering any changes that can be made to the JCDS and/or infrastructure which do not require changes to Jamf Pro. If changes can be made sooner, they will be.
- Finally, we'll also be looking at what additional changes to Jamf Pro are required.
Thanks for the updated info - even sub 1MB uploads have been failing on me, and I cannot trust AutoPkg right now. Putting a bit of egg on my face because I have a VP and Director in my POC Testing ring...
@drhoten Thanks for the update. The "next several weeks" and "considering" part is a bit concerning. At this point your customers need action and not words. You mention that you had some remediation already planned. Since this has been an ongoing issue for at least a year, why was nothing done last year to prevent the current major problem?
I was hoping the overnight maintenance would have made a difference, but I still cannot get a package uploaded. If there was a workaround, or chance of success this issue would be tolerable, but I literally cannot do my work.
@hrhnick @drhoten I just had success uploading a few files using my normal procedure (Safari, underscores) so maybe the maintenance resolved the issue? The files were fairly small so I will do another test with a large file after lunch.
Just wanted to let people know.
I have also been able to upload two packages today. One small and One large (Catalina installer) without having to change names or try a second or third time.
Thanks for the updates @erichughes and @tomt.
To clarify, we are continuing to review and consider possible changes and are implementing what we can now vs. other options that may require longer maintenance windows for the JCDS or a coordinated upgrade of both Jamf Pro and the JCDS at the same time.
I have a 1.5gb Adobe package that is still failing yesterday and this morning, as well as a second smaller <100mb file that’s also failing.
After the Jamf Cloud update yesterday, I decided to try and do an initial sync of my on-prem distribution point to the Jamf Cloud Distribution point. It seems faster now, and I’ve gotten further along compared to the first time I tried syncing.
However, it looks like after about 24 hours of syncing I’m seeing constant retries of the same package again. It doesn’t matter if it is a 16 GB pkg file or a 1.5 MB file. I’m going to let it to continue for the time being, but thought I would chime in on my current experience.
Update: Now it's doing retries on 5kb files! (pkgs with Launchagents in them)
Hello Everyone,
So folks are aware, we will be performing urgent maintenance to the JCDS for US Cloud customers overnight starting at 1 AM CST.
Emails with additional details about the changes and affected services have been sent out to all primary technical contacts and decision-makers for each Jamf Cloud customer serviced in our US Region.
To subscribe to status updates, see https://status.jamf.com/incidents/ksf6fsfttbfd.
Can someone please confirm if these issues have been officially been resolved? In the past two weeks I've received a total of 6 emails (although maybe only 3...each one seemed to be duplicated) regarding emergency maintenance to correct these issues. The status page is now showing "clear", but the latest email I got says more maintenance is going to be completed on 7/11. It would be nice to get a confirmation on if this issue has actually been confirmed fixed or if this is still in the monitoring or troubleshooting stage. Thank you.
Uploads are still failing for me on US-East as of today.
