Large pkg upload failures

[rpayne]

Has anyone else encountered failures when uploading large files? I have been attempting to upload a pkg containing the installer for Catalina to our instance and the upload keeps failing.

We have a cloud instance. I have renamed the file multiple times. I have tried in Jamf admin AND directly through the Jamf pro Webb interface. I've tried it remote from my home office (gigabit fiber) both wired/wireless, and from both wired/wireless in office. I have had this happen in the past and it was never resolved. The pkg in question is 7.7gb. It has done the same with 2+gb files as well (office pkgs).

[tomt]

@mscottblake Good call, I didn't notice the number. That's what I get for posting before coffee...

[hdsreid]

Having this issue today...joy
can confirm its any size, mine is only 15MB

[jschlimmer]

I too put in a ticket and was told that it's being tracked in PI-005548... I dont think these are the same issues though.

[tomt]

I think these are all symptoms of the same general issue. There seems to be something broken in the communication between the Jamf Cloud web servers and their JCDS buckets in AWS.

The only common thing I have noticed with multiple postings is that everyone who has mentioned it is hosted on US-East.

[hrhnick]

Yup, US-East here as well...

[john-hsu]

Even though I'm on the west coast, I was provisioned with Jamf Cloud on US-East when we migrated from on-prem to cloud recently. So me too, US-East.

[gachowski]

I know, this might not help, but I change to Chrome for .pkg uploads and saw improvements. I was trying to upload 10.15.5 (8+ GB)

C

[tomt]

At this point I would use IE on XP if it would work.

[hdsreid]

i am us-east, interesting...
@gachowski my custom 20MB pkg from Composer keeps failing

[tomt]

Ok, I combined a couple of the workarounds posted and it seems to be working for me.

1. Replace any underscores with hyphens.
2. Use only Chrome to upload.

I usually use Safari for everything and I have been using underscores in package names for a decade. I had my first success with a hyphenated name in Chrome. I then tried an underscore name in Chrome and it failed. I then tried a hyphenated name in Safari and it failed.

I have now done half a dozen new uploads using the hyphen/Chrome combination. Quitting Chrome in between just to reset the connection as a test. They were all 100mb or smaller since those were what I had waiting to go.

Can a few others give this combo a try so I know if I just got lucky or if this is really a valid workaround?

[Polargeek]

For me. I've only ever been using chrome. Hyphen, underscores and spaces whether there or not seem to make no difference. Size makes no difference (had a 4Gb package work earlier), renaming after deleting the failed package makes no difference. No matter the combination used only about 1 in 15 uploads works.

[tomt]

That's why I'd love someone else to test a few files using the Chrome/hyphen combination. I'm going to try a larger package here in a few.

[Polargeek]

That's my point I've tried it and it failed at the same rate as any other combination.

[jmuchow]

@tomt I tried your suggestion to get the latest TeamViewer installer package (49 MB) uploaded to our JCDS. I named the package TeamViewer-15.7.6.pkg and uploaded with Chrome. It failed. I deleted this package from JamfPro, renamed the installer package to TeamViewer_15.7.6, uploaded with Chrome and it worked. I don't think there is a magic formula, at least I have not found one yet. It is just hit or miss and mostly miss.

[tomt]

[donmontalvo]

Checksum issue?

[drhoten]

Hello Everyone,

Thanks for all the comments and feedback, as the technical product owner working on the JCDS I wanted to reach out with an update after we've had some engineers take a closer look at what is going on and have been able to plan out our next steps.

While we are still working to identify all the underlying causes, we've seen a large increase in IOPs in our AWS infrastructure particularly in the last week. As a result, we are going to need to make some changes that will most likely include a planned outage of the JCDS in the near future. At the moment, we are not anticipating this change will need to be linked or associated with an upgrade or hotfix for Jamf Pro but will let you know if we learn otherwise.

Please know we are investigating these issues, planning our next steps, and have started actively working on the JCDS as this is currently the top priority for the teams working on this.

Once we have more details including a target date range, we’ll reach back out with an update.

[hrhnick]

@drhoten Are we looking at 48 hours, a week, a month? None of the workarounds like zipping or renaming the files are working for me.

[tomt]

@drhoten This is not a new issue, it has been happening sporadically for at least a year. Now that it blew up it's finally moved off the "ignore" list. What kind of a timetable are we looking at?

[swapple]

we have tried all the tricks and still cannot upload our 65mb installer pkg that we made in composer.

[sschramm]

My organization has also been impacted on this since late January 2020. Its now late June 2020 - FIVE MONTHS with no fix!
Yesterday I spent several hours trying to upload a simple 50MB package with a simple three character file name (e.g. JC7.pkg)
Safari, Firefox, Chrome, all failed. After about 8 tries I got it to work. It feels like I'm trying to buy tickets to a sellout concert from Ticketmaster when millions of people are trying to access the page at the same time.
I'm very disappointed in Jamf's transparency on this issue. At a minimum, the status page should at least be updated to reflect this. It's clear a lot of customers are having this same issue and it seems like Jamf is trying to hide that fact.

[ctworow]

I am having this upload issue, persists regardless of .pkg size. More frustrating, though, is packages are unable to be downloaded to endpoints from our JC Distribution Point, so any package policies fail to run with a 403 when attempting to download. This is true even of packages that were uploaded and functioning fine before this week.

[adam_macy1]

Same issues here. The only workaround I've found to be somewhat reliable is to upload directly through Jamf Pro (not Jamf Admin) on an Ethernet connection, after 9pm EST.

[drhoten]

Hello Everyone,

Yes @tomt, I agree this is definitely not all new and we did have work planned but the recent issues have now made it our priority. So to answer your, @hrhnick, and other's questions on timing here are our current plans. 

• Overnight, we're planning on performing server maintenance during non-peak hours.
• From there we'll continue to monitor the performance and error rates, and make additional changes if needed.
• Over the next several weeks, we'll be considering any changes that can be made to the JCDS and/or infrastructure which do not require changes to Jamf Pro. If changes can be made sooner, they will be.
• Finally, we'll also be looking at what additional changes to Jamf Pro are required.

[davidi4]

Thanks for the updated info - even sub 1MB uploads have been failing on me, and I cannot trust AutoPkg right now. Putting a bit of egg on my face because I have a VP and Director in my POC Testing ring...

[tomt]

@drhoten Thanks for the update. The "next several weeks" and "considering" part is a bit concerning. At this point your customers need action and not words. You mention that you had some remediation already planned. Since this has been an ongoing issue for at least a year, why was nothing done last year to prevent the current major problem?

[hrhnick]

I was hoping the overnight maintenance would have made a difference, but I still cannot get a package uploaded. If there was a workaround, or chance of success this issue would be tolerable, but I literally cannot do my work.

[tomt]

@hrhnick @drhoten I just had success uploading a few files using my normal procedure (Safari, underscores) so maybe the maintenance resolved the issue? The files were fairly small so I will do another test with a large file after lunch.

Just wanted to let people know.

[erichughes]

I have also been able to upload two packages today. One small and One large (Catalina installer) without having to change names or try a second or third time.

[drhoten]

Thanks for the updates @erichughes and @tomt.

To clarify, we are continuing to review and consider possible changes and are implementing what we can now vs. other options that may require longer maintenance windows for the JCDS or a coordinated upgrade of both Jamf Pro and the JCDS at the same time.

[hrhnick]

I have a 1.5gb Adobe package that is still failing yesterday and this morning, as well as a second smaller <100mb file that’s also failing.

[john-hsu]

After the Jamf Cloud update yesterday, I decided to try and do an initial sync of my on-prem distribution point to the Jamf Cloud Distribution point. It seems faster now, and I’ve gotten further along compared to the first time I tried syncing.

However, it looks like after about 24 hours of syncing I’m seeing constant retries of the same package again. It doesn’t matter if it is a 16 GB pkg file or a 1.5 MB file. I’m going to let it to continue for the time being, but thought I would chime in on my current experience.

[john-hsu]

Update: Now it's doing retries on 5kb files! (pkgs with Launchagents in them)

[drhoten]

Hello Everyone,

So folks are aware, we will be performing urgent maintenance to the JCDS for US Cloud customers overnight starting at 1 AM CST.

Emails with additional details about the changes and affected services have been sent out to all primary technical contacts and decision-makers for each Jamf Cloud customer serviced in our US Region.

To subscribe to status updates, see https://status.jamf.com/incidents/ksf6fsfttbfd.

[sschramm]

Can someone please confirm if these issues have been officially been resolved? In the past two weeks I've received a total of 6 emails (although maybe only 3...each one seemed to be duplicated) regarding emergency maintenance to correct these issues. The status page is now showing "clear", but the latest email I got says more maintenance is going to be completed on 7/11. It would be nice to get a confirmation on if this issue has actually been confirmed fixed or if this is still in the monitoring or troubleshooting stage. Thank you.

[hrhnick]

Uploads are still failing for me on US-East as of today.

[tomt]

So far so good today, uploads and replications working normally. Cloud Primary, US-East.

[john-hsu]

Over the past few days, I've been trying to get my Jamf Cloud Distribution point sync'd with my on-prem primary distribution point again since I've been experiencing a lot of issues. Things have been faster syncing overall, but any new files it's taking 4 to 5 times to upload to the JCDS – doesn't matter the size. One thing I keep noticing is that I get a warning message in the Casper Admin Sync Log that there is a "connection failure: "The operation couldn't be completed ( error 502.)"":

At the end of sync, I got a list of files that didn't upload - which was 6 this time (a lot better than in the past). I decided to run a sync again to see what would happen. It took about 15 seconds to run and I got a message saying "Replications of the drives complete." Huh? I don't know what to believe on this one. This is the first time I actually got a favorable message after syncing and am a little leery on the integrity of what's uploaded to JCDS.

[pipo]

Hi, we have the same issue. I am trying to upload Office 16.38 package to Jamf Pro - Akamai via webapp.

I am receiving the error: Timed out uploading icon.

We are in central Europe.

The change https://status.jamf.com/incidents/ksf6fsfttbfd. was only related to US?

br

[drhoten]

Hello @pipo

The changes for last week were specific to the US-based regions. We are in the process of planning the maintenance windows for all other regions and you should expect an email being sent out soon with the details for your region.