Large pkg upload failures

rpayne
Contributor II

Has anyone else encountered failures when uploading large files? I have been attempting to upload a pkg containing the installer for Catalina to our instance and the upload keeps failing.

We have a cloud instance. I have renamed the file multiple times. I have tried in Jamf admin AND directly through the Jamf pro Webb interface. I've tried it remote from my home office (gigabit fiber) both wired/wireless, and from both wired/wireless in office. I have had this happen in the past and it was never resolved. The pkg in question is 7.7gb. It has done the same with 2+gb files as well (office pkgs).

84 REPLIES 84

tomt
Valued Contributor

@mscottblake Good call, I didn't notice the number. That's what I get for posting before coffee...

hdsreid
Contributor III

Having this issue today...joy
can confirm its any size, mine is only 15MB

jschlimmer
New Contributor III

I too put in a ticket and was told that it's being tracked in PI-005548... I dont think these are the same issues though.

tomt
Valued Contributor

I think these are all symptoms of the same general issue. There seems to be something broken in the communication between the Jamf Cloud web servers and their JCDS buckets in AWS.

The only common thing I have noticed with multiple postings is that everyone who has mentioned it is hosted on US-East.

hrhnick
New Contributor III

Yup, US-East here as well...

john-hsu
New Contributor III

Even though I'm on the west coast, I was provisioned with Jamf Cloud on US-East when we migrated from on-prem to cloud recently. So me too, US-East.

gachowski
Valued Contributor II

I know, this might not help, but I change to Chrome for .pkg uploads and saw improvements. I was trying to upload 10.15.5 (8+ GB)

C

tomt
Valued Contributor

At this point I would use IE on XP if it would work.

hdsreid
Contributor III

i am us-east, interesting...
@gachowski my custom 20MB pkg from Composer keeps failing

tomt
Valued Contributor

Ok, I combined a couple of the workarounds posted and it seems to be working for me.

  1. Replace any underscores with hyphens.
  2. Use only Chrome to upload.

I usually use Safari for everything and I have been using underscores in package names for a decade. I had my first success with a hyphenated name in Chrome. I then tried an underscore name in Chrome and it failed. I then tried a hyphenated name in Safari and it failed.

I have now done half a dozen new uploads using the hyphen/Chrome combination. Quitting Chrome in between just to reset the connection as a test. They were all 100mb or smaller since those were what I had waiting to go.

Can a few others give this combo a try so I know if I just got lucky or if this is really a valid workaround?

Polargeek
New Contributor II

For me. I've only ever been using chrome. Hyphen, underscores and spaces whether there or not seem to make no difference. Size makes no difference (had a 4Gb package work earlier), renaming after deleting the failed package makes no difference. No matter the combination used only about 1 in 15 uploads works.

tomt
Valued Contributor

That's why I'd love someone else to test a few files using the Chrome/hyphen combination. I'm going to try a larger package here in a few.

Polargeek
New Contributor II

That's my point I've tried it and it failed at the same rate as any other combination.

jmuchow
New Contributor III

@tomt I tried your suggestion to get the latest TeamViewer installer package (49 MB) uploaded to our JCDS. I named the package TeamViewer-15.7.6.pkg and uploaded with Chrome. It failed. I deleted this package from JamfPro, renamed the installer package to TeamViewer_15.7.6, uploaded with Chrome and it worked. I don't think there is a magic formula, at least I have not found one yet. It is just hit or miss and mostly miss.

tomt
Valued Contributor

56e541d22e634f2b8ed86c5d20ccb05f

donmontalvo
Esteemed Contributor III

Checksum issue?

--
https://donmontalvo.com

drhoten
Contributor II

Hello Everyone,

Thanks for all the comments and feedback, as the technical product owner working on the JCDS I wanted to reach out with an update after we've had some engineers take a closer look at what is going on and have been able to plan out our next steps.

While we are still working to identify all the underlying causes, we've seen a large increase in IOPs in our AWS infrastructure particularly in the last week. As a result, we are going to need to make some changes that will most likely include a planned outage of the JCDS in the near future. At the moment, we are not anticipating this change will need to be linked or associated with an upgrade or hotfix for Jamf Pro but will let you know if we learn otherwise.

Please know we are investigating these issues, planning our next steps, and have started actively working on the JCDS as this is currently the top priority for the teams working on this.

Once we have more details including a target date range, we’ll reach back out with an update.

hrhnick
New Contributor III

@drhoten Are we looking at 48 hours, a week, a month? None of the workarounds like zipping or renaming the files are working for me.

tomt
Valued Contributor

@drhoten This is not a new issue, it has been happening sporadically for at least a year. Now that it blew up it's finally moved off the "ignore" list. What kind of a timetable are we looking at?

swapple
Contributor III

we have tried all the tricks and still cannot upload our 65mb installer pkg that we made in composer.

sschramm
New Contributor II

My organization has also been impacted on this since late January 2020. Its now late June 2020 - FIVE MONTHS with no fix!
Yesterday I spent several hours trying to upload a simple 50MB package with a simple three character file name (e.g. JC7.pkg)
Safari, Firefox, Chrome, all failed. After about 8 tries I got it to work. It feels like I'm trying to buy tickets to a sellout concert from Ticketmaster when millions of people are trying to access the page at the same time.
I'm very disappointed in Jamf's transparency on this issue. At a minimum, the status page should at least be updated to reflect this. It's clear a lot of customers are having this same issue and it seems like Jamf is trying to hide that fact.

ctworow
New Contributor

I am having this upload issue, persists regardless of .pkg size. More frustrating, though, is packages are unable to be downloaded to endpoints from our JC Distribution Point, so any package policies fail to run with a 403 when attempting to download. This is true even of packages that were uploaded and functioning fine before this week.

adam_macy1
New Contributor II

Same issues here. The only workaround I've found to be somewhat reliable is to upload directly through Jamf Pro (not Jamf Admin) on an Ethernet connection, after 9pm EST.

drhoten
Contributor II

Hello Everyone,

Yes @tomt, I agree this is definitely not all new and we did have work planned but the recent issues have now made it our priority. So to answer your, @hrhnick, and other's questions on timing here are our current plans. 

  • Overnight, we're planning on performing server maintenance during non-peak hours.
  • From there we'll continue to monitor the performance and error rates, and make additional changes if needed.
  • Over the next several weeks, we'll be considering any changes that can be made to the JCDS and/or infrastructure which do not require changes to Jamf Pro. If changes can be made sooner, they will be.
  • Finally, we'll also be looking at what additional changes to Jamf Pro are required.

davidi4
Contributor

Thanks for the updated info - even sub 1MB uploads have been failing on me, and I cannot trust AutoPkg right now. Putting a bit of egg on my face because I have a VP and Director in my POC Testing ring...

tomt
Valued Contributor

@drhoten Thanks for the update. The "next several weeks" and "considering" part is a bit concerning. At this point your customers need action and not words. You mention that you had some remediation already planned. Since this has been an ongoing issue for at least a year, why was nothing done last year to prevent the current major problem?

hrhnick
New Contributor III

I was hoping the overnight maintenance would have made a difference, but I still cannot get a package uploaded. If there was a workaround, or chance of success this issue would be tolerable, but I literally cannot do my work.

tomt
Valued Contributor

@hrhnick @drhoten I just had success uploading a few files using my normal procedure (Safari, underscores) so maybe the maintenance resolved the issue? The files were fairly small so I will do another test with a large file after lunch.

Just wanted to let people know.

erichughes
Contributor II

I have also been able to upload two packages today. One small and One large (Catalina installer) without having to change names or try a second or third time.

drhoten
Contributor II

Thanks for the updates @erichughes and @tomt.

To clarify, we are continuing to review and consider possible changes and are implementing what we can now vs. other options that may require longer maintenance windows for the JCDS or a coordinated upgrade of both Jamf Pro and the JCDS at the same time.

hrhnick
New Contributor III

I have a 1.5gb Adobe package that is still failing yesterday and this morning, as well as a second smaller <100mb file that’s also failing.

john-hsu
New Contributor III

After the Jamf Cloud update yesterday, I decided to try and do an initial sync of my on-prem distribution point to the Jamf Cloud Distribution point. It seems faster now, and I’ve gotten further along compared to the first time I tried syncing.

However, it looks like after about 24 hours of syncing I’m seeing constant retries of the same package again. It doesn’t matter if it is a 16 GB pkg file or a 1.5 MB file. I’m going to let it to continue for the time being, but thought I would chime in on my current experience.

john-hsu
New Contributor III

Update: Now it's doing retries on 5kb files! (pkgs with Launchagents in them)

drhoten
Contributor II

Hello Everyone,

So folks are aware, we will be performing urgent maintenance to the JCDS for US Cloud customers overnight starting at 1 AM CST.

Emails with additional details about the changes and affected services have been sent out to all primary technical contacts and decision-makers for each Jamf Cloud customer serviced in our US Region.

To subscribe to status updates, see https://status.jamf.com/incidents/ksf6fsfttbfd.

sschramm
New Contributor II

Can someone please confirm if these issues have been officially been resolved? In the past two weeks I've received a total of 6 emails (although maybe only 3...each one seemed to be duplicated) regarding emergency maintenance to correct these issues. The status page is now showing "clear", but the latest email I got says more maintenance is going to be completed on 7/11. It would be nice to get a confirmation on if this issue has actually been confirmed fixed or if this is still in the monitoring or troubleshooting stage. Thank you.

hrhnick
New Contributor III

Uploads are still failing for me on US-East as of today.

tomt
Valued Contributor

So far so good today, uploads and replications working normally. Cloud Primary, US-East.

john-hsu
New Contributor III

Over the past few days, I've been trying to get my Jamf Cloud Distribution point sync'd with my on-prem primary distribution point again since I've been experiencing a lot of issues. Things have been faster syncing overall, but any new files it's taking 4 to 5 times to upload to the JCDS – doesn't matter the size. One thing I keep noticing is that I get a warning message in the Casper Admin Sync Log that there is a "connection failure: "The operation couldn't be completed ( error 502.)"":

fd95c922ca9d47a8b22b594285599bf0

At the end of sync, I got a list of files that didn't upload - which was 6 this time (a lot better than in the past). I decided to run a sync again to see what would happen. It took about 15 seconds to run and I got a message saying "Replications of the drives complete." Huh? I don't know what to believe on this one. This is the first time I actually got a favorable message after syncing and am a little leery on the integrity of what's uploaded to JCDS.

pipo
New Contributor II

Hi, we have the same issue. I am trying to upload Office 16.38 package to Jamf Pro - Akamai via webapp.

I am receiving the error: Timed out uploading icon.

We are in central Europe.

The change https://status.jamf.com/incidents/ksf6fsfttbfd. was only related to US?

br

drhoten
Contributor II

Hello @pipo

The changes for last week were specific to the US-based regions. We are in the process of planning the maintenance windows for all other regions and you should expect an email being sent out soon with the details for your region.