Random "network connection was interrupted" errors

[jmahlman]

JSS 9.100, RHEL6. HTTP distribution point and failover

For the past few weeks (2-3) we've been seeing random policy errors coming in. All errors are

The network connection was interrupted while downloading the package from http://<SERVER/PACKAGE>. Attempting to reconnect... Downloading http://<SERVER/PACKAGE>... Error: <PACKAGE> is not available on the HTTP server.

It seems to happen mainly with laptops over wireless but we've also been seeing it happen with some wired clients now. The odd thing is that we'll check the logs and notice that the user would run that policy 2-3 more times after the failed attempt and it would succeed every subsequent time (below seems to be a more excited user).

It's not just self service, it's also recurring policies. It's also not happening on one specific network/subnet, it seems to be very random.

Anyone have any ideas or suggestions?

[robertojok]

We've also had this issue since upgrading to the latest version. Packages are stated as not being present via http...

[steve_summers]

Put me in the I've seen this ghost too, category. Happens mostly when/while my New Build process is running. We run thin images, stuff triggers as soon as the machine is joined to the JSS. I moved away from the Enrollment Complete trigger for the bulk of my package installs and have the "Login" trigger. Anyway, I see this too, annoy's the crap outta me..

[marklamont]

we get this quite a lot, but as HTTP downloads self resume it usually fixes itself. We put in the HTTP distros, running on SMB distros, to improve reliability. We still get it on a new RHEL that only does distribution and has no jss or anything else on.

[jmahlman]

Funny enough, caching seems to work flawlessly...

[jmahlman]

Spoke too soon. Caching also does this sometimes. Just not as much.

Very frustrating!

[tlarkin]

Are these packages flat packages?

[jmahlman]

@tlarkin It's DMGs and PKGs.

[tlarkin]

I guess my next question would be, when this happens can you curl down the package manually, or copy the URL into a browser and download?

[jmahlman]

It downloads fine.

We've been trying on our different subnets that show errors and it seems to work fine every time.

It doesn't seem like an issue with networking because the same policy will have 1 error (on subnet 10.48.x.x) and 50 successes on the same subnet.

We thought it was an issue with the user not letting the policy finish by either closing the computer or disconnecting but if we flush the policy on the machine it'll error again and we can repeat that and it might succeed.

Nothing on our distro point has changed either, it's been working fine for over a year now.

[Asnyder]

I definitely see this as well. I just assumed it was from students closing their laptops between classes. Usually flushing errors will fix the issue. You can curl the file fine and copy/paste the url and it downloads fine.

[jmahlman]

@Asnyder We're thinking that as well, we typically don't push out to student laptops because of this but we switched AVs and need to get it out.

I wonder if power nap will cause a machine to check in and fail...

[jmahlman]

Still getting these randomly throughout the day. I opened a ticket with jamf but since I cannot replicate it on-demand with a verbose policy run the ticket went stale and closed. Not really sure what to do, it's very frustrating for our users.

[Asnyder]

@jmahlman We still see it as well. I just try to be proactive about flushing errors. Sometimes that can be a hard task with only two in our district's IT department.

[jmahlman]

We haven't seen any errors since we upgraded to 9.101. Not sure if it's because things are dying down here or because it's actually fixed.

[analog_kid]

I'm seeing it on 9.101 calling a manually triggered package install policy from a Self Service policy. Not all machines do it but I have a test laptop that I can reproduce it almost 100% of the time.

[donmontalvo]

Cough...cough...loadBalanceYourRHL6distributionPoints...cough...cough

[jmahlman]

So I'm re-opening my post, we're seeing tons of failures again, even when it's a small amount of policies being run. It seems to happen mostly with self service policies.

Log 1: Self service policy, laptop on wireless:

Actions from policy log:
    [STEP 1 of 9]
    Executing Policy Xerox Printer Drivers and Utilities
    [STEP 2 of 9]
    Downloading HP-PrinterDrivers-3.0.pkg...
    Downloading http://[DISTRO1]:8008/CasperShare/Packages/HP-PrinterDrivers-3.0.pkg...
    Could not connect to the HTTP server to download HP-PrinterDrivers-3.0.pkg.
    Retrying using distribution point [DISTRO2]...
    Downloading HP-PrinterDrivers-3.0.pkg...
    Resuming download of http://[DISTRO2]/CasperShare/Packages/HP-PrinterDrivers-3.0.pkg...
    Error: Could not connect to the HTTP server to download HP-PrinterDrivers-3.0.pkg.
    [STEP 3 of 9]
    Downloading Pharos-Popup-9.0.8.pkg...
    Downloading http://[DISTRO1]:8008/CasperShare/Packages/Pharos-Popup-9.0.8.pkg...
    Could not connect to the HTTP server to download Pharos-Popup-9.0.8.pkg.
    Retrying using distribution point [DISTRO2]...
    Downloading Pharos-Popup-9.0.8.pkg...
    Downloading http://[DISTRO2]/CasperShare/Packages/Pharos-Popup-9.0.8.pkg...
    Error: Could not connect to the HTTP server to download Pharos-Popup-9.0.8.pkg.
    [STEP 4 of 9]
    Downloading http://[DISTRO1]:8008/CasperShare/Packages/Xerox-3300MFP-Driver.dmg...
    Could not connect to the HTTP server to download Xerox-3300MFP-Driver.dmg.
    Retrying using distribution point [DISTRO2]...
    Downloading http://[DISTRO2]/CasperShare/Packages/Xerox-3300MFP-Driver.dmg...
    Error: Could not connect to the HTTP server to download Xerox-3300MFP-Driver.dmg.
    [STEP 5 of 9]
    Downloading Xerox-Phaser-3260-Driver-1.02.pkg...
    Downloading http://[DISTRO1]:8008/CasperShare/Packages/Xerox-Phaser-3260-Driver-1.02.pkg...
    Could not connect to the HTTP server to download Xerox-Phaser-3260-Driver-1.02.pkg.
    Retrying using distribution point [DISTRO2]...
    Downloading Xerox-Phaser-3260-Driver-1.02.pkg...
    Downloading http://[DISTRO2]/CasperShare/Packages/Xerox-Phaser-3260-Driver-1.02.pkg...
    Error: Could not connect to the HTTP server to download Xerox-Phaser-3260-Driver-1.02.pkg.
    [STEP 6 of 9]
    Downloading Xerox-PL-Print-Driver-2.0.15.pkg...
    Downloading http://[DISTRO1]:8008/CasperShare/Packages/Xerox-PL-Print-Driver-2.0.15.pkg...
    Could not connect to the HTTP server to download Xerox-PL-Print-Driver-2.0.15.pkg.
    Retrying using distribution point [DISTRO2]...
    Downloading Xerox-PL-Print-Driver-2.0.15.pkg...
    Downloading http://[DISTRO2]/CasperShare/Packages/Xerox-PL-Print-Driver-2.0.15.pkg...
    The network connection was interrupted while downloading the package from http://[DISTRO2]/CasperShare/Packages/Xerox-PL-Print-Driver-2.0.15.pkg. Attempting to reconnect...
    Downloading Xerox-PL-Print-Driver-2.0.15.pkg...
    Downloading http://[DISTRO2]/CasperShare/Packages/Xerox-PL-Print-Driver-2.0.15.pkg...
    The network connection was interrupted while downloading the package from http://[DISTRO2]/CasperShare/Packages/Xerox-PL-Print-Driver-2.0.15.pkg. Attempting to reconnect...
    Downloading Xerox-PL-Print-Driver-2.0.15.pkg...
    Resuming download of http://[DISTRO2]/CasperShare/Packages/Xerox-PL-Print-Driver-2.0.15.pkg...
    Verifying package integrity...
    Installing Xerox-PL-Print-Driver-2.0.15.pkg...
    Successfully installed Xerox-PL-Print-Driver-2.0.15.pkg.
    [STEP 7 of 9]
    Downloading Xerox-Print-Driver-3.68.0.pkg...
    Downloading http://[DISTRO1]:8008/CasperShare/Packages/Xerox-Print-Driver-3.68.0.pkg...
    Could not connect to the HTTP server to download Xerox-Print-Driver-3.68.0.pkg.
    Retrying using distribution point [DISTRO2]...
    Downloading Xerox-Print-Driver-3.68.0.pkg...
    Resuming download of http://[DISTRO2]/CasperShare/Packages/Xerox-Print-Driver-3.68.0.pkg...
    Verifying package integrity...
    Installing Xerox-Print-Driver-3.68.0.pkg...
    Successfully installed Xerox-Print-Driver-3.68.0.pkg.
    [STEP 8 of 9]
    [STEP 9 of 9]

Log 2: Self service policy, hard wired iMac.

Actions from policy log:
    [STEP 1 of 6]
    Executing Policy Adobe Acrobat DC 18.0
    [STEP 2 of 6]
    Running script removeJAMFdownloadFolder.sh...
    Script exit code: 0
    Script result: 
    [STEP 3 of 6]
    Downloading http://[DISTRO1]:8008/CasperShare/Packages/Adobe_Acrobat_DC_18.0_Install.pkg.zip...
    Verifying package integrity...
    Installing Adobe_Acrobat_DC_18.0_Install.pkg.zip...
    Installation failed. The installer reported: installer: Package name is Adobe_Acrobat_DC_18.0
installer: Installing at base path /
installer: The install failed (The Installer encountered an error that caused the installation to fail. Contact the software manufacturer for assistance.)
    Retrying using distribution point [DISTRO2]...
    Downloading http://[DISTRO2]/CasperShare/Packages/Adobe_Acrobat_DC_18.0_Install.pkg.zip...
    The network connection was interrupted while downloading the package from http://[DISTRO2]/CasperShare/Packages/Adobe_Acrobat_DC_18.0_Install.pkg.zip. Attempting to reconnect...
    Downloading http://[DISTRO2]/CasperShare/Packages/Adobe_Acrobat_DC_18.0_Install.pkg.zip...
    Error: Adobe_Acrobat_DC_18.0_Install.pkg.zip is not available on the HTTP server.
    [STEP 4 of 6]
    [STEP 5 of 6]
    Inventory will be updated when all queued actions in Self Service are complete.
    [STEP 6 of 6]

The last log is from our Adobe install policy which works fine for many users but we still see a lot of errors like this. Even though this machine has several failures, the application is still installed.

[jmahlman]

Oh and we have random load balancing enabled. Both DP servers are more than capable of handling the loads.

[tep]

I'm seeing this as well. FWIW, I have an AWS cloud distro as master. I've had to actually re-upload certain packages for days in a row.

[swapple]

been getting this for the last 2 weeks or so. Jamf support was saying that their cloud storage was partially to blame. Just got this error today.