Large amounts of Apple Update Traffic

[Andrew_Kuntz1]

Hello all,

  I'm wondering if anyone has had any issues currently/in the past with random large amounts of Apple traffic. Yesterday our network got down to 2 MB/s (usually 200+). After some investigation with our ISP, we saw large amounts of traffic coming from Apple's 17.x.x.x IP range and our Firewall CPU usage reaching well over 80% at times and staying. Our ISP specifically said it was from update servers grabbing Apple Img. files. The weird part is we have an update deferral going to all iPads/Macbooks. We are currently going to be throttling our bandwidth to these ranges, but does anyone have any other suggestions? Thanks!

[garybidwell]

Deferrals config profile only work with supervised devices, if they are un-supervised then these deferral policies will be ignored.

[dwbergstrom]

It may also be worth considering setting up a content caching server, if you haven't already:

https://support.apple.com/guide/mac-help/what-is-content-caching-on-mac-mchl9388ba1b/mac

[DBrowning]

you may want to consider setting up a mac mini or some other mac with Content Caching. Even when updates are past the deferral date, devices would first look to the internal Caching server for the updates rather than all devices pulling straight from Apple.  

[Andrew_Kuntz1]

Hello, we've looked in the past at content caching, and it's something I was looking at setting up last summer. The problem is I'm not sure what would be causing these updates to go through? I work at a school district, so all devices are managed and supervised, so deferral being in place is not an issue. I've contacted support and they suggested limiting our bandwidth to Apple's IP ranges, but we are leary since we don't want to mess with Apple Push Notifications.

[atomczynski]

Any changes to your environment?
How large is your fleet of devices?
Do you have a Guest network for external devices to connect to?

Research your needs and spin up an Apple Content Cache server. How is your network configured? How many physical buildings and how are they connected? How is the wireless traffic traveling between the device, access point, and controller?

Do you have any history of this happening in the past?
I would want to know what is the issue and not throttle down the traffic to 17.x as a fix. as that is not a solution.

[Andrew_Kuntz1]

We haven't made any significant changes to our environment of 6500+ mobile devices and 700 computers. We are think we may have narrowed down the problem. It would seem as if a bunch of devices (nearly all of our students) were having all their apps re-downloaded everyday (Looks like this is a whole different issue now).

I've found the Content Cache documentation is a little lack luster on both Jamf and Apple. We have 5 buildings in our school district each with around 1500 users at each, so we were planning to spin up a server at each building. We were primarily wanting the content caching server to handle OS updates, so our users didn't have an excuse to not update. Would you have any videos/documentation that you care pass along?

In our 5 years we've used Jamf we haven't had this issue, besides the "thundering stampede" as my peers so lovingly called it haha. We are continuing to work with Jamf Support on investigating the issue. This workaround would be perfect for a smaller district, but at our level it would be more hassle then it is worth.

[DBrowning]

@Andrew_Kuntz1 I would look at Apples documentation.  Its actually pretty good.  https://support.apple.com/guide/mac-help/what-is-content-caching-on-mac-mchl9388ba1b/mac We have 20 of them across the US and Canada.  Pretty simple to setup/configure.  You can even use the Content Caching Config Profile payload to do a lot of the config.  

[PaulHazelden]

Been hit with huge amounts of Apple Update traffic today. Going to a couple of labs that are fully up to date already. All supervised and all have deferral set. And the traffic from Apple updates is supposed to be throttled to prevent it from killing the whole network, but that too has failed in this case, and last weekend when I sent a Mass action to update 350+ Macs to 11.6.

[atomczynski · Accepted solution]

What is Content Caching on Mac
https://support.apple.com/guide/mac-help/what-is-content-caching-on-mac-mchl9388ba1b/mac

Intro to Content Caching
https://support.apple.com/guide/deployment-reference-ios/intro-to-content-caching-ior3da85399f/web

Set up content cache clients, peers, or parents on Mac
https://support.apple.com/guide/mac-help/set-content-cache-clients-peers-parents-mac-mchl9b56e1cf/11...

Configure Advanced Content Caching Settings on Mac:
https://support.apple.com/guide/mac-help/configure-advanced-content-caching-settings-mchl91e7141a/11...

[Andrew_Kuntz1]

Thank you so much!

[MrRoboto]

Content Caching Servers will cache and serve all macOS/iOS/tvOS updates and apps on demand. They are really a 'set it and forget it' appliance unless you are doing parent & child servers across connected sites. I recommend getting new M1 Mac mini's as they have fast internal storage, run cool, can be customized with 10GbE network, and run great headless.

We have 110 Mac mini's, one at each building and the amount of traffic saved each month is incredible. Without caching servers our sites with limited bandwidth would have Macs/iPads updates and app installations stalled or taking several hours to complete. Jamf Pro captures the caching stats with inventory. Here’s an example from one site: 

Cache Details

Mac Software 90.04 GB

Other 73.97 GB

iOS Software 75.09 GB

Total Bytes are Since 2021/09/16 at 7:21 AM

Total Bytes Returned to Clients 686.27 GB

Total Bytes Stored from Origin 33.62 GB

Almost 700 GB of bandwidth save in the past 2 weeks!

[Andrew_Kuntz1]

Amazing! We have 5 of the new Mac mini's reserved to the side for content caching. It was getting pushed from other projects, but it looks like that may no longer be the case.

[Andrew_Kuntz1]

Hey, everything is going awesome with Content caching! Is there a command to show the cache details as you have (I.E Mac Software, Other, iOS Software). Or do you have a separate cache machine for each category?

[MrRoboto]

Good to hear. No, we have a single Mac mini at each building location. The Content Caching details are collected as part of Jamf Inventory and stored in the computer record... https://docs.jamf.com/10.33.0/jamf-pro/administrator-guide/Computer_Inventory_and_Criteria_Reference...

[atomczynski]

Last 30 days:

Last 30 days:
Over 6TB of data and under 200GB from origin

[foobarfoo]

(Humble brag and demo of capability) - Last 30 days::

Data served to clients: 111TB

Data from origin: 738GB

Network peak - 15 gigabit aggregated

Servers: 2x Mac Pro, 4TB SSD (3TB configured as cache size), 96GB RAM, 2x10GE uplink (LACP port channel) each

Configured for only shared caching.

Estate size: 64k iOS devices, 2300 macOS devices spread across approx 170 geographical sites