Content Caching Woes

[MSBV]

Hello all,

We want to host caching servers in each site, and we've run into a few roadblocks in this area.

The main one, is we want to tell the computers to only use the cache on the servers, and not cache locally. We could not find such an option.

Another issue, we think we solved, was the fact the actual server had to be excluded from the policy that distributes the caching profile in order to configure it differently.

Can anyone share their experience with caching servers? We just want all macs under the same public IP to reach the server that's on that site.

[foobarfoo]

We have two content caches that serve approx 70k clients. Each is a Mac Pro with a 2x10GE uplink using LACP. We have multiple external IP's and configure the client settings by using DNS TXT records. And it works very well!

Now, in your case, this seems fairly straight forward. First a few questions:

The clients and your content caching servers, do they all have the same egress/external IP? If not, you probably need to configure client settings using DNS.

Do you block caching on clients? If not, you can do this via policy from JAMF. Keep in mind, depending on the size of your estate, you can bet that someone has caching enabled anyway. And if you don't configure authorized caches via DNS, your cache servers are as official as any other mac with content caching enabled. The likelyness of this happening increases if you have unmanaged, private, guest (etc) macs on your network that shares the external IP. You can see if this is happening in real time by shutting down your official cache server, and then using AssetCacheLocatorUtil on any client mac. If it finds other servers, well, the user experience for both macOS and iOS will probably suck.

We've found that the best way is to use multiple methods to block unauthorized caches. These are:

a) Configure official caches via DNS

b) Disable client content caching on managed devices though a JAMF configuration profile

c) Block the actual DNS record that the content cache service uses for registration on all client networks by modifying DNS server(s) responses to a particular record (check Apples enterprise network host names docs for which one it is, can't remember right now)

So to summarize, it's imperative that your intended cache is the actual ONLY cache that the clients use. And as usual, ensure you have enough bandwidth for the client <-> cache, ensure you have enough CPU and disk I/O on the cache etc. This is best checked by checking utilization with either the activity monitor or by your installed third party monitoring tool (we use Zabbix).

It's also a good idea to check/test different use cases by using AssetCacheLocatorUtil on clients, and AssetCacheManagementUtil status on the server. Also keep track of the logs from the AssetCache, it can give you essential information on various topics, such as why cache registration fails.

Hope that you and others find this useful, sorry for the wall-of-text with additional info.

[MSBV]

Hello and thank you for your detailed reply!

Your setup sounds way more complex than mine, it's impressive that it's working.

Now to answer your questions.

1. The clients should have a range of IPs that they can egress from, it's one physical router/firewall. I've configured the VLANs to reach each other, so that's not the issue.

2. I do not block caching on clients, on the contrary, I was under the assumtion that caching needs to be enabled for this to work, am I wrong? 

2.a. when you say configure official caches via dns, is that on the machines or network? also what do you mean by official?

2.b How will they know to use the caching server I have setup on the network then? will it automatically answer?

3. I have a mac mini with 8GB of RAM and 512GB of storage, as a proof of concept that should be fine, I can monitor the usage when it works, and upgrade when needed.

Thank you!

[foobarfoo]

1. All possible egress IP's must be listed explicitly and you need a DNS TXT record for this - see https://support.apple.com/guide/deployment/intro-to-content-caching-depde72e125f/web

2. Content caching as a client is always on. You don't need to enable the content cache on the client. So given what you probably meant, you're wrong in that case. Content caching is also used by iOS devices.

2a. On the resolver(s) that clients use. By official, I mean the cache that the client sees and connects to. The ones you want your clients to use.

2b. The clients will query for those DNS records by default and find it that way.

[MSBV]

One thing that comes up in this, you mention DNS a lot, but I use IPs as a way to direct them to the cache, do I need to switch to DNS Records?

[foobarfoo]

As per Apple documentation, if you use multiple egress IP's for your clients, or if your cache uses a different external IP than your clients, then yes. If not, Apples cache registration handles this automatically and clients discover the cache without using DNS.

[MSBV]

I have 2 or more external IPs per site, so I need to look into the dns txt records for my setup to work?

Also, when I run the locator, the cache of my server shows up as unhealthy, is that relevant to the DNS? I have ping to the server, and from the server to the client.

[AJPinto]

We retired all of our content caching servers a few years ago. Apple has been steadily killing the work flows of managing macOS updates in any way shape or form. 

• There is no way to stop Macs from caching updates.
  • MacOS installs updates from the system cache.
• A Content Caching Server must be on the same subnet as the Mac in order for the Mac to use it.
  • You could spoof this with VLANs. However if the Mac sees getting OS updates from Apple as a faster solution it will skip using the Content Caching Server and download directly from Apple. If you block the Apple traffic, it will also break the Content Caching Server as MacOS must call home to validate OS updates even if they come fro ma Content Caching Server
  • Depending on the size of your organizations footprint this may be functionally impossible or require many more than two content caching server.
• You would need two Configuration Profiles for Content Caching and scope accordingly
  • One on your production devices to disable Content Caching
  • One for your two devices enabling Content Caching

I suppose anything is possible with enough effort. Just know you will be going against the grain of what Apple wants, and Apple products tend to not function too well unless they are function as "Apple Intends". I will leave the network shenanigans for those who understand that side of things better than I.

[MSBV]

Thanks for your input. I have followed foobarfoo recommendations, and now I have caching enabled only on the server, but the clients see it as unhealthy.

As for going against Apple, it's my daily burden, in every thing you do with macs, you're going against Apple, since they don't even know what they're doing. So far, I've succeeded in most of the things, including software update, which was annoying.

[AJPinto]

I work in a 99% Windows environment, practically everything I am told to do goes against Apple recommendations also. I have been in my current role for 5 years and I am finally starting to win hearts and minds, it takes time but eventually people listen. I just keep beating the drum, "you cannot manage macOS like Windows as they are different platforms". Whenever I get someone who pushes back I use the example: 

"Would you tow a boat with a Corvette? Just because the Corvette can do it does not mean it should do it. You would get a Truck to tow the boat, right? Use the right tool for the job, and you wont have problems".

---

Software updates by and large are complete total garbage to manage.

[MSBV]

I don't envy you, we also have about 50% Windows machines, but someone chose to use Intune as our MDM, and that is garbage.

Since apple blocked updates without user password in Ventura, we found a compromise where we prompt the users for their password, and capture that and pass that to the installer.

When I feel I've reached a stable enough state, I would like to open source the scripts I use for software deployment through Jamf. Is that something you think will help people?

[AJPinto]

We use Intune to manage our iOS devices, mainly because of how Intune is licensed. I thankfully have managed to keep Intune the heck away from our Macs. I'd probably resign if I was told to move Macs over to Intune.

OS updates "should" be able to install without a PW so long as the user has a secure token. MDM issued OS updates should install automatically so long as the MDM has a secure token, and the device was enrolled with Automated Device Enrollment. OS upgrades are another beast entirely as users need Admin Access to install those. MDM can still use MDM commands to install OS upgrades. I'm not counting using the softwareupdate binary on Intel devices anymore. I have done some crafty stuff with expect functions in the past also. Apple really needs to figure out a better way to identify organization owned vs user owned device other than enrollment method. There are times were you need to reenroll a device in MDM, and if you don't reinstall macOS you lose management options.

By all means, share away. With how many issues admins have with OS updates, I am sure someone will find it useful. I have most of the scripts I use on github, though admittedly I have retired all my scripts for OS updates.

[MSBV]

Oh, it's not just software update for macOS itself, it's also for deploying fixes and software like chrome and other apps. I have not encountered a situation where I needed to reinstall a mac to re-enroll it, I would love to know what the specifics of those are.
As for MDM updates, we tried that, it was a hit or miss but mostly miss.

[foobarfoo]

No, content caches do not need to be on the same VLAN. The main and in fact only purpose of these caches is to save Internet traffic. ANd keep in mind, while Internet access is fairly cheap, passing traffic though an enterprise grade firewall is pretty expensive. Usually, content cache traffic doesn't traverse an expensive firewall in most scenarios. They can't be used as some sort of proxy so you can block Apple/Internet traffic otherwise.

[MSBV]

Yeah that's my goal to save bandwidth out of the sites. We tested Jamf's update thing, and we bogged down the internet out of the office due to the download sizes happening at once.

We want to avoid that.

[MSBV]

So does anyone know why other macs on the network see the server cache as unhealthy? is there a log that tells you why?