The great (caching) purge

[bentoms]

Check your caching servers!

Cache might be gone :(

[St0rMl0rD]

Yep, all gone here (3,6 TB of data). We had something similar on Friday, 16th as well, but back then the caching log showed no errors. So, two big purges in 2 weeks.

[mssaffan]

Here too Ben!

[al_platt]

Just re enabled ours. Now giving the message that clients need to be rebooted but will find the service over time.

Cache kaput though.

Wha!?

[jamesv_uasd]

Here too. I can't to hear the "why" this happened.

[elund]

Here is what I found out-

All,

Yesterday afternoon, the caching server registration service at Apple had some difficulty.

Caching Servers re-register themselves every 55 minutes with Apple. This is done so that if a server fails to re-register, Apple stops re-directing clients to the local caching server for app updates and downloads. When the registration service started having difficulty, some caching servers in the world were not able to re-register themselves and so the caching server service on those servers stopped. In some cases, for reasons I don’t know, the servers then purged their cache as well. I know mine here in my home office did.

The good news: The registration service is now functioning normally again. It appears that for most people, the servers that were not able to re-register themselves kept trying and were able to resume without any intervention.

I have had reports that some servers didn’t automatically restart the caching server service. That was the case with my server here in my home office. I suspect this has to do with me trying to manually restart it after I got notified that it stopped.

Here’s your homework: Double check that your caching servers are indeed running as they should be.

For the future: If you ever see a problem with your caching servers being able to register themselves, please contact AppleCare and let them know right away. The sooner AppleCare is aware that there is an issue, the sooner that our engineers can figure out what the problem is and get it resolved.

—Pete

 Pete Markham, Systems Engineer
Apple Education
Apple Inc. (763) 218-0949
www.apple.com/education

[steveevans]

Had this happen to us as well...got it registrered by removing all the specific networks to target and indicating all networks...nice to see it was a bigger issue then just us....

[mottertektura]

Same here!

[djdavetrouble]

Same here, 4 buildings caching server off at each. At my last JUG someone mentioned a sierra command that checks if a caching server is available on your network. Does anyone know what it is?

[donmontalvo]

Caching Server is on our to-do list. @bentoms, great The Great (Caching) Purge blog!

I don't suppose Caching Server is on JAMF Software's roadmap for NetBoot/SUS? #AGuyCanWish

[lindell]

Yep we saw the same thing yesterday afternoon. Since our caching servers are being monitored, we started receiving emails at 1:30 Central that caching was disabled on our 60 caching servers. The first notification I received, I connected to the box and tried to fix it. I couldn't fix it and started seeing the others were being disabled as well. I called AppleCare and got the usual "hmm, I don't think it's an Apple issue, it's probably a problem on your network..."

[AVmcclint]

So how exactly is Caching Server supposed to be superior to Software Update Server? I don't remember ever having to re-download the entire catalog of updates if there was a problem on Apple's end. These things are supposed to help alleviate bandwidth and throughput limitations. It doesn't really serve that purpose if Apple burps and causes terabytes of data to just disappear and it has to be downloaded all over again.

[RobertHammen]

@donmontalvo Given that Caching Server uses certificate-based communications, and everything relating to how it works is not documented/uses private API's and frameworks, I would consider it very unlikely that it will ever be available on a non-Apple server platform.

[JAMFatJESD]

Bummer!

My understanding and two cents regarding on this issue is that Apple Engineers have been working on issues pertaining to Caching not working on iOS devices and VPP apps mainly.

The purge happened because Apple applied a "Fix" but it did not work. Apparently there is a fix in the Beta 10.2 iOS.