Apple TV - timed - ntp error

[dptech]

Good Afternoon All,
We're currently having an issue with Apple's time servers that appear to me as if we're being blocked.

The problem was noticed earlier this week when we tried to activate a new Apple TV on our network and Activation failed. We moved the Apple TV to a separate network, the time got set and activation occurred as one would anticipate.

On my network, I can see tons of errors similar to what I'll paste below in italics showing a cert error and a time set error. You might be connecting to a server that is pretending to be "iphonesubmissions.apple.com" which could put your confidential information at risk.", NSErrorFailingURLKey=https://iphonesubmissions.apple.com/convert.jsp}
Dec 31 16:21:46 Apple-TV timed[27] <Notice>: (Note ) CoreTime: Received source unavailable from "NTP"
Dec 31 16:21:46 Apple-TV timed[27] <Notice>: (Note ) CoreTime: Want active time in -0.00min. Need active time in -0.00min. Remaining retry interval: 0.000000min.
Dec 31 16:21:53 Apple-TV addaily[101] <Warning>: addaily ended
Dec 31 16:21:55 Apple-TV OTACrashCopier[102] <Warning>: Scheduling on exit
Dec 31 16:22:15 Apple-TV AppleTV[40] <Error>: SecTrustEvaluate [leaf ValidLeaf] [ca1 ValidIntermediates] [ca2 ValidIntermediates] [root ValidRoot]
Dec 31 16:22:45 Apple-TV AppleTV[40] <Error>: SecTrustEvaluate [leaf ValidLeaf] [ca1 ValidIntermediates] [ca2 ValidIntermediates] [root ValidRoot]
Dec 31 16:22:50 Apple-TV timed[27] <Notice>: (Note ) CoreTime: Received source unavailable from "NTP"

We did a packet capture to watch network traffic from the apple tv to apple's time servers (time.apple.com), we could see traffic going to Apple's servers, but no response was ever noted in the PCAP file.

This seems to me like we've been blocked or blacklisted by apple, but our account reps are not really interested in going out of their way to provide assistance.

So my question for the group is really twofold:
1. Is this happening to anyone else? If so, how are you getting your apple tv's activated?
2. Does anyone know how to contact the right people at Apple to determine whether or not, I have in fact been blacklisted?

Thank you in advance.

[Look]

Personally I'd start looking at the security hardware/software/configuration between the device and internet. I can't imagine Apple blocking an individual network from making NTP requests, more likely something happening in transit that doesn't happen on the alternative network. Alternatively see if you can arrange a connection from just outside the security (DMZ) and see if that works, would tell you pretty fast where the problem was occurring.

[dptech]

I appreciate the response. We definitely tried this. I've placed our test Apple TV on an interface outside our firewall, assigned a static IP and the PCAP results are the same. We can see packets going out to apple.time.com but never a response. This is definitely one of the stranger things I've noticed.

[Michael_Meyers]

We are having the issue of not being able to activate or update our Apple TV's via ethernet or wireless going through our firewall. We are not blocking anything going to Apple but still not able to connect. If we try bypassing the firewall or connect to a personal hotspot, updating and activation work without issue. Apple sent us IP addresses to whitelist and sites to add to the firewall. We are testing them now.

Here is what we received from Apple:

"Hi Mike

Thanks for reaching out to us. I understand you are having issue activating Apple TV on your internal network.

Apple’s servers utilize load balancing technology. The IP addresses vary depending on your network location, our network load, device type trying to activate.

Apple recommends opening the entire 17.0.0.0/8 address block in your firewall. The entire 17.0.0.0/8 address block is assigned to Apple.

Have you tried inspecting your firewall during the times of the activations to see what servers the Apple TVs are activating?

Some of the servers our devices will contact will be:

phobos.apple.com
deimos3.apple.com
albert.apple.com
gs.apple.com
gg*.apple.com
itunes.apple.com

ax.itunes.apple.com
evintl-ocsp.verisign.com
evsecure-ocsp.verisign.com
.symcd.com
.symcb.com
*.amazonaws.com"

[dptech]

Mike, did this work for you?

[Michael_Meyers]

Nope. After doing some packet captures here is what we are seeing. We are sending traffic out to Apple and are not getting any response back unless the source and destination port are both 123. We use PAT inside so our source is not always 123 for the source in packets. This is a pretty common practice and is probably being seen on other networks as well. Since I never get a response from Apple, it is something outside of our network. I suspect apple has some sort of security appliance that is doing deep packet inspection and is denying any packets that aren’t both source and destination port 123.

[Michael_Meyers]

We received a solution from Apple and it is now working. We had an issue with our firewall, where it was not accepting the changes. We had to put in a ticket with Cisco to get that resolved. NAT was on for ports 1024 and below. We set it for ports above 1024. Once it was done, we are now able to activate and update our Apple TVs.

Here is what they suggested:
Hi Mike,

I heard back from my NTP team and it appears there is an issue with your PAT configuration.

All of the requests in the packet capture are coming from ports below 1024. As stated in the conference call, we are only responding to requests from port 123.

Any port below 1024 is considered a privileged port and we will not respond to any requests unless the protocol is sending over it's default port (NTP using 123). When the PAT your network is using reassigns the a port number, the process/service is not considered privileged and needs to be on a port higher then 1024, but your current configuration has them below.

Network best practices state that ports below 1024 are reserved for privileged processes and should not be used for client connections.

Hope that helps!

[dptech]

Mike,
This is perfect. Thank you.

[bentoms]

@Mike_Meyers So reading your reply, this was network related your end?

NTP works on UDP 123, & your firewall as translating the port which in turn then through out the request?

[dptech]

@bentoms @Mike_Meyers we appear to have some sort of resolution on our side. In the end, all we did was change our PAT pool range.

Once the IP address range was changed Apple TV's started getting their time and we were able to activate them.

It feels like we were blacklisted on our original PAT range. I only mention this because we have a very fast internet connection and traffic coming from our IP's can look like a DOS attack.

[bentoms]

@dptech glad you got it sorted. Thanks for advising what it is.

[brandonkahler]

@Mike_Meyers Any chance you could share the config changes TAC had you enter? An example would be greatly appreciated.

[BK]

@dptech You can always activate through Apple configurator, Mac & a Micro-USB as a workaround. You can benefit with this method by installing payloads such as wi-fi, passwords, or the profiles to use with Casper Focus. It will also update the OS and give you an opportunity to name them. It will take 4-6 min for each Apple TV. You can connect via a USB hub through multiple Apple TV's the only drawback with this method is it will name them the same name or sequentially. We named them by room #'s. But you can always go back and rename them using the a bluetooth keyboard or typing with the dreaded remote control :)

[DanielJusta_FTN]

@brandonkahler

dptech referred to a PAT Pool Range. On the FortiGate it's called a Central NAT Table entry to
force the NTP session from the TV IP addresses to use the same source port 123 and not translate it to anything else.
Keep in mind that this will only allow one session simultaneously as you can only have one IP address source port active concurrently. If you have one public IP address per TV, this would work fine.
In my scenario, I was able to swap out the customer's IP address of each TV after it activated one by one until all of our TVs activated.
It seemed as we only needed to activate once on each TV, and we were OK.