Push Notification Blues

[franton]

I'm utterly stumped on this one. Push notifications work on enrolled devices INSIDE our firewall but not outside.

So i'm in the middle of a replan and reimplement our JSS architecture. I've got a master JSS sitting inside the firewall. I've another restricted JSS sitting in the DMZ. There's a split DNS address that feeds both servers. You get the DMZ JSS if you're external to our network and vise versa.

I've already made sure ports 2195, 2196 and 5223 are open on both servers to the *.push.apple.com DNS names.

So i've got a test laptop with a valid current MDM profile. If it's inside our network on either ethernet or wifi and I push a lock command to it, within five minutes it's locking itself. Try the same thing outside our network and nothing. I even exposed the test laptop on my home router (and put it in it's own DMZ) and still nothing.

What else should I be doing?

[blackholemac · Accepted solution]

Uhh...so 8443's open...I'm going to guess you checked both inbound and outbound.

In short the way we do it probably won't work totally for you, but hopefully it can give you some food for thought about the whole communication triangle. When I looked at the two ways to get coverage to our iOS devices externally, I discarded the limited access JSS method...primarily because I don't really have the authority to place any machine in the DMZ here and was told that was not something our network guys wanted to do.

What I did ultimately do was have my production JSS face the outside world. (complete with a good backup scheme) We contacted our outside DNS provider and had them add an entry to make jss.yourorganizationhere.com map to an xxx.xxx.xxx.xxx (a publicly-facing external IP address) that was in our organization's pool.

We then added at NAT entry that forwarded queries made to that external IP address inside the firewall to the internal server. On the firewall side, the only two things for our situation that are important to note are: 1) we don't have a firewall on outbound traffic internally, 2) to make our situation work, the only inbound port we had to allow was 8443 and our firewall guy even placed some firewall rules that will ONLY open traffic on 8443 inbound through the firewall devices trying to access https://jss.yourorganizationhere.com

I am willing to share with you the network diagram I used to persuade our network guys to make firewall changes with you. Being that you are using the DMZ method, it probably isn't relevant, but it does try to document the whole communication model. If interested, I will email you the document that I used to help me get a complete understanding of the process and present to our network guys. I used to be a newspaper artist for a living so it's documented as best as I can. Let me know if it helps anyone. I will note here that I should probably give credit to both JAMF and Apple as both my clipart sources and my research sources.

external image link

[blackholemac]

I'm betting you are forgetting port 8443. It has to be open incoming to either your limited access JSS or the internal JSS. It's an easy one to overlook, but the most important one. In our environment, we don't block outgoing, just incoming and I almost overlooked making sure traffic from the mobile device was getting to port 8443 on the server.

Hope it helps, I am willing to share our network diagram for those that might be interested but warn of one thing. I don't have the limited access JSS documented as we don't use one...we have all traffic go to the internal one. Some days I don't like that...in backups I trust.

[franton]

Just checked, we didn't. 8443 is open to our externally visible server for inward traffic and we've just opened 443 too.

[franton]

Sigh. Does the server that sends the push message have to be the one receiving it? (we've two JSS tied to the same database).

[blackholemac · Accepted solution]

Uhh...so 8443's open...I'm going to guess you checked both inbound and outbound.

In short the way we do it probably won't work totally for you, but hopefully it can give you some food for thought about the whole communication triangle. When I looked at the two ways to get coverage to our iOS devices externally, I discarded the limited access JSS method...primarily because I don't really have the authority to place any machine in the DMZ here and was told that was not something our network guys wanted to do.

What I did ultimately do was have my production JSS face the outside world. (complete with a good backup scheme) We contacted our outside DNS provider and had them add an entry to make jss.yourorganizationhere.com map to an xxx.xxx.xxx.xxx (a publicly-facing external IP address) that was in our organization's pool.

We then added at NAT entry that forwarded queries made to that external IP address inside the firewall to the internal server. On the firewall side, the only two things for our situation that are important to note are: 1) we don't have a firewall on outbound traffic internally, 2) to make our situation work, the only inbound port we had to allow was 8443 and our firewall guy even placed some firewall rules that will ONLY open traffic on 8443 inbound through the firewall devices trying to access https://jss.yourorganizationhere.com

I am willing to share with you the network diagram I used to persuade our network guys to make firewall changes with you. Being that you are using the DMZ method, it probably isn't relevant, but it does try to document the whole communication model. If interested, I will email you the document that I used to help me get a complete understanding of the process and present to our network guys. I used to be a newspaper artist for a living so it's documented as best as I can. Let me know if it helps anyone. I will note here that I should probably give credit to both JAMF and Apple as both my clipart sources and my research sources.

external image link

[franton · Accepted solution]

Turns out we had the correct firewall settings for our organisation. What we didn't have was the correct tomcat certificate running on the externally visible server. *groan*

I spent 40 minutes on the phone today with our account supervisor Drew. He's the one who found and pointed this out, so all credit to him!

(thanks for the advice blackholemac! appreciated nonetheless.)

[Bendelaat]

@franton am i correct in assuming that both the JSS in DMZ and JSS master have APN acces? and need APN acces to properly function?
also maybe you'd like to add your insights to the post here https://jamfnation.jamfsoftware.com/discussion.html?id=10306 where i'm trying to gather all the wisdom available for the move to a more robust Casper environment.

[franton]

Ok i've done the CJA since I posted this. Basically your cluster master server (or whichever server you're doing the management from) needs to be able to talk to APNS.

If you have three JSS servers and all can be used for management commands, then all need firewall rules to see out.

[mblair]

Thanks you all for sharing,this post help me cleared my network guys.

Franton, what were you talking about "correct tomcat certificate" The date was wrong? Did you get one from an outside vender?

[mkremic]

@franton's solution of the tomcat certificate worked for us too... it helps when you finish reading ALL of the instructions on how to set up a JSS in the DMZ! :o whoops The moment I did that, restarted tomcat on the DMZ facing JSS and issued the lock command again it locked in a matter of seconds.

https://jamfnation.jamfsoftware.com/article.html?id=174 - under "Additional Information"

Cheers!

[chriscollins]

One other minor thing I would also recommend you check is that both internal and external servers have the same time /synced to same ntp server.

We had an issue where internal push commands would work immediately but external ones were flakey and would be delayed 10+ minutes or sometimes indefinitely. When we'd look at the iOS logs on the device in real time we would see that the push was going out and hitting the device immediately but when it would check in with the external JSS it would report that there were no commands waiting for the device. So push was working but for some reason JSS didn't think there was anything wrong.

After checking the time on the external server which is a headless Linux VM, I realized the clock was 13 minutes behind. When we would trigger a push in the internal JSS's web interface that pending command gets time stamped. When the iOS device would receive the push notification and check in with the external JSS technically to the external JSS that command was 13 minutes in the future so it would respond that there was nothing to run. Eventually the external device would try again at a later time that was at least 13 min from the internal JSS pushing the command and so it would actually then work. Getting the times re synced fixed that.

That was a head scratcher.