APNs via Proxy

[fda-ctp-tims]

We are the organization that has the most restrict security policy. The Casper MDM Redhat Linux Server is not allowed to open port 2195 to extranet directly. APNs traffic has to be redirect to our Bluecoat proxy server. I am asking any one, who successfully sends APNs via proxy, to provide advice,
1) What type of http proxy are you using?
2) Does MDM server must have JAVA 7?
3) port 2195 on Proxy server must be bi-directional?
3) How do you configure the range of IP for gateway.push.apple.com? we currently only maps three IPs. Congiure the entire "17.0.0.0/8 address book"? Don't even think about it. That's 16 million IP address.

[bentoms]

We opened 17.0.0.0/8 to TCP 2195, 2196 & 5223 non-proxied.

As per: http://support.apple.com/kb/TS4264 & the linked documentation.

As the 17.0.0.0/8 range is owned by Apple, the ports "non-standard" & traffic SSL'd we felt it would be ok.

If you cannot still open the firewall, then you'll need to look @ local profile delivery.

[fda-ctp-tims]

Hi Bentoms,

Thank you for your answer. May I bother you for more inofrmation
1) What is the Java version in your JSS? Our RHEL server has Java 6. It has been said that Java 7 is must for APNs via Proxy.
2) What type of proxy you have? Our web server catalina.sh configured the proxy at 8080. JSS can search AppStore apps ( https call does direct to proxy). The trace tool in the proxy server show none APNs TCP traffic ever been received.

[evarona]

We've struggled with this same question ourselves. We have a bluecoat proxy server and use port 80 for redirection but our security team will not allow the ports that @bentoms][/url mentions above. Apple's comment was "we support millions of APNs daily to iOS devices and it works. The protocol does not support proxies". So I'm reduced to creating CPs, downloading manually from the JSS and scripting a policy to apply using the 'profiles' command.

Do I like it? No. Does it work? Mostly yes, but is more work that I'd care for. Good luck.

[dpertschi]

@evarona][/url - your post almost mirrors my problem with this too...

Security dept. say's 'I don't want the Macs constantly talking to Apple'. Then I get our Apple account engineer on the phone with them to hopefully ease the fear, and he said verbatim what yours said: 'we support millions of devices via APNs without issue'

So let me ask the listening public here, for those of us hitting a brick wall trying to open up direct access to APNs:

• Over the last several years, have there been any reported hacks, breaches, spoofs, highjacks etc. in the APNs communications channels?

• Those of you in high security environments: how were you able to get cooperation from your networking/security departments to let this traffic through?

Venting now: I'm no security expert, but this does seem like a silly case of old school unjustified nerves. As I understand it, that communication between the Mac and Apple is an empty and encrypted notification telling the device 'hey, go talk to Casper and get your orders'.

@FDA][/url-ctp-tims - Vivian, if there are other government agencies permitting direct access to APNs, can that not be leveraged in your business case to allow it in your agency?

[bentoms]

Hi fda-ctp-tims

1) What is the Java version in your JSS? Our RHEL server has Java 6. It has been said that Java 7 is must for APNs via Proxy.

We're using Java 6 & 7 on our 8.73 JSS hosted on 10.9.

2) What type of proxy you have? Our web server catalina.sh configured the proxy at 8080. JSS can search AppStore apps ( https call does direct to proxy). The trace tool in the proxy server show none APNs TCP traffic ever been received.

We use a kerberized web sense proxy.

------

Honestly, i think you'll need to allow the ports in/out even outside of the 17.0.0.0/8 range as i've been advised that in some instances Apple MAY used akamai servers that fall out of that range.. OR you can manually deploy profiles.

[evarona]

@bentoms][/url: It's little things like this that my security people hang their hats on

Honestly, i think you'll need to allow the ports in/out even outside of the 17.0.0.0/8 range as i've been advised that in some instances Apple MAY used akamai servers that fall out of that range

They're concerned, and maybe rightfully so, with the nebulous nature of Apple's "just allow all of 17/8 through your firewall", even though it's only a few ports. When you have critical company data and are held to government regulations, this stuff doesn't fly. Yes, we're manually deploying Config Profiles and I wish we didn't have to but if the choice is between making my life easier or the security of the company, I will (albeit begrudgingly) pick the company. Usually.

Perhaps other readers here don't have the same security concerns but when you're dealing with PAI data, HIPPA regulations or other highly classified information, you take very few risks.

With this in mind that I suggest you take a look at the feature request I suggested to have JAMF develop APN for OSX without relying on Apple to address the security element of APN. I envision it working more like GPOs than the current APN model. It's currently at negative votes but I think it's worthwhile if you have data you want to protect.

[fda-ctp-tims]

Hi @bentams and @everona,
Thank you very much for the information. The JSS instance in our center only manages a thousand iPhones. Probably the security might allow outbound-only from JSS directly to apple.com.

[wmateo]

@evarona can you elaborate how are you getting this to work on your network using profiles command? maybe a sample scripts, or steps taken? I am having same issue with our Security Team. I am going to have to resort to this method also.

[evarona]

@wmateo][/url
Honestly, I've moved on to another company since then and no longer manage Macs (slight tear + big sigh). But here's what I did:

• Create my Config Profiles using the JSS as normal. Make sure to include any root/intermediate certificates for AD, SCEP, ssh, whatever else you need. IMHO, this is the easiest place to update them from.
• Download your profile from the JSS. If memory serves me, it's something like <jss_url>/exportOSXConfigurationProfiles.html
• I chose to copy the CPs to /Users/Shared/company.com because that was my local repo for Composer and it was on every machine.
• Write a script to install the CPs and put it in the same folder. My script went something like this

#!/bin/sh
CFG_PROF="***your_downloaded_CP***.mobileconfig"
SCRIPTDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
/usr/bin/profiles -i -F $SCRIPTDIR/$CFG_PROF

• Launch Composer or Packages and manually create a .DMG package with these files.
• Import/Deploy this like any other JSS package.
• Setup a policy to check for the script and it if it's present, run AT EACH BOOT

You need to create a .DMG because you're using the package to copy the files, not install anything. And because you're using the JSS to deploy, the install context is root so this should work once you get your tweaks worked out. Lastly, make sure to set the policy to occur at every boot so that you can update/deploy the policy and it will get applied. Alternatively, you can setup a launch control package but that's beyond the scope of this discussion.

I recommend you thoroughly test your CP before you go through these shenanigans because much of my issues were solved there. HTH.

[bradtchapman]

@fda-ctp-tims, it has been almost three years since the last reply to this thread. I would love to get an update as to whether you and the network administrators at the FDA were able to reach an amenable solution. Or if you're still working there at all, given the current administration's trajectory (your LinkedIn profile still shows you working at the FDA).

I found a solution and have shared the details below. This may help you or anyone else that stumbles upon this thread from Google or Jamf Nation.

My current employer has a secure network with multiple layers that employs a "zero trust" design for security. There are multiple DMZ networks that serve as buffers between the internal network and the public Internet. Devices on the corporate network are not allowed to reach the Internet except via proxy server, and Apple's push notification service does not ordinarily permit the use of proxy servers for push notification providers.

It is possible to get APNS working in a highly secured network environment, but it wouldn't have been completely possible without some changes on Apple's part as well: at WWDC 2015, Apple announced that a significant update to the APNS protocol was coming. The biggest change was the default delivery method to HTTP/2. This effectively enables clients to connect to APNS via an HTTP/HTTPS proxy server while retaining the client/server trust mechanism inherent to APNS.

Here is how we got APNS working in this environment:

1. Create a cluster with an additional JSS in the DMZ network;
2. Designate this DMZ JSS as the "Master" in the cluster;
3. Configure the DMZ firewall to allow the JSS to communicate with Apple;
4. Configure the Macs with an explicit proxy for HTTP & HTTPS.

Bear in mind these key facts about Clustering, and the roles of the Master JSS:

• In a cluster, only the Master JSS talks to APNS (but admins can log in and work on any JSS);
• The Master JSS queues pending MDM commands every 5 minutes;
• The Master JSS delivers device lock & wipe instructions immediately;

Additional infrastructure considerations:

• The DNS server(s) in the DMZ network must be configured for DNS forwarding to external resolvers;
• The DMZ firewall must allow outbound from the DMZ JSS to 17.0.0.0/8 on TCP 2195 + 2196;
• The DMZ network might require the JSS to have its own public IP for NAT purposes;
• The Mac process "apsd" connects to APNS and does not respect proxy PAC files, hence the explicit proxy setting.

After adding the JSS to the DMZ, you will need to open the firewall from DMZ to Corporate on port 3306 to allow MySQL traffic. Or, you can set up a third, independent server functioning as the database in a segregated part of the network, and link both of your JSS'es to it. Open TCP port 3306 as needed.

BTW, you can set this up with physical servers or virtual servers. The DMZ instance can talk to Apple while your database remains safe inside a secure network. All the JSS cluster nodes will coordinate their operations through the database. The DB can stay in the corporate network or be put on a dedicated server in a "Secure" DMZ network (one that is not in corporate, but not capable of reaching the Internet either). That is an infrastructure choice entirely up to you.

Lastly, and this is absolutely essential to remember when making your case to InfoSec about APNS ("The Service"):

• The Service is just a conduit for delivering short messages to devices.
• The Service is already widely used by providers like Facebook, Twitter, Breaking News apps, iMessage, and so forth.
• The Service network at Apple (17.0.0.0/8) only listens for incoming connections.
• The Service does not require your network to listen for unsolicited connections from Apple.
• The Service uses a TLS 1.2 handshake protocol.
• The Service authenticates all transactions with device tokens and payload tokens, and validates all SSL certificates.

[hunter990]

Bradtchapman, that is a great post and we may be actually going that route. Currently we are behind a load balancer due to a security requirement at the time.

We are also having issues connecting to APNS. We have ports 2195 and 2196 open on both servers and have a brand new push cert. I can telnet out to the apple push gateway url without any issues. We have also verified that nothing is happening to the outgoing traffic such as SSL decryption, etc. There is simply basic url filtering applied and *.apple.com is allowed. The issue we are having is that we cannot get profiles to download. Thus no MDM functionality, etc.

Could it be that the issue is not outbound but inbound. We opened 5223 and we know we have 8443 and 443 traffic back through the load balancer. 443 is odd as it only allows you to connect via the URL, we cannot enable HTTPS for the share as it will fail.

Anyway, we are stumped and have been working with out TAM and Apple Enterprise Engineering and they are stumped as well. We also have a test domain with far more freedom there and it has the same issue. Just thrown off as we can talk to APNS just not via the JSS.

If you have any ideas please post them.

[bradtchapman]

@hunter990 for the computers that can't download configuration profiles, let's check a few things:

Typically you want a load balancer to receive on :443 or :8443, then forward traffic to :8080 to the nodes. It's better for the load balancer to handle the SSL heavy lifting because it is a dedicated appliance. Note that you must log onto each and every JSS node to configure Tomcat as follows, then restart Tomcat on each node after doing so.

1. Tomcat proxy / valve settings. Image Link
2. Tomcat settings overview. Image Link
3. Logging into the node with http. Image Link

Here are some additional questions:

• Do you have a load balancer to the JSS that faces the Internet?
• Can your Macs check into the JSS from outside the company network?
• Are there DNS records for "jss.mycompany.com" published internally and externally?
• Are you using a self-signed JSS certificate, or a public cert that is trusted by Verisign, Entrust, Symantec, etc?
• Is the load balancer forwarding incoming traffic to :8080 to the nodes?
• Is the load balancer configured to have SSL connections terminate at the LB?
• Is the load balancer configured with "SSL Session Persistence"? See Example.
• Do Macs inside the network require a proxy server to reach the Internet?
• Is the proxy server configured as a PAC file, or a static entry on HTTP & HTTPS?
• Do you have an infrastructure diagram handy that I can review? (obfuscate IPs and hostname details before posting)

[hunter990]

@bradtchapman, I'll answer all of your questions right down the list. Thank you for the help.

We do have the load balancer configured to receive on 443 and 8443 and then forward to 8080.

1. We have the proxy/ valve settings configured but using 8443
2. That is set the same
3. We cannot log into the node with http we still have to use https

Do you have a load balancer to the JSS that faces the Internet?
- Yes
Can your Macs check into the JSS from outside the company network?
- Yes
Are there DNS records for "jss.mycompany.com" published internally and externally?
- Yes
Are you using a self-signed JSS certificate, or a public cert that is trusted by Verisign, Entrust, Symantec, etc?
- It's a self signed cert internally to our organization. Is the load balancer forwarding incoming traffic to :8080 to the nodes?
- Yes
Is the load balancer configured to have SSL connections terminate at the LB?
- Yes
Is the load balancer configured with "SSL Session Persistence"? - Yes, the team that handles it also turned on Source Address Persistence as well
Do Macs inside the network require a proxy server to reach the Internet?
- No
Do you have an infrastructure diagram handy that I can review? (obfuscate IPs and hostname details before posting)
- I can work on modifying the one that we have. It will probably be another day before i can post.

[hunter990]

Just to add another interesting bit to this saga....

In testing with our security team that manages the outgoing firewall, they can see traffic over 2195 when we telnet to an APNS url. As soon as we try to enroll a system, and it fails on creating the MDM profiles, not traffic at all. It looks as though nothing is happening from the JSS. We tested in the dev environment we setup and the same thing.

Wondering if there is software that is blocking the communication such as Bit9 or BigFix (these are windows boxes).

To further testing we are setting up on a Mac test workstation and will test external of the network and then move it inside and see the result. My guess right now is it will work for both. In the prod environment we are trying to get working we are now looking at standing up a RHEL box to see if that would do better. We just need to find out if the JSS works with 7.1.

[bradtchapman]

You might need to enable debug logs on the master JSS to confirm whether the profiles are getting to Apple or not. If they are stuck in "Pending" then it's up to the client (Mac) to connect to APNS.

Have you tried enrolling your Macs while outside the company network? Have you tried setting an explicit proxy for HTTPS when they are inside?

[bradtchapman]

Also, let's clear up a couple of things:

1. The Jamf Servers will only connect on ports 2195 and 2196. They do not use ports 5223 or 443.
2. Macs & iOS devices will only connect on ports 5223 or 443. They do not use ports 2195 or 2196.
3. All traffic to APNS must be initiated by the device or the app server, so all connections are outbound from device to Apple (17.0.0.0/8).
4. The APNS service at Apple does not make any unsolicited connections. Ever.

[bradtchapman]

Hey @hunter990 , whatever became of this?

[hunter990]

@bradtchapman after successfully testing in our test domain, which is just a simple simple single server, we knew that there was no issue with outgoing traffic. We went back to our servers in the production domain to look at them further. By isolating the load balancer out of the system we got everything to work.

Digging into the issue further we determined that the traffic on our load balancer over port 8443 incoming was the issue. We get out to 2195 and 2196 just fine, it was the request to download the profiles that was failing. 8443 was not providing a valid cert to the clients. What we figured out is either we remove the load balancer or upload a self signed cert from the JSS for port 8443 only and maintain the separate signed SSL cert from our internal authority for port 443.

Either way we now have a path to move forward.

[bradtchapman]

Awesome!

By the way, is your JSS URL set to the FQDN of the load balancer?

[Rokas]

Hi Both @bradtchapman and @hunter990

Very nice and helpful thread you have here!
We are in progress of planning and building up macOS management platform in our company using Casper Suite/JAMF PRO. Our company is also security concerned, so the only option we to get APNs is this one, but even this one can be challenging when having a talk with our security guys.

As I understand by creating this clustered JSS and DMZ environment will only remove need to have two ports (2195 and 2196) open to Apple's 17.0.0.0/8 our internal network, but we still need to open 5223 and 443 (For DEP, not sure we really need it) to Apple's 17.0.0.0/8?
Is there a chance if you could share a diagram of such solution (of course I don't need to know any related information to your company like IPs/hostname and etc.

[bradtchapman]

Hey @Rokas, et al:

I've made a generic "ACME" version of our current infrastructure diagram with hostnames and IP addresses changed.

https://www.dropbox.com/s/dt0rovt9hmge4x0/2017_jamf_infrastructure-ACME.pdf?dl=0

This diagram includes an illustration of tunneling APNS client traffic on port 443 through a proxy server.

As for DEP and activation, one option is to have a dummy WPA2 Wi-Fi network that only allows traffic to Apple's activation servers and connect devices to this network on first boot. Once DEP enrollment occurs, your device will be instructed to obtain one of its first payloads from the Jamf Pro server: the "real" Wi-Fi configuration for your institution's network. Your JSS must have an instance visible to either the dummy Wi-Fi network, or to the outside world (if the Wi-Fi network is physically or logically air-gapped from the internal network).

[spraguga]

@bradtchapman @hunter990 We are having an APNS issue with a load balanced setup as well. We can ncat out on 2195/6 ports on the tomcat server just fine. We then migrated to this new load balanced JSS and all previously MDM capable Macs work fine. However, anything added new will not become MDM capable. There tons of connection error logs to feedback.push.apple.com and gateway.push.apple.com even though we can ncat to those ports on that server.

[bradtchapman]

@spraguga : which server have you elected as your "Master" in the cluster? In the configuration example that I posted in my previous post, I had to designate one of the DMZ instances as the master. The Master instance is the only one that sends APNS commands.

See this KB article: Understanding Master and Child Web Application Responsibilities

[spraguga]

@bradtchapman Yes, our internet facing master is setup to send the APNS commands.

[bradtchapman]

Ugh. Sorry. Duplicate.

[bradtchapman]

@spraguga : You're saying "anything added new" is not MDM capable and that you're seeing errors.

1. Are you talking about newly added cluster instances, or newly added Macs?
2. Do the Macs require the use of a proxy server to reach the Internet?
3. Can your Macs reach your JSS'es outside the institutional network?
4. Can you post some of the sample logs from the master? (please sanitize the logs and rename the server addresses)

[spraguga]

@bradtchapman 1. Are you talking about newly added cluster instances, or newly added Macs? Macs
2. Do the Macs require the use of a proxy server to reach the Internet? Sort of, the connection is, LB to Web server to App server
3. Can your Macs reach your JSS'es outside the institutional network? Yes
4. Can you post some of the sample logs from the master? (please sanitize the logs and rename the server addresses) [ERROR] [at-http--33] [PushNotificationUtility ] - Unable to send push notification com.jamfsoftware.jss.pushnotification.notifications.AppleMDMCheckInNotification@3fa28d62. Unable to connect to APN server "gateway.push.apple.com:2195"
[ERROR] [at-http--26] [APNSConnection ] - Exception (class java.net.UnknownHostException : gateway.push.apple.com) in getAPNSSocket (357838168)
java.net.UnknownHostException: gateway.push.apple.com

[ERROR] [at-http--32] [APNSConnection ] - Unable to connect to APN server (550039706)
[ERROR] [at-http--32] [APNSConnection ] - Exception in APNSConnection: Unable to connect to APN server "feedback.push.apple.com:2196" (550039706)
com.jamfsoftware.jss.pushnotification.connection.APNSConnectionException: Unable to connect to APN server "feedback.push.apple.com:2196"

[bradtchapman]

Those log entries indicate that your servers are not able to connect to APNS anymore.

Run the following commands from your master:

nslookup gateway.push.apple.com
telnet [any returned IP] 2195
nslookup feedback.push.apple.com
telnet [any returned IP] 2196

If the DNS lookup portion succeeds, but the telnet fails, then you have to talk to your firewall admin.
If the DNS lookup fails, talk to your DNS admin or check the server DNS configuration.

For reference, on a system that has functioning MDM, the MDMCommandQueueMonitor subroutine on the JSS will send MDM commands every 5 minutes to the Apple Push Notification Service. The commands are queued by Apple for up to 7 days, or until they are delivered to the endpoint, which happens the instant that device makes a successful connection to APNS.

[spraguga]

@bradtchapman As I stated in my other post I can ncat to APNS just fine over 2195/6 ports from the app server.

[bradtchapman]

This year at JNUC, I'm giving a presentation on APNS — how it works, and why allowing communication with those IP addresses and those protocols is safe. My hope is that many jittery network admins will be less concerned about it afterwards, and those of you in 'high-security' environments will be sufficiently armed to answer your bosses' concerns.

[jtarantino]

Hi bradtchapman,

This thread is really helpful and clear on proxy requirements in a clustered & balanced environment.

I still have a point not crystal clear. Do the SSL must be terminated on the Load Balancer ?
Can it be re-encrypted to the JSS node ?

[bradtchapman]

You shouldn’t do this because you add a serious amount of encryption overhead. Just pass unencrypted to the nodes.