APNS Issues...Among Other Things

[nathan_thornhil]

Hello all. This upgrade to 10.22.1 has come with some challenges. I have an open ticket with Jamf, but I'm in a bind and on a time crunch so I'm reaching out to the community for ideas.

We have Jamf Cloud and starting yesterday no device can be enrolled. Automated Device Enrollment through Setup Assistant fails and User Enrollment fails. The profiles are unable to be installed.

The Jamf Server Logs show APNS issues. I called Jamf and per their suggestion I renewed the APNS push cert early and removed the devices from the Prestage Enrollment, then assigned them to it again. Still no success. Now I'm also starting to see VPP is unable to verify licenses.

I double checked with my networking team and they still have all of the ports available for Apple's services. Nothing changed on their end. It just stopped working.

Here's a sample error

2020-06-30 19:19:03,288 [WARN ] [eralPool-18] [ApnsPushQueueManager ] - Error sending push notification com.jamfsoftware.jss.pushnotification.notifications.AppleMDMCheckInNotification@f189d3fe to connection com.jamfsoftware.jss.pushnotification.connections.ApplePushNotificationServiceConnection@76963a8f. Remote host terminated the handshake 2020-06-30 19:19:03,430 [ERROR] [eralPool-19] [ApnsFeedbackConnection ] - IOException getting and entering feedback data: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake at java.base/sun.security.ssl.SSLSocketImpl.handleEOF(SSLSocketImpl.java:1321) at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1160) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1063) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402) at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:716) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:799) at java.base/java.io.InputStream.read(InputStream.java:205) at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2314) at org.apache.commons.io.IOUtils.copy(IOUtils.java:2270) at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2291) at org.apache.commons.io.IOUtils.copy(IOUtils.java:2246) at org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:765) at com.jamfsoftware.jss.pushnotification.connection.ApnsFeedbackConnection.getFeedbackData(ApnsFeedbackConnection.java:34) at com.jamfsoftware.jss.pushnotification.connection.ApnsFeedbackConnection.run(ApnsFeedbackConnection.java:88) at org.springframework.security.concurrent.DelegatingSecurityContextRunnable.run(DelegatingSecurityContextRunnable.java:84) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Suppressed: java.net.SocketException: Broken pipe (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110) at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150) at java.base/sun.security.ssl.SSLSocketOutputRecord.encodeAlert(SSLSocketOutputRecord.java:81) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:352) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:264) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:405) ... 16 more Caused by: java.io.EOFException: SSL peer shut down incorrectly at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:167) at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:108) at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1152) ... 18 more

Any help would be appreciated.

[IT-Chris]

yes this is screwing me up. I cant even enroll units

[mainelysteve]

There is another thread about this as well.

[nathan_thornhil]

Thanks for that. I was wondering why I hadn't seen another post about this issue, but I was so focused on looking for posts about APNS I guess I missed it. Glad to know I'm not crazy.

[kevin_v]

I have a 13" MBP (2019) that hangs at enrollment, however my VMWare DEP enrolled DEV system enrolls just fine.

[seankwalker]

Yeah all my remote mgmt commands are failing right now. I flipped for a second and thought I had screwed up our cert renewal, thank god it's not that at least

[snowfox]

Go on the Mac App store and look for a free utility called Push Diagnostics by Twocanoes software. It will tell you if all the APNS hosts and ports are reachable/open on your network. Then you can definitely rule out anything on your end.
https://apps.apple.com/us/app/push-diagnostics/id689859502?mt=12
take these service status pages with a grain of salt:
https://www.apple.com/support/systemstatus/
https://developer.apple.com/system-status/
they can sometimes tell you if theres a service issue denoted by a red dot etc.
Also check the Jamf cloud service status:
https://status.jamf.com/
https://status.jamf.com/incidents/ksf6fsfttbfd
There is critical maintenance scheduled for July 1st. This may be related to your issue...

[Jason33]

The only devices we're having issues with enrolling are the latest MacBook Air's. I've got a ticket opened with Jamf as well, and the engineer mentioned APNS yesterday

[jrafferty]

I'm also having issues with APNS but in Jamf Cloud. In most cases, I don't think your firewalls or connectivity are to blame here.

[korzeniowskin]

ditto

[CGundersen]

Same issue here w/ Jamf Cloud (Test and Prod environments). I don't see us keeping up w/ flushing of pending/failed management commands ... too much random and not enough scalable from my testing of that workaround. I very much hope the unrelated "urgent" AM maintenance outage has side benefit of correcting this issue.

[Cayde-6]

https://status.jamf.com/incidents/5xzj5xy6nk2x

Known issue for US East

[hengover28]

i can confirm that i have the same issue.

It seems that my command will work after a while ( up to 30 min of wait )

[Jason33]

Is this still happening for people? I got an email that the issue was identified and resolved on 7/3; I dont have any machines to test with at the moment

[galionschools]

No, it was resolved. Devices enroll without issue and already enrolled ones don't experience failed commands.

[nathan_thornhil]

This issue has not been resolved for me. Enrollments now happen sometimes, which is better then a 100% failure rate, but the issue persists.

[mainelysteve]

@nathan.thornhill That stinks. I assume you're on us-east-1? Have you submitted a support case yet?

[nathan_thornhil]

Yes, and I've sent a variety of logs and requested information. Currently waiting on the next response from Jamf.