APNS Issues...Among Other Things

nathan_thornhil
New Contributor II

Hello all. This upgrade to 10.22.1 has come with some challenges. I have an open ticket with Jamf, but I'm in a bind and on a time crunch so I'm reaching out to the community for ideas.

We have Jamf Cloud and starting yesterday no device can be enrolled. Automated Device Enrollment through Setup Assistant fails and User Enrollment fails. The profiles are unable to be installed.

The Jamf Server Logs show APNS issues. I called Jamf and per their suggestion I renewed the APNS push cert early and removed the devices from the Prestage Enrollment, then assigned them to it again. Still no success. Now I'm also starting to see VPP is unable to verify licenses.

I double checked with my networking team and they still have all of the ports available for Apple's services. Nothing changed on their end. It just stopped working.

Here's a sample error

2020-06-30 19:19:03,288 [WARN ] [eralPool-18] [ApnsPushQueueManager ] - Error sending push notification com.jamfsoftware.jss.pushnotification.notifications.AppleMDMCheckInNotification@f189d3fe to connection com.jamfsoftware.jss.pushnotification.connections.ApplePushNotificationServiceConnection@76963a8f. Remote host terminated the handshake 2020-06-30 19:19:03,430 [ERROR] [eralPool-19] [ApnsFeedbackConnection ] - IOException getting and entering feedback data: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake at java.base/sun.security.ssl.SSLSocketImpl.handleEOF(SSLSocketImpl.java:1321) at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1160) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1063) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402) at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:716) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:799) at java.base/java.io.InputStream.read(InputStream.java:205) at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2314) at org.apache.commons.io.IOUtils.copy(IOUtils.java:2270) at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2291) at org.apache.commons.io.IOUtils.copy(IOUtils.java:2246) at org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:765) at com.jamfsoftware.jss.pushnotification.connection.ApnsFeedbackConnection.getFeedbackData(ApnsFeedbackConnection.java:34) at com.jamfsoftware.jss.pushnotification.connection.ApnsFeedbackConnection.run(ApnsFeedbackConnection.java:88) at org.springframework.security.concurrent.DelegatingSecurityContextRunnable.run(DelegatingSecurityContextRunnable.java:84) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Suppressed: java.net.SocketException: Broken pipe (Write failed) at java.base/java.net.SocketOutputStream.socketWrite0(Native Method) at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110) at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150) at java.base/sun.security.ssl.SSLSocketOutputRecord.encodeAlert(SSLSocketOutputRecord.java:81) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:352) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:264) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:405) ... 16 more Caused by: java.io.EOFException: SSL peer shut down incorrectly at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:167) at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:108) at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1152) ... 18 more

Any help would be appreciated.

17 REPLIES 17

IT-Chris
New Contributor III

yes this is screwing me up. I cant even enroll units

mainelysteve
Valued Contributor II

There is another thread about this as well.

nathan_thornhil
New Contributor II

Thanks for that. I was wondering why I hadn't seen another post about this issue, but I was so focused on looking for posts about APNS I guess I missed it. Glad to know I'm not crazy.

kevin_v
Contributor

I have a 13" MBP (2019) that hangs at enrollment, however my VMWare DEP enrolled DEV system enrolls just fine.

seankwalker
New Contributor

Yeah all my remote mgmt commands are failing right now. I flipped for a second and thought I had screwed up our cert renewal, thank god it's not that at least

snowfox
Contributor III

Go on the Mac App store and look for a free utility called Push Diagnostics by Twocanoes software. It will tell you if all the APNS hosts and ports are reachable/open on your network. Then you can definitely rule out anything on your end.
https://apps.apple.com/us/app/push-diagnostics/id689859502?mt=12
take these service status pages with a grain of salt:
https://www.apple.com/support/systemstatus/
https://developer.apple.com/system-status/
they can sometimes tell you if theres a service issue denoted by a red dot etc.
Also check the Jamf cloud service status:
https://status.jamf.com/
https://status.jamf.com/incidents/ksf6fsfttbfd
There is critical maintenance scheduled for July 1st. This may be related to your issue...

Jason33
Contributor III

The only devices we're having issues with enrolling are the latest MacBook Air's. I've got a ticket opened with Jamf as well, and the engineer mentioned APNS yesterday

jrafferty
New Contributor

I'm also having issues with APNS but in Jamf Cloud. In most cases, I don't think your firewalls or connectivity are to blame here.

korzeniowskin
New Contributor II

ditto

CGundersen
Contributor III

Same issue here w/ Jamf Cloud (Test and Prod environments). I don't see us keeping up w/ flushing of pending/failed management commands ... too much random and not enough scalable from my testing of that workaround. I very much hope the unrelated "urgent" AM maintenance outage has side benefit of correcting this issue.

Cayde-6
Release Candidate Programs Tester

https://status.jamf.com/incidents/5xzj5xy6nk2x

Known issue for US East

hengover28
New Contributor

i can confirm that i have the same issue.

It seems that my command will work after a while ( up to 30 min of wait )

Jason33
Contributor III

Is this still happening for people? I got an email that the issue was identified and resolved on 7/3; I dont have any machines to test with at the moment

galionschools
Contributor

No, it was resolved. Devices enroll without issue and already enrolled ones don't experience failed commands.

nathan_thornhil
New Contributor II

This issue has not been resolved for me. Enrollments now happen sometimes, which is better then a 100% failure rate, but the issue persists.

mainelysteve
Valued Contributor II

@nathan.thornhill That stinks. I assume you're on us-east-1? Have you submitted a support case yet?

nathan_thornhil
New Contributor II

Yes, and I've sent a variety of logs and requested information. Currently waiting on the next response from Jamf.