Java Errors in Serverlog

[dwilliams304]

Getting the following java errors in our serverlog:

2016-01-28 14:21:02,341 [WARN ] [oolThread-2] [PushQueueManager ] - Error sending push notification com.jamfsoftware.jss.pushnotification.notifications.AppleMDMCheckInNotification@68bff97f to connection com.jamfsoftware.jss.pushnotification.connections.ApplePushNotificationServiceConnection@6676d8ac. java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty 2016-01-28 14:21:02,383 [ERROR] [oolThread-3] [APNSFeedbackConnection ] - IOException getting and entering feedback data: javax.net.ssl.SSLException: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty at sun.security.ssl.Alerts.getSSLException(Unknown Source) at sun.security.ssl.SSLSocketImpl.fatal(Unknown Source) at sun.security.ssl.SSLSocketImpl.fatal(Unknown Source) at sun.security.ssl.SSLSocketImpl.handleException(Unknown Source) at sun.security.ssl.SSLSocketImpl.handleException(Unknown Source) at sun.security.ssl.AppInputStream.read(Unknown Source) at java.io.InputStream.read(Unknown Source) at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1792) at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769) at org.apache.commons.io.IOUtils.copy(IOUtils.java:1744) at org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:462) at com.jamfsoftware.jss.pushnotification.connection.APNSFeedbackConnection.getFeedbackData(APNSFeedbackConnection.java:34) at com.jamfsoftware.jss.pushnotification.connection.APNSFeedbackConnection.run(APNSFeedbackConnection.java:89) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty at sun.security.validator.PKIXValidator.<init>(Unknown Source) at sun.security.validator.Validator.getInstance(Unknown Source) at sun.security.ssl.X509TrustManagerImpl.getValidator(Unknown Source) at sun.security.ssl.X509TrustManagerImpl.checkTrustedInit(Unknown Source) at sun.security.ssl.X509TrustManagerImpl.checkTrusted(Unknown Source) at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(Unknown Source) at sun.security.ssl.ClientHandshaker.serverCertificate(Unknown Source) at sun.security.ssl.ClientHandshaker.processMessage(Unknown Source) at sun.security.ssl.Handshaker.processLoop(Unknown Source) at sun.security.ssl.Handshaker.process_record(Unknown Source) at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source) at sun.security.ssl.SSLSocketImpl.readDataRecord(Unknown Source) ... 13 more Caused by: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty at java.security.cert.PKIXParameters.setTrustAnchors(Unknown Source) at java.security.cert.PKIXParameters.<init>(Unknown Source) at java.security.cert.PKIXBuilderParameters.<init>(Unknown Source) ... 26 more

I have a ticket open with JAMF, need to get this resolved by tomorrow morning. I can enroll devices, but cannot do anything else. I have about 300 elementary students patiently waiting for their Ipads. My environment is a Windows Server 2012 instance running on VMWare.

Thanks!

Don

[jpellet2]

We noticed the same issue on our JSS this morning. Any word from Jamf? Ours is on virtual a Linux server and not a Windows server.

[brettml]

I too am having a similar issue on a virtual Linux server - RHEL 6.5, manual install of JSS 9.82, Tomcat 7.0.67, Java 8u72. My error:

2016-02-03 04:13:01,256 [WARN ] [oolThread-0] [PushQueueManager ] - Error sending push notification com.jamfsoftware.jss.pushnotification.notifications.AppleMDMCheckInNotification@868d8314 to connection com.jamfsoftware.jss.pushnotification.connections.ApplePushNotificationServiceConnection@2c464bc0. Connection reset
2016-02-03 04:13:01,262 [ERROR] [oolThread-6] [APNSFeedbackConnection ] - IOException getting and entering feedback data: java.net.SocketException: Connection reset

[tobiaslinder]

Hi dwilliams304
With me the restart of the server solved the problem.

[usd230it]

On both 1/28 and 2/3, we logged a bunch of errors as shown below. At the same time, our JSS load average topped 200!!! It dropped to normal after about 3 to 5 minutes. Still looking for a cause.

2016-01-28 08:01:07,404 [ERROR] [lThread-866] [APNSFeedbackConnection ] - IOException getting and entering feedback data
javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:902) at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1208) at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:838) at sun.security.ssl.AppInputStream.read(AppInputStream.java:94) at java.io.InputStream.read(InputStream.java:101) at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1792) at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769) at org.apache.commons.io.IOUtils.copy(IOUtils.java:1744) at org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:462) at com.jamfsoftware.jss.pushnotification.connection.APNSFeedbackConnection.getFeedbackData(APNSFeedbackConnection.java:34) at com.jamfsoftware.jss.pushnotification.connection.APNSFeedbackConnection.run(APNSFeedbackConnection.java:89) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:701)
Caused by: java.io.EOFException: SSL peer shut down incorrectly at sun.security.ssl.InputRecord.read(InputRecord.java:482) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:883) ... 16 more

[ACousino]

I am having a similar issue as well on a virtual Linux server. Did you ever see what happened with it? Restarting the server hasn't fixed it for us like tobiaslinder suggested.

[tom_londy]

Has anyone solved this issue? Rebooting our server hasn't resolved it and JAMF support doesn't seem to know what the cause is.

[tobiaslinder]

Hi
Could it be related to the combination of tomcat7 and Java8? I had serious issues with this combo running on Ubuntu 12.0.4 LTS. Downgrading to java7 solved the problem for me.

With best regards, Tobias Linder

[bumbletech]

This probably doesn't help anyone who's had the issue out of the blue on their server, but I recently migrated our JSS to a Ubuntu cluster and hit this wall. LDAP tests failed immediately, push notifications wouldn't work correctly to get iOS devices to check in, VPP sync issues...

I copied /etc/ssl/certs/java/cacerts from the old server to the new servers (and they didn't have that at all which was odd, maybe?) and restarted tomcat. Devices are checking in and ldap lookups are happening again!

[michaelhusar]

We also saw the error "...the trustAnchors parameter must be non-empty..." on our fresh Ubuntu 14

we found out that somehow installing Java JDK did not correctly link the cacerts

/var/lib/dpkg/info/ca-certificates-java.postinst configure

did fix the issue - seems to be already reported as an ubuntu bug

Hope that helps!

[hunter990]

I am getting this same issue here. These are servers that were recently re-IP'd. We had APNS working last night and then it started this early this morning. Haven't been able to use config profiles, remote lock, or remote wipe a system since. Reaching out to Jamf tomorrow but figured I would ask first.

Have restarted tomcat, re-installed java with the same results. Funny thing is if I enroll a system I am getting the profiles, self service, and showing MDM capability. Already verified all of my ports, etc.so I know that is working as it should.

[ifbell]

So with our issue which is similar to what everyone else is stating I put wireshark on my server and captured the 2195 traffic and compared it to what I consider a clean 2195 activity using a Mac laptop and the app push diagnostics. The activity is markedly different.

So from our windows server we see the following currently. this traffic was generated while trying to send a blank push and a lock command to a machine. as well as just normal idle traffic.

1 0.000000 x.x.x.x 17.188.164.77 TCP 66 53027 → 2195 [SYN, ECN, CWR] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1 2 0.051519 17.188.164.77 x.x.x.x TCP 66 2195 → 53027 [SYN, ACK, ECN] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 SACK_PERM=1 WS=128 3 0.051671 x.x.x.x 17.188.164.77 TCP 54 53027 → 2195 [ACK] Seq=1 Ack=1 Win=131328 Len=0 4 0.123240 x.x.x.x 17.188.164.77 TCP 293 53027 → 2195 [PSH, ACK] Seq=1 Ack=1 Win=131328 Len=239 5 0.154702 17.188.164.77 x.x.x.x TCP 60 2195 → 53027 [RST, ACK] Seq=1 Ack=240 Win=65664 Len=0 6 0.232200 x.x.x.x 17.188.164.77 TCP 66 53029 → 2195 [SYN, ECN, CWR] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1 7 0.289507 17.188.164.77 x.x.x.x TCP 66 2195 → 53029 [SYN, ACK, ECN] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 SACK_PERM=1 WS=128 8 0.289630 x.x.x.x 17.188.164.77 TCP 54 53029 → 2195 [ACK] Seq=1 Ack=1 Win=131328 Len=0 9 0.291171 x.x.x.x 17.188.164.77 TCP 293 53029 → 2195 [PSH, ACK] Seq=1 Ack=1 Win=131328 Len=239 10 0.340243 17.188.164.77 x.x.x.x TCP 60 2195 → 53029 [RST, ACK] Seq=1 Ack=240 Win=65664 Len=0
This repeats constantly and consistantly.
on my Laptop as I said using push diagnostics as my known good I see this. 1 0.000000 x.x.x.x 17.188.166.23 TCP 68 55741 → 2195 [SYN] Seq=0 Win=65535 Len=0 MSS=1310 WS=32 TSval=788600035 TSecr=0 SACK_PERM=1 2 0.051075 17.188.166.23 x.x.x.x TCP 64 2195 → 55741 [SYN, ACK] Seq=0 Ack=1 Win=28960 Len=0 MSS=1310 SACK_PERM=1 TSval=2344741226 TSecr=788600035 WS=128 3 0.051123 x.x.x.x 17.188.166.23 TCP 56 55741 → 2195 [ACK] Seq=1 Ack=1 Win=131072 Len=0 TSval=788600085 TSecr=2344741226 4 0.064551 x.x.x.x 17.188.166.23 TCP 56 55741 → 2195 [FIN, ACK] Seq=1 Ack=1 Win=131072 Len=0 TSval=788600097 TSecr=2344741226 5 0.113336 17.188.166.23 x.x.x.x TCP 56 2195 → 55741 [ACK] Seq=1 Ack=2 Win=29056 Len=0 TSval=2344741288 TSecr=788600097 6 0.113386 x.x.x.x 17.188.166.23 TCP 56 [TCP Dup ACK 3#1] 55741 → 2195 [ACK] Seq=2 Ack=1 Win=131072 Len=0 TSval=788600145 TSecr=2344741288 7 0.113796 17.188.166.23 x.x.x.x TCP 56 2195 → 55741 [FIN, ACK] Seq=1 Ack=2 Win=29056 Len=0 TSval=2344741289 TSecr=788600097 8 0.113981 x.x.x.x 17.188.166.23 TCP 56 55741 → 2195 [ACK] Seq=2 Ack=2 Win=131072 Len=0 TSval=788600145 TSecr=2344741289 9 55.849478 x.x.x.x 17.188.165.212 TCP 68 55761 → 2195 [SYN] Seq=0 Win=65535 Len=0 MSS=1310 WS=32 TSval=788655540 TSecr=0 SACK_PERM=1 10 55.908882 17.188.165.212 x.x.x.x TCP 64 2195 → 55761 [SYN, ACK] Seq=0 Ack=1 Win=28960 Len=0 MSS=1310 SACK_PERM=1 TSval=2348380012 TSecr=788655540 WS=128 11 55.908936 x.x.x.x 17.188.165.212 TCP 56 55761 → 2195 [ACK] Seq=1 Ack=1 Win=131072 Len=0 TSval=788655596 TSecr=2348380012 12 55.914713 x.x.x.x 17.188.165.212 TCP 56 55761 → 2195 [FIN, ACK] Seq=1 Ack=1 Win=131072 Len=0 TSval=788655601 TSecr=2348380012 13 55.963307 17.188.165.212 x.x.x.x TCP 56 2195 → 55761 [FIN, ACK] Seq=1 Ack=2 Win=29056 Len=0 TSval=2348380066 TSecr=788655601 14 55.963359 x.x.x.x 17.188.165.212 TCP 56 55761 → 2195 [ACK] Seq=2 Ack=2 Win=131072 Len=0 TSval=788655647 TSecr=2348380066