Lost Contact with JSS on Port 8443

[akahn]

Hello,

After updating certificates on the JSS console, I unloaded the Tomcat process from the command line to do a full backup (it's quicker that way). I reloaded it but was unable to connect using a browser or any of the Casper Suite apps. I am able to connect through vnc, afp, smb, ssh, scp, and web hosting on port 80 with multiple computers.

Here's what I've done so far for troubleshooting:
- tested networking
- turned the server off and on again
- restarted Tomcat
- restarted MySQL
- restored an earlier database backup
- reinstalled JSS and rebooted
- updated to JDK 7 u71 with Tomcat unloaded then rebooted

I'm on 9.62 and the server is on 10.9.5. I'm about to do a decent-sized rollout of classroom Macs that need management starting Friday. Any suggestions?

Happy Holidays!

-Andrew

[akahn]

Here's what the Tomcat startup looks like:

Using CATALINA_BASE:   /Library/JSS/Tomcat
Using CATALINA_HOME:   /Library/JSS/Tomcat
Using CATALINA_TMPDIR: /Library/JSS/Tomcat/temp
Using JRE_HOME:        /Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home
Using CLASSPATH:       /Library/JSS/Tomcat/bin/bootstrap.jar:/Library/JSS/Tomcat/bin/tomcat-juli.jar
Tomcat started.
Dec 24, 2014 12:53:33 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-bio-9006"]
Dec 24, 2014 12:53:33 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-bio-8443"]
Dec 24, 2014 12:53:33 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-bio-8443"]
Dec 24, 2014 12:53:33 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["ajp-bio-8009"]
Dec 24, 2014 12:53:33 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["ajp-bio-8009"]
Dec 24, 2014 1:35:45 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /Users/administrator/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
Dec 24, 2014 1:35:46 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-9006"]
Dec 24, 2014 1:35:46 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-8443"]
Dec 24, 2014 1:35:47 PM org.apache.coyote.AbstractProtocol init
SEVERE: Failed to initialize end point associated with ProtocolHandler ["http-bio-8443"]
java.io.IOException: Alias name 744c97ffd73d4577b005f81c90f37a25 does not identify a key entry
    at org.apache.tomcat.util.net.jsse.JSSESocketFactory.getKeyManagers(JSSESocketFactory.java:579)
    at org.apache.tomcat.util.net.jsse.JSSESocketFactory.getKeyManagers(JSSESocketFactory.java:517)
    at org.apache.tomcat.util.net.jsse.JSSESocketFactory.init(JSSESocketFactory.java:462)
    at org.apache.tomcat.util.net.jsse.JSSESocketFactory.createSocket(JSSESocketFactory.java:209)
    at org.apache.tomcat.util.net.JIoEndpoint.bind(JIoEndpoint.java:398)
    at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:646)
    at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:434)
    at org.apache.coyote.http11.AbstractHttp11JsseProtocol.init(AbstractHttp11JsseProtocol.java:119)
    at org.apache.catalina.connector.Connector.initInternal(Connector.java:978)
    at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
    at org.apache.catalina.core.StandardService.initInternal(StandardService.java:559)
    at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
    at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:821)
    at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
    at org.apache.catalina.startup.Catalina.load(Catalina.java:638)
    at org.apache.catalina.startup.Catalina.load(Catalina.java:663)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:280)
    at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:454)

Dec 24, 2014 1:35:47 PM org.apache.catalina.core.StandardService initInternal
SEVERE: Failed to initialize connector [Connector[HTTP/1.1-8443]]
org.apache.catalina.LifecycleException: Failed to initialize component [Connector[HTTP/1.1-8443]]
    at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:106)
    at org.apache.catalina.core.StandardService.initInternal(StandardService.java:559)
    at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
    at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:821)
    at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
    at org.apache.catalina.startup.Catalina.load(Catalina.java:638)
    at org.apache.catalina.startup.Catalina.load(Catalina.java:663)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:280)
    at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:454)
Caused by: org.apache.catalina.LifecycleException: Protocol handler initialization failed
    at org.apache.catalina.connector.Connector.initInternal(Connector.java:980)
    at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
    ... 12 more
Caused by: java.io.IOException: Alias name 744c97ffd73d4577b005f81c90f37a25 does not identify a key entry
    at org.apache.tomcat.util.net.jsse.JSSESocketFactory.getKeyManagers(JSSESocketFactory.java:579)
    at org.apache.tomcat.util.net.jsse.JSSESocketFactory.getKeyManagers(JSSESocketFactory.java:517)
    at org.apache.tomcat.util.net.jsse.JSSESocketFactory.init(JSSESocketFactory.java:462)
    at org.apache.tomcat.util.net.jsse.JSSESocketFactory.createSocket(JSSESocketFactory.java:209)
    at org.apache.tomcat.util.net.JIoEndpoint.bind(JIoEndpoint.java:398)
    at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:646)
    at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:434)
    at org.apache.coyote.http11.AbstractHttp11JsseProtocol.init(AbstractHttp11JsseProtocol.java:119)
    at org.apache.catalina.connector.Connector.initInternal(Connector.java:978)
    ... 13 more

Dec 24, 2014 1:35:47 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["ajp-bio-8009"]
Dec 24, 2014 1:35:47 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 2575 ms
Dec 24, 2014 1:35:47 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service Catalina
Dec 24, 2014 1:35:47 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.57
Dec 24, 2014 1:35:47 PM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deploying web application archive /Library/JSS/Tomcat/webapps/ROOT.war
ClassLoaderLeakPreventor: Settings for se.jiderhamn.classloader.leak.prevention.ClassLoaderLeakPreventor (CL: 0x6b8e2779):
ClassLoaderLeakPreventor:   stopThreads = true
ClassLoaderLeakPreventor:   stopTimerThreads = true
ClassLoaderLeakPreventor:   executeShutdownHooks = true
ClassLoaderLeakPreventor:   threadWaitMs = 5000 ms
ClassLoaderLeakPreventor:   shutdownHookWaitMs = 10000 ms
ClassLoaderLeakPreventor: Initializing context by loading some known offenders with system classloader
Dec 24, 2014 1:36:00 PM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deployment of web application archive /Library/JSS/Tomcat/webapps/ROOT.war has finished in 12,539 ms
Dec 24, 2014 1:36:00 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-9006"]
Dec 24, 2014 1:36:00 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["ajp-bio-8009"]
Dec 24, 2014 1:36:00 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 12774 ms

[justinrummel]

Want to make sure I understand your environment:

You are running the JSS on a Mac Server, but which OS? Yosemite or Mavericks? You can VNC, AFP, SMB, Web, etc to the Mac Server so you know it's up and running

Try to stop/start the tomcat service via your JSSDatabaseUtil.jar file, but this time take note of the time. Then see if there are any clues in the JAMFSoftwareServer.log or catalina.out files.

- Justin

[justinrummel]

I now see "server is on 10.9.5"

[akahn]

I restarted the service a few times for consistency.

So this shows up every 10 seconds in catalina.out:

Dec 25, 2014 11:56:06 AM org.apache.catalina.core.StandardContext listenerStop
SEVERE: Exception sending context destroyed event to listener instance of class com.jamfsoftware.jss.server.JAMFContextListener
java.lang.NullPointerException
    at com.jamfsoftware.jss.threading.ThreadPool.submit(ThreadPool.java:141)
    at com.jamfsoftware.jss.pushnotification.connection.PushQueueManager.getInstance(PushQueueManager.java:68)
    at com.jamfsoftware.jss.server.JAMFContextListener.contextDestroyed(JAMFContextListener.java:67)
    at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:5035)
    at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5687)
    at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232)
    at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1591)
    at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1580)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

And after it passes initializing then enabling LDAP pooling, I get these messages with different components listed below (in JAMFSoftwareServer.log):

2014-12-25 11:56:06,538 [ERROR] [startStop-1] [DataSource               ] - Error requesting connection: java.lang.NullPointerException
java.lang.NullPointerException

2014-12-25 11:56:06,791 [FATAL] [Thread-2   ] [InitializeServerThread   ] - Fatal error logged during server initialization: Java Cryptography Extensions Not Installed

Here's the full JAMFSoftwareServer.log of the restart:

2014-12-25 11:56:06,478 [INFO ] [startStop-1] [InitializeServer         ] - Initializing JSS...
2014-12-25 11:56:06,490 [INFO ] [Thread-2   ] [LDAPUtils                ] - Enabled LDAP pooling
2014-12-25 11:56:06,538 [ERROR] [startStop-1] [DataSource               ] - Error requesting connection: java.lang.NullPointerException
java.lang.NullPointerException
    at com.jamfsoftware.jss.database.DataSource.prepareStatement(DataSource.java:197)
    at com.jamfsoftware.jss.database.DataSource.prepareStatement(DataSource.java:186)
    at com.jamfsoftware.jss.database.DataSource.prepareStatementForExecuteQuery(DataSource.java:291)
    at com.jamfsoftware.jss.objects.user.AccountHelper.getUsersForNotificationEvent(AccountHelper.java:274)
    at com.jamfsoftware.jss.usernotification.UserNotificationService.notify(UserNotificationService.java:103)
    at com.jamfsoftware.jss.usernotification.UserNotificationService.notify(UserNotificationService.java:45)
    at com.jamfsoftware.jss.server.JAMFContextListener.sendShutdownNotification(JAMFContextListener.java:83)
    at com.jamfsoftware.jss.server.JAMFContextListener.contextDestroyed(JAMFContextListener.java:53)
    at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:5035)
    at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5687)
    at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232)
    at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1591)
    at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1580)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
2014-12-25 11:56:06,541 [ERROR] [startStop-1] [AccountHelper            ] - java.lang.NullPointerException
java.lang.NullPointerException
    at com.jamfsoftware.jss.objects.user.AccountHelper.getUsersForNotificationEvent(AccountHelper.java:275)
    at com.jamfsoftware.jss.usernotification.UserNotificationService.notify(UserNotificationService.java:103)
    at com.jamfsoftware.jss.usernotification.UserNotificationService.notify(UserNotificationService.java:45)
    at com.jamfsoftware.jss.server.JAMFContextListener.sendShutdownNotification(JAMFContextListener.java:83)
    at com.jamfsoftware.jss.server.JAMFContextListener.contextDestroyed(JAMFContextListener.java:53)
    at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:5035)
    at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5687)
    at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232)
    at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1591)
    at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1580)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
2014-12-25 11:56:06,542 [ERROR] [startStop-1] [DataSource               ] - Error requesting connection: java.lang.NullPointerException
java.lang.NullPointerException
    at com.jamfsoftware.jss.database.DataSource.prepareStatement(DataSource.java:197)
    at com.jamfsoftware.jss.database.DataSource.prepareStatement(DataSource.java:186)
    at com.jamfsoftware.jss.database.DataSource.prepareStatementForExecuteQuery(DataSource.java:291)
    at com.jamfsoftware.jss.objects.user.AccountHelper.getUsers(AccountHelper.java:221)
    at com.jamfsoftware.jss.objects.user.AccountHelper.getUsersForNotificationEvent(AccountHelper.java:287)
    at com.jamfsoftware.jss.usernotification.UserNotificationService.notify(UserNotificationService.java:103)
    at com.jamfsoftware.jss.usernotification.UserNotificationService.notify(UserNotificationService.java:45)
    at com.jamfsoftware.jss.server.JAMFContextListener.sendShutdownNotification(JAMFContextListener.java:83)
    at com.jamfsoftware.jss.server.JAMFContextListener.contextDestroyed(JAMFContextListener.java:53)
    at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:5035)
    at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5687)
    at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232)
    at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1591)
    at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1580)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
2014-12-25 11:56:06,543 [ERROR] [startStop-1] [AccountHelper            ] - java.lang.NullPointerException
java.lang.NullPointerException
    at com.jamfsoftware.jss.database.DataSource.executeQuery(DataSource.java:390)
    at com.jamfsoftware.jss.objects.user.AccountHelper.getUsers(AccountHelper.java:222)
    at com.jamfsoftware.jss.objects.user.AccountHelper.getUsersForNotificationEvent(AccountHelper.java:287)
    at com.jamfsoftware.jss.usernotification.UserNotificationService.notify(UserNotificationService.java:103)
    at com.jamfsoftware.jss.usernotification.UserNotificationService.notify(UserNotificationService.java:45)
    at com.jamfsoftware.jss.server.JAMFContextListener.sendShutdownNotification(JAMFContextListener.java:83)
    at com.jamfsoftware.jss.server.JAMFContextListener.contextDestroyed(JAMFContextListener.java:53)
    at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:5035)
    at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5687)
    at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232)
    at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1591)
    at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1580)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
2014-12-25 11:56:06,757 [INFO ] [Thread-2   ] [InitializeServerThread   ] - JSS Version: 9.62
2014-12-25 11:56:06,791 [FATAL] [Thread-2   ] [InitializeServerThread   ] - Fatal error logged during server initialization: Java Cryptography Extensions Not Installed
2014-12-25 11:56:06,792 [INFO ] [Thread-2   ] [InitializeServer         ] - Finished JSS Initialization.

[keaton]

Hey @akahn

Looking at the latest log you posted we may just need to install the JCE files on the server. Please refer to step 3 under "OS X v10.7 or later" in our KB below.

Installing Java and MySQL
https://jamfnation.jamfsoftware.com/article.html?id=28

Hope that helps!

Thanks,

--

Keaton Svoma
JAMF Support

[davidacland]

Look like that to me as well, from the line:

Java Cryptography Extensions not installed

Not quite sure how they have disappeared from the original change (updating certificates on the JSS console).

I'd be pretty sure the database isn't at fault. If you do get database problems the web interface normally loads up and displays an error of some kind.

For what it's worth, whenever I've had a problem with tomcat starting up, or being unreachable on 8443, it's been Java related.

[akahn]

So I did replace the JCE files in the new JDK installation (U71) as per instructions but I guess JSS isn't using it.

After replacing the files on U68, I was able to get through on port 9006 and get a new error (attached). The password should be correct and I tried replacing it, only to get the same error.

external image link

[akahn · Accepted solution]

I was on with JAMF support and it came down to a bad SSL certificate in Tomcat. I recreated the certs and everything came back. Thanks for your help, everyone.

[jamfvijju]

Hello Akahn,

Can you please tell me the process of creating the certs for Tomcat.

Thanks
Vijendra