Jamf Failing On Tomcat & Server Restart

[MBrownUoG]

Hello folks.

We currently run one JSS server and one database server, both on Ubuntu 18.04, and Jamf version 10.16.1. This has been humming along without any issue for the past few months.

However, I noticed this morning that if I restart the JSS, or even just restart Tomcat (I was looking to resolve the DEP sync TLS issue), I now just get a "connection refused" message when trying to browse back to the front-end. This occurs on a server restart also, and even if I reboot both the database and front-end, it's still the same.

EDIT: To clarify, I didn't even get as far as making any changes to the JSS server, just a reboot and then the website refuses connections

Using the Jamf server tools I can start/stop/restart the tomcat service without any errors reported (but still no website availability), but I'm not entirely sure where to start troubleshooting this issue. Does anybody have any tips on areas to look at?

[kerouak]

check the following file /usr/local/jss/tomcat/webapps/ROOT/WEB-INF/xml/DataBase.xml
Ensure that the correct username and password are defined.
On the Database Server (local or remote)
>mysql -u root -p
use datbase_name
SHOW GRANTS FOR 'your db username as specified in Database.xml'@'localhost';

it should show something like: GRANT SELECT, INSERT, UPDATE ON databasename.* TO username@localhost

if not, just type :

GRANT SELECT, INSERT, DELETE, UPDATE ON database name TO 'username'@'localhost identified by 'password';

Then restart tomcat

[bentoms]

@MBrownUoG any messages in the JAMFSoftwareServer.log or Tomcats logs?

Can you revert the change you made to TLS to get the service running once more?

[MBrownUoG]

@kerouak Checking the MySQL config, the database user has "ALL PRIVILEGES" set, so should be ok. The JSS & DB servers are currently up and running without any issue so the database config should be all fine (this hasn't changed in years), it just falls over on restart of Tomcat for some reason.

@bentoms No changes were made to revert unfortunately. I logged on to the JSS server (out of hours thankfully) and decided to snapshot and then restart it before making any changes, which is when I noticed the issue. Poking through the JAMFSoftwareLog I can see the following from when I initiated the connection:

[ERROR] [duledPool-0] [VppLicenseUpdater ] - Error updating licenses
[ERROR] [duledPool-0] [VppLicenseMonitor ] - Error running license monitor
[ERROR] [Thread-10 ] [DataSource ] - Error requesting connection: java.lang.NullPointerException
[ERROR] [Thread-10 ] [ComputerHelper ] - Error deleting by computer_id for table installed_printers
[ERROR] [Thread-10 ] [DataSource ] - Error requesting connection: java.lang.NullPointerException
[ERROR] [Thread-10 ] [ComputerHelper ] - Error inserting installed_printers
[ERROR] [Thread-10 ] [lientCommunicationServlet] - Error processing communication content - org.springframework.beans.factory.BeanCreationNotAllowedException: Error creating bean with name 'runningServiceHelper': Singleton bean creation not allowed while the singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!)
[ERROR] [Thread-10 ] [JAMFHttpServlet ] - The JSS received an error (Error creating bean with name 'artifactVersionManager': Singleton bean creation not allowed while the singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!)) for a request made by device.

Unsure what to make of that...

EDIT: Also seeing a few instances of the following when the server starts:

[ERROR] [duledPool-5] [ntInstanceSyncCommService] - com.jamfsoftware.jss.objects.streamlinedenrollment.service.DeviceEnrollmentProgramException: OAuth communication failed
[ERROR] [duledPool-5] [OAuthConnection ] - javax.net.ssl.SSLHandshakeException: Received fatal alert: decode_error

Thank you both for your responses btw, hugely appreciated.

[bentoms]

@MBrownUoG So the messages you're seeing when the server starts are related to the TLS issue.

But the bean issue I've not seen before.

How did you make the TLS change?

[MBrownUoG]

@bentoms Looking at other threads on the same issue, I was going to follow along with PI-007522 on the known issues page. However I didn't even get that far. I just ran a snapshot, rebooted the JSS server and noticed the site wasn't responding.

At that point I tried a Tomcat restart as a quick fix, and when that didn't work (no errors reported on the command line, but site still not responding), I rebooted the database server to see if that had any issues. As that didn't work I had to revert the JSS snapshot quickly as we were close to core work hours.

It's fully functional as we speak, just falls over on restart. No changes have been made to it for months. Strange!

[kerouak]

can you post you setenv.sh please??

[kerouak]

if you are editing the setenv.sh file manually, it's required that the addition is added thus: export JAVA_OPTS="$JAVA_OPTS -Xmx4096M -Xms512M -Djava.awt.headless=true -Djdk.tls.client.protocols=TLSv1.1,TLSv1.2"

otherwise, Tomcat will not startup.

as soon as I added it, bingo! We're back communicating again...

[MBrownUoG]

@kerouak Thank you for this. Very interesting, as that's the change I was about to make when I initially discovered this problem... however I just assumed the communication issues were isolated to communication with DEP, and not that Tomcat itself wouldn't start without the edit. The server has been restarted in the last few months also, so unsure as to how we haven't run into this before now, odd!

For reference, our setenv.sh is currently as follows:

#! /bin/sh
# ==================================================================
#       Created: 03/21/2014
#       Updated: 11/14/2014
# ==================================================================
# Place any and all customizations to your Tomcat instance here
# e.g. export JAVA_OPTS="$JAVA_OPTS -Xmx1024M -Xms256M -Djava.awt.headless=true"
# You may also choose to place any customizations to your Catalina Options here as well
# e.g. export CATALINA_OPTS="$CATALINA_OPTS -XX:MaxPermSize=256m"
# Tomcat temp directory can also be set within this file
# e.g export CATALINA_TMPDIR=/tmp
#
# This file will be backed up and restored on upgrades

JAVA_OPTS=" -Xmx10240M -XX:MaxPermSize=512M"
export CATALINA_OPTS="$CATALINA_OPTS -XX:PermSize=256m"
# Sets JAVA_HOME if it is NOT set
. /usr/local/jss/tomcat/bin/set-java-home.sh

[MBrownUoG]

Resolved! Turns out the Tomcat server.conf file had become corrupt and was showing as blank. Restored the previous version from the last upgrade backup I took (with help from Jamf support, who were excellent on this), and now all is good.

Cheers for the responses all.

[bentoms]

awesome.. nice sleuthing @MBrownUoG

[MBrownUoG]

To be fair the sleuthing was 50/50 with the Jamf Support operator, I'm still a relative newbie and wearing many hats that stop me focusing on reinforcing my Linux skills, but all useful lessons! Cheers for the help :)

[keopraseuths]

We recently ran into this issue with the server.xml file becoming corrupt/empty after a server reboot. Resolved it by copying the file form our last update backup and the webapp started up again.