Tomcat7 fails to start after ubuntu and java upgrades

[dnelson2813]

Hoping someone can shed some light on this. We're preparing to move to JAMF Cloud and have to upgrade our JAMF Pro server from 10.12 to the latest release (10.22.1). To do that, we have to go to 10.14 and then to 10.22.1.

10.14 requires Java 11, which requires Ubuntu 18 LTS so I had to upgrade our server from 14 to 16 to 18 and also upgraded to Java 11. Everything seemed fine until I tried to do the manual JAMF Pro upgrade.

I copied the root.war file over and when I went to start tomcat, it just fails. Code=exited status=1/FAILURE Then it says failed to start LSB: Start tomcat. That's it. I can't find any other details about why it fails to start anywhere. I reset JAVA_HOME after the java upgrade and confirmed its pointing to the right folder so I have no idea what else to check at this point. I have a ticket in with support but also posted here in case anyone has any ideas that may come faster.

[dnelson2813]

I rolled back my snapshot to just before the Java 11 upgrade and tomcat stops and starts without an issue so it seems that the Java 11 upgrade caused the issue, but I'm still not sure what specifically.

[DaveMB]

I'm seeing a similar issue, both with the installer and doing it manually with root.war. Strangely, the JAMF installer seems to be the only thing that can't see $JAVA_HOME. Tomcat sees it, but fails to start. Right now, I'm stuck at 10.12 until I can figure this out.

[dnelson2813]

I have a support ticket with JAMF. Since we've been doing manual upgrades (even before I was hired), tomcat hasn't been getting updated. Our servers are still on tomcat7 and JAMF support thinks that may be part of the problem because tomcat 8.5.40 is recommended and possibly required for 10.14. If you use Jamf's installer instead of the manual process, tomcat is automatically updated. Great. So if you're also on tomcat7, you're probably in the exact same boat as me.

[jared_f]

@dnelson2813

I assume your SQL is on a separate VM? If so, Jamf Pro Tomcat VMs are disposable - bring up a new VM, shutdown old Tomcat VM & move IP, install Jamf Pro using the installer (don't do a manual upgrade) & Install Java 11, point it to your database & start Tomcat.

Ideally you would install the version of Jamf Pro that your database is on and then again do the upgrade.

Watch this video, while you may not be clustered the same principles apply when it comes to Tomcat. https://www.youtube.com/watch?v=I1fo7_geOsU

I run a test instance in Windows Server 2019 VMs (Two TC VMs and one SQL VM). I could help you troubleshoot, but I am not too Linux savvy for Jamf hosting. Anyways, my email is flitt.j@gmail.com.

[dnelson2813]

SQL is on a Windows VM and needs to be upgraded to mySQL 8.0 also. Maybe that is part of the problem? I think it's on mySQL 5.6.

My environment is clustered (though I really don't know why because it seems like overkill) so I would have to rebuild 3 more servers, which isn't ideal. I tried installing Tomcat8 using the instructions JAMF has floating around, but it still wouldn't start even though I pointed it to the correct JAVA_HOME directory. I'm going to upgrade the SQL server to see if that's part of the problem.

[jared_f]

@dnelson MySQL 5.6 is compatible with Jamf Pro 9.98 - 10.16.0 (https://www.jamf.com/jamf-nation/articles/667/installing-java-and-mysql-for-jamf-pro-10-14-0-or-later). As long as you fall inside the compatibility matrix, I suspect your SQL is not the problem.

I assume all Tomcat VMs are upgraded to Ubuntu 18.0? Why not just stop all tomcat services on the 3 nodes, uninstall Jamf Pro and Java on the master node and re-install using the packaged installer from Jamf Nation. Then start the master node and see if it comes up, then do the same for the other 2 nodes. To me, this sounds like a corrupted install of Tomcat having to do with a mix os OS upgrades on the VMs and manual install of Jamf Pro over the years. Just my two cents.

Make sure you are using the version your database currently was left off on. If it is 10.12.0, install 10.12.0 on all three fresh and restart services. Then start upgrading again.

If you want to Zoom let me know.

[dnelson2813]

In that case, the only thing left to upgrade is Tomcat which is what JAMF suggested.

All of the VMs were upgraded to Ubuntu 18 this week in preparation for the move to 10.22.1. I asked support if I could do what you suggested and they told me since the upgrade process has been manual on these servers, it needs stay manual. It does sound like a corrupt Tomcat install but Tomcat starts on all 3 servers with 10.12, even after the Ubuntu upgrades. The web application is accessible and I can use it normally. The Java 11 update kills it.

[jared_f]

I disagree with support there - it should not make a difference whether it is a manual upgrade or upgrade via installer. It is all pointing at the same database. I would stop Tomcat, stop SQL, snap all 3 VMs, take a DB backup, and then try the above (removing Jamf Pro Web App & Java & re-install using installers).

[dnelson2813]

They claim that since we're ultimately going to the cloud, it could make a difference because the manual upgrade and upgrade via installer do different things to the server environment in general. They didn't specify what though. I guess I can talk to our migration specialist to confirm that. I've snap shotted all 4 servers so I can at least try it. If it works but the migration specialist says it could be a problem, I can revert back.

[dnelson2813]

I'm editing this comment because I realized in my frustration, I didn't run the installer as the superuser and screwed up the permissions on the jss folders.

@jared_f Your suggestion of uninstalling JAMF Pro and reinstalling with JAMF's installer package seems to be working! Thanks so much for the suggestion. I'm just going to say for people who come across this post in the future, make sure you save your server.xml file somewhere on the server so you can copy the 8443 settings into the new server.xml file.

[sdagley]

@dnelson2813 In addition to server.xml, it's also a good idea to back up DataBase.xml before upgrading the JSS app in case you need to revert