Tomcat/JSS going unresponsive. Have to restart entire server.

[nadams]

Hello, all. I am a new JAMF customer, in the middle of my 3 day JumpStart. We're experiencing an issue with JSS and Tomcat, where after actively using JSS for some time (setting up policies, packages, etc), JSS will suddenly go unresponsive. Checking the server, Tomcat is typical idle, with maybe 3-400mb of memory usage.

Server is 2012R2, and we're running JSS 9.81. When the service goes unresponsive, I am not able to restart Tomcat... it won't respond to a stop request at all. Restarting the server fixes the issue for a short time. During the jump start today, we had to restart twice. Once around noon, and again right at the end of the day.

Since we're still in implementation, there are only 8 devices assigned to Casper, and minimal check-in traffic. I'm the only person signed in and using the server, so the web interface shouldn't be hit hard either.

The last thing we did today was increase the Tomcat max memory from 1024mb to 2048, but I have a feeling that won't help since every time we look at it, it's well under 1gb in the first place.

Any other things we should look at before I contact JAMF support?

[bbot]

We had a similar issue where our jss would become unresponsive every 3-5 days. Usually restarting the tomcat service would fix it, or a full reboot of the server.

Not the best fix, but we ended up setting up scheduled reboots on the server and haven't seen the issue yet...

[nadams]

I was hoping to avoid this "solution"... I've had to do it in other cases, but I always felt that it was just ignoring the underlying problem. I found other references to the issue and there was some discussion about clearing out the Tomcat logs more frequently. But really, this is a brand new system just being set up... shouldn't be much in the way of logs. If we continue to have the issue, I will reach out to JAMF support next week. For now, I should focus on getting through the Jumpstart and retaining as much of that information as I can.

[jasonjrjune]

We also had this issue in our environment and after working with JAMF we came to realize it was an issue with our smart computer groups we had to many search criteria options being checked which caused a big issue. Since we have deleted the smart groups we have not experienced the problem.

[BenDenham]

@jasonjrjune you mean smart groups?

[jasonjrjune]

@BenDenham

Yes Ben my apologies correct i meant smart groups.

[davidacland]

The two things I've had causing this have been memory (in almost every case), and in one case CPU. In the CPU case, the server (virtual) only had one core. We added a second and all was well.

In the case of memory, it is still quite possible for the server to spike and then crash tomcat. When you check, it may just be running normally.

If you look at Settings > JSS Information > Memory Usage, how much RAM does it have available?

Other potential issues can be problematic Java versions but that is less likely.

[nadams]

In the CPU case, the server (virtual) only had one core. We added a second and all was well.

I thought about the one core thing... I've often created VMs and forgot to go back in after and raised the core count. In this case, however, it's allocated 4 cores.

In the case of memory, it is still quite possible for the server to spike and then crash tomcat. When you check, it may just be running normally. If you look at Settings > JSS Information > Memory Usage, how much RAM does it have available?

Right now it has 1.8gb free out of 2gb total. We did just increase it to 2gb at the end of the day yesterday, so maybe that will help. We're about to start day 3 of the Jumpstart, and I believe we're also going to start by upgrading from JSS 9.8.1 to 9.8.2. Not sure that's going to help, but why not?

[andrew_nicholas]

I experienced the same thing as @jasonjrjune. We had two Smart Groups that were extremely complex and causing major slow downs. Is your DB on the same box as well or is it on another device? You might want to take a look at the slow query logs just incase.

[were_wulff]

@nadams

One other thing to look at that I didn't see mentioned in your post is the version of MySQL that you're running. If you're using MySQL 5.7.x, that may be the problem.

The JSS only officially supports MySQL versions 5.5 and 5.6 (recommended); this is mentioned on page 26 of the Casper Suite Administrator's Guide. We clarified in the latest release of that documentation (9.82) that it was for those two versions only; previous versions simply said "MySQL 5.1 or later" which was outdated wording.
I do apologize if the wording in our older documentation caused confusion in this instance; the current documentation has been changed to reflect that the supported versions of MySQL are "MySQL 5.5.x or 5.6.x (MySQL 5.6.x is recommended)".

While some customers do run MySQL 5.7.x with no issues, it is not an officially supported version yet, and we can't guarantee that it'll work as intended.

We have seen, in support, several cases in which MySQL 5.7 causes the JSS to go randomly unresponsive. The solution in that case is to make sure you have a good backup, then remove MySQL 5.7 and go back to either 5.6.x (recommended) or 5.5.x.

Thanks!
Amanda Wulff
JAMF Software Support

[nadams]

We have seen, in support, several cases in which MySQL 5.7 causes the JSS to go randomly unresponsive. The solution in that case is to make sure you have a good backup, then remove MySQL 5.7 and go back to either 5.6.x (recommended) or 5.5.x.

Well, that could be the explanation then. We're running MySQL 5.7, which is what was installed by a JAMF technician during our demo. We were having issues with the cloud-based demo, so he installed it for us on an internal VM to get us up and running. Perhaps the issues with 5.7 were not well-known when this was done in early December.

We're opening a case with JAMF support to schedule a WebEx session to downgrade to 5.6 after the Jumpstart is complete.

[scottb]

Could it be that this is also at play?
Posted yesterday...

[wayfaircasper]

We just increased the allocated RAM using this guide.

https://jamfnation.jamfsoftware.com/article.html?id=139

We are still waiting to see if it happens again but so far so good, our JSS seems faster and more responsive as well!

[were_wulff]

@nadams

My guess is the confusion came in part because of how our documentation was worded; for the version that the technician set up for you at the time, it would have only read "MySQL 5.1 or later" which could easily lead to the conclusion that 5.7.x was compatible, as it falls within the scope of 5.1 or later.

This was an unfortunate error in our documentation, which is something we corrected in the 9.82 documentation and it now specifies, correctly, that MySQL 5.5 and MySQL 5.6 are the two compatible versions that are recommended for use with the JSS.

I have reached out to our Professional Services team to let them know so they can take steps to make sure our on site technicians are aware of the documentation change/correction.

Sorry again for the confusion!

Amanda Wulff
JAMF Software Support

[nadams]

@scottb

Not the same issue, but I did run into that one too! We were on the phone w/ tech support and he mentioned that there are undocumented steps to complete the upgrade from 9.81 to 9.82 on a Windows server. We ended up having to roll back to a snapshot made of the VM prior to the upgrade attempt in order to get the server back up to finish the Jumpstart. That's when we decided to leave it alone until next week after the Jumpstart is over.

So far, the increase in available memory to the Tomcat service seems to have helped, although we have had to restart the VM for other reasons several times.

[scottb]

@nadams Yeah, just wondered if that might have compounded the problems you have (had).
I always light incense and pray right before each Windows upgrade ;)
We also upped the memory on all the JSS stuff a while back (dedicated server, not a VM) and it did help a lot.
We have less than 1200 Macs, but they're all over the globe and have a wide range of crappy networks connections. More memory settled most of those issues as did moving to IP-addresses for the SMB shares and getting rid of special characters in the two JSS account passwords (!@&, etc.).

[jaharmi]

Is the situation still true for any JSS hosted on Mac OS X/OS X/macOS?

From my reading of MySQL :: Supported Platforms: MySQL Database, OS X v10.11+ is not officially supported by MySQL 5.5.x-5.6.x. I didn't see that mentioned in this thread.

Doesn't the documentation — regarding version 5.5 or 5.6 (I checked the v9.93 release notes to confirm) — leave those running a JSS database server (for whatever reason, could be development/testing) on El Capitan and newer in a different unsupported state? Presumably, organizations will also be under pressure to keep up-to-date with macOS upgrades because they are often security patches for the last major version.

Is there a roadmap/timeframe for supporting JSS with MySQL 5.7.x?