weird JSS issue with TomCat

Hi,
I have the same problem! JSS server crashes a few times a week. No access to the JSS web backend. Need to restart Tomcat. Don't know yet if it's a memory configuration problem of Tomcat... My JSS is on Windows Server 2008 R2, 8 Gb RAM, cpu dual 2.7 GHz Intel Xeon...

Same issue. Ubuntu Linux server with JSS 9.25. Updating to 9.31 today..

Since upgrading our test environment to 9.x (now 9.32), Tomcat has crashed on us a few times over the course of a month or so. @joe.farage][/url or @lehmanp00][/url - any luck on this issue? Support isn't seeing anything in our logs.

Edit - our JSS is Windows 2008R2

@CasperSally

I haven't had a chance to look at your case, mostly because I'm guessing your contact name in our ticket system isn't CasperSally. :)

However, there are frequent things that need changing that won't show up in log files but, instead, show up in the JSS Summary. A full summary will show everything but allocated Tomcat memory, that, we have to look at manually.

I'm not certain how many devices you're managing, so I'll go with settings on the higher end of what might be strictly (and minimally) necessary.

MySQL Related Things

- Is your max_allowed_packet set to something under 512MB? If so, it should be changed. Default max_allowed_packet ranges from 1MB to 16MB, and all are too small. When it's too small, we can see intermittent failures in backups, package installs (especially printer drivers), and imaging. USUALLY, there is a specific error in a log file that will reference it being too small, but that doesn't always happen; sometimes it just shows up as a 'lost connection to database' style error.
- How about your max_connections? Still at 151? Try bumping it up to 601.

If Change MySQL Settings is lit up under the Utilities menu of the JSS Database Utility, we can edit these settings there. If not, for Windows, the default location for the configuration file is in Program DataMySQLMySQL Server 5.x and is called my.ini (it may just say my if displaying file extensions is turned off).

In Linux and Mac environments, we should see it in /etc/my.cnf. If you don't have my.cnf in /etc, copy the file over from /path/to/mysql/support-files

Ex: sudo cp /usr/local/mysql/support-files/my-huge.cnf /etc/my.cnf

If my-huge doesn't exist, try my-default, or just ls the directory. Once you have a my.cnf in /etc edit it with elevated privileges; I usually use nano because I've long since forgotten all of my vi keys.

In my.ini we'll see variables that will start with max_allowed_packet and max_connections.
We'll change those so they look similar to:
max_allowed_packet=512M
max_connections=601

If we see a line that starts with sql_mode in 5.6, comment that line out with #. Strict mode and the JSS don't often play nice together, even in 9.x.

Save and restart the MySQL service for those changes to take effect.

Tomcat Related Things

- Stop Tomcat first, Windows gets grumbly if you try to edit Tomcat config files while Tomcat is still running sometimes.

- How much memory is allocated to Tomcat? The bare minimum we recommend, and only for smaller environments (under 1k devices) is 2GB. For environments larger than that 4GB is the minimum recommended with 8GB being closer to the ideal. In general, we want to allow Tomcat to take half of the server's available RAM for its maximum amount. Note: There is a cosmetic defect in the JSS Database Utility for Windows; it will always show 512MB no matter what you set it to. It DOES save your setting, and you can verify that by opening up tomcat7w.ext and checking the amount of memory on the Java tab.
For instructions on changing Tomcat memory allocation on a Linux server: https://jamfnation.jamfsoftware.com/article.html?id=139

- In your server.xml (program filesjss omcatconf for Windows, /path/to/jss/tomcat/conf for other installs done using the installer), check your maxThreads. We want them to be 2.5 times the amount of MySQL max_connections. If your max_connections are set to 601, we want 1502 maxThreads.

- In your DataBase.xml (program filesjss omcatwebapps ootweb-infxml), up the max pool size to 300 or so.

Save & restart Tomcat for those changes to take effect.

Back to the Summary: What does your database size look like in relation to the amount of devices you're managing? If we see something abnormal like, say, 700 devices and a 2GB database, we'll know we need to do some house cleaning, which is a nicer way of saying "flush your logs."

Common tables that explode in size due to infrequent log flushing or oddly frequent, constant inventory updates:

applications
fonts
plugins
usage_logs log_actions (policy logs end up here, failed or successful)

When any of these tables get bloated, we can see the JSS become sluggish, unstable, crash-y, taking forever to bring up inventory tabs for devices, policy logs that take ages to finally show, searches taking a long time, etc...

These can be kept under control by setting the JSS to automatically flush logs on a regular basis AND by making sure the log flushing time doesn't butt heads with the backup time. Both are defaulted to midnight, and both can't really run at the same time, so we need to have some time offset between the two. Whether you set yours to flush before or after the backup runs is up to you.

It can also help to, in the summary, search for policies that have Update Inventory set to True; the more there are, the more redundant data you may have. Inventory reports in the JSS do not just add to the record if something has changed and do not overwrite. They append.
So, if you have, as an example, 20 policies that all have Update Inventory checked and 700 computers, on any given day you have the potential to get up to 14000 inventory updates and the JSS will only be using the most recent one for its scoping purposes. Even if the policies run once per computer, they will still typically re-run if that computer is reimaged, so you still have the potential for a lot of redundant inventory reports.
Over time, it all builds up, kind of like someone stacking file folders on your desk over and over and over and never cleaning it off because you might need them. After awhile, your desk will be a disaster and it'll be hard to get any other work done. If record retention is necessary per policy or by law for what you're doing, we usually recommend taking an End of Month or End of Year style backup, saving it somewhere else, and labeling it clearly so you know what it is; if you need the old data for auditing or reporting, spin up a sandboxed JSS, restore that database, and get the info you need that way.

General rule of thumb: If a policy doesn't absolutely, 100% NEED to update inventory immediately and it can wait until the next regular check in for the device(s) scoped, uncheck that box.

Hopefully some of the above info helps.

Thanks!

Amanda Wulff
JAMF Software Support