Tomcat / Server Performance Tuning?

[musat]

Hello,
We are running our JSS on a dedicated Linux server with 8GB of memory. Lately the JSS has been running really slowly and after doing some searching I ran into some older post about Tomcat memory settings. After chekcing the JSSDatabaseUtility I saw that we were still running at the initial defaults of 512MB max memory and 64MB max permgen. I increased both of those, but since there isn't any sort of guideline, I feel like I just randomly increased them. I increased the memory to 3072MB and the permgen to 256MB. The server is definitely running much smoother with much better response times, but I still feel like there should have been some "thought" put into changing those settings instead of just me raising them.
I saw some old posts referencing a post from @amanda.wulff. But that post was from 2014 and she has since removed it, understandably. I'm just wondering if there are some new guidelines for those settings based on checking some metric.

Thanks, Tim Musa

[millersc]

Tuning has been an ongoing target as jamf matures. Reach out to your buddy and work with them as every environment is unique.

[senior_techs_sd]

A report or alert generated when the server is low on Java memory is all we really need to tune ourselves. I'm happy to keep raising the numbers as we add devices but I have to wait for users to report issues before I know there is a problem and it seems like the sort of thing that could reasonably be alerted on since alerting (such as backups, services starts/stops) is already built into the product.

[tlarkin]

Hello everyone,

There have never been any magic numbers that Jamf or anyone else can really give you on far as what settings go where and how much CPU, RAM and disk I/O you need to fine tune your deployment. The reason is that everyone uses the product differently and workflows, environment, fleet size, and everything else I will lump into "other factors," are at play.

Really you should scale with data. Get some monitoring software and start to collect metrics on your environment. Trend them into a system and just collect data over time. Sure there could be a huge spike in resources needed, but you need to figure out is that a workflow issue, or is that a scaling issue? Perhaps it is neither and it is a network issue (say DNS lag or something as an example). If you have 1,000 devices checking in every 5 minutes and submitted inventory constantly, versus 10,000 devices that check in every hour and submit inventory once a week you could get very different data and the larger environment might even run better on the same spec infrastructure.

There are so many tools out there to do this. Here goes a quick list of ones you can try to see if you like or not.

• Zabbix
• Nagios
• Prometheus
• Sensu
• Zenoss
• New Relic
• Spice works

Those are just some of the common ones I have read or head about over the years. I have used several of those listed but not all of them. Also note that Tomcat is a Java Application. Meaning there are already tons and tons of Java dev/debug tools out there that can help you monitor Java Application health. JMX hooks is such a feature. VisualVM and jconsole are also tools that can work as well.

I have a test box with JMX enabled and I can run a simple set of command via a Java binary to get info out:

get -b java.lang:type=Memory HeapMemoryUsage
#mbean = java.lang:type=Memory:
HeapMemoryUsage = { 
  committed = 4098359296;
  init = 262144000;
  max = 10517741568;
  used = 874578912;
 };

$>

So it looks like the JMX binary just spits out a dictionary output there and you could maybe natively send that to a solution that can consume that or with a bit of code have it scrape and escrow the data to a tool.

However with lots of monitoring tools there may be a cost of entry of writing some simple code to parse and collect it. A lot of them have built in tooling that just makes this work out of the box, and others just have frameworks/APIs and leave the data collection up to you. Out of the box most tools will give you CPU, RAM, and Disk I/O out of the box, JMX just lets you view the data directly from the Java App. So even if you don't use JMX hooks to get app specific data you could at the very least get basic hardware resources and monitor it that way.

Just remember that when you run into scaling issues you should take everything into consideration. Not just server spec, but also workflows, network flows, firewalls, QoS, and anything else that could possibly impact performance. The great thing about getting tools like this is that it isn't just about Tomcat either. You can collect metrics on OS, MySQL, other apps/services, and so forth. I could blindly tell you to double your $JAVA_OPTS memory but that may not fix the problem if the problem is actually something else. Instead you are just temporarily filling a pot hole in the road, instead of fixing the actual road.

It can be a lot of work and a full time job (hence why Orgs hire Site Reliability Engineers) but the benefits are very great for having systems like this in place and gathering metrics for you.