Jamf Crashing. Smart Groups Calculating to often.

[mcmeekin91]

This has been going on for almost 2 months. So here is the background of my problem. We are a 1-1 school with macbooks and iPads. We have a little over 2100 computers and 1200 mobile devices. Back in September when the new iOS released we upgraded from 9.96 to 9.101 after the upgrade we added about 140 ipads via DEP enrollment.

Since the upgrade we started seeing some issues with our server. At the time we were running our MySQL, Tomcat, and JSS on a mac mini. Got a support ticket open and seen that we have been having troubles with smart group calculations. Mostly on the Mobile Devices side. We then decided to move out MySQL to a Windows Server and migrated the Database to use InnoDB. After that we were still having troubles so I cleaned up and deleted unnecessary smart groups and fixed any Nested smart groups to not be nested anymore. Still no fix. We then found that the pending install app commands were in the 20,000. Cleared all those and it seem to be working better for a bit but went back to the same problems of crashing. The commands stayed at a low count though. At this point when it crashes it really is just locking up the tomcat server. Tomcat is still running but you just can't get into it via self service or the web portal. If you restart the tomcat service it comes back up untill you try editing something and then it crashes again. Around here is when we also started seeing after a tomcat restart it would get stuck loading at "Altering table mobile_devices_denormalized..." and you would have to restart the MySQL service to get it back up.

After that we found that there was an app that was stuck in pending for a group of about 170 ipads that were added right before the upgrade in September. Not ever iPad had them but most of them did and the app wasn't scoped to those iPads and it wasn't even set to auto install. So even though we just cleared the pending install application commands we target that app ID and cleared all pending commands for it. Cleared them all out but tomcat was locking up and we were seeing more of the mobile_devices_denormalized table locking. I've been sending Java thread dumps and MySQL logs and they say they are still seeing to many mobile device group smart group calculation. Was looking through the group of 140 ipads that I added after the update and i've notice that random ones would have apps that are not scoped to them in the pending install list. I even found two that had around 20 apps pending that were not scoped to the iPad. We are going to try and removed all pending app installs for that group and see what happens.

If anyone has some ideas on what might be going on any and all input would be helpful. I'll keep this updated as I go. Thanks.

[millersc]

We also did the same upgrade Monday, seeing all sorts of ugly. Still waiting for support.

[FastGM3]

We had similar issues with our Tomcat server. With over 6000 macOS and 5500 iOS devices. I could restart the server and watch Tomcat eat up all the memory until it was no longer responding, then crash the server. Windows 2016 VM server I might add. It didn't matter how much we increased the RAM, Tomcat would use it all up and crash.

Long story short after many many support calls, we spun up another Tomcat VM server and clustered the JamF server. Now we have 2 Tomcat servers, and a MySQL server. We no longer have those Tomcat crashes.

JamF has great support, patience is a virtue with those logs but it does help them. Ask them about Clustering, they'll help you with the setup.

Also we had apps not auto-populating in the scope for iOS devices and found that if you deleted unused apps, it can cause apps not to populate. We had to use a workaround that required us to move unused apps into a category we named it "not in use". Not sure if you've deleted apps in Jamf but thought it worth mentioning.

Chuck

[bburdeaux]

What version of Java are you running? I believe one of the new JAMF releases included an update to Tomcat that doesn't play nicely with Java 7. We were having all sorts of optimization issues with our Tomcat that were massively improved by upgrading to Java 8 and enabling some of Java 8's garbage collection features.

As for clustering, the generally accepted guideline is 1 webapp per 5000 devices, so you may be fine with a single webapp, but every environment is different and it is certainly worth discussing with your JAMF buddy.

[mcmeekin91]

Update from support last night. We upgraded to Jamf 10 and we changed tomcat memory settings. Early on in the support ticket one of the technicians told me to turn up the Tomcat Memory to 8 gigs which is how much RAM is in the machine. Tomcat never maxed out the systems memory and the server was saying it was only using 7GB total out of everything when it was hit hard. We never though twice about it. Either way the newest person we had on support saw this and we turned it down to 4GB and it seem to have stopped the crashing. It's running slower when it hit hard but no crashing for now. We are going to order more memory for it and see how it goes. Thanks for the input everyone. @bburdeaux I'll be sure to check out to make sure we have an updated java on there just in case.

[mcmeekin91]

Update: Server is still crashing. Cleared out all the pending commands and still nothing. We are going to move tomcat to a windows machine and go from there.

[jared_f]

@mcmeekin91 Did you ever resolve this issue? Running into a similar issue.

[martenblank]

Similar issue here on Ubuntu 18. Tireing to have this problem over and over again.