This has been going on for almost 2 months. So here is the background of my problem. We are a 1-1 school with macbooks and iPads. We have a little over 2100 computers and 1200 mobile devices. Back in September when the new iOS released we upgraded from 9.96 to 9.101 after the upgrade we added about 140 ipads via DEP enrollment.
Since the upgrade we started seeing some issues with our server. At the time we were running our MySQL, Tomcat, and JSS on a mac mini. Got a support ticket open and seen that we have been having troubles with smart group calculations. Mostly on the Mobile Devices side. We then decided to move out MySQL to a Windows Server and migrated the Database to use InnoDB. After that we were still having troubles so I cleaned up and deleted unnecessary smart groups and fixed any Nested smart groups to not be nested anymore. Still no fix. We then found that the pending install app commands were in the 20,000. Cleared all those and it seem to be working better for a bit but went back to the same problems of crashing. The commands stayed at a low count though. At this point when it crashes it really is just locking up the tomcat server. Tomcat is still running but you just can't get into it via self service or the web portal. If you restart the tomcat service it comes back up untill you try editing something and then it crashes again. Around here is when we also started seeing after a tomcat restart it would get stuck loading at "Altering table mobile_devices_denormalized..." and you would have to restart the MySQL service to get it back up.
After that we found that there was an app that was stuck in pending for a group of about 170 ipads that were added right before the upgrade in September. Not ever iPad had them but most of them did and the app wasn't scoped to those iPads and it wasn't even set to auto install. So even though we just cleared the pending install application commands we target that app ID and cleared all pending commands for it. Cleared them all out but tomcat was locking up and we were seeing more of the mobile_devices_denormalized table locking. I've been sending Java thread dumps and MySQL logs and they say they are still seeing to many mobile device group smart group calculation. Was looking through the group of 140 ipads that I added after the update and i've notice that random ones would have apps that are not scoped to them in the pending install list. I even found two that had around 20 apps pending that were not scoped to the iPad. We are going to try and removed all pending app installs for that group and see what happens.
If anyone has some ideas on what might be going on any and all input would be helpful. I'll keep this updated as I go. Thanks.
