9.62 HIgh CPU and Memory Usage

[brushj]

We upgraded to 9.62 to fix our iPad app installation issues when the App store is locked down, that seems to be fixed now.

Everything was fine yesterday, but today we are seeing 100% CPU usage and double the amount of memory usage as we normally do. Has anyone else seen issues with High CPU usage after upgrading to 9.62?

EDIT: According to Task Manager it looks to be mysql and tomcat that are killing everything.

[GabeShack]

Out of curiosity, what version of mysql are you using?

Gabe Shackney
Princeton Public Schools

[davidacland]

Although normally associated with an increase in devices, I've seen this behaviour when the tomcat permgen settings are too low. On one of our systems the CPU usage would ramp up to 100% and then hang there. As soon as we allocated more memory all was well again.

[brushj]

We got in touch with support and looks like during the upgrade we lost our wireless profile configuration. This prompted it to attempt to push a profile removal from all of our computer clients, over 11k machines, to remove the profile and it was getting caught.

We are using mysql 5.6

We did allocate more memory to tomcat but it didn't help us out. I just rebooted the server because the web portal went down and it seems to be back to high CPU usage again. Time to contact support again.

[andyinindy]

@brushj

What platform/OS are you using for your JSS?

(please say windows, please say windows)

-Andy

[brushj]

@andyinindy yes, windows server 2008 r2

[GabeShack]

So I have a working theory about mysql 5.6....its awful and breaks with newer versions of Tomcat. At least thats what I keep coming to while looking at the discussions on here. We have had nothing but issues since upgrading the version of mysql we were using.

We have not had Tomcat go down, but we are also using a new MacPro 6 core with 64gb of ram which helps a little I think...
Gabe Shackney
Princeton Public Schools

[brushj]

I haven't upgraded in a while because of issues we had in the past with stuff like this and nothing was really "broken" besides the app store apps not pushing out correctly with the app store locked up.

This fixes that but seems to have broken something else. I am just trying to find out if anyone else is running into any issues.

[cdenesha]

I've been having trouble with high Java and MySQL since before Thanksgiving, JSS 9.61 on a Mac Mini w/ 4 GB RAM, managing 1000 iPads.

It became constant so I brought it down. Have upgraded from Mountain Lion to Mavericks, Java 1.6 to 1.7, and am on the highest version of mysql 5.5. Still not much better, so I changed my log retention from 1 year to 6 months, which helped. Still having trouble for 30-60 seconds every 10 minutes. Support tweaked my memory and max connection settings but we didn't touch Tomcat permgen.. Am now on 9.62 which doesn't fix this.

Full debug by changing log4j.properties indicates a recalculation of user groups. Since I've started heavily using VPP and Managed Distribution (and Self Service Mobile) I'm working a lot with user objects. My smart groups take advantage of the fact that our staff and student email addresses are different, so the criteria is LIKE '@student'. I'm going to try changing to a EA, although I'm sad to have to click several hundred times.

So while different from your environment, I wanted to share that I've been seeing high cpu usage as well.

Thanks,

chris

[cdenesha]

@brushj are you using Self Service Mobile or web clip?

Support opened a defect because the App Store doesn't stay enabled long enough for the app to install if the server is under high load. Only happens with Mobile, not the web clip.

[brushj]

@cdenesha Self Service Mobile. That problem is solved since the update. It's now just the issue of the high CPU and RAM usage.

[Chris_Hafner]

Interesting. Anyone else seeing this? I'm about to pull the trigger on 9.62 so... if not I'll just in with both feet shortly and share.

P.S. If you use the supported JAMF installer it will reset any changes you've made to tomcat and MySQL regarding maxThreads. FYI. I know that i should use the root folder method but I still tend to use the JAMF installer (OS X for me) and re-modify my maxThreads and Connections each time I upgrade.

[bloree]

I just upgraded our JSS to 9.62 last night. This morning we experienced high CPU utilization and high disk IO. We checked the MySQL logs and saw references to the downloadable_file_chunk_data table. Our JDS replication has been a bit troublesome. Yesterday we had some Adobe installers fail to replicate from our master CasperShare to the JDS.

Truncating the downloadable_file_chunk_data, downloadable_file_chunk_metadata, and downloadable_files tables resolved the issue. Something in the update did not like the corrupt Adobe files that were stuck in the database.

[Chris_Hafner]

Yea. I'm not up to 9.62 yet (still 9.61) however, any time there is anything corrupted like that you're going to have issues after an upgrade if not before! I had this happen early in the 9.x release cycle.

Did removing these packages as well as the MySQL record clear up your issues?

[bloree]

Yes. We have not had any alerts from the database server since yesterday morning.

I replicated this morning and discovered my folly. When we updated from Tomcat 6 to Tomcat 7 about a month ago, our symbolic links pointing the /var/share/tomcat/temp folder to a larger volume were wiped out. I fixed that this morning and we are able to replicate large files again. Sorry for the tangent. But we have definitely not had any issues since truncating those tables. **knocks on wood

Your mileage may vary. I suppose it's possible that if there is corruption in the database in a different table, you could see similar behavior. If you are experiencing this issue, I would recommend digging into the MySQL logs.

[Chris_Hafner]

@Loree nice catch on Tomcat!

[GabeShack]

So we are seeing swings of Java using 250% to 1000% of our cpu at any given moment. mysql is not spiking as much but sometimes in the 25%-100% range.

To solve a problem with our auto backups not running we reverted from mysql 5.6 to the most current 5.5 which fixed that issue, but did not resolve the java/mysql cpu usage issue.

We are not seeing any issues regarding speed or access times so I'm not too concerned(again I think the MacPro helps a lot), but just wondering what the cause could be.

Gabe Shackney
Princeton Public Schools

[brushj]

We have a support call with our rep in about 30 minutes to move our database to another server.

We are still seeing high CPU usage, we went from a dual core VM to 6 cores and it seems to be manageable now.

We also have nearly 12,000 computers and 2,000 iPads, so we probably need to separate the two.

I still feel like something isn't right with the database, but hopefully this helps.

[cdenesha]

@Loree
Location of mysql logs? On Mac I found /usr/local/mysql/data/host_name.err are there more?

In my case 9.62 didn't help. I created an EA and marked each user as 'Student' or 'Staff', and changed the criteria away from a substring. No improvement. So I threww hardware at it, and I now have a 2012 Mac Mini w/ 8 GB RAM instead of a 2010 with 4 GB. Much better, although I can still see small humps in the Activity Monitor every 10 minutes or so.

chris

[powellbc]

We have been bitten with this in the past days (Windows Server 2012 R2 JSS). After running 9.62 without any problems for a month Tomcat is now chewing up CPU and memory. Classes started here this week so I suspect the increase in users here is a contributing factor.

Does anyone know if 9.63 helps with this?