9.62 HIgh CPU and Memory Usage

brushj
New Contributor III

We upgraded to 9.62 to fix our iPad app installation issues when the App store is locked down, that seems to be fixed now.

Everything was fine yesterday, but today we are seeing 100% CPU usage and double the amount of memory usage as we normally do. Has anyone else seen issues with High CPU usage after upgrading to 9.62?

EDIT: According to Task Manager it looks to be mysql and tomcat that are killing everything.

19 REPLIES 19

GabeShack
Valued Contributor III

Out of curiosity, what version of mysql are you using?

Gabe Shackney
Princeton Public Schools

Gabe Shackney
Princeton Public Schools

davidacland
Honored Contributor II
Honored Contributor II

Although normally associated with an increase in devices, I've seen this behaviour when the tomcat permgen settings are too low. On one of our systems the CPU usage would ramp up to 100% and then hang there. As soon as we allocated more memory all was well again.

brushj
New Contributor III

We got in touch with support and looks like during the upgrade we lost our wireless profile configuration. This prompted it to attempt to push a profile removal from all of our computer clients, over 11k machines, to remove the profile and it was getting caught.

We are using mysql 5.6

We did allocate more memory to tomcat but it didn't help us out. I just rebooted the server because the web portal went down and it seems to be back to high CPU usage again. Time to contact support again.

andyinindy
Contributor II

@brushj

What platform/OS are you using for your JSS?

(please say windows, please say windows)

-Andy

brushj
New Contributor III

@andyinindy yes, windows server 2008 r2

GabeShack
Valued Contributor III

So I have a working theory about mysql 5.6....its awful and breaks with newer versions of Tomcat. At least thats what I keep coming to while looking at the discussions on here. We have had nothing but issues since upgrading the version of mysql we were using.

We have not had Tomcat go down, but we are also using a new MacPro 6 core with 64gb of ram which helps a little I think...
Gabe Shackney
Princeton Public Schools

Gabe Shackney
Princeton Public Schools

brushj
New Contributor III

I haven't upgraded in a while because of issues we had in the past with stuff like this and nothing was really "broken" besides the app store apps not pushing out correctly with the app store locked up.

This fixes that but seems to have broken something else. I am just trying to find out if anyone else is running into any issues.

cdenesha
Valued Contributor II

I've been having trouble with high Java and MySQL since before Thanksgiving, JSS 9.61 on a Mac Mini w/ 4 GB RAM, managing 1000 iPads.

It became constant so I brought it down. Have upgraded from Mountain Lion to Mavericks, Java 1.6 to 1.7, and am on the highest version of mysql 5.5. Still not much better, so I changed my log retention from 1 year to 6 months, which helped. Still having trouble for 30-60 seconds every 10 minutes. Support tweaked my memory and max connection settings but we didn't touch Tomcat permgen.. Am now on 9.62 which doesn't fix this.

Full debug by changing log4j.properties indicates a recalculation of user groups. Since I've started heavily using VPP and Managed Distribution (and Self Service Mobile) I'm working a lot with user objects. My smart groups take advantage of the fact that our staff and student email addresses are different, so the criteria is LIKE '@student'. I'm going to try changing to a EA, although I'm sad to have to click several hundred times.

So while different from your environment, I wanted to share that I've been seeing high cpu usage as well.

Thanks,

chris

cdenesha
Valued Contributor II

@brushj are you using Self Service Mobile or web clip?

Support opened a defect because the App Store doesn't stay enabled long enough for the app to install if the server is under high load. Only happens with Mobile, not the web clip.

brushj
New Contributor III

@cdenesha Self Service Mobile. That problem is solved since the update. It's now just the issue of the high CPU and RAM usage.

Chris_Hafner
Valued Contributor II

Interesting. Anyone else seeing this? I'm about to pull the trigger on 9.62 so... if not I'll just in with both feet shortly and share.

P.S. If you use the supported JAMF installer it will reset any changes you've made to tomcat and MySQL regarding maxThreads. FYI. I know that i should use the root folder method but I still tend to use the JAMF installer (OS X for me) and re-modify my maxThreads and Connections each time I upgrade.

bloree
New Contributor II

I just upgraded our JSS to 9.62 last night. This morning we experienced high CPU utilization and high disk IO. We checked the MySQL logs and saw references to the downloadable_file_chunk_data table. Our JDS replication has been a bit troublesome. Yesterday we had some Adobe installers fail to replicate from our master CasperShare to the JDS.

Truncating the downloadable_file_chunk_data, downloadable_file_chunk_metadata, and downloadable_files tables resolved the issue. Something in the update did not like the corrupt Adobe files that were stuck in the database.

Chris_Hafner
Valued Contributor II

Yea. I'm not up to 9.62 yet (still 9.61) however, any time there is anything corrupted like that you're going to have issues after an upgrade if not before! I had this happen early in the 9.x release cycle.

Did removing these packages as well as the MySQL record clear up your issues?

bloree
New Contributor II

Yes. We have not had any alerts from the database server since yesterday morning.

I replicated this morning and discovered my folly. When we updated from Tomcat 6 to Tomcat 7 about a month ago, our symbolic links pointing the /var/share/tomcat/temp folder to a larger volume were wiped out. I fixed that this morning and we are able to replicate large files again. Sorry for the tangent. But we have definitely not had any issues since truncating those tables. **knocks on wood

Your mileage may vary. I suppose it's possible that if there is corruption in the database in a different table, you could see similar behavior. If you are experiencing this issue, I would recommend digging into the MySQL logs.

Chris_Hafner
Valued Contributor II

@Loree nice catch on Tomcat!

GabeShack
Valued Contributor III

So we are seeing swings of Java using 250% to 1000% of our cpu at any given moment. mysql is not spiking as much but sometimes in the 25%-100% range.

To solve a problem with our auto backups not running we reverted from mysql 5.6 to the most current 5.5 which fixed that issue, but did not resolve the java/mysql cpu usage issue.

We are not seeing any issues regarding speed or access times so I'm not too concerned(again I think the MacPro helps a lot), but just wondering what the cause could be.

Gabe Shackney
Princeton Public Schools

Gabe Shackney
Princeton Public Schools

brushj
New Contributor III

We have a support call with our rep in about 30 minutes to move our database to another server.

We are still seeing high CPU usage, we went from a dual core VM to 6 cores and it seems to be manageable now.

We also have nearly 12,000 computers and 2,000 iPads, so we probably need to separate the two.

I still feel like something isn't right with the database, but hopefully this helps.

cdenesha
Valued Contributor II

@Loree
Location of mysql logs? On Mac I found /usr/local/mysql/data/host_name.err are there more?

In my case 9.62 didn't help. I created an EA and marked each user as 'Student' or 'Staff', and changed the criteria away from a substring. No improvement. So I threww hardware at it, and I now have a 2012 Mac Mini w/ 8 GB RAM instead of a 2010 with 4 GB. Much better, although I can still see small humps in the Activity Monitor every 10 minutes or so.

chris

powellbc
Contributor II

We have been bitten with this in the past days (Windows Server 2012 R2 JSS). After running 9.62 without any problems for a month Tomcat is now chewing up CPU and memory. Classes started here this week so I suspect the increase in users here is a contributing factor.

Does anyone know if 9.63 helps with this?