Server Performance - Java Memory

[#1]

I'm looking into options to get better performance from my JSS. Its a
dual cpu quad core xserve with 20GB of memory.

Every morning we have about 1500 computers startup at the same time which
puts a heavy load of both Java and MySQL. The Java process runs at 300%
or more CPU. It uses 2GB of System Memory and over 10GB of virtual
memory. Tomcat is configured for 64 bit and set to the max of 8GB memory.
Any ideas?

I've tried to work with Jamf on tweaking mysql tuning, but they aren't
very supportive of this. I don't really want to make changes without
their backing.

Thanks,
Chad

[dustydorey]

We have just shy of 5k computers reporting in and noticed that our
biggest issue with this was when they were running their Scheduled Task
to report in. We divided the district up into 5 sections and set them
to only run that scheduled task once a week. Though not ideal for
reporting it did the trick for us. But the thing I found interesting is
that you said it was only using 2GB of System memory and then defaulting
over to virtual. Ours when set to 64 bit mode takes up to 8 GB of
System Memory if we give it that. I wonder why yours is not.

-Dusty-

Dustin Dorey

Technology Support Cluster Specialist

Independent School District 196

Rosemount-Apple Valley-Eagan Public Schools

dustin.dorey at district196.org

651|423|7971

[Bukira]

What's ur bandwidth? How many gbits?

[tlarkin]

Wow you have 20gigs of RAM?

What is your max packet size for MySQL packets? How big is your MySQL database?

?xml version="1.0" encoding="ISO-8859-1"?>

[#5]

I'm only using 1 NIC so just 1 Gigabit a bandwidth. Bandwidth does not
appear to be a bottleneck here.

All my clients coming from the schools have gigabit bandwidth to the JSS.

[Bukira]

20gbs?

I have 3 servers all with 32gb's of ram

Not that I want to glote :-)

[Bukira]

And what bandwith are u getting? I have 2gb bonded and that's with clients on 100mb so personally I'd say if need more bandwidth

[#8]

Max allowed packet size is set to 1 megabyte.

Database is 879M

[John_Wetter]

I guess starting at basics, what are your performance issues you’re trying to tweak?

When the computers start up, are to sending packages at started such as putting down a fresh environment package every time, or is it just that they are all doing the manage framework items at startup?

We have about 4200 clients on our JSS, and it has 8GB RAM, and I have Tomcat set in 64-bit mode and at 4GB RAM and right now, it is using just over 800MB real Memory and about 6.85 GB virtual memory. I’m not concerned about the virtual memory usage as there isn’t a lot of disk activity, so it’s not thrashing the disk.

Do you also have Netboot, AFP, or any other services on the Server?

We have our Inventory run randomly once per week per computer and have a 30 minute check-in policy to look for jobs. Our server pretty much sits at about 20-60% utilization, so is quite comfortable. We spike up into the 150% plus sometimes, but it’s fairly rare. This is on a server with 2 quad-core processors and 8GB RAM.

Thanks,
John

--
John Wetter
Technology Support Administrator
Educational Technology, Media & Information Services
Hopkins Public Schools
952-988-5373

[tlarkin]

1MB? Yeah no wonder it is slow. You need to open that puppy up. I allow 64megs per a packet on my JSS and my database is about 6 gigs in size, maybe 7gigs in size. My server is only a JSS though, my ODM is a different box.

-Tom

?xml version="1.0" encoding="ISO-8859-1"?>

[#11]

I'm having performance issues too. 8 GB RAM, 7GB for tomcat in 64 bit mode. Packet size is 16 MB. Database is 1.2 GB
dedicated JSS, 1GB connection. Intel 8-core

Do you think increasing the packet size to 64 MB will really help? I've been working with JAMF support and we've been looking at decreasing the number of Smart Groups. It seems to help a little, but I haven't gotten rid of that many. We've also scaled back Policies, seems to help some, but not as much as I hoped for.

Nathaniel Lindley

++++++++++
Information Technology
Saint Paul Public Schools
Saint Paul, Minnesota
nathaniel.lindley at spps.org
phone: 651-248-6861

[#12]

Thanks Tom. I changed my max packet to 64M and my key_buffer_size from
256M to 4096. We'll see if that helps. Mysql is already running at 800M
system memory which is more than it was allowed to previously use.

My server is also a JSS only. It doesn't do AFP or OD.

[dustydorey]

Totally forgot about the max packet size issue, that was the first thing
JAMF told me to do when we were having performance issues. Well that
and then I upped it's RAM to 32GB and split up the reporting. BTW we
did reset the reporting back to the default for the scheduled task and
since upgrading our ram and upping the max packet we've been ok.
Good Call Tom I totally forgot we did that!

-Dusty-

Dustin Dorey

Technology Support Cluster Specialist

Independent School District 196

Rosemount-Apple Valley-Eagan Public Schools

dustin.dorey at district196.org

651|423|7971

[tlarkin]

I can only tell you what my experience was when I had database issues too. My database at the time was around 17gigs in size. What had happened was, I had Casper Policy in the JSS that ran a command using the find binary. That find binary basically created a huge log file. Those log files got so big so quick that MySQL could not delete them because they were actually larger than the max allowed packet size. So, my database which is suppose to be around 5 to 6 gigs was triple in size.

Once I increase the max packet size, I was able to clear out all those log files and flush them out reducing the size of my database and improving performance. The max packet size feature allows MySQL to use a specific amount of RAM when running a single query through the database. So, if you are trying to search for a table that is larger than 16megs, it will just time out.

My server is about a year old, dual quad-core Xeons with 8gigs of RAM with 5gigs going to TomCat in 64bit mode. Once I increased the max packet size for the MySQL database I noticed a rather large increase in performance. I think by default it only allows for 16megs, and I bumped mine up to 64megs and then down to 32megs once I thought it was performing well enough for me to bump it down a bit.

You cannot configure this setting in the JSS setup utility. I downloaded the GUI tools from the MySQL project page and in the advanced networking set up modified my database. You can also do it from the mysql command line.

I can't promise it will fix your problems, but it sure as heck helped with my performance issues. I mean some of those policy logs were so large I could not even delete them, because the max packet size was smaller than the actual database table in question.

Hope that helps

-Tom

---

Thomas Larkin
TIS Department
KCKPS USD500
tlarki at kckps.org
blackberry: 913-449-7589
office: 913-627-0351