Tomcat running hot

[Lincoln]

Hi all,

I have contacted support on this but as it is the middle of the night there and the middle of the day here (Melbourne, AU) I thought I'd ask here in case anyone has struck this before.

We are running 8.52 on a Win2008R2 VM. It has been running fine up until sometime yesterday when we noticed that the JSS was responding very slowly. A look at Task Manager on the VM shows that the Tomcat process is ramping up to the point where it is using 99% of CPU and it's memory allocation goes through the roof. Restarting the Tomcat service fixes things for a bit but then it starts ramping up again.

The VM has been fully patched but this has not helped (or made things worse). We think that there is something else causing the Tomcat service to behave this way, i.e. we don't think that the problem is Tomcat itself but we really don't know.

Any suggestions welcome. At present I am having to sit here with a remote desktop connection to the VM open and restart the service every so often, which is getting a bit old... 80)

Thanks in advance.

Lincoln

[McAdams]

Hi Lincoln,

We just went through something like that. In our case we had clients trying to submit backlogs to the jss. The logs were too big for mysql to handle and that caused Tomcat to hang.

Support had us create an extension attribute (Settings -> Inventory Options -> Inventory Collection Preferences) that would report on the size of logs. However we found that the clients could not report back if they had these logs.

The Extension Attribute looks like this:

if [ -d "/Library/Application Support/JAMF/Logs/" ]; then
size=$(du -sh "/Library/Application Support/JAMF/logs/" | awk '{print $1}');
echo "<result>$size</result>";
fi

We put that into a script and ran it with Casper Remote. Clients that had backlogs would fail with a message that they were uploading old logs. We then went and manually removed logs from those clients.

sudo rm -r /Library/Application Support/JAMF/logs/*

So far so good.

[haydio]

Hi,

I'm having the same issues on my OS X Server hosting our JSS. I'm going to try your McAdams, hopefully that resolves our issues.

Haydn

[marcusransom]

we are having similar issues here as well. TomCat is freezing sometimes a couple of times a day. This is manifesting itself in a strange situation where users (offsite or onsite) are unable to log off or on to their machines as they get stuck on checking for policies unless I restart tomcat. We have it set to automatically restart twice a day now which is getting a little ridiculous.

we are still running 8.31 on Windows Server 2008r2 and I was hoping the update to 8.5.2 might flush out some of the issues.

[Lincoln]

Hi Marcus,

Your issue sounds a little different to ours but just as frustrating.

I am still working on this with JAMF Support. At this stage we have not gotten to the bottom of it. Under their direction I have tried a few things including:

• Increasing Tomcat's memory allowance.
• Uninstallaing Java, Tomcat and JSS.
• Flushing logs.
• Optimising database tables.

These last two were after noticing during a database backup that one particular table seemed exceptionally large. This lead to the thought that our logs etc may have blown out and be causing the problem.

Still no joy. The whole thing is taking longer because of the 13 hour time difference meaning that I'm asleep when they are replying and they're asleep when I am trying the latest thing.

We've also been trying a few things ourselves but again to no avail. I'm just about ready to get up at 1am to try speeding the process along.

Will post again when a solution is found.

Regards

Lincoln

[Lincoln]

OK so I promised an update on this. After much head scratching and early mornings and working from home in the evenings testing various things including JSS installation on a new Windows VM and on our Xserve I was able to prove that the problem definitely originated from our clients.

It seems I had managed to inadvertently managed to create a 'perfect storm' where I had a number of policies using the 'any' trigger and frequency of 'ongoing.' These policies were applied to smart groups so that membership of the group would cause the policy to execute, and execution would cause the machine to leave the smart group. This seems to have been the major factor in the issue though I can't that this was all there was to it. In the end to get the machines to behave I had to either re-image or remove framework. This settled things down so that I could actually work with the JSS. Since school finishes for the winter break tomorrow afternoon I am happy with this for now. I will be re-imaging all classroom machines over the holidays so crisis averted.

A new problem which has now arisen is that machines do not check in after re-imaging. The settings in the configuration are correct yet the machines act like they were imaged with a config not set to ensure management.

Lincoln