Tomcat's Launch Daemon question

[donmontalvo]

We're having to restart Tomcat every so often, so I'm wondering if there is a problem with the Tomcat launch daemon?

Does the below match anyone else's Tomcat launch daemon? If not, care to post yours? :)

We're running a 1st generation Intel Xserve (you know the one that's supposed to be 64 bit but isn't <g>)...soon to be replaced with a new(ish) Xserve that is really 64 bit (with eventual migration off this cr@ppy Apple hardware and on to proper virtualized Windows Server environment). Wondering if the 64/32 fiasco (thanks Apple!) is causing this endless Java/Tomcat issue (seen it in other environments that ran 1st generation Xserve as well).

Thanks,
Don

/Library/LaunchDaemons/com.jamfsoftware.tomcat.plist

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Disabled</key>
    <false/>
    <key>KeepAlive</key>
    <true/>
    <key>Label</key>
    <string>com.jamfsoftware.tomcat</string>
    <key>OnDemand</key>
    <false/>
    <key>ProgramArguments</key>
    <array>
        <string>/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bin/java</string>
        <string>-Xms256m</string>
        <string>-Xmx3072m</string>
        <string>-XX:PermSize=64m</string>
        <string>-XX:MaxPermSize=128m</string>
        <string>-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager</string>
        <string>-Djava.util.logging.config.file=/Library/JSS/Tomcat/conf/logging.properties</string>
        <string>-Djava.awt.headless=true</string>
        <string>-Djava.endorsed.dirs=/Library/JSS/Tomcat/common/endorsed</string>
        <string>-classpath</string>
        <string>/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/tools.jar:/Library/JSS/Tomcat/bin/bootstrap.jar:/Library/JSS/Tomcat/bin/tomcat-juli.jar</string>
        <string>-Dcatalina.base=/Library/JSS/Tomcat</string>
        <string>-Dcatalina.home=/Library/JSS/Tomcat</string>
        <string>-Djava.io.tmpdir=/Library/JSS/Tomcat/temp</string>
        <string>org.apache.catalina.startup.Bootstrap</string>
        <string>start</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>ServiceIPC</key>
    <false/>
    <key>UserName</key>
    <string>_appserver</string>
</dict>
</plist>

[hkim]

Ever since moving to a ton more RAM, I haven't seen issues with Tomcat needing to be restarted. I have 16GB dedicated just to Tomcat.

[donmontalvo]

@hkim Yep, had same experience at previous job, once we migrated to real 64bit Xserve, we pumped up the RAM for Tomcat and all the issues went away. The first Intel Xserve sucked (not that I'm an Xserve fan but I digress).

PS, mind posting contents of your Tomcat launch agent so we can compare?

Don

[spowell01]

anything I can post from my 2008 R2 instance Don? sounds like I've got the same issue as your having with apple hardware

[donmontalvo]

@spowell01 Thanks, we should be OK, since we're moving to Windows Server vm soon (infrastructure expansion), right now we're moving a dying 1st generation Xserve with 12 GB of memory to a last generation Xserve with 36 GB of memory. Hoping moving from a not-quite-64-bit legacy box to a better one will help tame Tomcat/Java.

[donmontalvo]

So today I took a closer look at the launchd daemon and noticed the file it is looking for is missing:

/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bin/java -Xms256m -Xmx8192m -XX:PermSize=64m -XX:MaxPermSize=128m -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/Library/JSS/Tomcat/conf/logging.properties -Djava.awt.headless=true -Djava.endorsed.dirs=/Library/JSS/Tomcat/common/endorsed -classpath /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/tools.jar:/Library/JSS/Tomcat/bin/bootstrap.jar:/Library/JSS/Tomcat/bin/tomcat-juli.jar -Dcatalina.base=/Library/JSS/Tomcat -Dcatalina.home=/Library/JSS/Tomcat -Djava.io.tmpdir=/Library/JSS/Tomcat/temp org.apache.catalina.startup.Bootstrap start

**$ ls -l /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bin/java**
-rwxr-xr-x  1 root  wheel  100768 Mar  8 08:20 /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bin/java
**$ ls -l /Library/JSS/Tomcat/conf/logging.properties**
-rw-r--r--  1 _appserver  _appserveradm  3352 Sep  2  2012 /Library/JSS/Tomcat/conf/logging.properties
**$ ls -l /Library/JSS/Tomcat/common/endorsed**
ls: /Library/JSS/Tomcat/common/endorsed: No such file or directory
**$ ls -l /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/tools.jar**
ls: /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/tools.jar: No such file or directory
**$ ls -l /Library/JSS/Tomcat**
total 200
-rwxr-xr-x   1 _appserver  _appserveradm  57862 Sep  2  2012 LICENSE
-rwxr-xr-x   1 _appserver  _appserveradm   1228 Sep  2  2012 NOTICE
-rwxr-xr-x   1 _appserver  _appserveradm   9054 Sep  2  2012 RELEASE-NOTES
-rw-r--r--   1 _appserver  _appserveradm  16011 Sep  2  2012 RUNNING.txt
-rw-r--r--   1 _appserver  _appserveradm   4167 Mar 11 15:14 TomcatSSLKeystore
drwxr-xr-x  26 _appserver  _appserveradm    884 Mar 11 14:57 bin
drwxr-xr-x  10 _appserver  _appserveradm    340 Mar 11 14:57 conf
drwxr-xr-x  21 _appserver  _appserveradm    714 Mar 11 14:57 lib
drwxr-xr-x  38 _appserver  _appserveradm   1292 Mar 14 08:34 logs
drwxr-xr-x   2 _appserver  _appserveradm     68 Mar 13 10:20 temp
drwxr-xr-x   4 _appserver  _appserveradm    136 Mar 11 14:57 webapps
drwxr-xr-x   3 _appserver  _appserveradm    102 Mar 11 14:57 work
**$ **

Notice that these paths do not exist, but the launchd daemon call on them:

ls: /Library/JSS/Tomcat/common/endorsed: No such file or directory

and

ls: /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/tools.jar: No such file or directory

Could it be that the JSS Installer is not setting up the correct launchd daemon to keep Tomcat running?

If this turns out to be a bug, do I get a JSS Bacon t-shirt? :D

Don

[acdesigntech]

Hey Don,

Would the 2007 Xserve be the one you are referring to as legacy? We're having the exact same issue and are running on a 2007 Xserve, soon(ha!) to be migrated to a Win 2008R2 VM, but not soon enough.

Tomcat just crashes every so often. Seems to be when there are a lot of check ins at once, but there always were a lot of check-ins at once and this didnt start until we migrated to 8.62 in late Dec 2012. Tomcat will eventually recover (after 20 or so minutes - real fun when you're in the middle of a refresh :/), but I find that restarting it manually kicks it in the pants- that is if it hasn't crashed the server first. Also, killing the runaway java process consuming over 5GB of real mem and 200+ threads wakes it back up too. I recently increased tomcats available ram to 8GB, and that's seemed to help but it still crashes about once or twice a week. Hasn't happened since Monday of this week.

I'd be happy to post my launchd file, but I'm stuck in SCCM 2012 training all week, and away from my management consoles :( It's actually pretty painful, and I'm drowning in emails from my backups.

[SeanA]

@Don... Interesting observation that bears further investigation!

My current speculation is if the correct launch daemon is not being set, then I am wondering how Tomcat works at all, or that it would be failing more than the intermittent period that you alluded to earlier?

[Chris_Hafner]

Just an update in case this happens to anyone else. We just finished up a support ticket with JAMF regarding very similar issues. It turns out that we were maxing our threads for the JSS web application and had to adjust them up from 150 (151 really) to 300 (301). A quick change of two XML files, an adjustment in the JSS database utility and a tomcat restart.

P.S. Thanks Mitch! I've always loved JAMF support and you've kept our high regard!

[acdesigntech]

Chris, that actually sounds like the number of MySQL database connections, not the Tomcat threads. Jamf support had me increase the connection limit to those exact numbers from those exact numbers and increase the number of threads Tomcat can create to 600 from 150.

We landed on a happy medium of 6GB RAM for tomcat. I'm still getting some weird feedback at about 9:10-ish pm at night in the form of OS upgrades that run overnight missing 9 or 10 steps in the policy claiming that "the casper share could not be found."

I've tracked it down to possibly the 9pm policy we have to submit inventory. So i've temporarily disabled that to see if we have any improvement tonight with about 15 clients undergoing an OS upgrade tonight.