How to manually purge JSS before restoring from known good backup?

[donmontalvo]

We're trying to purge JSS from an Xserve, so we can restore from a backup. When we try to uninstall JSS using the JSS Setup Utility, it sits there for (many) hours spinning away.

Is there a way to purge JSS without having to rely on the JSS Setup Utility? I looked for a JAMF KB article, without luck.

Thanks,
Don

[ernstcs]

My guess is your database is massive? How big?

[donmontalvo]

Hi Craig,

It's bigger than a bread box. :) Over 50G. Mostly due to our disabling purging of logs, but we're going to start purging once it's back up. Only need to keep a month or so of logs.

We know the db is corrupt beyond repair so we simply want to purge it. Since JSS Setup Utility isn't working, I tried stopping Tomcat and MySQL and doing a manual purge (mysql> drop database jamfsoftware) but that isn't working either.

JAMF asked me to run permissions repair, reboot and try JSS Setup Utility again. Will shout back with results.

Thanks,
Don

[tlarkin]

Change your max packet size to allow more ram to access mysql then you can delete it probably

[tlarkin]

Alternatively you can drop the database form the mysql command line and then clear out all the bin files as well.

[tlarkin]

Sounds like a job for a wipe and reload job. Only takes about an hour to wipe and reload a server, and if you got a good back up plan this should be something you can do

[donmontalvo]

Hi Tom,

Thanks for all your ideas, I've been running around so haven't been
able to respond. I'm interested in finding out what might have made
the database go kaboom, so we're going to hold off on hitting the
server with a BFH for the time being. :)

JAMF support has us running a few troubleshooting routines to see if
we can get the MySQL database to drop. I hope we can, then we'll just
need to restore from a good backup to be back on track.

Thanks,
Don

[ernstcs]

I too am having database issues because of size apparently. Jake's been
spending some quality time with me.

Had a policy that was enforcing management framework every hour, so that
beast was apparently adding some records. However with 7.3 stuff that just
happens all the time when the box talks back.

I also had plug-ins turned on. Turned that off so that should help too.
However, my database not nearly as large, around 3GB. We purge logs at three
months.

My issues started when I would delete a system and it would take 10-20
minutes for that to complete. Still working on that a bit...

I like gin. =)

Craig E

[tlarkin]

This has happened to me a few times since my environment is rather large. 8,000 clients for 1 JSS then add in all the policies I do and you got a HUGE database. I have many policies that run once per a day and are ongoing, that check for certain things and then manually trigger other policies. I have put in a feature request to have advanced logging abilities for policies, where on a per policy basis you can specify to auto flush the logs. Not all logs need to be flushed, but some do. I have had my database up near 50G in size before as well.

I also had inventory updating like crazy, which also caused huge logs. Over the past 3 years I have tweaked the JSS to allow for larger max packet sizes when using memory to query MySQL, added tons of RAM, given Tomcat 16G dedicated memory, tweaked policy and inventory settings, and now my database never gets over 10G in size. I also had Sam help me index a few things, which does create more bin files, but keeps the database easier to access.

I think if they add more modular logging to policy, where as if you can choose as an extra feature to always flush logs of a certain policy it would probably make my last database issues go away. Right now I have policies set to delete after 30 days because I do need some sort of history, however, not every policy needs logs.

If you do get larger and larger databases you may want to view the process list in MySQL and see what is timing out if anything.

[donmontalvo]

Ok, problem solved. Jake Mosey worked with us to fix JSS. It is back up and running again. Nice and snappy too!

Root cause analysis...database grew large because some inventory options were enabled that shouldn't have been (UNIX executables, Fonts and Plug-ins). Also, logs were being retained indefinitely. Combined, the database grew and grew...eventually KABOOM!

We made some adjustments, created an internal best-practices for settings (we'll tweak once we upgrade to latest JSS version).

Database is now 5G instead of 50G+. Did I mention it's snappy? :)

Craig, you were right when you hinted at the database size...let me know where to send the ether beer. :)

Don

[donmontalvo]

Hi Tom,

Thanks for all the valuable feedback.

Interesting, during our troubleshooting we reviewed the memory allocation for Tomcat. I always saw the 64bit checkbox but never tried it since the Xserve is 1st generation that doesn't support 64bit:

http://donmontalvo.com/jamf/64bit/Xserve_Late_2006_32bit.jpg <-- note "Architecture"
http://donmontalvo.com/jamf/64bit/Xserve_Early_2008_64bit.jpg
http://donmontalvo.com/jamf/64bit/Xserve_Early_2009_64bit.jpg

I checked the box and was able to get it to 8192 MB...and the setting is persistent:

Web Applications > Settings > Tomcat Settings > Maximum Memory: 8192 MB

I understand the value of increasing Tomcat memory...but not sure about the max packet size. It's set to 128 MB now. What's the benefit to increasing this? The Xserve has 16G of memory...

Don

[tlarkin]

I am no MySQL guru, but to my understanding the max packet size allows MySQL to use that amount of memory for any single query or string processed. So, in my instance I was having policies that logs were getting so big, that the manual maintenance settings could not flush them because I think the default was set to 16megs.

I am in a different environment. Academia needs way more management in my personal opinion than enterprises just because the nature of dealing with students being minors.

So, increasing it to 128M allows MySQL to use up to 128M to query MySQL. Since my database was getting too big, the packet size was smaller than the actual log files and thus, the logs would not flush. I assume this is because the blob that contained these logs was larger than 16megs at the given time. So, it would error out and just grow and grow in size.

If this is incorrect please correct me on this.

Thanks,

Tom