Backups are much faster when Tomcat isn't running

[lastdanstanding]

Hi JamfNation,

Generally our automated JSS backups take about three hours to run, from 3:00 am to about 6:00 am, and during that time the availability of the JSS is intermittent.

Last night we experimented with a couple of new automated jobs on the server to see what would happen. One unloads Tomcat just before the automated backups begin, at 2:50 am. Last night, with Tomcat disabled the backups completed in a speedy 49 minutes! Then of course we have a second automated job which reloads Tomcat about an hour later.

Our only hesitation is that because Tomcat was not running when the backup completed, we see the following in the backup log: Sending backup notification to <JSS-URL> Encountered an error while sending the backup notification to the JSS. Check your JSS URL Setting found in the Global Management section of your JSS. Connection refused

Is anyone else doing something like this? What are the ramifications of the JSS not receiving the backup notification? Is it just so the JSS can send email notifications on backup success or failure? Any other gotchas?

Thanks for taking a look!

-Dan

[donmontalvo]

If you have a spare server JAMF can set you up with MySQL replication so you can back up the replica and it won't matter if Tomcat is running.

[peineke]

I definitely second Don's reply. If you are having backups take that long you should look into the replicated MySQL server to offload the backup creation.

[hzimmerman]

Three hours does seem rather long to do a backup.

I would take a look at your server logs and table sizes to see if there is anything that catches your eye. I have seen the plugin table, for example, grow to many gigabytes inexplicably.

[lastdanstanding]

Hey cool, thanks for the replies.

Our backups are consistently only about 1.1GB, so I agree three hours is excessive.

We actually already have replication in place. We use it to run application usage queries, without bogging down the JSS. Perhaps we'll start using it for backups as well.

[bmak]

@lastdanstanding

Our DB backups were previously taking too long to backup roughly between 2-3 hours and the the size of the backup was similar to yours roughly 1GB or so.

Definitely do check the size of your table sand also how many rows in each table.
Our issue was that we our DB was about 32GB in size at one point and with help from Adam Sippl and Bram Cohen from JAMF Support we figured out that it was down to a 2 things

1. We had a few policies that were running at the every 15 min window (recurring check-in) on an "on-going" basis that had recon checked and were scoped to all workstations.

This was a big problem if you think about it that's all your workstations uploading their inventory 4 times per hour which then ends up being about 96. And this blew out the Applications Table for us. We audited all our policies so that we didn't have any of these types of policies anymore

1. The other issue was that we hadn't had any auto-log flush set up so our log files were also gigantic. So I had to manually flush the log files each night to bring it down in size before then settuping auto log flushing.

Fast forward to now, we now have our auto log flush set up before we our automated DB backups occur.
And this has worked really well for us.

The size of our DB went from 32GB down to now roughly 5.8GB
The time taken for DB backups used to be 2-3 hrs and now it roughly takes 30 mins
And the size of the backups are half of what they use to be which is roughly 580 MB

I'd also be interested in how you are replicating your MySQL DB. As I would like to introduce that into our environment here.

[donmontalvo]

@lastdanstanding wrote:

We actually already have replication in place. We use it to run application usage queries, without bogging down the JSS. Perhaps we'll start using it for backups as well.

+1