JDS Replication and Performance

[bajones]

I've been struggling with the JDS for a few days now and I was wondering if anyone had any tips on tuning or tweaking any settings to improve the performance. My JSS and root JDS are on the same xserve running 10.8.5. My master distribution point is an AFP server in the same rack, which has performed quickly and reliably pre-9.x.

From my observation, the JDS replicates in an interesting manner. It seems when I replicate using Casper Admin, the packages transfer from my master distribution point into the JSS mysql database (downloadable_file_chunk_data table), then once the file is completely transferred the JDS is notified to download it. While downloading, the package appears to sit in /tmp and move to the JDS share once it is fully "downloaded" from the local database.

My issue is that in attempting to perform my full first replication to the root JDS, the mysql database balloons up to 10-50 GB and mysql becomes mostly unresponsive. When this happens, the JAMFSoftwareServer.log gets flooded with repeated mysql connection errors and all JSS tasks slow to a snail's pace. These are the log entries I'm seeing, grep-ed to show only the error lines:

2013-10-10 09:58:37,372 [ERROR] [atch-thread] [BoneCP ] - Failed to acquire connection to jdbc:mysql://localhost:3306/jamfsoftware?characterEncoding=utf8&useUnicode=true&jdbcCompliantTruncation=false. Sleeping for 7000 ms. Attempts left: 5
2013-10-10 09:58:45,690 [ERROR] [atch-thread] [BoneCP ] - Failed to acquire connection to jdbc:mysql://localhost:3306/jamfsoftware?characterEncoding=utf8&useUnicode=true&jdbcCompliantTruncation=false. Sleeping for 7000 ms. Attempts left: 4
2013-10-10 09:58:52,693 [ERROR] [atch-thread] [BoneCP ] - Failed to acquire connection to jdbc:mysql://localhost:3306/jamfsoftware?characterEncoding=utf8&useUnicode=true&jdbcCompliantTruncation=false. Sleeping for 7000 ms. Attempts left: 3
2013-10-10 09:58:59,697 [ERROR] [atch-thread] [BoneCP ] - Failed to acquire connection to jdbc:mysql://localhost:3306/jamfsoftware?characterEncoding=utf8&useUnicode=true&jdbcCompliantTruncation=false. Sleeping for 7000 ms. Attempts left: 2
2013-10-10 09:59:06,704 [ERROR] [atch-thread] [BoneCP ] - Failed to acquire connection to jdbc:mysql://localhost:3306/jamfsoftware?characterEncoding=utf8&useUnicode=true&jdbcCompliantTruncation=false. Sleeping for 7000 ms. Attempts left: 5
2013-10-10 09:59:13,729 [ERROR] [atch-thread] [BoneCP ] - Failed to acquire connection to jdbc:mysql://localhost:3306/jamfsoftware?characterEncoding=utf8&useUnicode=true&jdbcCompliantTruncation=false. Sleeping for 7000 ms. Attempts left: 5
2013-10-10 09:59:20,732 [ERROR] [atch-thread] [BoneCP ] - Failed to acquire connection to jdbc:mysql://localhost:3306/jamfsoftware?characterEncoding=utf8&useUnicode=true&jdbcCompliantTruncation=false. Sleeping for 7000 ms. Attempts left: 4
2013-10-10 09:59:27,751 [ERROR] [atch-thread] [BoneCP ] - Failed to acquire connection to jdbc:mysql://localhost:3306/jamfsoftware?characterEncoding=utf8&useUnicode=true&jdbcCompliantTruncation=false. Sleeping for 7000 ms. Attempts left: 3
2013-10-10 09:59:34,755 [ERROR] [atch-thread] [BoneCP ] - Failed to acquire connection to jdbc:mysql://localhost:3306/jamfsoftware?characterEncoding=utf8&useUnicode=true&jdbcCompliantTruncation=false. Sleeping for 7000 ms. Attempts left: 2

If I interrupt the Casper Admin replication, the JDS eventually catches up overnight and retrieves the packages living in the database and JSS performance is back to normal. If I run another replication to get the remaining packages into the JDS, the files copy into the database but the JDS never starts to "download" them from the local database until I optimize the mysql tables. Optimizing seems to kickstart the JDS to download the packages waiting in the database. I changed mysql's max packet size to 256 MB and performance seems improved, but it might just be a placebo.

I would appreciate any feedback or tips that might improve my situation. I've been working for over a week to try and replicate about 250 packages (188 GB) to my root JDS instance. I'd be happy to provide any other relevant information about our setup to assist troubleshooting.

[sgorney]

Bump. I'm having this same issue.

[calum_carey]

yep me too i posted about it earlier.
i've decided to just forget about using the JDS at the moment. Probably should log a support case when I get around to it

[bajones]

I eventually got it resolved. Resolution involved mostly patience, as the replicated packages seemed to sit in "package purgatory" (mysql database? /private/tmp? I forget which) for quite some time before eventually "downloading" into the JDS repository. During this time, the JSS is extremely unresponsive. I was monitoring /Library/JSS/Logs/JAMFSoftwareServer.log and /Library/JDS/logs/jamf.log during this time to check for errors. JAMFSoftwareServer.log frequently (constantly) showed errors regarding lost connections to mysql. I ignored them and they eventually stopped once replication finished.

Twice during replication I encountered an issue where 40 or so remaining packages were successfully replicated via Casper Admin, but the JDS log was showing them as failed downloads due to an incorrect File ID. The only way I was able to get the files to replicate properly was to truncate the downloadable_file_chunk_data table, run a database repair and optimize, and possibly restarting Tomcat (it was a long weekend). Then I would replicate with Casper Admin again and the process completed slowly and successfully.

On a side note, I run my root JDS on the same server as my JSS. When the files were being replicated from the master DP over AFP to the JAMF database the files moved at acceptable speeds. When the files were being "downloaded" to the root JDS (remember, same box at the JSS), my transfer speeds were incredibly slow. Somewhere around 7 MB/s. I have since setup 3 more JDS instances and when they replicate from the root, they saturate the root JDS's gigabit connection to the tune of 112 MB/s.

[stevehahn]

I'm running my master DP on the same Mac mini that I want to turn into my root JDS. Does anyone know if there's a shortcut to avoid moving the files up to the JSS and then back down to the same box? Especially since when I started the replication process from master DP to JDS last night, it progressed smoothly for a while but eventually locked up my JSS and caused Casper Admin to freeze.

[egjerde]

Hi Steve - late comment to this thread, but the following article by JAMF might help others searching for the same answers:

https://jamfnation.jamfsoftware.com/article.html?id=351