Running out of HD space on Distribution Points

[#1]

Hey guys, I'm having a bit of a critical issue.

My Master Share clocks in at about 85GB. In addition to the normal stuff in there I also have keep 14 days worth of database backups.

I have about 12 remote Distribution points. I use remote sync on all of them. It runs every night after I do a DB Backup, this way the sync carries the latest backup to all the remote distribution points.

My problem is that I noticed that casper remote sync is not cleaning up after itself after it does a sync. It is not moving deleted packages into the deleted packages folder, but recopying the ones from the Master share that I've deleted since the backup, leaving two copies of the same package on the repository. Its also not deleting old database backups from the remote distribution points either, so when I last checked my smallest repository which has a 160GB Hard Drive in it, it was completely full.

How can I make this work?

[#2]

I've noticed that too, the Automatic Sync with the JSS utility copies
EVERYTHING in the master "CasperShare" folder to the remote distribution
points. However, all that needs to be on remote distribution points is
/Packages and /Scripts. So all the Deleted Packages get rsynced out, too.
Other problem I noticed is that when I replace a file in an /Extras folder
"Mac_WinXP.ntfs" for example, it doesn't replace on the remote dist. points
because the file name is the same, even though the size and modification
date is different. So I delete it on the master one day, wait a day or
two, then put the new one back which is then rsynced out.

Feature Request: Can we select which folders get synced or not? I'd
like some distribution points to get the /Packages and /Scripts and some
distribution points to get /Packages, /Scripts and /Extras. And can we
tell it to look at size and/or date in addition to filename? I did get
help from JAMF support on modifying the plist at each dist. point, but that
is a pain and gets reset when you change the scheduled sync time.
This added auto-sync feature has been great for me with many distribution
points, just needs a little more to be awesome.

Thanks,

Nathaniel Lindley

++++++++++
Educational Technology
Saint Paul Public Schools
Saint Paul, Minnesota
nathaniel.lindley at spps.org
phone: 651-248-6861

[ernstcs]

Hello,

I like the idea of expanding the functionality, but at the same time...if you don't want it copied, don't put it in there.

By default I thought rsync used mod and size, again according to the site for it:

"Rsync finds files that need to be transferred using a "quick check" algorithm (by default) that looks for files that have changed in size or in last-modified time"

That would concern me if your ntfs file isn't getting updated.

If it's not comparing by mod and date...I'm wondering why not. I started to sync to my second server recently, but just now have time to look into doing more. Like putting my backups in the CasperShare directory so it automatically goes to the other server in the event of a hardware failure and I need to setup the new server fast. Lazy way of not setting up my own task.

It should also be easy to set it to delete what's no longer on the master if that isn't being set, to help clean things up:

--delete-delay find deletions during, delete after --delete-after receiver deletes after transfer, not before

I believe those are the options in the current release. I would have looked more into what is actually being set by the JSS util but don't have time right now.

As for the comment about copying over Deleted Packages...true it doesn't need to, but Empty the trash if you are truly deleting something. I actually keep a separate subfolder of packages I want completely removed from being visible to the JSS, Package Archive. It requires me to do it manually, but that's fine by me. I know I've asked for that feature at one point. As always...different strokes for different folks. =)

Craig E

[#4]

Thing is, I *DO* want the deleted packages copied, and I do want everything in Casper Data copied, but when I clean things up on the Master Server (e.g. Emptying the deleted packages folder), those deletions don't carry over to the remote distribution points, and worse, those deleted packages are *STILL* in the main packages folder on the Distribution Points.

[milesleacy]

Perhaps having the ability to manually or automatically re-sync after
emptying the trash would be a good feature request to solve this?

----------
Miles A. Leacy IV

? Certified System Administrator 10.4
? Certified Technical Coordinator 10.5
? Certified Trainer
Certified Casper Administrator
----------
voice: 1-347-277-7321
miles.leacy at themacadmin.com
www.themacadmin.com

[tlarkin]

I would also like the feature to modify a script on the Master server
with say Casper Admin, and then have it sync a new copy of that script
down and over write the previous ones. That would be a small time
saver.

I would also like a sync list, where you can check packages you want to
sync out to all shares and then anything else you can manually sync
later on. For example at some of our smaller deployments at the middle
schools they still use PPC macs, and they are the only buidlings that do
so, and they are the only casper share points that would ever need PPC
packages. I only want PPC packages to sync to those buildings and no
where else.

A set of rules for package synchronizing would be a very nice feature. I'd vote for it, if this were a democracy.

[ernstcs]

So let me recap what I was stating and other things:

You can manually resync in Casper Admin anytime you want I wouldn't want it to happen automatically while in Casper Admin Sean, I know that you DO want what is in Deleted Packages Nathaniel wanted Deleted Packages ignored, but my comment about not wanting something copied and don't have it in the CasperShare directory was directed at him Sean, I think something is broken with Rsync that needs to be addressed to clean up so the items you deleted are no longer in the regular packages folder. My guess is that the none of the delete options are implemented, but again I haven't verified that. I think something is broken with Rsync which is causing Nathaniels ntfs file to not get updated. Again, perhaps a flag isn't right, but again I haven't verified what JAMF is doing. To add to Tom's email, unless Rsync is being configured differently for some reason, the default settings SHOULD overwrite the old files. Also to Tom's email, Rysnc is NOT a product of JAMF's and the ability to hand pick which files come and go may be difficult to implement, handy...most certainly.

Craig E

[scottb]

There is a request up somewhere asking for the capability to choose what packages/scripts/compiled images go to what DP's. That would be an epic change in the Casper Admin capability.

As it is, we use a combination of rsync for larger xfers and Casper Admin when I want to choose all 19 DP's for a smaller sync. Since the servers are located all over the world, and with varying degrees of network quality, being able to NOT send everything everywhere would be the best improvement to date for me.

[bentoms]

@boettchs, JDS allows selective replication as well as having parent & child JDS.

So you can not only sync what you want, but also choose where to sync from.

[zanb]

Do a "diff" between two text files on each server. Here's how you can print out a list of files in each CasperShare directory. FYI I've escaped some newlines in the script to help prevent word-wrapping on this website. If you have questions about these methods, please let me know.

#!/bin/bash
# Change "/Path-2-CasperShare/" to the directory of your CasperShare
# e.g. 'cd /Volumes/RAIDs0/CasperShare/'
cd /Path-2-CasperShare/
#  I used $2 for the computer name, assuming your master DP
#+ is managed by the JSS.
find ./{Packages,Scripts,CompiledConfigurations} -maxdepth 2 
| sed 's/.///' > jss_files_${2}.txt
[[ ! -d jss_files_${2}.txt ]] 
&& echo "Could not write file jss_files_${2}.txt" 
&& exit 1
exit

Now on each distribution point you can run the following script:
(Note: You will need to enter the 4th field for the FQDN of your master Casper server. e.g. "casper.pretendco.com" or IP address.
Enter the 5th field for the hostname of the master Casper server - whatever was defined as "$2" in the previous script)

#!/bin/bash
# Change "/Path-2-CasperShare/" to the directory of your CasperShare
# e.g. 'cd /Volumes/RAIDs0/CasperShare/'
cd /Path-2-CasperShare/
find ./{Packages,Scripts} -maxdepth 2 | 
sed 's/.///' > jss_files_${2}.txt
#  I'm assuming here that the URL to your CasperShare is similar to
#+ "http://casper.pretendco.com/CasperShare/"
curl -O http://${4}/CasperShare/jss_files_${5}.txt
#
diff jss_files${2}.txt jss_files_${5}.txt | 
grep "<" | 
sed 's/<  //' > jss_delete_files.txt
#  Now you have an array of files that exist on your distribution point and not on
#+ the master Casper server. From here it's up to you to write a loop to iterate 
#+ through these lines and either move them, compress them, or delete them.
#+ Remember to set your IFS to handle filenames with spaces!
[[ ! -d jss_delete_files.txt ]] 
&& echo "Could not write file jss_delete_files.txt" 
&& exit 1
exit

[scottb]

@bentoms: Thanks - I've heard that. No JDS and not likely coming soon for us, but thanks. Not sure if there's going to be any budget changes for 2015 at this point.
I'd still like to see that option with the basic JSS setup.

[bentoms]

@boettchs, JDS is free.

[scottb]

@bentoms: Budget includes lots of things that are out of my control here. In other words, it would be a project that includes other enterprise groups and they will have a charge associated with it. So, the option is free in the JSS, but not for me to implement, sadly.

[bentoms]

@boettchs, understood.

But since the OP's post 6 years ago.. I think JDS has been offered since for this functionality.

Have a look at any feature requests on this, JAMF may have supplied some more information in them.