RSync back up size allowing for exclusions

[May]

Hi all

Working on a script to back up the user's home folder, it will run from Self Service under a local admin login.

I'm using jamfhelper to notify the admin which home folder is backing up and also how much data will be copied, at the moment du -hs retrieves the size of the home folder but this will be an incorrect as i'm using an exclusion list with RSync, can anyone suggest a way to retrieve just the size of the data that will be copied allowing for the exclusion list ?

is there a way to use RSyncs' --dry-run and report back the data size before the copy ?

[hkabik · Accepted solution]

You're looking for :

rsync -an --stats

This will give you both "Total file size" and "Total transferred file size".
Total size being the total amount of the data and total transferred file size is the total amount to actually be synced over.

You probably just want to grep out the transferred file size and present that to the admin.

[markkenny]

We use a script to get home folder sizes individually once a day and then EAs report those sizes.

We use public/private key setup established in advance, (only 70 users), and a rsync that backups all most of home folder but excludes trash, downloads, spotify etc. over SSH to a ZFS network volume.

It's set to run daily on laptops if in to office, and then I have a smart group to warn me if it's been longer than 5 days without a backup.

We call it "Sup-Homes";-)

#!/bin/bash
# GET USER FOLDER SIZES AND WRITE TO PLIST
# Calculates storage (GB) used by a subfolder in logged-in user's homedir.
# This should be run periodically to update the folder size.

# Common JAMF Script Variables
DRIVE="$1"
COMPUTER_NAME="$2"
JAMF_USER="$3"

# Output plist and key for dictionary
RESULTS_FILE="$DRIVE/Library/Preferences/com.MYCOMPANYPREFERENCE.plist"
KEY="HomeFolderUsage"

# Folders to scan
SUBFOLDERS=(
    # Consider including entire home folder, which is not directly accessible
    '/'
    '.Trash'
    'Desktop'
    'Downloads'
    'Documents'
    'Movies'
    'Music'
    'Pictures'
)

# Attempt to auto-detect the last-logged-in user.
CONSOLE_USER=`defaults read "$DRIVE/Library/Preferences/com.apple.loginwindow.plist" lastUserName`

# Calculate size for each specified folder.
# If a meaningful end-user is not available, size is calculated from all user
# folders on the machine.
for SUBFOLDER in ${SUBFOLDERS[*]}; do
    if [ "$CONSOLE_USER" = '' ] || [ "$CONSOLE_USER" = 'CASPERUSERNAME' ]; then
        USAGE=`du -sg "$DRIVE"/Users/*/"$SUBFOLDER" | cut -f 1`
    else
        USAGE=`du -sg "$DRIVE/Users/$CONSOLE_USER/$SUBFOLDER" | cut -f 1`
    fi

    # Place findings in plist for future recovery
    defaults write "$RESULTS_FILE" "$KEY-$SUBFOLDER" "$USAGE"
done

# Display new contents for reference
# defaults read "$RESULTS_FILE"

#!/bin/bash
#SUP-HOMES BACKUP SCRIPT-V6
MOUNT_POINT="$1" # TODO Integrate into other variables
COMPUTER_NAME="$2"
# USER_NAME="$3" # only defined at login

# Convert global config from v5 to per-user config for v6
OLD_CONFIG=/Library/Preferences/com.MYCOMPANYPREFERENCE.plist
if [ -f $OLD_CONFIG ]; then
    for U in $( defaults read $OLD_CONFIG "Backup Users" ); do
        NEW_CONFIG="/Users/$U/Library/Preferences/com.MYCOMPANYPREFERENCE.plist"
        if [ -f "$NEW_CONFIG" ]; then
            echo User $U has already been converted.
            echo Consider removing the original config file from $NEW_CONFIG.
        else
            echo "Creating new entry for user $U at $NEW_CONFIG."
            defaults write "$NEW_CONFIG" "Enabled" -bool true
        fi
    done

    # TODO Once everyone is safely migrated, begin deleting the original file
    # rm "$OLD_CONFIG"
fi

# Run per-user sync
for LOCAL_HOME in /Users/*; do
    CONFIG_FILE="$LOCAL_HOME/Library/Preferences/com.MYCOMPANYPREFERENCE.plist"
    if [ -f "$CONFIG_FILE" ]; then
        START_TIME=$( date +'%Y-%m-%d %H:%M:%S' )
        defaults write "$CONFIG_FILE" "Last Run" -date "$START_TIME"

        # Extract User Name from home folder
        U=`basename $LOCAL_HOME`

        # Copy relevant files from user home to rsync folder in remote home.
        echo Starting backup for $U
        time rsync --archive --partial --numeric-ids --one-file-system --stats 
            --delete-after --delete-excluded 
            -e "ssh -i $LOCAL_HOME/.ssh/sup-homes" 
            --exclude=Downloads --exclude=.Trash --exclude=.FileSync 
            --exclude=Documents/Microsoft User Data 
            --exclude=Library/Application Support/MobileSync 
            --exclude=Library/Application Support/Adobe 
            --exclude=Library/Application Support/Google 
            --exclude=Library/Application Support/Firefox 
            --exclude=Library/Application Support/Skype 
            --exclude=Library/Application Support/iLifeAssetManagement 
            --exclude=Library/Mail 
            --exclude=Library/Logs 
            --exclude=Library/Caches 
            --exclude=Music/iTunes 
            --exclude=Library/Caches 
            --exclude=Library/Developer 
            --exclude=Dropbox 
            --exclude=Google Drive 
        "$LOCAL_HOME/" 
        $U@sup-homes:rsync/

        RSYNC_STATUS=$?

        END_TIME=$( date +'%Y-%m-%d %H:%M:%S' )
        if [[ $RSYNC_STATUS -eq 0 ]]; then
            defaults write "$CONFIG_FILE" "Last Success" -date "$END_TIME"
        fi
    fi
done

[May]

@hkabik @markkenny

rsync -an --stats suits what i need perfectly, i can run it before the copy to get the copy size, also found a nice snippet to convert it from bytes to a more palatable form.

#sure this ain't so pretty!
#bytes conversion from https://gist.github.com/mlegenhausen/9365461
syncsize=$( caffeinate rsync -an --stats --exclude-from=/var/tmp/temp_exclusions.txt /Users/bill.danforth/Desktop/READ/ /Users/bill.danforth/Desktop/Aqui | grep "Total transferred file size" | sed 's/[^0-9]*//g' | awk '{ sum=$1 ; hum[1024**3]="GB";hum[1024**2]="MB";hum[1024]="KB"; for (x=1024**3; x>=1024; x/=1024){ if (sum>=x) { printf "%.2f %s
",sum/x,hum[x];break } }}' )

echo "$syncsize"

Liking the Sup-Homes script, got me thinking about using keys instaed of prompting for the password each time.

Thanks for your help!