rsync script for backing up user data

[Rosko]

Okay, so we are getting ready for our 10.6 migration to 10.8 and with it we are doing a fresh install. Below is a rsync script I've been working on, but it seems to ignore the "--excludes" every time???

Any ideas?

#!/bin/bash

#########################################################################################################
# This script replicates files from the current logged in users home directory                          #
# to USMFS008 for temporary storage.                                                                    #
#                                                                                                       #
# This script was made as part of the 10.8 Upgrade Project & JAMF Implementation.                       #
#                                                                                                       #
# Users can restore their data from Self Service by click on Restore User Data.                         #
# This will be a one-time restore and files will be deleted from USMFS008 once restore is complete.     #
# See UserData_Restore.sh script                                                                        #
#                                                                                                       #
# The following is excluded in the backup:                                                              #
# - Downloads Folder                                                                                    #
# - Library Folder (except Safari, Chrome, Firefox and Stickies)                                        #
# - iTunes                                                                                              #
# - Microsoft User Data                                                                                 #
# - Trash                                                                                               #
#########################################################################################################

# File Name: UserData_Backup.sh
# Created By: Joshua Roskos
# Created On: Thursday, January 9th, 2014
# Modified On: Thursday, January 9th, 2014

# Determine Current Logged-In User
lastUser=`defaults read /Library/Preferences/com.apple.loginwindow lastUserName`
echo "$lastUser"

# Mounts USMFS008 UserData Folder
sudo mkdir /Volumes/UserData
sudo mount_afp afp://user:pass@usmfs008/UserData /Volumes/UserData

# Create Log File
# sudo touch /Volumes/UserData/_Logs/"$lastUser.log"
# sudo chmod 777 /Volumes/UserData/_Logs/"$lastUser.log"
# sudo echo " " >> /Volumes/UserData/_Logs/"$lastUser.log"
# sudo echo "=====Starting rSync of $lastUser @ $(date)=====" >> /Volumes/UserData/_Logs/"$lastUser.log"

# Backup Users Home Directory
sudo mkdir /Volumes/UserData/"$lastUser"
sudo rsync -avzrpog --update --delete --include=Library/Safari/** --include=Library/StickiesDatabase --include=Library/Applications Support/Firefox/** --include=Library/Application Support/Google/** --exclude=./Trash/** --exclude=/Music/iTunes/** --exclude=/Documents/Microsoft User Data/** --exclude=/Library/** --exclude=/Downloads/** --progress --log-file="/Volumes/UserData/_Logs/$lastUser.log" /Users/"$lastUser"/ /Volumes/UserData/"$lastUser"/
sudo umount /Volumes/UserData

exit 0

[davidacland · Accepted solution]

The one I use is:

--filter='merge /path/to/your/exclusion/list/exclusionlist'

Below is a snippet from an example file:

# Top level files & folders to exclude:

- Downloads
- .Trash
- .DS_Store
- Public
- Sites

# Library folder items to exclude:

- Address Book Plug-Ins
- Application Support
- Assistants
- Audio
- Autosave Information
- Caches
- Cookies
- Calendars
- ColorPickers

[pblake]

Is it because you have a / in front of the excludes. Making it no longer in the user library but the root?

Yours:
sudo rsync -avzrpog --update --delete --include=Library/Safari/* --include=Library/StickiesDatabase --include=Library/Applications Support/Firefox/* --include=Library/Application Support/Google/* --exclude=./Trash/* --exclude=/Music/iTunes/* --exclude=/Documents/Microsoft User Data/* --exclude=/Library/* --exclude=/Downloads/* --progress --log-file="/Volumes/UserData/_Logs/$lastUser.log" /Users/"$lastUser"/ /Volumes/UserData/"$lastUser"/

My Suggestion:
sudo rsync -avzrpog --update --delete --include=Library/Safari/* --include=Library/StickiesDatabase --include=Library/Applications Support/Firefox/* --include=Library/Application Support/Google/* --exclude=./Trash/* --exclude=/Music/iTunes/* --exclude=Documents/Microsoft User Data/* --exclude=Library/* --exclude=Downloads/* --progress --log-file="/Volumes/UserData/_Logs/$lastUser.log" /Users/"$lastUser"/ /Volumes/UserData/"$lastUser"/

[Rosko]

That's what I thought when I first stated it, but according to the man pages it treats it as the transfer root. Without the / would exclude any folder with that name.

But I have tired both ways with the same outcome. :(

[davidacland]

I would normally use the —filter option and reference a separate file with the exclusion list if there are multiple items just to make it a bit easier to read and edit.

Looking at the script, I think you have a few absolute paths pointing to the root of the hard drive:

/Music/iTunes/*
/Documents/Microsoft User Data/*
/Downloads/**

The /Library/** entry would work as the folder does exist but I would expect the others to not match.

As a separate test, you could try specifying the full path to a particular user and check that the exclude works as expected. If it does, you can just substitute it with /Users/$USER at the start of each one.

Hope this helps.

David

[Rosko]

I've tried adding the full path to the exclusions and it still skips over them. Now for some reason, when I point it to my second HDD versus a network locations, some of the options seem to work, but for no apparent reason.

sudo rsync -avzrpog --update --delete --include='/Library/Safari/**' --include='/Library/StickiesDatabase' --include='/Library/Applications Support/Firefox/**' --include='/Library/Application Support/Google/**' --exclude=/.Trash/** --exclude=/Users/$lastuser/Music/iTunes/** --exclude=/Documents/Microsoft User Data/** --exclude=/Library/** --exclude=/Downloads/** --progress --log-file="/Volumes/VAULT/UserData/_Logs/$lastUser.log" /Users/"$lastUser"/ /Volumes/VAULT/UserData/"$lastUser"/

With that, the Exclusions all seem to work, but now it doesn't include everything? It manages to include the stickies database and that is it. :(

I tried using an Exclusions and Inclusions file but can't seem to get those to work, no I'm looking at the --filter option, but can't find really any good documentation on how to structure the file itself?

sudo rsync -avzrpog --update --delete --filter='merge /Volumes/VAULT/UserData/.merge' --progress --log-file="/Volumes/VAULT/UserData/_Logs/$lastUser.log" /Users/"$lastUser"/ /Volumes/VAULT/UserData/"$lastUser"/

Anyone have some examples they can share?

[davidacland · Accepted solution]

The one I use is:

--filter='merge /path/to/your/exclusion/list/exclusionlist'

Below is a snippet from an example file:

# Top level files & folders to exclude:

- Downloads
- .Trash
- .DS_Store
- Public
- Sites

# Library folder items to exclude:

- Address Book Plug-Ins
- Application Support
- Assistants
- Audio
- Autosave Information
- Caches
- Cookies
- Calendars
- ColorPickers

[Rosko]

So I created a ".merge" file with all the Inclusions/Exclusions as below:

# INCLUSIONS
+ /Pictures/**
+ /Desktop/**
+ /Documents/**
+ /Movies/**
+ /Music/**
+ /Library/Safari/**
+ /Library/StickiesDatabase
+ /Library/Application/ Support/Firefox/**
+ /Library/Application/ Support/Google/**

# EXCLUSIONS
- /Music/iTunes/**
- /Documents/Microsoft/ User/ Data/**
- /.Trash/**

But it seems to be doing the same thing where it will exclude some lines but not others??? This is really starting to drive me nuts :(

[stevewood]

I'm curious as to why you are excluding things. I guess I can understand possibly excluding the Downloads folder, and I can definitely understand the Trash folder, but the rest I don't get. When I do a user migration, I grab the entire user folder. That way, when a user logs into their new machine, or updated machine, they have the same experience they had previously.

[Rosko]

Thanks David- I'll try removing the slashes and wildcards and see if that works better in this scenario.

[mpermann]

@Roskos][/url, in your second to last post about the merge file a few of your paths seem like they may not be correct. Shouldn't the slashes between Application and Support and Microsoft, User and Data be the (backslash) rather than the / (forward slash)? I don't think it will interpret the path correctly if you use the forward slash in those instances. Not sure if that is what is ultimately causing your problem or not though.

[Rosko]

Thanks David! So much closer now :D

So i use the following rsync command:

sudo rsync -avzrpog --update --delete --filter='merge /Volumes/VAULT/UserData/.merge' --progress /Users/jbrosko/ /Volumes/VAULT/UserData/jbrosko/

In conjunction with this file:

# INCLUSIONS
+ Safari
+ StickiesDatabase
+ Firefox
+ Google

# EXCLUSIONS
- iTunes
- Microsoft User Data
- Downloads
- Library
- .Trash
- Public
- .ssh

The only issue I am having now is that it is excluding the Library folder right away and not grabbing the Safari folder, StickiesDatabase or Firefox and Google folders within the App Support folder?

Any ideas? Or would I just have to exclude all the the folders individually versus the entire Library folder and including the ones I want? I know in the man pages it said the command is prioritized by what comes first, I was hoping that carried over the the merge file, but maybe not...maybe a combination of the file and include commands?

And Steve, we've had problems with backing up the Microsoft User Data folder and restoring it when the user tries to use it. We also limit the number of Library folders to eliminate issues when upgrading from one OS to another. If this was a simple backup script we would treat it differently, but since we're going from 10.6 to 10.8 we are being cautious. We also do not support iTunes and are not going to waste Terabytes of company storage for somebodies personal music ;) lol.

[MrP]

The only issue I am having now is that it is excluding the Library folder right away and not grabbing the Safari folder, StickiesDatabase or Firefox and Google folders within the App Support folder?

Sounds like you need a couple more rsync commands, or is there some reason you have to do it all in one command?

[nessts]

As far as backing up the Microsoft User Data (MUD) folder, if you quit outlook and make sure the microsoft database daemon is shut down you can back it up and restore it quite well. And since you are scripting it you might as well shutdown those things to back it up since restoring local folders in Outlook is less than easy. Seems like somebody found a way to do it, but I cannot find it again. I would think you would want to backup Library/Keychains as a matter of course, since it usually has some handy data in it. get more picky on exclusions like leave out Caches, Cookies, Preferences? etc. For preferences i would personally migrate them and then rename the folder to Prefs-10.6 so if people wanted to pull some specific app preferences in they could.

[davidacland · Accepted solution]

I think you will need to allow the Library folder and then exclude the specific items beneath it. This is the full exclusion list I use for the user library. It's for a slightly different purpose but should get you some of the way there:

- Address Book Plug-Ins
- Application Support
- Assistants
- Audio
- Autosave Information
- Caches
- Cookies
- Calendars
- ColorPickers
- Compositions
- Favorites
- FontCollections
- Fonts
- Icons
- iMovie
- Input Methods
- Internet Plug-Ins
- Keyboard Layouts
- Keychains
- Logs
- PreferencePanes
- *com.apple.LaunchServices
- Printers
- PubSub
- Screen Savers
- Sounds
- Spelling
- Voices

[acdesigntech]

If you're only rsyncing several folders under /Library, make a separate rsync statement for each folder you want to include. Put these statements AFTER the main rsync command that excludes /Library/.

@davidacland, you could do it your way too, but since we're only concerned with 4 folders, it's less typing to exclude everything then only sync the ones you want.

I believe excludes supersede includes when using rsync. Correct me if I'm wrong.

[acdesigntech]

and as I'm sitting waiting to post my comment I remembered you're using a merge file. disregard my previous advice :)

[Rosko]

Thank you to everyone for your help! Below is the final product :)

UserData_Backup.sh

#!/bin/bash

#########################################################################################################
# This script replicates files from the current logged in users home directory                          #
# to USMFS008 for temporary storage.                                                                    #
#                                                                                                       #
# This script was made as part of the 10.8 Upgrade Project & JAMF Implementation.                       #
#                                                                                                       #
# Users can restore their data from Self Service by click on Restore User Data.                         #
# This will be a one-time restore and files will be deleted from USMFS008 once restore is complete.     #
# See UserData_Restore.sh script                                                                        #
#########################################################################################################

# File Name: UserData_Backup.sh
# Created By: Joshua Roskos
# Created On: Thursday, January 9th, 2014
# Modified On: Friday, January 10th, 2014

# Determine Current Logged-In User
USER=`defaults read /Library/Preferences/com.apple.loginwindow lastUserName`

# Mounts USMFS008 UserData Folder
sudo mkdir /Volumes/UserData
sudo mount_afp afp://user:pass@usmfs008/UserData /Volumes/UserData

# Create Log File
if [ ! -f /Volumes/UserData/_Logs/"$USER".log ]; then
    sudo touch /Volumes/UserData/_Logs/"$USER.log"
    sudo chown dataadmin:staff /Volumes/UserData/_Logs/"$USER.log"
    sudo chmod 777 /Volumes/UserData/_Logs/"$USER.log"
fi

# Stamp Log File - Starting rsync
sudo echo " " >> /Volumes/UserData/_Logs/"$USER.log"
sudo echo "=====Starting rSync BACKUP of $USER @ $(date)=====" >> /Volumes/UserData/_Logs/"$USER.log"

# Backup Users Home Directory
if [ ! -d /Volumes/UserData/"$USER" ]; then
    sudo mkdir /Volumes/UserData/"$USER"
fi

sudo rsync -vzrpog --update --delete --ignore-errors --force --filter='merge /Volumes/UserData/Exclusions' --progress --log-file="/Volumes/UserData/_Logs/$USER.log" /Users/"$USER"/ /Volumes/UserData/"$USER"/

# Stamp Log File - rsync Complete
sudo echo " " >> /Volumes/UserData/_Logs/"$USER.log"
sudo echo "=====Completed rSync BACKUP of $USER @ $(date)=====" >> /Volumes/UserData/_Logs/"$USER.log"

# Unmount USMFS008 - UserData
sudo umount /Volumes/UserData

exit 0

UserData_Restore.sh

#!/bin/bash

#########################################################################################################
# This script replicates files from USMFS008 back to the current logged in users home directory.        #
#                                                                                                       #
# This script was made as part of the 10.8 Upgrade Project & JAMF Implementation.                       #
#                                                                                                       #
# Users can restore their data from Self Service by click on Restore User Data.                         #
# This will be a one-time restore and files will be deleted from USMFS008 once restore is complete.     #
#########################################################################################################

# File Name: UserData_Restore.sh
# Created By: Joshua Roskos
# Created On: Friday, January 10th, 2014
# Modified On: Friday, January 10th, 2014

# Determine Current Logged-In User
USER=`defaults read /Library/Preferences/com.apple.loginwindow lastUserName`

# Mounts USMFS008 UserData Folder
sudo mkdir /Volumes/UserData
sudo mount_afp afp://user:pass@usmfs008/UserData /Volumes/UserData

# Check for Existing Backup
if [ ! -d /Volumes/UserData/"$USER" ]; then
    sudo jamf displayMessage -message "No Backups Found for Current User. Restore Cancelled."
else
    # Stamp Log File - Starting rsync
    sudo echo " " >> /Volumes/UserData/_Logs/"$USER.log"
    sudo echo "=====Starting rSync RESTORE of $USER @ $(date)=====" >> /Volumes/UserData/_Logs/"$USER.log"

    # Restore Users Home Directory
    sudo rsync -vzrpog --update --ignore-errors --force --progress --log-file="/Volumes/UserData/_Logs/$USER.log" /Volumes/UserData/"$USER"/ /Users/"$USER"/

    # Stamp Log File - rsync Complete
    sudo echo " " >> /Volumes/UserData/_Logs/"$USER.log"
    sudo echo "=====Completed rSync RESTORE of $USER @ $(date)=====" >> /Volumes/UserData/_Logs/"$USER.log"

    # Deletes Remote Backup of Users Files
    sudo rm -fdr /Volumes/UserData/"$USER"
fi

# Unmount USMFS008 - UserData
sudo umount /Volumes/UserData

# Run CHOWN
sudo chown -R $USER:staff /Users/$USER/

exit 0

Exclusions

# Home Directory Exclusions
- iTunes
- Microsoft User Data
- Downloads
- .Trash
- Public
- .ssh

# User Library Folder Exclusions
- Assistants
- PhotoshopCrashes
- Audio
- Caches
- Calendars
- ColorPickers
- Colors
- Compositions
- Containers
- Cookies
- Favorites
- FileSync
- FontCollections
- Fonts
- GoogleSoftwareUpdate
- iMovie
- Input Methods
- Internet Plug-Ins
- Keyboard Layouts
- Keychains
- LaunchAgents
- LaunchDaemons
- Logs
- Mail
- Parallels
- PDF Services
- PreferencePanes
- Preferences
- Printers
- PubSub
- Saved Application State
- Screen Savers
- Sounds
- Spelling
- SyncedPreferences
- Voices

# User Library Application Support Folder Exclusions
- AddressBook
- Citrix
- Citrix Receiver
- com.apple.QuickLook
- CrashReporter
- Dock
- Microsoft
- Mozilla
- PowerRegister
- Preview
- SyncServices

Hopefully this will help someone else as well. Next up is to combine it all a little to be an actual backup client that will work across multiple systems for those users that use more than one Mac :)

Thanks Again!

[nessts]

do you need all of the sudo calls?
mounting the disk and such should be able to be done by the user.
the user should own all the files they are copying right? Just curios.
Looks nice.

[Rosko]

@nessts

Technically...I probably shouldn't since Casper runs the script as root anyway, but I just play it safe. And the user does own all the files that are being copied, but they do not have access to the network share normally.

[mm2270]

It may not be anything to be concerned about for this scenario, but be aware that all sudo calls get logged to the /private/var/log/system.log, including any usernames, passwords, fileshare paths, etc that may be included on the sudo line, essentially they get logged in plain text. These are fully readable by your end users from the Console.app and get stored in the log until its turned over.
Again, for your rsync script, probably not that big a deal, but do be aware of that. For this reason i avoid using sudo in any script I know will be running from a Casper Suite policy. There are however times when it can't be avoided.

[Rosko]

@mm2270

Thank you for that reminder...I had forgotten about that. I did just remove all the sudo commands with the exception of the one that calls the jamf binary on the restore script and it still worked beautifully, so I will probably keep it this way.

Thanks again!

[msnowdon]

@Rosko , How well did the script you created work for you? I am in the same situation where we are replacing all teacher MacBooks and need an easy way for them to backup and restore their files. We have someone creating a set of directions to do it manually but a script would be better and less confusing to the users.

Thanks

Mark

[dvasquez]

Anyone see Self Service crash after the back up script finishes?

I have tried sleeping between commands for a few seconds and before the created mount is removed (which I think is the issue).

I know Self Service can be sensitive with how scripts end.

Thanks.