Script to fix OneDrive file, folder, path errors

[DougE]

We are working on a script that will fix file naming issues for users new to OneDrive. We have some users who have all kinds of MS illegal characters in their file names and folder names. One user has 800+ such names to fix.

Rather than have them do this manually, or buy third party software we decided to write some scripts that will find all these illegal characters/names and change them. This will be a policy in Self Service that will:
1. Run a script that looks over all the file names and generates a report. 2. Make the changes. 3. Create two reports. One that lists changes made, and a second that lists changes that could not be made and why.

Some questions for those more experienced than we are (we are new to Jamf as of July):

• Has this been done/attempted before? I don't see anything, but want to make sure we are not re-inventing the wheel or going down a rabbit hole that will ultimately not work for some reason we are missing.
• We could share this with others, would there be interest?
• Any suggestions on how to handle the find/replace logic? We can use legal characters, but need to be mindful that we could change an illegal name to a legal, but existing name. For example if "/" was replaced with "-" we could have example/1.doc changed to example-1.doc which may already exist?

With OneDrive being so new for Mac users I can see how many will have this issue. HOpefully this works out and could be helpful to the larger Jamf community.

[blackholemac]

While I don't have a script, I have a wonderful Mac app that helped me to weed out the irregular characters in a user's home folder prior to the migration to OneDrive. It's called Name Mangler <https://itunes.apple.com/us/app/name-mangler-3/id603637384?mt=12>.

Check it out...if you have any questions how we used it to solve this problem feel free to email me...it's not automated, but cuts the time on character correction down to mere minutes instead of hours if one was to do it by hand.

[DougE]

$18.99 each, thats the deal breaker with that. As we scale that gets expensive.

[blackholemac]

Two things on that... we only bought two copies for the whole district to give to the guys in the field that were migrating end users.. if you did want to buy it for more than that it's half price at 20 copies if I recall correctly . There is no need to give the software to all the end-users though

[blackholemac]

In Fact if you wanted to get really creative, you could map to home folder share for your entire user base on one machine run this utility on it ... it Will give you a report and then you could decide what to do .... if I had our migration to do over again I would've done that .

[DougE]

The script is basically done now. Works great. If there is an illegal character it will remove it. If the resulting file name or folder name already exists (at that level or anywhere in the OneDrive folder I'm not sure) it will not make a change.

Two text files are created in a folder placed on the Desktop:
-a log of all errors found and the before and after name
-a log of all errors found that could not be fixed (ie if the name already exists) so the user can follow these up manually

We'll see how it works once we test it a bit more and then run it for the guy with 800+ illegal names. For that I'll copy his OneDrive folder to our test machine so we don't risk his data and can repeat the testing as needed.

[BOBW]

Hi @DougE

Any chance you want to share your script?

I have been playing round with getting this working and thought I might not have to re-invent the wheel

[mark_mahabir]

We're about to roll out OneDrive for Business here, and I would also love to see some example scripts if at all possible!

[blackholemac]

I don't have scripts to share as I wasn't the lead engineer on this in our district, but I do have some advice that's worthwhile. If you're dealing with a mix of platforms, you will find the Mac users, especially the older ones, tend to be loosey-goosey in their file naming conventions. They tend to use special characters in their names specifically. OneDrive for Business doesn't like that and barfs when trying to sync such files.

We actually did not try to fully automate the transition but worked with each building site one by one (we have 14).

What I did for some of our more problematic Mac users with file name issues just download a product called Name Mangler to my laptop, mounted their share volume and did pattern searching/replacement of all their file names to fix issues before we migrated that user to OneDrive. You could probably do that on a more mass scale for the whole home folder share if you want to deal with a large group of users.

[blackholemac]

I don't have scripts to share as I wasn't the lead engineer on this in our district, but I do have some advice that's worthwhile. If you're dealing with a mix of platforms, you will find the Mac users, especially the older ones, tend to be loosey-goosey in their file naming conventions. They tend to use special characters in their names specifically. OneDrive for Business doesn't like that and barfs when trying to sync such files.

We actually did not try to fully automate the transition but worked with each building site one by one (we have 14).

What I did for some of our more problematic Mac users with file name issues just download a product called Name Mangler to my laptop, mounted their share volume and did pattern searching/replacement of all their file names to fix issues before we migrated that user to OneDrive. You could probably do that on a more mass scale for the whole home folder share if you want to deal with a large group of users.

[EricClarcks]

"Long path tool is the very good program for error, unlock solution. Try it and solve your problem."

[Cornoir]

I have made an Applescript app that will find the illegal characters in a user selected folder and output the files/folders with illegal characters to a text file on the user's desktop. There are a few scripts on the web that make the changes but I wanted the end user to make the changes themselves.

property searchCharacters : "'[<>:|?]*'"
set folderpath to POSIX path of (choose folder with prompt "Chose a Folder")
do shell script "find " & folderpath & " -name " & searchCharacters & " > " & quoted form of POSIX path of (path to desktop as text) & "IllegalCharactersList.txt"

[sanbornc]

Hey Doug-

Would you be willing to share your script? Im having to do the same thing. Let me know!

Thanks!

[dsavageED]

I've got a shell script which will fix illegal characters by turning them into dashes, check for trailing spaces and find long file names. There may be a better way of doing this, but the script has worked for us for a while...

You can have a look here: https://github.com/UoE-macOS/jss/blob/master/utilities-fix-file-names.sh

[KyleEricson]

@DougE Will you please share your script? @dsavageED I want to scan the more folders than Desktop and Downloads is that easy to add to this script?

[JoshRouthier]

I believe Microsoft is/has been working on this problem: https://support.office.com/en-us/article/renaming-items-with-invalid-characters-99333564-c2ed-4e78-8.... I have not tested this yet, so YMMV.

[samspade21]

Thank you for your script examples, we have incorporated some of these suggestions into our MacOS OneDrive KFM tool: https://github.com/synapsepd/MacOS-OneDrive-KFM

[teodle]

Was playing around with this earlier. Microsoft OneDrive Sync Client for Mac doesn't appear to be able to fix filenames yet. Instead it simply errors out and forces the user to find and fix all their own illegal filenames. Remember that total path length of deeply nested files might be problematic as well. I've tested several scripts that were posted here and none of them worked very well. What did work (for file and folder names with illegal characters), is the miracle of zmv. Run this at the top level of your user's OneDrive Folder and it will fix illegal characters. I have it set up to replace them with underscores. This is such a powerful command. Use it at your own risk. First time I botched the regular expression and forgot to "allow" the "." character, so the command replaced all dots with underscores (xx.jpeg became xx_jpeg) and destroyed Finder file associations....you don't want that!!
Here's the example. First you have to switch to zsh, then load zmv. When you run the command from the top level of a user's OneDrive Sync folder it will recursively remove every single illegal character and replace those characters with underscore. It works. And it works fast. I used this command for a case where user has three four, even 5 levels deep with thousands of instances of illegals. OneDrive tells them the names are illegal, but it doesn't really do a very good job of telling them where the files are past the top level. After running this command, sync errors disappeared. Now we just need wrap it up into a script to make it easy for end users to run in Self-Service and test it thoroughly. All I can say is zmv is pretty cool.

zsh

autoload zmv

What is up with display text encoding on the forums? I included the screengrab because if you copy paste the snipped below, it will most certainly return a bad pattern error from zmv

```

zmv '(*/)()' '$1${2//[^A-Za-z0-9.]/_}'
```

[jasonhollis]

This is awesome, and to your point ran across 22,000 files pretty quickly on my SSD, but it replaces all of my spaces with underscores and spaces are not illegal. How do I leave the spaces there and just replace the illegal characters: " * : < > ? / | with a space?

[AdamCraig]

posted twice. deleted

[AdamCraig]

@jasonhollis

Not super familiar with zsh, and haven't tested this yet but...

#!/bin/zsh
loggedInUser=$( echo "show State:/Users/ConsoleUser" | scutil | awk '/Name :/ && ! /loginwindow/ { print $3 }' )

if [[ -z "$loggedInUser" ]] || [[  "$loggedInUser" == 'root' ]] || [[ "$loggedInUser" == "loginwindow" ]] ; then
    loggedInUser="$3"
fi
autoload zmv

oneDriveFolder="/Users/$loggedInUser/OneDrive - CompanyName"

if [[ -d "${oneDriveFolder}" ]] ; then

cd "${oneDriveFolder}"

zmv '(**/)(*)' '$1${2//[^A-Za-z0-9. ]/_}'

else

echo "OneDrive Folder not Found"

fi

[nexus0000]

This script woks well but it doesn't remove the leading or trialing whitespace from files.

I have been working on this for a while but haven't been able to workout how to remove the white space only from the start and end of filenames.

I will update here if I work out how to do it.

Any idea would be great :)

[nexus0000]

so I worked out the leading space....

zmv '(/)()' '$1${2//[^A-Za-z0-9. ]/-}' zmv '(/) ()' '$1$2'

Tailing space is still proving difficult

[teodle]

I've been off jamfnation for a while, and was surprised to see so much activity on this thread. It certainly is an issue that a lot of users who want to start synching legacy data with illegal characters (well, illegal to OneDrive anyway) in files/foldernames, etc. Working within the limitations of OneDrive has been a challenge for these users.

[elijahfrankinso]

Use LongPathTool it will solve your problem of deleting and copying long path files

[rcorbin]

@strayer your script works perfectly. Thanks for that ! It would be just great if it could also do leading or ending spaces. Did you or @Heading ever figure that out ?

[HenryOzsoy]

her you go champ

zmv '(**/)(*)' '$1${${2##[[:space:]]#}%%[[:space:]]#}'

[HenryOzsoy]

#!/bin/zsh
###################################################################
#
# Script to check user's OneDrive folders and files
# for any illegal characters, leading or trailing spaces and
# overlong path names and to correct them to help allow smooth
# synching in OneDrive.
#
# Date: Wed 28 Jul 2021 12:08:33 AEST
# Version: 1
#
##################################################################

loggedInUser=$( ls -l /dev/console | awk '{print $3}' )
	
	if [[ -z "$loggedInUser" ]] || [[  "$loggedInUser" == 'root' ]] || [[ "$loggedInUser" == "loginwindow" ]] ; then
		loggedInUser="$3"
	fi
	autoload zmv
	
	oneDriveFolder=/Users/$loggedInUser/OneDrive\ -\ Oganisation\ Name
	
	if [[ -d "${oneDriveFolder}" ]] ; then
		
		cd "${oneDriveFolder}"
		
		zmv '(**/)(*)' '$1${2//[^A-Za-z0-9. ]/_}'
		zmv '(**/)(*)' '$1${${2##[[:space:]]#}%%[[:space:]]#}'
		
	else
		
		echo "OneDrive Folder not Found"
		
	fi
exit 0

[teodle]

@HenryOzsoy 

That command used to work for me but now I keep getting: 

zmv: error(s) in substitution: 

Only it doesn't tell me what the error(s) actually is (are). 

[HenryOzsoy]

Hi @teodle 

I have this script still working. Please mind it does run in zsh

#!/bin/zsh
###################################################################
#
# Script to check user's OneDrive folders and files
# for any illegal characters, leading or trailing spaces and
# overlong path names and to correct them to help allow smooth
# synching in OneDrive.
# Must run in terminal as: sudo zsh scriptname.sh
# Date: Wed 28 Jul 2021 12:08:33 AEST
# Version: 1
#
###################################################################

loggedInUser=$( ls -l /dev/console | awk '{print $3}' )
	if [[ -z "$loggedInUser" ]] || [[  "$loggedInUser" == 'root' ]] || [[ "$loggedInUser" == "loginwindow" ]] ; then
		loggedInUser="$3"
	fi

autoload zmv

oneDriveFolder=/Users/$loggedInUser/OneDrive\ -\ Your\ Organisation\ Name

	if [[ -d "${oneDriveFolder}" ]] ; then
		cd "${oneDriveFolder}"
		zmv '(**/)(*)' '$1${2//[^A-Za-z0-9. ]/_}'
		zmv '(**/)(*)' '$1${${2##[[:space:]]#}%%[[:space:]]#}'
	else
		echo "OneDrive Folder not Found"	
	fi
exit 0

[winntech]

I'm sorry for the dumb question, I don't know where to add my onedrive folder name and I am having errors like OneDrive Folder not Found. Could you please show me an example assuming my one drive folder name is OneDrive - WinnTech.

Thank you!

[teodle]

you have to escape the spaces that Microsoft OneDrive uses when setting up default folders for each org. In your case, I believe your variable declaration would look like this: 

oneDriveFolder=/Users/$loggedInUser/OneDrive\ -\ WinnTech

[teodle]

I've got users who are trying migrate years worth of Mac file names still--just today I watched a customer move a hierarchy into the OneDrive folder and immediately OneDrive reported 1,677 Sync errors. And the self-service policy that called a script similar to above that was tested and worked fine only a year ago now fails. What did you do, Tim Apple? 

[teodle]

@HenryOzsoy 

I keep getting those substitution errors returned by zmv. My script has run under zsh, just like yours. In fact I copied yours just now and substituted the path for our org's OneDrive folder and it still errors. M1 Mac running Big Sur 11.5.2. Tearing out my hair. Just had to disable the Self Service Policy because it won't work. Can't figure it out. 

[teodle]

Okay everyone, I figured it out. It's not that the commands don't work. It's that the commands won't work on the OneDrive Folder. I cannot figure it out. 

 If I create a folder on my desktop called test,  nest a bunch of illegal file and foldernames there, then modify the variable in the script above so that it points to that folder, the script works perfectly. But when I point it at the OneDrive folder, I always get the following error, and no illegal  file/folder names are changed:

zmv: error(s) in substitution:

file exists: Icon_

I can run the zmv command from within  subdirectories in the OneDrive folder and it does what I expect it to do. 

If I run the command straight from terminal within the top level directory of the OneDrive folder, I get the same error as above. 

This worked last spring. Obviously both Mac OS X and the OneDrive app have been updated since then. 

There's something about the top level of the OneDrive Folder that's stopping zmv from doing what I want it to do. 

[teodle]

SOLVED: zmv won't do anything if there's a file called "Icon_" in the top level directory that you're wanting to do pattern matching/substitution.  Why on earth would it care? It's just a file. I removed that file (have no idea how it got there or what it's for (some sort of legacy resource fork artifact?) and then all my commands and scripts proceeded to execute without error. 

[HenryOzsoy]

you my want to incorporate rm -rf Icon_ after 

cd "${oneDriveFolder}"

as OneDrive creates this file in senarios where you would have signed into a pc, then signed into your OneDrive to sync your profile on the pc. Your correct with this file located in the root folder zmv seems to do nothing.

[teodle]

going to add that to the script. I suppose it would be best practice to test for the existence of  the file, then remove it if it's there. 

Thanks for the explanation of how/why that "Icon_" file is created!!

[teodle]

got it into a policy in our TEST environment and it works great. (as long as the OneDrive folder doesn't have a file named "Icon_")...... KUDOS to @HenryOzsoy for the zmv syntax that strips leading and trailing whitespace. 

KUDOS to @franton earlier for helping tighten up my scripts with regards to notifications.