Very bad Active Directory bug in OSX

[bollman]

So, yesterday we found what must be one of the scariest "bugs" I've yet to come across in OSX.

A user failed to get a proper home folder when logging on to a AD-connected OSX Server and closer examination revealed that his home folder and all contents where owned by another account.
Turns out, Apple translates the 128-bit "GeneratedUID" from AD to a 32-bit "UniqueID" which is used in (at least) the filesystem.
And, it's even worse, they don't translate the entire 128bit UID, but directly uses the first 32 bits. Like this:
My user in our AD:
GeneratedUID: 0D7EB1D5-657B-4D24-B8E4-FB08AF84CEB2
UniqueID (on a mac): 226406869
Proof:
echo "obase=16; 226406869" |bc
D7EB1D5
Which means that there might be more than one user in the AD that has the same UID on a mac! In our AD, 6 users are affected, they share UID on a mac. So far, only one user has encountered problems, but this might change at anytime... Our AD contains 108000 user accounts and grows steadily (university)

And worse: the same principle is seemingly used for all objects the mac looks up in the AD.
Mac OS X only uses the first 32 bits of any obejct and searches the AD and presents the first reply. So, if suddenly a new group in the AD gets the same first 32 bits as the group "Domain Users", files might be owned by any of those 2 groups and there's no way of telling which one the Mac will use, or it might change the next time it contacts the AD. This might of course lead to an amazing array of strange behaviours...

Not confirmed but likely: computers connected to AD might not be able to update it's computer password since OSX might not be able to present itself as the correct computer.

Scariest part: I have yet to figure out a way to fix this. If Apple changes the way UID is calculated from AD, that means that every mac user connected to the AD will get a different UID and GID, and all versions of Mac OS X prior to the fixed version will be in trouble... (and there'll be a lot of work changing ownership on all files on all clients...)
The only way to fix this for now is to use the ability in the AD-plugin to use another attribute as UID for users and GID in groups. That, on the other hand means that we have to construct new UID/GID and populate our AD, and change ownership on every mac connected anyways... not fun.

I've just posted a bugreport and by connections it's on it's way upwards within Apple. For now we are scanning our AD for potential conflicts and will be holding a meeting with the AD-people and management to try to get a plan on the road in case something breaks bad.

[alexjdale]

That's interesting, for sure. I know Apple does read the entire GeneratedUID string (which is used as the uuid for things like FV users and other OS functions), but I suppose since it can't use that as the UID for the user account it has to turn that into an integer that can fit in 32 bits.

I'm actually surprised you have collisions in that space, though.

[bollman]

Yes, some kind of conversion is needed, but the one chosen back in the early days of OSX (I'm pretty sure they've used the same principle for years) might nor be a clever one.
Back then, ADs weren't as big perhaps and this was seen as highly unlikely, but these days, I don't think 100k users in an AD is something completely unheard of.
We have 3 collisions over 108k, that means one collision for about 35k users.

I know the UUID is there. We use mobile accounts and the entire UUID is cached, but everytime it needs a UID, it converts the 128 bits down to 32.

[calumhunter]

Hmmm interesting
Might have to watch out for this
we have about a million users in AD

I can only see this objectGUID/GeneratedUID to UID happening for User accounts - not groups

If i look at the objectGUID in AD for our Domain Users group - i am unable to use the same ibase/obase piped to BC to get the group id as shown by dscl

For example.

gid=686305078(DOMAINDomain Users)

The Domain Users object in AD has a objectGUID of

objectGUID = {a8e82f36-d943-4ed3-a834-bb95110dbf25}

Trying to convert a8e82f36 to decimal fails

echo "ibase=16; a8e82f36" | bc

Similarly if i go the other way trying to convert the gid to hex

echo "obase=16; 686305078" | bc
28E82F36

Searching through AD for an object that contains 28E82F36 i get 0 results

But I was able to confirm that it is the case for User accounts

So I think its not quite as bad as you think

I'll have to make up some kind of script that pulls the first section of the objectGUID and checks for duplicates

[bofh]

I can see the same behaivor on OSX 10.10.
Still the 1st position of the ObjectGUID doesnt match the first position of the UID of OSX (the first 4 bits).

But I also can say, that there is an workaround implemented... you can map the UID to another attribute within the AD.

[mjsanders]

@bollman: are you sure you used the (default) apple connector, not your own objectGUID-uniqueUID translator?
I forgot the details, but I was assured (in the 10.6 days) that the conversion done by the Apple connector was smart enough to avoid duplicate uniqueUID based on the references below.
from this Apple whitepaper

Mappings By default, OS X dynamically generates unique UIDs and GIDs for Active Directory accounts on a system. Ordinarily this is sufficient. However, if there’s a need to manage the UIDs and GIDs, you can map to the appropriate fields in the user record in Active Directory that contain the values.

from Peachpit Directory Services 10.6 book from Arek Dreyer

By default, the Active Directory connector generates a unique user ID, or UID—dsAttrTypeStandard:UniqueID—for an Active Directory user record based on that user’s GUID attribute. The calculated UniqueID is unique across the domain, yet consistent across every Mac OS X computer in the domain. Likewise, the Active Directory connector generates a unique integer for each Active Directory group record as well (dsAttrTypeStandard:PrimaryGroupID). However, if you have extended your Active Directory schema, or if you have appropriate values populated in the RFC2307 attributes (which are already part of the Active Directory schema in domains hosted by Windows Server 2003R2 and later, but are not populated by default), then you can use the Mappings pane to access these attributes.

But this was 10.6, and maybe Apple did not forsee 100K+ users in AD?
There is the Mappings solution mentioned before...

[bofh]

@mjsanders Yes, it's the default. We bind to the Directory while Imaging and have the same "feature".
And there's more potential Duplicates in our environment, as soon is I find some time, I'll play around with it ;-)

[bollman]

All our connections to the AD is done plain standard with the built in connector, no fancy stuff. I've tested on 10.7.5-10.10.5 and they all show these 2 users as having the same UID.
The alternative mapping is of course a workaround, but to convince the AD-people to actually get a new, unique ID into every user in the entire AD, and implement the function to ensure it's "uniqueness", well, I'm pretty sure that's not gonna happen...

In our AD, the GID is created using the first 32 bits of the GeneratedUID from AD:
GeneratedUID: 542C0361-18CC-41CF-B2DA-8C6F54E1635E
PrimaryGroupID: 1412170593

echo "obase=16; 1412170593" | bc
542C0361

So, as far as I can tell, there's huge chance that someday, every mac user will belong to a new group...

[ZachB]

I would suggest installing the extended schema in AD "Unix Attributes". This globally assigns a UID/GID to each user/group. You can then map the UID to uidNumber & GID to gidNumber (if needed) or which ever attribute you want to use, from OS X as @bofh stated. Make sure you set whatever attribute you decide to map to to replicate to global catalog in AD so that the authentication payload returned from AD includes these attributes or you will get some very strange results.

[bradtchapman]

@mjsanders: Nice find. I bought that same book about 5 years ago when I was working on the ACSA—still have it, in fact. Then Apple pulled the plug on that whole certification track, and it made me a sad panda.

I feel like there is a huge divide between pre-Lion Mac sysadmins, and post-Lion Mac sysadmins. I cut my teeth on the Macs between 2007-2010, so I got a lot of exposure to the under-the-hood stuff, but get really frustrated by the dwindling documentation and support for advanced administration tasks from Apple.

[bollman]

@ZachB , well, that can be done, but what will happen to all our current clients? I'm suspecting that if we move to a different UID/GID that will mess up current clients beyond our wildest dreams. I'm not pulling that thread until I've saved up for a loooong vacation ;)

@bradtchapman When I got A/UX 3, I got a huge box full of documentation in pretty binders. When one buys OSXS today, even most of the web documentation is outdated... Even manpages are forgotten about (try man languagesetup)

[bofh]

@bollman This will only affect the local UserID. For Fileservers you are usually using the Kerberos feature -> Files get created modified using your SID.
If you are not using Domain-joined/kerberos, you may have other UIDs on that system anyway.

[bollman]

Ok, I've got reply from Apple engineering:

This is a courtesy email regarding Bug ID# 22557489.

Engineering has the following feedback for you:

We have no way to resolve this and it is a rare situation in our experience. This can be resolved by using UNIX services for Windows and assign a UID/GID to users accordingly.

Engineering has determined that there are no plans to address this issue.

We are now closing this bug report.

Well, perhaps Apple has a different definition of "rare" than most of us? Hm, wonder if I even want to go down the path of making new UID/GIDs? What will happen to computers already in AD? More specifically: what will happen to the local files on AD-connected computers if one changes how UID/GID is calculated?

[bollman]

Funny that Apple refer to UNIX services for Windows. It not available for Windows Server 2012r2, and was deprecated with windows 8.

[Hafiz]

@bollman This adds fuel to the fire that Apple is no longer interested in the Enterprise as it's not shiny and cool.

This discussion in general I am finding hard to follow. We use Centrify Express (the Free Version) in my environment to bind to the domain (we were having horrendous problems at log-in and Macs not booting properly when using Apple's built-in binding utility so I switched the company over to using Centrify Express) and we only have around 700 actual AD Users, as in real users, I don't think we are in any way close to experiencing the problems that you are seeing with possible collisions. However, a friend of mine recommended getting the paid version of Centrify precisely because it adds a plug-in to AD which can modify I think the UID/GID values stored in AD. Guys, is there any good documents or videos in understanding AD UUID/UID/GID values and how they tie into domain binding the Macs with their local UID/GID values? I would like to understand this area better. I think it might be important to buy-into the paid version of Centrify but I need to understand why it's important and the benefits of doing so. After that I can sell the idea of getting the paid version of Centrify to the company once I understand things better. Thank you.

[calumhunter]

FWIW i finally got around to testing this in our environment

We found 274 collisions ..... yet to see anything "in the wild" as not all users will ever use a mac.

But it's not a great result. This is definitely an issue

[eWhizz]

calumhunter, can you share your testing process? Do you have a script that iterates through all your accounts?

[calumhunter]

@eWhizz sorry for the delay, i dont get on jamfnation much and just noticed this

i wrote about it here: https://themacwrangler.wordpress.com/2016/11/29/reversing-the-ad-plugin-uid-algorithm/

hope it helps