Keychain Not Found on Lab Macs

[mark_mahabir]

I'm suddenly seeing the above error on a number (most) of our lab-based iMacs running macOS Big Sur v11.6+ which are bound to AD.

I noticed on the affected systems that our root certificate is no longer trusted in the System Keychain even though it has an expiry date in 2037. However setting this to be trusted again doesn't improve things. We are also seeing similar messages when attempting to launch other applications such as Chrome etc. and Adobe Creative Cloud applications no longer launch, claiming they are unable to contact the Adobe licencing servers. Users don't seem to be able to write to the Desktop either. I've tried removing the computer objects in AD and rebinding, and also removing previously created user directories in /Users and rebooting. 

We are about to begin a trial of Jamf Connect and come away from binding but for now the issue is disrupting our teaching, so any advice around how I might overcome these errors would be greatly appreciated!

[SGill · Accepted solution]

We use SPSS with a license server, so no FUT or FEU needed, but not sure if you have that option on your site.

The User Template has moved to /Library.  Be very sparing about what (if anything) you put in it -- I only have 1 app where I still use that option, but there are other options for this, including the app Outset.  Permissions should generally get set automatically by macOS without modifications, so if a script is modifying them, you might want to take a look at updating or retiring that option.

[SGill]

Do those labs a favor and upgrade them to Monterey 12.1+ as soon as you can.  Its keychain management has been more reliable overall, and we found Big Sur to be somewhat of a mess with many enterprise management aspects.

AD logins have been working as expected with Monterey in general.

[mark_mahabir]

The annoying thing is everything has been working perfectly for about a year. 

[mark_mahabir]

I've just upgraded a couple of machines to macOS Monterey v12.1 and the problem remains after rebinding - numerous keychain related pop-ups after removing everything under /Users.

[mainelysteve]

Haven't bound anything since 2014/15, but back then when I saw the issues you're experiencing it's most likely the home folder doesn't have the proper permissions which in turn means the machine can't properly query AD.

Check the home folder on an effected machine and see if the folders owner is properly populated. Additionally try id username in Terminal to see if it can query any directory db for that user. 

[SGill]

Yes, see if you get expected results back from dsconfigad queries.  Also we run this script as a logout item in the Loginwindow payload.  It's been working well on keychain headaches in our labs for years now (not recommended for single user macs, though, of course):

#!/bin/bash

user=$(ls -la /dev/console | cut -d " " -f 4)
rm -rf /Users/$user/Library/Keychains/*

exit 0

Also, our AD logins aren't mobilized in labs - AD creates network users in labs that require an active connection to our AD --not sure if that could be a factor for you, as well.

[mark_mahabir]

dsconfigad -show and id user both show expected results, home folder permissions look as they should

[tomt]

Do you have the full certificate chain installed and trusted on these machines? That's been the fix for a number of issues with certs in macOS 11. Any chance you have a policy that is creating strange permissions on the keychain files themselves?

[mark_mahabir]

I've tried removing and replacing the root and intermediate certificates, then rebinding. The only policies that I've applied recently have been for WordPress and Office 2019 updates which should be fairly run of the mill?

I did notice that the permissions were 600 on /Users/user/Library/Keychains, rather than 755. Correcting this seems to improve things, but I'm puzzled about what process might have reset them and why they get set to this when the user logs in for the first time.

[mark_mahabir]

OK, so I think I've narrowed it down to a IBM SPSS installation policy where I've used the FUT and FEU options - I'd accidentally included the Keychains folder under /Users/user/Library whilst repackaging in Composer.

There's no /System/Library/User Template/English.lproj/Library/Keychains folder on the affected systems, so how would I go about ensuring folders get set with 755 permissions rather than 600 going forward?

[SGill · Accepted solution]

We use SPSS with a license server, so no FUT or FEU needed, but not sure if you have that option on your site.

The User Template has moved to /Library.  Be very sparing about what (if anything) you put in it -- I only have 1 app where I still use that option, but there are other options for this, including the app Outset.  Permissions should generally get set automatically by macOS without modifications, so if a script is modifying them, you might want to take a look at updating or retiring that option.

[mark_mahabir · Accepted solution]

Many thanks for all the suggestions 👍

It was indeed my repackaging of IBM SPSS Statistics in Composer which caused the issues. I'd left in various directories such as Group Containers, Photos and Keychains under /Library/User Template/Non_localized/Library, so I scripted removal of these and everything worked fine after the script applied (on 12.1, not on 11.6.x though for whatever reason) and the Macs were rebooted.

The silver lining was that I was able to upgrade to macOS Monterey a few months earlier than scheduled - we usually upgrade our labs in the summer vacation period.