User Keychain Destroyed after AD Password Update

[dfeulner]

We have reoccurring issues where most users can't successfully update their keychain after their AD password has been updated. You've probably heard this one before, so let me explain a few twists in my circumstances.

1. Our AD passwords are managed by a third party Identity Management Solution, so we can not have users change their password through built-in macOS features. They have to do it through a web portal.
2. We have user certificates in the user keychain that are required for connecting to the company network including VPN. These are deployed by configuration profiles.
3. The whole Mac workforce is now remote due to social distancing. They need VPN for everything.

Whenever a user changes their password, at some random point they will be prompted to update their keychain. I've instructed users to always, always, always select "update" and not "create new", however, it's very rare a user is able to update their keychain on their own successfully. Many users swear up and own that they had my instructions right in front of them as they attempted to update their keychain, but that selecting "update" instead of "create new" did not help. Although it's still possibly a user training issue, I've completely failed over the past several years to train anyone on how to avoid this problem, so I don't think more user training is the way forward.

The end result is that every time a user must change their password, within a few days their user keychain is wiped, and they can no longer connect to VPN. I have to have them temporarily connect to a VPN without certificate-based authentication. I then remove the MDM profiles, and re-enroll the machine so that the user configuration profiles come down again. This whole process typically takes one to two hours per user, and I'm the only one who understands JAMF enough to handle the tickets. This has become close to a full time job and is clearly unsustainable.

Does anyone have any advice? My ideal solution is to prevent macOS from ever creating a new user keychain.

[MTFIDjamf]

You may be seeing the issues called out in the links below.

We have to instruct all Mac users to change their PW from the Mac itself or they are struck with this bug. If they change it from a Windows device, or our internal change PW web portal, or a secondary Mac, they end up with the blank keychain.

We have had some success with the workaround in the first link.

10.14.4 - First Time Broken
10.15.1 - Broken Again!

[dfeulner]

@MTFMRCO Thanks for your response. That would explain why some of the users are having new keychains created even though they click the "Update" button. Unfortunately, the workarounds aren't really practical at the scale of hundreds of users, and doesn't address the root issue. It's good to know my users aren't crazy, though!

[dmichels]

I feel your pain especially with working remotely. When my user's corrupt their Keychain after changing AD password. I put a link in /Applications/Self Service called Fix Keychain. It deletes user's keychain and requires a reboot which on next log in creates a new one based on the new password. Please TEST, TEST, TEST but this works for me.

Keychain Fix SCRIPT

!/bin/bash

Deletes the user keychain folder.

user=$(ls -la /dev/console | cut -d " " -f 4)
rm -rf /Users/$user/Library/Keychains/*

exit 0

[dfeulner]

@dmichels Thanks for your response. I used to have an almost identical script available in our Self Service. However, now we use certificates to authenticate on VPN, so deleting the Keychain means deleting the certificate. Since VPN is the only method users have to connect to JAMF in our current environment, deleting the keychain actually creates the problem I'm describing. What I'm looking for is a way to prevent users from creating new keychains, inadvertent or otherwise.

[MTFIDjamf]

@dfeulner According to Apple, they fixed this whole thing (again) in 10.15.4. As we have not been able to get on-prem since Mid-March, I have been unable to confirm as we are holding the 10.15.4 update and nobody wants to be the poor out of luck remote tester that forces a PW change and then ends up running 10.15.4 but still loses their keychain (our setup uses user-based certs for network Wi-Fi, VPN, and some internal site pass-through).

[LeidenUniv]

We've had several user that changed their password in 10.15.4.
I can confirm the keychain stays, the problem has indeed been fixed (for now)

[dfeulner]

@MTFMRCO @LeidenUniv It's nice to know the future may be a bit more rosy, but unfortunately we have only a few users currently on Catalina. There are compatibility issues with some of our security applications on Catalina which has been holding us back. Also, I'm not particularly comfortable with upgrading users remotely which limits our options. Thanks for the information though.

[AdamCraig]

I have this as a Self Service policy to run after the computer restarts with the broken keychain. It replaces the new empty keychain with the old one with the user's cert in it.
https://github.com/theadamcraig/jamf-scripts/blob/master/Troubleshooting%20-update_keychain_password...

you do need to pass it a local admin's username and password as variable $4 and $5

[Chris_Hafner]

This is one of the things that I love about Enterprise Connect. Sorts that out completely. Are you moving to Apple SSO?

[dfeulner]

@strayer Thanks, this is the closest thing to a workable solution I've seen so far. I have two main concerns about it however. Once the user has a new keychain created, they are no longer able to connect to VPN, so they would not have access to Self Service at that time. (We're discussing a cloud implementation, but that's a ways off.) If we can find a way for a user to use this script while offline, that could work. My other concern would be keeping a copy of the local admin password in a plain text document that we'd also expect the user to run. Can you think of a way we could accomplish this offline while also preserving our security?

[dfeulner]

@Chris_Hafner Thanks for your suggestion. We actually do use Enterprise Connect in a different part of the business that doesn't use the same password management tool as us. Previously, we had not implemented it because it didn't appear to function well with third party password management tools such as ours. I have considered reevaluating this, however, the password change functionality is currently broken for the side of the business that users Enterprise Connect. We have a ticket in with Apple, but we've had trouble getting support. This doesn't exactly fill me with confidence, but once it's up and running again for the other side of the business I'm willing to test it once more.

[AdamCraig]

@dfeulner that is a pickle. We use jamfcloud. I would definitely not store this policy offline.
There are some options for encrypting the passwords within the script that I have not had great luck with. But even then if the script and all the pieces are stored locally then you're just making it a little more work to get the admin password.

However the only thing the admin password is used for is to updating the filevault password (in my experience is the keychain broke then filevault probably broke too) So you could remove that chunk from the end of the script (roughly lines 164 through 188) and then you won't need the admin username or password, and you could store it locally because it just uses the user credentials.

[dfeulner]

@strayer Although we do have similar Filevault issues, I don't consider it as severe as the keychain issue. Also, I'm a bit reluctant to remove and re-add Filevault users when they don't have an option to return to the office. I'll remove those pieces and experiment with using the script to resolve the keychain issue only. I'm thinking maybe if I can place the script in one of the $PATH folders, then I can instruct users to open terminal and type a simple command. Do you think this would work run as a user without admin access? I could probably simplify the $loggedInUser bits too.

[gabester]

I’m having a similar issue for users. I wish my IT team would allow us a VPN that didn’t require a cert!

Our environment made changing passwords challenging for a variety of reasons not the least of which is the seeming impossibility for users to correctly answer the “update keychain password” prompt at next logon - but I see that mrmacintosh page clearly indicates this is a known issue for mobile account users; so I've submitted a bug report to Apple. If they can issue 1.5+GB security updates for Mojave they can fix this one simple little bug. One challenge was that all to frequently users could change their mobile account’s password while offline... and there was no easy way to get things synced up then. To improve things we trained users to only change their passwords via a Self Service policy that launched the method by which they were supposed to change their password, thus ensuring that they would have network connectivity as it was required for Self Service to reach the JSS. I don’t think it would be too hard to add a script step to that policy to export the user’s cert from keychain so they could more easily recover. Could you export your users’ certs before or soon after they change their password?

@strayer When you say your certs "cannot" be exported is that a limitation of technology - they're simply not exportable - or is it a policy, they are NOT to be exported?

[AdamCraig]

@Sterritt Our certs cannot be exported from the keychain. @dfeulner I wrote the script to be run as root via a Self Service policy. But without testing (too sucked into a lot of projects to jump down the rabbit hole) I'm like 80% sure it could be re-worked and run from non-admin.

[dfeulner]

@strayer I've been having some success with your script. I removed the FileVault stuff, but I was getting errors with the osascipt calls to System Events. I removed all the osascript stuff and made it a pure terminal script. It can be a bit of a hassle to walk users through launching it, but it works! I will test with a few more users, but I'm going to look into deploying to all computers so its available offline moving forward. I'm undecided on where I should stage the file, but I'm leaning towards /usr/local/bin. I'll let you know how it goes once I start deploying.