mounted EFI partition after imaging causing FileVault kernel panics

[AVmcclint]

I've been dealing with this for a long time and I'm still struggling to find a cure.

After imaging all brand new MacBook Pros with a fresh 10.10.x image when I login to the computers with the admin account 9 times out of 10 the EFI partition is automatically mounted on the desktop. SOMETIMES repeated reboots will make it go away, but it will come back at some randomly selected boot up at a later time. After many months of troubleshooting, I have linked the presence of the EFI partition at startup to random Kernel Panics caused by FileVault. How do I know it's FileVault? Any and everything relating to FileVault can cause it to crash - including fdesetup commands, the simple act of enabling FileVault, or even clicking on the FileVault tab of the Security preference pane. There are other events that link it to FileVault. if I completely disable FileVault, I am guaranteed to never have a kernel panic. If I re-enable it, the kernel panics return. At first I thought FileVault was causing the EFI partition to be mounted somehow at bootup, but I removed the enabling of FileVault from the imaging and enrollment process. The EFI partition is present right after imaging. Every so often the EFI partition won't automatically mount and the computer remains completely stable for the duration, but if they reboot, there is a chance that it will come back. I have had SOME success in using Create Recovery Partition Installer.app with preventing the kernel panics but they do eventually return. The problem with that tool is that if the computer is in an unstable state, installing a new recovery partition will trigger a kernel panic. if I can boot and reboot and reboot over and over until I happen to catch it on a boot without the EFI partition mounting, then I can safely run it and it will work for weeks or even months, but the kernel panics WILL return.

I found this post that suggests running the recon command, but that has no effect on the EFI partition.

*further details: we are running JSS 9.65, the problems have persisted since 10.9.x, the kernel panics only happen on Macs with SSDs (we have slowly transitioned from HDD Macs to SSD Macs and don't have many left), HDD Macs occasionally mounted the EFI partition, but even when they did, they NEVER crashed. I've made and remade the images from the brand newest Macs that we got with all the updates etc. I've made sure that all the Macs have all the latest firmware/EFI/storage updates applied. I have noticed that the contents of the EFI/APPLE/ folders sometimes vary from mac to mac. Sometimes they are empty, sometimes they are filled with what look like firmware updates. However, since the EFI partition is read/only, I can't do anything with them.

I want to call Apple about this, but I can't get my hands on a Mac that is 100% reproducible. What usually happens is that I'll reboot a couple or few dozen times and the EFI partition will stay hidden and it'll stabilize. If the EFI partition auto-mounts and I just unmount it, it won't make a difference. The conditions exist that made it mount in the first place and the crashing will happen eventually.

Has anyone else dealt with this or anything even close? Every search I've done for FileVault kernel panics either go completely unsolved or promise "it'll be fixed in the next update" going back to Lion. The only true fix is to disable FileVault which we can't do because of HIPAA requirements. Every search I've done for EFI partition doesn't cover anything about this at all.

[perrycj]

How exactly are you making your base OS image? Or are you just using the OS that ships with each Mac?

[cdev]

I'd also be curious to see the output results when you run the command: diskutil list

diskutil list
/dev/disk0
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *500.3 GB   disk0
   1:                        EFI EFI                     209.7 MB   disk0s1
   2:          Apple_CoreStorage                         499.4 GB   disk0s2
   3:                 Apple_Boot Recovery HD             650.0 MB   disk0s3
/dev/disk1
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:                  Apple_HFS Macintosh HD           *499.1 GB   disk1
                                 Logical Volume on disk0s2

[AVmcclint]

I'm making the base image by taking a brand new Mac out of the box, creating an admin account, run software update until there are no more updates to get, set ARD and SSH, boot up from an external drive that has all my Casper tools on it, use Composer to make a new OS image of the Macintosh HD, copy the resulting .dmg to the JSS via Casper Admin. If I then take that image and put it right back on the same machine I created it from it will display the EFI partition when there was none displayed before. I've tried with and without "Erase target drive" when putting the image on the machine.

I'l see if I can get an output of diskutil list and paste here for inspection.

[AVmcclint]

This is the output from a Mac that has the EFI partition automatically mounted on the desktop. I ran the command on a Mac that just happens to not have the EFI partition mounted at the moment and it is identical with the exception of the UUID

Last login: Fri Aug 28 07:37:28 on console
$ diskutil list
/dev/disk0
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *500.3 GB   disk0
   1:                        EFI EFI                     209.7 MB   disk0s1
   2:          Apple_CoreStorage                         474.6 GB   disk0s2
   3:                 Apple_Boot Recovery HD             25.1 GB    disk0s3
/dev/disk1
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:                  Apple_HFS Macintosh HD           *474.2 GB   disk1
                                 Logical Volume on disk0s2
                                 64F89905-7698-4D95-A456-FBB4320484B8
                                 Unlocked Encrypted

[perrycj]

If I had to guess, I would say it's something with your OS image creation. Look into AutoDMG and create an OS image using that. It's a very easy but very useful tool to use and creates a never-booted, clean OS image from an Apple installer. Try it out: AutoDMG

[AVmcclint]

Here's a screen shot of the contents of the EFI partition on this particular Mac. The exact contents vary from computer to computer. Some will have all the files listed here, others may have just the FIRMWARE folder with something in it, others may only have an EXTENSIONS folder. etc.

For some brand new machines when I took them out of the box, I went ahead and ran software update on them BEFORE imaging to make sure that any and all firmware updates were applied AND VERIFIED. Some still end up with the EFI partition mounting with various contents. Just for the hell of it, I downloaded the firmware updates from Apple's website and applying them manually. Some updates said they weren't needed because it was already updated. One update told me that I needed to be running OSX 10.9.5 to install it. Regardless, According to System Profiler, all updates are applied and Software Update finds no more to get.

[mjsanders]

About the content of the EFI partition I discovered that this is usually the content:

-On each EFI hardware there should be an EFI partition (required by EFI specs)
-Sizes are between 209 and 300+ MB on newer macs.
-Each vendor is allowed to use it as it likes.
-PC vendors typically store some utilities on it, Apple seems to use it (temporarily) for firmware and smc updates, and usage varies with different hardware.

-After a clean install the EFI partition is usually empty.
-OS updates like 10.10.2 (thunderstrike), 10.10.4 and probably others too leave firmware updates behind on applicable Macs (like your example), the same OS will not put anything in EFI on an iMac 2007. See Allister Banks blog post on afp548

-on 2013+ Mac's with bootcamp and windows8 or 10, Windows will use UEFI boot, and there will be a /EFI/Boot and /EFI/Microsoft folder with many files.

for everyone: mounting the EFI is simple:

diskutil mount disk0s1

( 1 being the usual partition, but it may vary on your system)

[AVmcclint]

So why would the EFI partition be populated with items if I've already applied all thefirmware updates to the Macs? Imaging and re-imaging and re-imaging again doesn't always remedy the problem. Sometimes troublesome Macs will stabilize after 5 or 6 re-imaging sessions. When I use Composer to create the image, is it somehow also grabbing the EFI partition of the "master mac" that I created the image on?

[mjsanders]

@AVmcclint: It is only Apple who can answer the 'why' questions (:-), but to me it looks like leftovers from firmware updates. (which could/should be removed after applying the updates imho)
In the case of Bootcamp with UEFI boot the files are used...
Most imaging tools I know do NOT capture EFI partitions, and I guess Composer does not too (since the content is not required by Mac's, and as we found out the content varies by model this makes sense)
I have not checked if composer grabs the EFI however.

[merps]

I second @perrycj suggestion and recommend going with AutoDMG to create the base installer. If you want to have ARD & SSH enabled by default, you can do so with a script run during Casper Imaging.

AutoDMG will also download the available updates when you create the base installer. It's a great tool & has cleaned up all of the issues we've had regarding OS deployment.

Edit: To create your local admin account, you can use CreateUserPkg. Works great.

[AVmcclint]

More info: Today I decided to take a Mac fresh out of the box, setup the admin account and then make an image of it with Casper Composer 9.65. Then I took the resulting .dmg file and restored it to another Mac of the exact same model... however I did not use Casper Imaging to do it. I used Disk Utility to restore the .dmg onto the drive of the other computer. After the restore completed, I booted up that 2nd computer from the freshly laid down system. Then I ran the RecoveryPartition installer to make sure that the system had a recovery partition created of the same OS version (10.10.5). After that I booted and rebooted it several times just to make sure there was no EFI partition mounted on the desktop. After about 15 reboots with no EFI automounting, I decided to netboot it and launch Casper Imaging. I picked the standard image set I've been using with 10.10.5 with the exception that I did NOT choose to erase the drive and I removed the OS .dmg from the set. Basically it is just installing some apps (MS Office, VLC, TextWranger, and a few others like that) onto the already proven working OS and enrolling into JSS - that's it. After it finished installing the apps and rebooted, I logged in with the admin account and sure enough... there's the &$(@$#! EFI partition mounted on the desktop. I go poking around the EFI partition to see what's there and I see a file in /EFI/APPLE/CACHES/ called "CAFEBEEF" Whaaa??? I booted and rebooted about a dozen times and the EFI partition always automounted. And while the EFI drive was automounted, any and everything I did that related to FileVault caused a kernel panic even clicking on the FileVault tab in the Security preference pane crashed it.

I've managed to confirm that this weirdness is being generated at some point in Casper Imaging, but I'll be damned if I can figure out what specifically is doing it.

Just for the heck of it, I made a new image with AutoDMG and I got the same results after imaging.

[mpermann]

@AVmcclint do you mind posting a sanitized screen shot of your Casper Imaging workflow so we can take a look at what is being done?

[AVmcclint]

[mpermann]

@AVmcclint what is the unmount_recovery_part.sh doing? If you are using an AutoDMG created OS, which I am not certain that is what you are doing, there is no reason to do anything with the recovery partition. If you aren't using an AutoDMG created OS, would you mind posting a sanitized version of that script so we can take a look at it? Are you doing any custom partitioning that you aren't showing in the screen shot? You can click on Show Custom and click the advanced tab to check that out. Also, have you tried using the latest version of Casper Imaging (9.73) with your existing workflow. I believe the version of Casper Imaging doesn't necessarily have to match the JSS version. I know they used to say it did, but I think some things have changed in that regards.

[AVmcclint]

The unmount_recovery_part.sh is a script that my predecessor put in place and it didn't seem to cause problems under 10.9 so I left it alone. The script in full is:

#!/bin/sh
/usr/sbin/diskutil unmount /dev/disk0s3
/usr/sbin/asr adjust -target /dev/disk0s3 -settype Apple_Boot

I'll take that out of the flow and see if that helps anything.
Not doing any custom partitioning. I've tried using the newer 9.7x Casper tools with our older JSS but it throws up an error telling me I can't use it because the versions are different. I held off on updating to 9.7x when I read all the problems that a lot of people were having. I'm hoping 9.8x fixes those problems so I can update soon.

[mpermann]

@AVmcclint if you use AutoDMG to create a new Base 10.10.5 image you shouldn't need that script in your workflow. If you don't want to use the Accounts tab located under the Show Custom button to create your local admin account you can use CreateUserPKG to create the account. The only other thing in your workflow that might be something to look at would be your McAfee Agent. If you used the Composer application to make that DMG, you might want to consider taking a look at this knowledgeable article. Many times antivirus clients require some special handling when installing. None of your other packages struck me as anything that might potentially be causing this issue. Our workflow is a bit different from yours. See attached image. We typically use the manufacturers .pkg files and install them at the first boot. We don't use many Composer .dmg files unless we need to put down some plist files or some other user customization. We're currently using 9.72 but will be upgrading to 9.73 soon.

[AVmcclint]

When I mentioned earlier that I tried an AutoDMG image, I was in error. Today I tried it and I hated that I still had to set EVERYTHING up. I know some of it can be scripted, but it is just infinitely faster for me to create the admin account and set everything up from the very beginning on a master image and be done with it, but besides that... When the computer rebooted after the imaging process, the enrollment process and post-enrollment policies never ran. I'm sure AutoDMG is a great tool. I'm just not seeing it useful for us here. But the good news is that the EFI partition didn't automount.

I went back to the old imaging workflow and removed the unmount_recovery_part.sh script from the imaging process but now the Recovery HD is automounted on the desktop ... along with the EFI partition. However, I am beginning to suspect something in the post-imaging. McAfee is always suspect but I removed that from the process months ago when I first started troubleshooting this with no change in symptoms. I'm going to have to tear this down one. policy. at. a. time.

[mpermann]

@AVmcclint what kinds of things are you setting up in your local admin account that you're having to manually setup when you tried the AutoDMG created OS? Are your end users logging in with an AD account on the computers? Our normal workflow includes creating a single admin level user account that the end user uses. The only other account on the system is the hidden casper admin account which generally only gets used by Casper. If we have an end user that is having problems we do all of our diagnostic and troubleshooting from their account. That way we know whatever problem they are having is resolved in their account when we are finished. So I guess I'm just wondering what kinds of things you are using the local admin account for.

[AVmcclint]

FOLLOWUP: This is a rundown of the Kernel Panic problem that has affected many of the Macs. So far my fix is holding up well. I hope this info can help anyone else experiencing this same problem.
... Symptoms:
- When the EFI partition automounts at startup, there is a 75% chance the computer will kernel panic
- when the EFI partition does NOT automount, there is maybe less than 5% chance that the mac will kernel panic
- the appearance of the EFI partition is more common on some macs than others. Some may see it once in 3 months. Others see it every day.
- the kernel panics can usually be triggered by putting the computer to sleep and then waking it up and entering the PW to unlock it.
- Zapping the PRAM seems to help on some computers but only for a few days or weeks
- on rare occasions, kernel panics can happen while running any app or while doing absolutely nothing but sitting at the desktop
- zero percent of the Macs with spinning platter HDDs ever experience the kernel panics but they do still occasionally see the EFI partition.
- manually unmounting the EFI partition does not change anything. Once the conditions are present that make the partition automount in the first place, the potential for kernel panics still exists
- 100% of the crashes seen have only happened on Macs with SSDs.
- OSX 10.9 and 10.10 are both affected

I still do not know the underlying cause, but my working theory (after an exhaustive investigation) is that it has something in the way the partitions were created and the images put down with Casper 9.65. I have not had a chance to build a new imaging process with Casper 9.81 yet to see if the problem persists.
Applying the fix: (There may be ways of doing this purely in the command line but this is easier for me)
- completely decrypt FileVault
- reboot
- verify the drive is decrypted
- you can launch Terminal and run "diskutil list" to see that EFI, Macintosh HD, and Recovery HD are all present
- boot up from a Netboot image that has a full OSX install (running 10.9 or 10.10. The Disk Utility in 10.11 does not look like it would work) so I can access Terminal and Disk Utility. (although theoretically, booting from an OSX install USB could provide the same functionality)
- while booted from Netboot image, launch Terminal and run:

defaults write com.apple.DiskUtility DUDebugMenuEnabled 1

- Launch Disk Utility and click on the Debug menu and choose "Show hidden partitions" (or something to that effect)
- click on the EFI partition on the internal drive and choose to erase it. you'll get an error, that's ok
- click on the Recovery HD partition and choose to erase it. you'll get an error, that's ok
- (those 2 previous steps may seem unnecessary, but you can't proceed unless you do them)
- Click on the drive and then click on the Partition tab
- Click on the EFI partition in the visual map of the partitions, then click the [-] button to delete.
- scroll down the visual map of the partitions until you see the Recovery HD partition. Click on that then click the [-] button to delete
- click Apply
- When finished, restart from the internal drive.
- launch Terminal and run: "diskutil list" to verify that the EFI and Recovery partitions are gone.
- This next steps requires a pre-built installer package made using "Create Recovery Partition Installer" https://github.com/MagerValp/Create-Recovery-Partition-Installer
- I make sure I build the pkg with the same version that is on the target Mac I'm trying to fix.
- I copy the pkg to the Mac and run it. it will then create a proper Recovery HD partition.
- run "diskutil list" in Terminal to verify that it worked. (you will not see EFI here but it will be created automatically after FileVault is re-enabled)
- enable FileVault and reboot.
Start to finish takes about 2 hours because of the length of time it takes to decrypt and encrypt the drive. YMMV.

So far i've run this on 8 Macs and none of them have crashed since. I can't say with 100% certainty that this is the cure because historically, some macs have gone months without crashing at all. The ones I've applied this to have only been crash-free for about a month or so, but it looks a lot better and so far the EFI partition has not automounted even once on those macs.

[AVmcclint]

I think maybe, perhaps, there is a chance I may have found the underlying cause of the kernel panics. We have a Config Profile that limits the read/write to USB drives. One of the settings for Internal Disks was checked for Allow and Read-Only. I'm not sure why that was selected since all our macs are either laptops or iMacs - all with single drives and no extra drives installed.

One day recently I was going over every setting in our config profiles to see if I could clean up or optimize things and I saw that setting and I decided to clone that Profile and uncheck the box for Read-Only on the internal drives. I pushed it out to a handful of Macs. I rebooted a few Macs that were having the problem and I discovered that the EFI partition stopped mounting at login. I rebooted the computers several times just to make sure... the EFI partition still wasn't mounting! I went back to the JSS and removed that fixed profile from these Macs and re-pushed the active profile with the setting enabled and the EFI partition returned! I rebooted several machines a couple dozen times to make sure and it was mounted 90% of the time. I removed the Profile and re-pushed the fixed profile to those same machines and EFI NEVER returned.

I unchecked the Read-Only box on the Profile that all the Macs get and one by one as the Macs rebooted, the EFI partition disappeared from desktops. Unfortunately I think the damage was done to a lot of these computers because a few continued to kernel panic, so I fixed them according to the steps described above. For Macs I've imaged since using this corrected Config Profile, they have never once displayed the EFI partition nor have they kernel panicked. I'm keeping my finders crossed.