2018-001 & Safari Update causing crashes on 10.12.6

[jwojda]

As an FYI - we have Apple auto-updating and they released the specter/meltdown fixes. I've started getting a few people bringing me machines that are now crashing on boot up after installing, incl rebuilds that ran the updates.

Anybody else seeing similar behavior?

[ShadowGT]

Same here, but its due to our DLP software that has its digital hooks in the kernel. The 2018-001 update must of changed the kernel once again. So any DLP or tough Virus/Malware software that uses kernel hooks can cause the system to crash. We ultimately have to uninstall prior or go into safemode and remove after.

SMH. Is this installing automatically on your side? Because I have auto install security updates turned off in my enviroment.

-Frank J

[ShadowGT]

If it becomes epidemic, what I would suggest ignoring it with the softwareupdate command:

softwareupdate --ignore "macOS Security Update 2018-001-10.12.6"

But if you do so, I believe you will have to deploy manually as the softwareupdate will ignore this update.

[adam_mcdowell]

I can confirm we are also experiencing the same issues for users on 10.12.6
Yet to test on 10.13.3 ( High Sierra )

[mtapal]

We have 8 users experiencing the same issues.

Any fixes?

[jwojda]

@mtapal we reinstalled macOS from the recovery partition.

[pcrandom]

@ShadowGT, they'll actually want to run softwareupdate --ignore "Security Update 2018-001". Per the man page for softwareupdate:

--ignore identifier ... Manages the per-machine list of ignored updates. The identifier is the first part of the item name (before the dash and version number) that is shown by --list. See EXAMPLES.

You can test it by running softwareupdate --list command twice, once before the ignore command and once after, and you should see the update not appear on the second run if the ignore was successful.

The nice thing is that then this ignore command covers both Sierra and El Capitan, since the update identifier is the same, without the OS version number after the last dash.

(Anyone else having trouble editing comments? Like it doesn't keep the original text and instead brings up a blank field and I have to copy and paste the original text, reformat it, and add the edit.)

[pcrandom]

Any commonalities? We have Macs that have installed this that hasn't had problems, and Macs that have. Any anti-malware software installed? Any kernel extensions mentioned in the kernel panic message (if you're getting kernel panics)?

[dgreening]

Any additional info (specifics from kernel panic messages) is much appreciated. I ran the updates on my 10.12.6 MBP, am now on build 16G1212, have Symantec Endpoint Protection.app version 14.0.2332.0100, and haven't had any issues so far.

[rqomsiya]

Does anyone know if this affects 10.11.6 clients that have the 2018-001/Safari 11.0.3 patch?

Thanks!

[mcampbel]

I've had six 10.12.6 MBP's run the update, all 6 had the issue. One user claims the issue fixed itself after a couple of reboots, two others had to have 10.12.6 reinstalled. The other three we have yet to try the reinstall. We were not running DLP on these machines but run a lot of other security agents that might be conflicting. I'll do more testing tomorrow.

[PatrickD]

We were running an older version of Trend AV 3.0.1098, updated to 3.0.3044 and the issue is resolved for us. Likely issue was incompatible kernel extensions. I have tested successfully on two seperate 10.12.6 machines.

Who is using what AV and what version?

Try updating to latest available version.

This is very similar to Microsoft when they released their Meltdown patch which caused BSODs because of incompatible AVs.

[sahmed]

I had 2 users on 10.11.6 who ran this update and doing a kernel panic on their MacBook Pro

[mcampbel]

I'm running Trend 3.0.1106. I'll play with that tomorrow.

[sahmed]

we are using Sophos Antivirus ver. 9.6.2, also Carbon Black. going to uninstall it and see if it fixes it.

[sahmed]

@pcrandom I am having the same issue as well for editing comments

[mcampbel]

Upgrading Trend to 3.0.3044 didn't work for me :(

[PatrickD]

Hey @mcampbel ,

How did you upgrade to 3.0.3044? did you install over the top or did you run Trend's specific uninstaller and then run the installer? I have had issue with just trying to install over the top.

Cheers,

Pat

[mcampbel]

@PatrickD , I "checked for updates" and let it update itself. Nothing's ever easy I guess. Thanks Pat, I really appreciate the help.

[rqomsiya]

For those running SEP14:

10.12.6- no kernel panics to report (installed security update and Safari 11.0.3)
10.11.6- will test to tomorrow

[daz_wallace]

To add a further point, both 10.11 and 10.12 patches change some things in the Kernel (required to patch the Meltdown vulnerability), so possibly causes could be anything that has a kernel extension. This would include AV, but could also include:
- Carbon Black and other deep level security solutions
- Some printer drivers (yea I know, right?)
- Data Loss Prevention software (an example is EndPoint Protector (EPP), but I don't know if that specifically causes, or indeed doesn't cause, an issue).

Safe Boot (hold down "Shift" on startup) should boot without third party Kernel Extensions, but may take longer to boot too. That might get you in enough to start troubleshooting causes

Good Luck, and I hope that helps

Darren

[jwojda]

@daz_wallace I can confirm carbon black's Bit9 v7 patch 8 is causing problems - and that version was released a week or two ago.

[daz_wallace]

@jwojda That seemed to be the common feeling on the MacAdmins Slack channels too. Fun Times

[mojo21221]

@daz_wallace and @jwojda I can also confirm the Carbon Back agent is causing the crash 10.12.6 latest security patch. Currently testing 10.13.3 to see if it has the issue as well.

[StoneMagnet]

@pcrandom and @sahmed The editing bug for Jamf forum posts is a known issue, and hopefully will be fixed when they next update the forum software (tentatively next week)

[jmahlman]

@rqomsiya Oh that is a relief. I accidentally left my test machines set to install updates overnight and forgot to remove the security updates from our SUS.

I'm hoping the machines that got the update don't crap out...it was only a few thankfully.

[rlee]

We're running Sophos AV 9.6.6 and Ensilo 2.0.1.21 here and seeing the issue. We do not use any Carbon Black products here.

Will test to see if it's Ensilo.

@sahmed Did you uninstall Sophos - if so, any luck?

[garryjoshi]

I am running 10.11.6 as well and encountering this same crashing error. Anyone able to resolve the issue?

Update: Reinstalling MacOS just got this error fixed. Thanks to rlee for the solution.

Garry Joshi https://dltutuapp.com/ https://show-box.ooo/ https://tutuappx.com/

[rlee]

Forgot to mention that re-installing MacOS from the Recovery partition worked for me.

Give that a shot @garryjoshi

[JPDyson]

Let's all open cases w/the vendors for confirming this compatibility. I'm hoping the message will be clear - we expect this kind of validation during betas if possible, or day zero otherwise.

[mojo21221]

Carbon black has informed us that a patch is a week out. Their recommendation was to block the Apple update. Another solution is safe boot the mac (Hold Shift on startup), log in as an admin, then Uninstall the CB agent via terminal (sudo sh /Applications/CarbonBlack/sensoruninst.sh). Reboot and your Mac's Kernal will be happy again.

[Jessy111]

We have Carbon Black running and that caused the crashes here

Boot in Safe Mode and delete the Carbon Black .kexts in /Library/Extensions

[adam007]

Removing Carbon Black was also the solution for me. Thanks to @daz_wallace, @jwojda, and @mojo21221 ! Mac updated to 10.12.6 with the 2018-001 update for Sierra, rebooted, and could not boot up normally. I was able to get into safe mode, then remove CB using Terminal and Admin account.

Of note: simply trashing the Carbon Black folder does not solve the issue. If you have done this, reinstall CB, then use @mojo21221's solution: Uninstall the CB agent via terminal (sudo sh /Applications/CarbonBlack/sensoruninst.sh) and reboot.

[mcampbel]

For me I had to uninstall Carbon Black and Trend to get the machine running again. I have not tried updating Trend to a newer version yet.

[pcrandom]

We couldn't boot into Safe Mode or Single User Mode on nearly all of the systems affected by this, perhaps because we have both Carbon Black Response and Carbon Black Protect installed on our Macs. I figured out the files that need to be removed before the Mac could boot normally:

For Response, I deleted:

/Library/Extensions/CbOsxSensorNetmon.kext
/Library/Extensions/CbOsxSensorProcmon.kext
/Library/LaunchDaemons/com.carbonblack.daemon.plist
/Applications/CarbonBlack/CbOsxSensorService

For Protect, I deleted:

/Library/Extensions/b9kernel.kext
/Library/LaunchDaemons/com.bit9.Daemon.plist
/Applications/Bit9/Daemons/b9daemon

I either booted the affected Mac to Target Disk Mode and connected it to a working Mac, and used the Finder to delete the files, or I booted into the Recovery partition and use Terminal to delete them (remembering to target "/Volume/Macintosh HD" in the commands). After removing those files, the Mac should be able to start up without kernel panicking, and the uninstall scripts for each product still remained, which I then ran in Terminal:

sudo /Applications/CarbonBlack/sensoruninst.sh
sudo /Applications/Bit9/uninstall.sh

This worked on all affected Macs that I had yesterday.

[jconte]

I am seeing this on 10.11 and 10.12, kernel panics after patch and restart. Booted in Safe Mode, removed Carbon Black, machine works again.

[AVmcclint]

I've installed it on about 90 Macs so far and none of them have crashed. We're running McAfee ePO 10.2.2, and all computers are FileVaulted.

[cizdziem]

Just FYI, the only anti-virus/malware software my organization uses is Sophos. Our Sophos Central Endpoint clients are on 9.6.6 and we are not experiencing any boot-loops or kernel panics in my testing of our Macs running 10.11/10.12. I will continue testing and update you all if anything changes.

Just wanted to add another data point to the thread since I've gleaned some useful information from it. Thanks, all.

[rlee]

It is Ensilo on our systems and not Sophos or OpenDNS. Did testing this morning with each software installed prior to installing 2018-001 to isolate which software was the issue.

After a macOS reinstall, the App store does show 2018-001 as being installed. I haven't seen any further issues on these systems. Even Ensilo console is reporting back properly.

[ifbell]

Okay so we have seen the same issues with Carbon Black. We are removing the b9kernel.kext by going into recovery mode, then using disk utility and terminal to remove the offending kext. On reboot this has shown to work well.