2018-001 & Safari Update causing crashes on 10.12.6

jwojda
Valued Contributor II

As an FYI - we have Apple auto-updating and they released the specter/meltdown fixes. I've started getting a few people bringing me machines that are now crashing on boot up after installing, incl rebuilds that ran the updates.

Anybody else seeing similar behavior?

72 REPLIES 72

PhillyPhoto
Valued Contributor

10.11.6 and 10.12.6 installed the updates (SecUpd2018-001 and Safari 11.0.3, both as .pkg in Self Service) fine. We run SEP 14 and CrowdStrike Falcon Host, no Carbon Black.

PeterClarke
Contributor II

2018-001 & Safari Update & 10.12.6 & Sophos Vn 9.6.7 ?

Anyone seen issues with that combination ?
I have tried to block for now…

But haven't yet really run a full set of tests.
Sophos app says Vn 9.6.2 but SAV is Vn 9.6.7

I'll run some tests tomorrow…

roiegat
Contributor III

Just wanted to report the issue is also happening with Digital Guardian. They are working on a hot fix.

mslaughter
New Contributor

Forgot to include the we fixed all of the ones adversely affected by booting into safe mode and the reinstalling the OS. Frustrating but at least we didn’t have to rebuild them or loose peoples ‘stuff’.

bradtchapman
Valued Contributor II

Has Symantec SEP 12.1.6* for Mac been updated to support Spectre/Meltdown?

mark_mahabir
Valued Contributor

Not seeing any issues (yet) with just Sophos v9.6.6 installed (and not the other products mentioned above).

vinny83
New Contributor III

Have McAfee Endpoint Security 10.2.3 and ePO Agent 5.0.6.347:
- Sierra Test VM 10.12.6 - No issues so far
- High Sierra MacBook 10.13.13 - No issues so far

Osmond
New Contributor

I was experimenting this problem, but was the ESET NOD32, so I uninstalled it. I dunno if I should install it again

PeterClarke
Contributor II

mac OSX 10.12.6 & Safari 11.0.3 & Sophos 9.67 - No problems so far…

We have not done much testing yet… But so far no problems with this combination…
We need to test with more Software applications still…

It seems to me the 'most likely' products we have that might suffer issues are:
Video related software - especially drivers of various sorts
Audio related software - again including various drivers etc.
We need to ensure that these continue to work - so testing is going to be awkward…
So start with the simpler things first… and use test equipment wherever possible
- an alternative where specialised equipment is concerned - when we get to it…
- will be to test while booted from an alternate system disk…

PhillyPhoto
Valued Contributor

My test worked ok, so I tried on another machine (10.3.3 upgrade) and now it reboots once I get through FileVault authentication... I tried removing all traces of Symantec and Tanium but no luck. I also tried reinstalling 10.13.3 from a USB drive after unlocking the drive with the same results. I'm working on rebuilding now.

AHolmdahl
New Contributor III

Hi, a client of ours is running Trend Micro Security on their Macs.
We experienced the same problem - after the 2018-001 security update the machine crashes on reboot.
Unfortunately the official Trend Micro Security Uninstaller doesn't completely remove everything.

We fixed it this way:
boot into safe mode (shift key pressed)
run the following commands to completely remove the Trend Micro Security client:
(as root)

launchctl unload /Library/LaunchDaemons/com.trendmicro.icore.av.plist
rm /Library/LaunchDaemons/com.trendmicro.*
rm -r "/Library/Application Support/TrendMicro"
rm -r /Library/Frameworks/TMAppCommon.framework
rm -r /Library/Frameworks/TMAppCore.framework
rm -r /Library/Frameworks/TMGUIUtil.framework
rm -r /Library/Frameworks/iCoreClient.framework
rm -r /Applications/TrendMicroSecurity.app

killall -kill TmLoginMgr
killall -kill UIMgmt

Update!

The Jamf framework has limited functionality when booted into Safe Mode however It does work to manually trigger a policy.
Create a policy with a custom trigger f.x UninstallTMS , boot the Mac (in Safe Mode), and run sudo jamf policy - event UninstallTMS

Hope this helps.

jwojda
Valued Contributor II

Looks like Carbon black's update for bit9 is out.
haven't tried it yet.

easyedc
Valued Contributor II

Just as a +1 we're seeing this with our DLP - Digital Guardian. We've got an update received from the vendor, but haven't gotten the patch out there yet.

JohnG
New Contributor

Sorry if this is a naive question, but do conflicting kexts only conflict with the installation of 2018-001, or do they conflict period with a system that's installed that update?

In other words, if I determine that the version of CarbonBlack we're running is resulting in kernel panics, would it work to remove CarbonBlack, run the update and then install CarbonBlack again?

Or is it necessary either to leave it off entirely or get an updated version that doesn't conflict?

AHolmdahl
New Contributor III

@JohnG This refers to Trend Micro Security
If an old version of TMS is present on the Mac when applying the 2018-001 update it will crash the system.
If you have a Mac which has already done the update, trying to Install an old version of TMS will break TMS.

jconte
Contributor II

@johng i would remove it and wait for a new version, reinstalling should produce the same error.

adp
New Contributor

We have multiple machines with the same boot problem after 10.12.6 updates. I was able to boot in safe mode and disable all LaunchAgents and LaunchDaemons by moving them into a duplicate folder, then put them back a few at a time and rebooting. I isolated the specific daemon to com.verdasys.dgagent.plist (now Digital Guardian, as mentioned above).

In the near-term, removing that daemon from the folder resolves the inability to boot the machine until an update is released.

easyedc
Valued Contributor II

@adp check with your infosec or whoever owns DG. They've provided a patch to us for both the latest and legacy agents.

adp
New Contributor

Yeah, we have a patch but it's getting pushed from within the network and our remote workers (sales and some tech) are having trouble accessing the VPN in safe mode. My fix was an improvisation to allow our traveling folks to get back into their machines, to then update properly.

milesleacy
Valued Contributor
Any fixes?

Don't install these types of software on an Apple device.

This is not a troll. This is the only foolproof approach. This is also not a comment for the frontline techs and engineers being asked to deploy these software titles. It is for security teams and leadership.

  • From an Apple perspective, KEXTS are bad. If a software title requires a KEXT, it's doing things in the OS that it shouldn't be doing. This invariably degrades performance and UX, causing frequent work stops. It's not a matter of if, but when these stops will occur, and an OS patch or update is one of those fairly certain "whens".
  • 3rd-party antivirus tools are a net negative when installed on macOS. They don't protect the Mac, but they do cause problems with the OS and other application software. Apple has antivirus functionality covered demonstrably better than any 3rd party via Xprotect, MRT, Gatekeeper, and a few other protections built-in to macOS.
  • DLP starts with properly securing the applications and services that handle sensitive information and ensuring that sensitive information cannot be downloaded to a client device outside of active memory or a closed application sandbox. If it is possible for someone to download the company's payroll details or customer PII, etc. to the local disk in a format readable by anything or anyone other than the intended tools or accounts, then the application/workflow design was a failed project.
  • VPN can be achieved without KEXTS. Dump vendors that require them.

Nobody argues with the fact that this type of software shouldn't be installed on iPhones or iPads, because Apple made it impossible to do so.

Just because Apple hasn't yet made it impossible to do so on macOS yet does not make it a good or supportable idea.

milesleacy
Valued Contributor

One last point...

I hold that the premise/title of the original post is misdirected.

"2018-001 & Safari Updates" did not "cause crashes on 10.12.6".

The DLP, antivirus, VPN, etc. vendor/title caused the crash by not following Apple developer best practices.

If the rebuttal is "but we can't create this software without using KEXTS or violating Apple practices in some way", then see my previous comments. If you can't do it correctly, you can't do it. One of my primary and inviolable acceptance criteria for any software to be deployed is that the software in question cannot, in normal and expected usage, break the OS and/or cause work stops.

easyedc
Valued Contributor II

@milesleacy How do you manage the L of DLP then? Given that we have marketing staff, for example, who have access to real-live PHI, PII, PFI data in-order to proof created content. If the P_I data can get to your computer, on purpose, due to your job, we have to secure it's possible exfiltration.

I'm an anti-champion of DG, for example, due to the heavy handed approach to everything we do, however I respect it as a necessary evil to protect not just our .org, but also your info that our .org may have.

milesleacy
Valued Contributor

@easyedc wrote:

How do you manage the L of DLP then?

@milesleacy wrote:

ensuring that sensitive information cannot be downloaded to a client device outside of active memory or a closed application sandbox. If it is possible for someone to download the company's payroll details or customer PII, etc. to the local disk in a format readable by anything or anyone other than the intended tools or accounts, then the application/workflow design was a failed project

This is straightforward, if not facile, to achieve in an iOS workflow. It is achievable, if not quite as simple, on macOS. It requires the org's leadership to provide firm direction that internally-developed and 3rd party software and workflows follow this paradigm.

PatrickD
Contributor II

Hi @milesleacy ,

Would you be able to provide some examples of Enterprise grade Anti-Virus solutions that are built without KEXT's for us? It not an easy topic to search.

I would be interested in raising this with our org if there are other suitable solutions.

Cheers,

Pat

milesleacy
Valued Contributor

@PatrickD XProtect, MRT, Gatekeeper, etc. Apple has the best malware protection available for the platform built in to macOS.

The phrase “enterprise grade” is a weasel word used by proponents of the Windows-centric status quo or sales teams working for a security software vendor.

Study and compare performance. IBM famously did and shared their results. You and your org can conduct your own study if you/your key stakeholders remain unconvinced. The fact is that 3rd Party antivirus software titles provide zero benefit to the Macs they are installed on while inflicting issues up to and including ‘bricking’ the computer. They are a 100% net negative in practice. If you have requirements to have an antivirus tool, write ‘XProtect’ in that box on the form. You have it, and it’s the best thing available.

PatrickD
Contributor II

@milesleacy, the reason I used the "weasel word" Enterprise grade is because we require a centralised reporting dashboard for virus and malware detection/removal so you can identify risk areas/staff. Computers maybe perfect (they're not) but people most certainly are not.

Restricting Gatekeeper to "App Store Only" will stifle productivity of staff by preventing them from using the software that they work best with. This is why we offer macOS as an option to staff is to allow them the be more productive with the operating system they prefer.

milesleacy
Valued Contributor

An open request to Apple is to provide easier reporting around these malware protection tools. In the meantime, I recommend building in-house reporting against these tools. There are several plists that contain the pertinent info on macOS.

I do not recommend setting Gatekeeper to “App Store only”, but “App Store and Identified Developers”.

Users must be free to select software, but within some sanity guidelines, including that vendors and internal developers must be educated in and commit to Apple best practices, including code signing.

Berzinji
New Contributor II

What happened:

I applied SecUpd2018-001Sierra.pkg via policy to all machines after it tested fine in my test environment. In production, however, it caused systems to crashes after becoming unstable with certain programs like Chrome not responding properly, then the system restarts and you get a black screen stating “Your computer restarted because of a problem. Press a key or wait a few seconds to continue starting up.”

Solution:
We resorted to re-installing macOS Sierra via thumb drive or using our netboot server. We did not wipe the hard drive, Macintosh HD. We kept the same file structure, all we did is reinstalled the OS using a thumb drive or the netboot server. That has resolved the issue. The total time the fix takes approximately about half an hour.

Notes:
- If file vault is turned on, you will need the file vault key to unlock the Macintosh HD befroe continuing.
- This process does not affect the user's profile, programs, or data.

DenysB
New Contributor

I have this problem too. Apple sent an auto update, and then my El Capitan went into a reboot loop. Runs for 2 and a half minutes, then the mouse freezes, then the screen goes blank.

I reloaded El Captian via Recovery, and then it took an overnight auto-download which repeated the fault.

I have reloaded El Capitan again, and turned off Auto Update, and it's still working. I note that there is a Safari update pending, so I suspect that is may be the cause of the problem.

In case there is any correlation with anti-Virus, I am running ClamXav.

bradtchapman
Valued Contributor II

When you “recovered” El Capitan, you noted that the issue repeated, but not whether you reinstalled ClamXav.

Antivirus products are a major source of conflict with the Spectre and Meltdown patches. On Windows, if you failed to update Symantec before applying the MS patch, your system would BSOD.

Try disabling or uninstalling ClamXav before installing the Security Update 2018-001. If it works, there’s your answer. At that point you would need to grab an update to ClamXav.

bgrant11
New Contributor III

I had to remove the jamf framework to get a problem iMac going again. My Jamf policy that kicked off softweareupdated made the Mac unable to complete a final boot. Had to take snaps of the screen in verbose boot. Jamf was what the Mac was choking on. Other Mac's were fixed with a PRAM zap or two. Seems like there are multiple reasons for the failures.

macdsl
New Contributor III

We are also experiencing the Boot loop issue per this thread here:
https://www.jamf.com/jamf-nation/discussions/27101/macs-getting-stuck-in-boot-loop

But I wanted to reply here as it seems more watched.

Not installing this is not an option for us, has to be done, end of story.
Of 300 or so so far, only 6 machines borked, but I 'think' I have a reason.

It "feels" like this is happening to machines that are having the SecUpdate2019-001 installed in the background.
What I mean is, if JAMF runs this update, it installs with the user fully on the machine, everything running, and then restarts when done.
But, if I manually download the .dmg, and open, and run it locally, enter the admin PW and go, the machine fully logs out first, and THEN runs the updates.
I get the feeling that most of the time this is not a problem, but once in a while, sometimes, possibly, having the machine fully up and running and a user on it while this massive Security Patch runs, something can go wrong. It's touching so much low-level stuff that I got a feeling that's part of the problem.

For anyone here having the Boot-Loop or broke machines after update issue.
Are you running it in the background and then forcing a reboot?
Or, are you opening the .pkg/.dmg and manually running it where it logs everything off then installs?

I'm just trying to find a commonality in how it might be getting broke.

Thanks!

nwagner
Contributor

.