Avoid Security Update 2016-003 for OS X El Capitan 10.11.6

[RobertHammen]

We, and several others, are seeing major performance issues on El Capitan 10.11.6 after installing Security Update 2016-003 (OS build 15G1212). The issue is with WindowServer going "out to lunch" after some period of uptime. The issue causes GUI freezes and system lags until the Mac is restarted.

Specifically:

a) the user will be using the system, and the GUI will appear to "freeze" for some period of time (mouse will still move, but nothing appears to respond to mouse or keyboard input)
b) eventually, the system will "catch up" and all queued UI actions (window drag/close, etc.) will happen at once
c) system will still appear to lag intermittently until it's restarted - only fix I've found so far (uninstalled AV/all third-party kexts/LaunchAgents/LaunchDaemons, cleared caches, etc, no change in behavior)
d) you'll find WindowServer userspace_watchdog_timeout.spin logs in /Library/Logs/DiagnosticReports. The logs don't really say much.

watchdogd[569]: [watchdog_daemon] @(_wd_daemon_service_thread) - service (com.apple.WindowServer) reported as unresponsive
spindump[2113]: Saved userspace_watchdog_timeout.spin report for WindowServer version ??? (???) to /Library/Logs/DiagnosticReports/WindowServer_2017-01-03-152435_computername.userspace_watchdog_timeout.spin
[2113]: Removing excessive log: file:///Library/Logs/DiagnosticReports/WindowServer_2016-12-20-112659_computername.userspace_watchdog_timeout.spin
watchdogd[569]: [watchdog_daemon] @(__wd_service_report_unresponsive_block_invoke) - spindump gathered for (com.apple.WindowServer) at (/Library/Logs/DiagnosticReports/WindowServer_2017-01-03-152435_computername.userspace_watchdog_timeout.spin)
watchdogd[569]: [watchdog_daemon] @(_wd_daemon_service_thread) - service (com.apple.WindowServer) reported as unresponsive
WindowServer[331]: send_datagram_available_ping: pid 2024 failed to act on a ping it dequeued before timing out.
WindowServer[331]: disable_update_timeout: UI updates were forcibly disabled by application "SystemUIServer" for over 1.00 seconds. Server has re-enabled them.
WindowServer[331]: common_reenable_update: UI updates were finally reenabled by application "SystemUIServer" after 4.22 seconds [0.24fps] (server forcibly re-enabled them after 3.88 seconds

Per: Apple Discussions I do see

kernel[0]: stalling for detach from IntelAccelerator
kernel[0]: stalling for detach from IntelAccelerator

on my rMBP at the relevant times.

Also see discussion of it here (since many other people are reporting/experiencing this, I do not believe it to be anything specific to our environment):

More Apple Discussions
Macintouch

Suspect there are bugs in the updated graphics drivers in SecUpd2016-003. I've examined the pkg and there most definitely are updated graphics drivers in the update.

We have opened a case with AppleCare Enterprise on this issue (100105893834). If anyone is seeing this in their environment, and has AppleCare, please open a case and reference the one above.

We have scientists and developers working with large data sets and having to reboot in the middle of work is problematic. We also have a number of users that access their work systems remotely, and we require smart card login. If the user must reboot they need assistance from IT in order to get logged back in, and sometimes they need to reboot outside of official support hours.

So this issue, which we did not discover in testing since it takes a few days to happen, is becoming very problematic for us, and many others. Strongly recommend that if you have not applied the security update, that you do not install it until this situation is resolved or addressed in a further update.

--Robert Hammen
Sr Mac Admin, NIH

[AVmcclint]

Thank you for confirming that I'm not the only one experiencing this, and narrowing it down to the Security Update. I was beginning to pull my hair out. Unfortunately I've already deployed that update to the majority of our Macs.

[onyx4]

I have the same exact issue on 3 of my mac at home (Macbook Pro 15" Retina mid-2015, another 2012, and macmini 2011). All pausing randomly due to windowserver hanging. Has apple provided any updates in your case? I find that odd that not more people are seeing this issue. Is there an apple forum thread for this issue? The only one I could find is different and talks about a complete freeze. Where as in this case it's a hang which last about 3-4 seconds and comes back and forth until reboot. The problem starts for me after 3-4 days of use sometime sooner.

[RobertHammen]

@onyx4 See the two "Apple Discussion" links in my post. Your experience is the same as mine... and @AVmcclint we've also deployed it everywhere. Not quite ready to roll out Sierra (waiting on an Endnote X8 upgrade purchase, which is in .gov purchasing hell), so we are stuck...

[rstasel]

This is good to know. I was seeing this on one machine, and was thinking it was something on my end!

[nkalister]

Just had a crash that fits this profile on a machine running 10.12.2- the behavior was exactly the same as what you describe, and the window server spin dump from the crash time is basically the same.
Has anyone seen this on 10.12.1?

[onyx4]

Maybe this is related to a conflict with a particular software ? Especially the video driver could be at fault where some security fixes were made. I have VMware fusion and Duet display installed on the Macs where I see the issue.

[bvrooman]

Is anyone else running McAfee EPM or ENS? We saw this same behavior until we rolled out ENS 10.2.1 and hotfix 1159589, and it has since gone away.

[RobertHammen]

Disabled SEP (A/V) and any other third-party software save for Casper Suite. Problem still occurred. Since we're seeing this on all our Mac hardware, and I'm seeing reports from "consumers" on Apple Discussions, I suspect that this is problem is widespread, at least on 10.11.6. Can't speak to Sierra as we're still testing it and it's not widely deployed.

[cpitchford]

I've had this problem on a 2013 Mac Pro since around the 2nd week in December 2016

Each time the graphics hangs, I see this line in the kernel log:

stalling for detach from AMDTahitiGraphicsAccelerator

The machine is otherwise (i.e via SSH) responsive. The problem seems exacerbated with large windows. Chrome or Safari full screen causes hanging, where as resized into a smaller window seems better. The problem can take a day or two to appear after a reboot.

My Mac Pro is waiting to be sent away for repair. However, if this is a software bug, I'll see if I can get upgraded to 10.12.2 to see if the problem goes away!

Really pleased I saw this thread!

[nkalister]

Downgraded my affected tester from 10.12.2 to 10.12.1 (16B2659 build) over the weekend, and I was no longer able to get this crash to happen on that build.

[RobertHammen]

If anyone is seeing the WindowServer freezes in their organization after applying SecurityUpdate 2016-003 for El Capitan, and has AppleCare Enterprise, please open a ticket, referencing ticket number 100105893834, and let Apple know how many systems you have that are being affected by this issue. My guess is that Apple's Product Engineering is attempting to prioritize the fix based on affected systems.

[willjac037]

Hi Robert

Did you download the update from https://support.apple.com/kb/DL1903?locale=en_US

I ask as the link wasn't available when it was first released and I had to download via swcdn link see

https://www.jamf.com/jamf-nation/discussions/22449/download-link-for-security-update-2016-003

The Apple kb has the update containing the 3 packages and the pkg is 486MB compared to the 331MB for the single pkg from swcdn.

I deployed the single package and I have the issues so I am now testing the bundled pkg to see if resolves the issue.

[Nix4Life]

+1 on this.

I deployed the 3 pkg version the week after Christmas. I have not seen the formentioned issue. I'm in .edu, so our machines are caffeinated at boot and shutdown nightly. The release difference as @willjac037 mentioned, seems to be a firmware update pkg.

Larry

[murph]

I've seen this happen also on a machine with the 2016-003 Security update that pretty much only runs iTunes 24x7. I was curious if the iTunes 12.5.4 update had any role in the problem too?

[raymondap]

I noticed this post on Apple Discussions regarding Incompatible Kernel Extension Configuration Data published on 01/04. I realized that our test machines, which aren't seeing the freezing problem anymore, received this update, while production machines haven't yet. I've rolled this out to all, and I'm hopeful it will take care of the problem.

[makken12]

After installing El Capitan on my iMac, I also got the trouble you describe, several times each day, it was really frustrating. After posting on a norwegian forum, I got this answer:
https://support.apple.com/en-gb/HT201295
Resetting the SMC. And for two days I have not had any problem. Hope it will last forever!

[michelangelo]

The issue may be also triggered by installed virtualization apps (Fusion, VirtualBox). in my case, I noticed powerd using up most of my late 2012 macmini cpu time. I was able to kill it from terminal and my macmini was restored to normal operations

sudo killall powerd

The process gets respawned after a few seconds, so no need to take further actions. It may be a workaround not to reboot every time the sluggishness comes up. ciao

[RobertHammen]

1) has nothing to do with the subsequent updates (Incompatible Kernel Extension Configuration Data)
2) has nothing to do with virtualization software
3) updates to our systems were applied via softwareupdate command from Apple's servers. I picked the 10 systems experiencing the issue the most and re-deployed the downloaded SecurityUpdate2016-003.pkg (from support.apple.com) to them
4) for most users the problem doesn't re-occur for 4-6 days, but for some users (have seen both Photoshop as well as Safari being triggers) it can take as little as an hour or two before the problem returns.

The posts from "hughmcintyre" in this thread on Apple Discussions (mentioned above) were most interesting. It does seem that some Sierra functionality was backported to 10.11.6, but not adequately tested...

Really really really hope Apple decides that this issue is worth fixing. We're going to Sierra in Feb/Mar (after 10.12.3 is out), but can't any sooner (in the middle of testing all of the ssh/logging/NFS changes in Sierra).

[tomt]

Has anyone noticed if this issue happens in Sierra? I have a few 10.11.6 machines exhibiting the freeze and all but 1 (a file server) use domain accounts. So the only machine I have available to test Sierra is a small file server due to the account locking bugs in 10.12. Not sure I want to sacrifice a server and have to rebuild it if Sierra does not resolve the freeze.

[tuinte]

@RobertHammen

Did the re-deploying in your third bullet point resolve for you, or is it just listed there as something you've tried without resolving?

[nkalister]

I've seen this on 10.12.2 with certain hardware- i.e. 15" touchbar had the issue, 12" MacBook did not. Have been unable to reproduce at all on any hardware with 10.12.1.

[RobertHammen]

@tuinte Have tried it, but not enough data to determine if it did anything. Off to go look at firmware versions now...

[Nix4Life]

@RobertHammen Thanks for your diligence on this. Although we were not affected, I have disable the update from going out

[BluesBrother]

I'm sure glad I finally stumbled across information on this problem! I have an iMac and a MacBook Pro (early 2011), both running El Capitan 10.11.6 and Security Update 2016-003, which was applied on December 20, 2016. Since that time, I've been experiencing what appeared to be "random" freeze-up's in the GUI - mouse would move, but mouse and keyboard input appeared to be ignored, and eventually things would finally unfreeze for a few seconds, then hang up again. The only "cure" was to forcibly reboot the afflicted machine, and resume working in fear that the problem would occur again within a few days. As I've experienced it, the problem is exactly as described by others in this forum thread.

Let's hope that Apple fixes this, and soon! Unfortunately, Apple doesn't have a great track record for acknowledging problems, and is often slow to provide fixes (and in some cases, never seems to get around to fixing certain problems). In the meantime, I guess our only recourse is to forcibly reboot the machine when this glitch rears its ugly head.

[RobertHammen]

Looks like Apple just posted a Supplemental Update to Security Update 2016-003 for El Capitan (per @timsutton):

Only available via App Store/softwareupdate right now...

• Security Update 2016-003 Supplemental Update-10.11.6 Security Update 2016-003 Supplemental Update (10.11.6), 364830K [recommended] [restart]

"The OS X El Capitan Security Update 2016-003 Supplemental Update fixes a kernel issue that may cause your Mac to occasionally become unresponsive."

[djdavetrouble]

Love to see some response from affected sites! I have been holding off on this update.

[rstasel]

So far so good on my one machine (Mac Pro 2013), but it usually took a week or so to crop up.

[bainter]

@RobertHammen My MBP Retina Mid-2014 pulled down a Firmware Update along with the SecUpd2016-003Supplemental.

[Nix4Life]

Installed in Media Lab. New OS Version is 15G1217. as @bainter confirmed. there are 3 pkgs. 1 being a firmware update. Have not seen anything for 10.12.x. Students coming in 5 mins, so I wil know if there are issues

Larry

[pcrandom]

Download page for updated update, for those who haven't installed the original Security Update 2016-003 (10.11.6) : https://support.apple.com/kb/DL1907?locale=en_US

Download page for supplemental update, for those who had installed the original: https://support.apple.com/kb/DL1908?locale=en_US

[dpertschi]

El Capitan Security Update 2016-003 Supplemental Update

600MB, Come on Apple!

[djdavetrouble]

> 600MB, Come on Apple!

use the Caching server, you won't regret it.

[Nix4Life]

Hi Guys,

No issues to report, but it's only been 24hrs. Heavy use this evening will update if there are issues

L

[ClassicII]

Safe to say this issue is resolved ?

[RobertHammen]

@ClassicII Yes!

[wboone66]

I cannot get this update to download. I have been having many issues, which I was hoping would be resolved with this update. The main one is having to do a hard reboot almost every 10 minutes! Everything stops responding. I cannot get iCloud to respond; I tried to delete and re-add it to my mail account, but it isn't recognized. I've called Apple and they say the error message has something to do with my network connection, but my iPhone can connect to it on the same network. How can I get this download?