10.9.5/10.10 Random Beachball

[alexjdale]

We're seeing a large number of reports of Macs randomly locking up and beachballing a couple times a day: the user can move the cursor around and click on apps (which can be repositioned) but can't actually do anything.

This started happening with the 10.9.5/Yosemite release timeframe and only affects 10.9.5 and 10.10, with freshly imaged as well as upgraded systems, and we have not been able to pin it on anything. I'm wondering if we are the only folks running into this issue, which would help us with our troubleshooting. Is anyone seeing something similar? Logs are not showing anything useful.

[waqas]

Alex,

Do you have the new Mcafee Agent and AV installed on these systems. We upgraded our MCAfee EPO servers to accommodate Yosemite and since then, we are seeing exactly the same behavior with a number of our macs. Removing McAfee not only improved the startup time of Yosemite clients, random beachball also stopped appearing.

[alexjdale]

We don't, we use Symantec Endpoint Protection 12.1.5 (although it could be a similar issue). Were you seeing this on 10.9.5 by chance?

[dpertschi]

@waqas.s.khan what exact version of McAfee agent and AV software are you using?

[waqas]

The McAfee Agent is 4.8.0.1816 and EPM is 2.2.

[alexjdale]

We've stripped down to a very basic configuration (OS install from USB installer, configured Active Directory and Wifi only) and we are able to reproduce this with no apps installed, so I think it's something in the AD plugin or wifi that changed when 10.9.5 and 10.10 were released. It must be something in our environment that is not playing nice with those changes because we aren't doing anything differently and it's not happening with earlier OS versions as far as I know.

[alexjdale]

I've been able to reproduce this on 10.10, pretty much at will, by toggling the network adapter on and off. I set up a script to turn Airport off/on every 30 seconds and it will lock up within a couple minutes.

It only occurs when the logged-in user is an Active Directory user, though. Strange stuff.

[mm2270]

@waqas.s.khan - any update on your particular situation regarding the McAfee EPO agent? Are you still seeing performance issues?
We're about to push out this same HF for McAfee EPO in our environment to accommodate Yosemite, so I'm curious to hear how its going. I always get wary of any new McAfee hot fixes. They sometimes fix one thing and break something else, but from what I understand, its a requirement to support Yosemite, so we don't have much choice but to deploy it to get ready for that.

[Anonymous]

@mm2270][/url @waqas.s.khan][/url We are seeing this issue too; The random beachball will only happen on certain machines however. I have currently put a hold on any Yosemite updates company wide until we get the issue resolved. I have been in contact with our McAfee rep, but have yet to get anything past the initial Yosemite hotfix.

Usually the random beachball can be fixed with a simple forced restart, but that's something your users probably won't want to deal with on a daily basis.

[mkremic]

Has anybody else seen these symptoms in their environment or had any luck resolving them?

We've got 4 sites around the world and one of them is experiencing this exact issue where almost all our Yosemite clients are hanging at various points of the day. All using AD accounts and FV2.

Users on 10.9 aren't having these issues though.

[davidacland]

Hi @mkremic

We normally set our used with mobile accounts and make sure there aren't any network home folders being mounted (if they are using laptops). How are these bits configured in your environment?

Do the Macs there have a good Internet connection?

[alexjdale]

To keep this conversation going, we're still seeing this problem with 10.9.5 up to and including the latest 10.10.3 beta. I can reproduce it easily with a script that toggles the NIC off and on every minute, causing systems to lock up (usually within a half hour).

It is triggered in our case within a minute of connecting to the corporate network. Starting a VPN connection, connecting to wifi, etc all have a chance to trigger this. Frequency varies, I usually have it happen once every week or two, and some users can have it happen multiple times in a day. Keeping a stable network connection (no sleep/wake, stay plugged in to Ethernet) staves it off.

If you have this issue, please file a support ticket/bug with Apple. I have spent months of back and forth with premiere support with absolutely no resolution, because the kernel locks up and you cannot capture much sysdiagnose data for their engineers to work with. I kept getting blown off because it's apparently not a common issue.

[mkremic]

@davidacland Yup all our accounts are mobile and not using network home folders. We have a 100MB fibre connection to the interwebs.

It's happening to most of our Yosemsite users at least once to twice a day. Even with wifi turned off and plugged in with LAN it's still happening. We've tried a couple of things like removing a couple of macs from the domain and others have tried local accounts and that seems to stop the issue, so we know it's an "AD" issue isolated to Yosemite.

And like you said @alexjdale , being in the corporate network triggers it. We've had some users work from home and not connecting to VPN and they're fine.

[mkremic]

@alexjdale FYI we've logged a case with Apple today regarding this too... will let you know if we get any movement with it.

Cheers

[donmontalvo]

Intriguing thread, just wondered what Apple Enterprise Support found?

We're seeing similar issues, and are still working with Apple Enterprise Support, nothing yet.

[mm2270]

Well, the overarching answer is really that Yosemite is a steaming pile of... bugs. Really. Worst OS release Apple has ever put out in my opinion.

We have several open cases with Apple enterprise support on all sorts of problems with 10.10.x in our corp environment. We've been going around and around with them now for months, attempting to capture verbose logs, uploading EDCs, etc. and have no solid resolutions yet on any of them, other than that they are actually acknowledging some of the issues (movement!). The latest emails we are getting in response to some of our cases are actually telling us to wait until the next version of OS X releases in beta form, test there and submit a new bug report. (!)
Translation:

Software engineering is tired and bored of trying to fix your silly bugs. They've moved on to the next great OS release, so let's see if its still a problem there, and if so, we can start this all over again from the beginning.

Nice Apple.

[donmontalvo]

We are still in 10.9, and disabling McAfee makes most of the problems away (as a test). We are finding that 10.9 still has some directory services lookup Issues with nested groups in AD, and still SMB volume navigation problems.

We've only done POC on 10.10 but as with 10.9, no McAfee, no spinning beach ball issues. We found fewer issues with directory services lookup Issues with nested groups in AD, and SMB volume navigation is better.

I'm guessing antivirus exclusions aren't where they need to be right now. Not a lot of effort went into the Mac exclusions yet. Based on experiences in other large environments, with slightly older OSes, we had tons of exclusions and didn't have antivirus related issues. Hoping the same applies to 10.9/10.10 but not betting the farm.

So, can't wait for 10.11! #not

[alexjdale]

In our case, Apple finally acknowledged there is a bug in opendirectoryd that causes excessive network and CPU usage which can lead to the beachball lockup. Basically, opendirectoryd is performing excessive querying of my AD groups whenever a connection is made to the corporate network: every member of every group I am a member of (pretty silly, I looked at some unencrypted packet captures, it's a ridiculous amount of data). For me, this ends up with hundreds of thousands of query results and 50+MB of data pulled from the global catalog. Not surprised opendirectoryd sometimes chokes! Most of our users have similar group memberships (for some reason we have lots of HUGE DLs that are in AD as security groups).

We finally found a fix: remove Active Directory from the authentication search path. This kills all the querying and the lockups, at the cost of being able to change or pick up a change to an AD password (luckily it is very easy to add back on the fly when needed). It breaks some other authentication stuff as well, but it's a fair tradeoff for users who are seeing multiple daily lockups.

Apple said they are working on a fix, but with no ETA or build I can test after 9+ months, I am expecting it to be in 10.11 at the earliest.

[andyinindy]

We had similar issues with beachballs/freezes in 10.9.5 and Yosemite, and Apple eventually provided the following "fix":

sudo launchctl unload -w /System/Library/Launch*/com.apple.ReportCrash.*

This fixed the issue for us. We were actually able to observe this in real time, as we happened to have a root shell open on one of the affected systems so we could still run commands. The minute that we unloaded ReportCrash, the system came back to life.

Apple has resolved this issue in 10.10.4, FYI.

[jaharmi]

Just out of curiosity, for those with AppleCare access, have you tried running the AppleCare EDC tool (v4.5) on a system that would normally be affected by this random beachball/freezing problem? Does it complete successfully?

[donmontalvo]

For anyone who doesn't know who this upside down guy is, @jaharmi (Jeremy Reichman) is one of two uber gurus we all bow to when we come to work. :)

Twitter
LinkedIN
PSU archives

[jjones]

One thing I'd like to chime in that we had an issue of in the school year. We were introducing 1to1 in our MIS school and had all machines bound to AD through Casper. Unknowingly Casper had an issue with configuration style AD binding and would bind in OD format instead of AD.

This created the users account as /Users/ADName/Username in the /Users/ file instead of /Users/Username which in turn caused the profile not able to open anything due to permissions were incorrect.

I don't know if this would be a similar issue, but figured it'd be worth mentioning in case.

[expphoto]

We're having a similar issue, though the beachball isn't happening on all programs. Primarily outlook and indesign. Which is odd, because the computer has a Xeon processor and 12 gigs of ram. I disabled the AD search path, and it was still lagging a bit. We've done everything from clear preferences, to reinstalling indesign, to building a new outlook profile. Any other thoughts of where to look on this one? It's on 10.10.4.