Stuck on startup

@LibertyJSS When I verbose boot I get the same thing.

This is starting to happen more frequently now and is super frustrating. Any one find anything that seems to help this issue ?

We're seeing similar issues here and seems to be the same kind of setup as others:

Running El Capitan 10.11.5
Late 2012 & 2013 imacs
All AD bound
All in high use open access areas with lots of different users

The boot issue is seemingly intermittent, sometimes it will fail to boot with just the grey loading screen and the loading bar will get about half way then just go extremely slow until it gets to the end but will never actually show the login window.

Tried disconnecting network cable, removing usb devices (keyboard, mouse etc) but it makes no difference.

once it does boot successfully you can keep clicking reboot and it will probably boot every time, there's no obvious pattern to it.

@allanp81 Yeah, very intermittent issue.

On 10.11.6 I was seeing this issue daily.

I've rolled machines back to 10.11.4 and have not seen the issue on the same machine that was previously running 10.11.6 so I'm hoping it continues to work with that image.

@Chuey that's annoying as 10.11.6 was my next test! It mentions a fix to slow AD authentication times.

@allanp81 Yeah, it is super annoying. They fix one issue and causes another. I'm not seeing many slow login issues just that issue where it hangs on startup.

I've seen this issue as well. I was testing 10.11 in a public area and some machines would randomly not complete the boot. Summer time rolled around and the problems went away. Problem is, the students went away for the summer so machine use was very limited. I even had my staff try to login to machines over and over and couldn't re-produce the problem. Sure enough, students came back after the summer and the problem has appeared again. Same types of fixes, sometimes single user mode boot works, sometimes resetting PRAM, sometimes I need to fully re-image the machine.
These are AD bound in high turnover public use areas. Only thing I have been able to find is that the previous shutdown seems to be dirty. Not sure how it is being shutdown, but I always catch error messages about this.

@amiller6 Yeah it's tough. Like you said there's never an overall fix. One time resetting PRAM another time I'm unplugging ethernet, then I may delete all the user accounts under the users folder and it will work, another time I netboot and restart and it works, other times I just have to re-image completely.

If you can, I'd make a 10.11.4 and see if that helps. Since re-imaging some machines that were 10.11.6 to 10.11.4 I've seen the issue less.

Is anyone doing account cleanups as part of the boot process?

We have a launch daemon that runs on bootup that clears all accounts except administrator etc. so wondered if that might be a factor. It's hard to test based on the fact it's so intermittent.

As far as I can tell the machines we're seeing the issue on are late 2012 and late 2013 imacs but that might just be coincedence.

@allanp81 If I have a machine that is stuck on startup I will boot single user mode and remove accounts. However, I am not removing accounts with a LaunchD. Are you seeing this issue less since you have been doing that ?

We're seeing the issue regardless of whether we clean up accounts or not.

It's now getting quite serious for us, it's totally intermittent but affecting pretty much all of the macs we have in our main open access areas.

All of these are late 2012 or late 2013 imacs. Can anyone confirm if they are seeing the issue on newer macs as at the moment we're thinking the only fix is going to be to replace this with newer models.

We do account cleanups on all the machines that were affected by this issue and it had no affect. Unfortunately the only thing that has alleviated this was when we moved to the newer 2015 hardware. We're still seeing the issue on 2011 iMacs, but the 2015's do not seem to have this problem (yet).

Apple is aware of this and we have an open ticket with them. We continue to gather and submit diagnostics with each OS release, but the extraordinarily random nature of the issue makes it very difficult to really decide what is and is not working.

@allanp81 I use to only see it with 2013 iMacs but now seeing issue with 2012 Mac Minis as well.

If you have a 10.11.4 image around or can create one I would do that and see if issue continues.

I have an open area with roughly 20 Macs and 17 of them are 2013 iMacs 3 are 2012 Mac Minis. About a week ago I re-imaged 10 of them from 10.11.6 to 10.11.4 and have not seen the issue back on EITHER os when previously I was seeing 5-10 startup issues a day. At another building we have about 20 or so 2013 iMacs. The technician re-imaged using 10.11.4 on a few of them and has not seen the issue on either OS either. Kind of weird . . . .

Doesn't help much, but all my issues are with the late 2012 iMacs. I'm going to try to find a new one to put in and see if that makes a difference. Unfortunately, I have some machines that have no issues so I really won't have a great feel as to if that resolves the issue(and upgrading all the machines isn't a fix either).

We do not purge users at all on the machines if that helps any troubleshooting. Also, has anyone tried Sierra at all to see if that resolves it? I'm hesitant to downgrade the machines to 10.11.4 as that wouldn't be a good long term fix.

@amiller6 I have not even entertained the idea of Sierra. . .

Clutching at straws here, but could it be something to do with user configuration profiles?

I've noticed that when I run "sudo profiles -P" on an affected machines it returns well over 100 user configuration profiles. An unaffected machine is generally about half of this.

Doing a "sudo jamf removemdmprofile" only clears the 2 computer level profiles, all of the user profiles remain. I can't find any combination of commands to remove these, it either complains they don't exist anymore in DS (if we're doing account cleanup) or that it can't be removed as it's non-removable.

The only method I've found to clear them is to trash the /var/db/ConfigurationProfiles directory and then re-apply MDM. Sadly all of the local macs to me that have the issue are all in use currently so I can't put this theory to the test to see if it solves anything.

Deleting the config profiles made no difference sadly, was worth a shot though. It really is a bizarre issue.

One of the machines I was looking at wouldn't boot at least 20 times in a row yesterday and then just randomly did. This morning it booted fine 2-3 times fine after logging in with AD accounts and then on the 4th boot it refused to boot.

There literally is no pattern to it!!! I'm just going to reimage it with 10.11.3 instead of 10.11.5 and wait and see. Failing to see what else we can do at this point.

Still trying to figure out what the issue could be.

Looking at our affected machines one of the main differences is that we install xcode 7.3.1 and then make all users part of the developers group so they can compile stuff.

Anyone else running xcode in an affected area?

@allanp81 No, I am not running xCode on any of these machines.

I've now got some of the technical staff imaging them back to 10.11.3 to see if the issue goes away, if yes then we'll have to look at rolling them all back to that version.

@allanp81 Seems when I reverted back to 10.11.4 issue stopped. I don't like rolling back but I'm also not prepared for Sierra yet without some good testing. When I see issue on 10.11.6 I just re-image it back to 10.11.4. Let me know if reverting helps your issue out too.

Thanks

Has anyone seen this issue on MacBook Airs? I've seen a few MacBook Airs with stuck on startup but not sure if it is related to the same issue with the iMacs / Mac Minis. . .

We don't have any macbook airs, only seen it on imacs so far and only late 2012/2013 models. That might be pure coincidence though as they're the only models we have in large usage areas.

We were seeing this issue on 10.11.5, Apple advised we upgrade to 10.11.6, and the issue seemed to go away. They wouldn't explain what the issue was.

More recently it's come back, and is instantly a problem if we do an upgrade from 10.10.5 to 10.11.6.

Newly imaged 10.11.6 (15G1004) machines also experience this issue, but only after 2-3 weeks of heavy use.

All the standard stuff doesn't seem to help. Even insane stuff like trashing the contents of /var/folders/ or /var/db/spindump/

Upgrading to 10.12 doesn't help, same issue. Can't test 10.12.1 yet as the Sierra installer in the App Store is still 10.12 for some reason.

Most machines (though not all, and not consistently) show the following when booting verbosely.

kauth external resolver timed out (1 timeout(s) of 60 seconds)

Even Safe Boot doesn't work properly any more on these machines. Very perplexing.

Interesting, seems to me like Apple have no interest in fixing this issue.

@draeconis Thanks for the information. I too am seeing the kauth external resolver error a lot.

@allanp81 Why would they care? They only care about 1-1 and not enterprise environments. Sierra is out but I haven't even entertained the idea because I've had zero time to test thoroughly.