Many OS X machines starting into Recovery Mode, System on Main Drive defective

[tobiaslinder]

Hi guys
We have the really weird situation that suddenly many OS X machines with several clients start to act weird and when trying to restart boot into recovery mode. The system that was installed on the main drive is defective and and the only troubleshooting step that helped so far was restoring the machine.

When looking at the system log the problem appears for the first time shortly after this entries:

Feb 8 06:28:21 iMac-225-C16-2.local nbagent[10168]: XPC Activity invoked with state=2 Feb 8 06:28:21 iMac-225-C16-2.local nbagent[10168]: NoticeboardUpdateOperation: Cancelling poll operation because Check for Updates Automatically is disabled in App Store prefs Feb 8 06:28:23 iMac-225-C16-2.local WindowServer[2009]: CGXDisplayDidWakeNotification [855555200113266]: posting kCGSDisplayDidWake Feb 8 06:28:23 iMac-225-C16-2.local WindowServer[2009]: handle_will_sleep_auth_and_shield_windows: Deferring. Feb 8 06:28:35 iMac-225-C16-2.local SecurityAgent[2013]: User info context values set for >power Feb 8 06:28:35 iMac-225-C16-2.local loginwindow[2008]: Login Window - Returned from Security Agent Feb 8 06:28:36 iMac-225-C16-2.local shutdown[92409]: halt by _usbmuxd: Feb 8 06:28:36 iMac-225-C16-2 kernel[0]: Kext loading now disabled. Feb 8 06:28:36 iMac-225-C16-2 kernel[0]: Kext unloading now disabled. Feb 8 06:28:36 iMac-225-C16-2 kernel[0]: Kext autounloading now disabled. Feb 8 06:28:36 iMac-225-C16-2 kernel[0]: Kernel requests now disabled. Feb 8 06:28:36 iMac-225-C16-2.local shutdown[92409]: SHUTDOWN_TIME: 1454909316 93358 Jan 1 01:00:09 localhost bootlog[0]: BOOT_TIME 978307209 0 Jan 1 01:00:24 localhost syslogd[19]: Configuration Notice: ASL Module "com.apple.appstore" claims selected messages. Those messages may not appear in standard system log files or in the ASL database. Jan 1 01:00:24 localhost syslogd[19]: Configuration Notice: ASL Module "com.apple.authd" sharing output destination "/var/log/system.log" with ASL Module "com.apple.asl". Output parameters from ASL Module "com.apple.asl" override any specified in ASL Module "com.apple.authd". Jan 1 01:00:24 localhost syslogd[19]: Configuration Notice: ASL Module "com.apple.authd" claims selected messages. Those messages may not appear in standard system log files or in the ASL database. Jan 1 01:00:24 localhost syslogd[19]: Configuration Notice: ASL Module "com.apple.bookstore" claims selected messages. Those messages may not appear in standard system log files or in the ASL database. Jan 1 01:00:24 localhost syslogd[19]: Configuration Notice: ASL Module "com.apple.eventmonitor" claims selected messages. Those messages may not appear in standard system log files or in the ASL database. Jan 1 01:00:24 localhost syslogd[19]: Configuration Notice: ASL Module "com.apple.install" claims selected messages. Those messages may not appear in standard system log files or in the ASL database. Jan 1 01:00:24 localhost syslogd[19]: Configuration Notice: ASL Module "com.apple.iokit.power" claims selected messages. Those messages may not appear in standard system log files or in the ASL database. Jan 1 01:00:24 localhost syslogd[19]: Configuration Notice: ASL Module "com.apple.mail" claims selected messages. Those messages may not appear in standard system log files or in the ASL database. Jan 1 01:00:24 localhost syslogd[19]: Configuration Notice: ASL Module "com.apple.MessageTracer" claims selected messages. Those messages may not appear in standard system log files or in the ASL database. Jan 1 01:00:24 localhost syslogd[19]: Configuration Notice: ASL Module "com.apple.performance" claims selected messages. Those messages may not appear in standard system log files or in the ASL database. Jan 1 01:00:24 localhost syslogd[19]: Configuration Notice: ASL Module "com.apple.securityd" claims selected messages. Those messages may not appear in standard system log files or in

Really weird is the fact that the machine changes the date to Jan 1st. After restarting the date is again correct but we have an endless list of errors:

Feb 8 08:10:25 iMac-225-C16-2.local loginwindow[42]: ERROR | -[Application hardKill:] | Application hardKill returned -600 Feb 8 08:10:26 --- last message repeated 1 time --- Feb 8 08:10:26 iMac-225-C16-2.local jamf[965]: open on /Library/Application Support/JAMF/tmp/anykey IT Client Logout: File exists Feb 8 08:10:34 iMac-225-C16-2.local cfprefsd[453]: cfprefsd failed to open /Users/schueler/Library/Preferences/com.microsoft.Word.plist.iDoVoFl for writing. Cancelling attempt to save preferences. Errno was 2 Feb 8 08:10:52 iMac-225-C16-2.local com.apple.usbmuxd[26]: stopping. Feb 8 08:10:52 iMac-225-C16-2.local com.apple.usbmuxd[1028]: log filter changed from 4 to 6 Feb 8 08:10:52 iMac-225-C16-2.local com.apple.usbmuxd[1028]: usbmuxd-364 on Aug 4 2015 at 14:36:18, running 64 bit Feb 8 08:16:58 iMac-225-C16-2.local com.apple.kextd[12]: Rescanning kernel extensions. Feb 8 08:17:11 iMac-225-C16-2 kernel[0]: Resetting IOCatalogue. Feb 8 08:17:11 iMac-225-C16-2.local com.apple.kextcache[1035]: rebuilding /System/Library/Caches/com.apple.kext.caches/Startup/kernelcache Feb 8 08:17:17 iMac-225-C16-2.local com.apple.kextcache[1037]: / locked; waiting for lock. Feb 8 08:17:21 iMac-225-C16-2.local coreservicesd[69]: Posted system configuration change notification Feb 8 08:17:51 --- last message repeated 2 times --- Feb 8 08:18:11 iMac-225-C16-2.local com.apple.kextcache[1039]: / locked; waiting for lock. Feb 8 08:18:48 iMac-225-C16-2 com.apple.launchd.peruser.503[299] (com.apple.ReportCrash[1041]): Job failed to exec(3). Setting up event to tell us when to try again: 2: No such file or directory Feb 8 08:18:48 iMac-225-C16-2 com.apple.launchd.peruser.503[299] (com.apple.ReportCrash[1041]): Job failed to exec(3) for weird reason: 2 Feb 8 08:18:50 iMac-225-C16-2.local com.apple.kextd[12]: Warning: /System/Library/Extensions: No such file or directory Feb 8 08:18:50 iMac-225-C16-2.local com.apple.kextd[12]: Can't stat source folders for cache file /System/Library/Caches/com.apple.kext.caches/Directories/System/Library/Extensions/KextIdentifiers.plist.gz. Feb 8 08:18:50 iMac-225-C16-2.local com.apple.kextd[12]: Rescanning kernel extensions. Feb 8 08:18:50 iMac-225-C16-2 com.apple.launchd[1] (com.apple.ReportCrash.Root[1043]): Job failed to exec(3). Setting up event to tell us when to try again: 2: No such file or directory Feb 8 08:18:50 iMac-225-C16-2 com.apple.launchd[1] (com.apple.ReportCrash.Root[1043]): Job failed to exec(3) for weird reason: 2 Feb 8 08:18:50 iMac-225-C16-2 kernel[0]: Resetting IOCatalogue. Feb 8 08:18:50 iMac-225-C16-2.local com.apple.kextd[12]: Can't stat /System/Library/Extensions - No such file or directory. Feb 8 08:19:29 iMac-225-C16-2.local com.apple.kextcache[1036]: IOSCSIParallelFamily.kext is invalid; can't resolve dependencies. Feb 8 08:19:29 iMac-225-C16-2.local com.apple.kextcache[1036]: IOPCIFamily.kext is invalid; can't resolve dependencies. Feb 8 08:19:29 iMac-225-C16-2.local com.apple.kextcache[1036]: PromiseSTEX.kext is missing dependencies (including anyway; dependencies may be available from elsewhere) Feb 8 08:20:24 iMac-225-C16-2.local com.apple.kextcache[1036]: IOSCSIParallelFamily.kext is invalid; can't resolve dependencies. Feb 8 08:20:24 iMac-225-C16-2.local com.apple.kextcache[1036]: IOPCIFamily.kext is invalid; can't resolve dependencies. Feb 8 08:20:24 iMac-225-C16-2.local com.apple.kextcache[1036]: HighPointRR.kext is missing dependencies (including anyway; dependencies may be available from elsewhere) Feb 8 08:20:44 iMac-225-C16-2.local com.apple.kextcache[1036]: IOSCSIParallelFamily.kext is invalid; can't resolve dependencies. Feb 8 08:20:44 iMac-225-C16-2.local com.apple.kextcache[1036]: IOPCIFamily.kext is invalid; can't resolve dependencies. Feb 8 08:20:44 iMac-225-C16-2.local com.apple.kextcache[1036]: HighPointIOP.kext is missing dependencies (including anyway; dependencies may be available from elsewhere) Feb 8 08:20:52 iMac-225-C16-2.local com.apple.kextcache[1036]: IOSCSIParallelFamily.kext is invalid; can't resolve dependencies. Feb 8 08:20:52 iMac-225-C16-2.local com.apple.kextcache[1036]: IOPCIFamily.kext is invalid; can't resolve dependencies. Feb 8 08:20:52 iMac-225-C16-2.local com.apple.kextcache[1036]: CalDigitHDProDrv.kext is missing dependencies (including anyway; dependencies may be available from elsewhere) Feb 8 08:21:03 iMac-225-C16-2.local com.apple.kextcache[1036]: IOSCSIParallelFamily.kext is invalid; can't resolve dependencies. Feb 8 08:21:03 iMac-225-C16-2.local com.apple.kextcache[1036]: IOPCIFamily.kext is invalid; can't resolve dependencies. Feb 8 08:21:03 iMac-225-C16-2.local com.apple.kextcache[1036]: ATTOExpressSASRAID2.kext is missing dependencies (including anyway; dependencies may be available from elsewhere) Feb 8 08:21:57 iMac-225-C16-2.local com.apple.kextcache[1036]: IOSCSIParallelFamily.kext is invalid; can't resolve dependencies. Feb 8 08:21:57 iMac-225-C16-2.local com.apple.kextcache[1036]: IOPCIFamily.kext is invalid; can't resolve dependencies. Feb 8 08:21:57 iMac-225-C16-2.local com.apple.kextcache[1036]: ATTOExpressSASHBA2.kext is missing dependencies (including anyway; dependencies may be available from elsewhere) Feb 8 08:22:05 iMac-225-C16-2.local com.apple.kextd[12]: Warning: /System/Library/Extensions: No such file or directory Feb 8 08:22:05 iMac-225-C16-2.local com.apple.kextd[12]: Can't stat source folders for cache file /System/Library/Caches/com.apple.kext.caches/Directories/System/Library/Extensions/KextIdentifiers.plist.gz. Feb 8 08:22:05 iMac-225-C16-2.local com.apple.kextd[12]: Rescanning kernel extensions. Feb 8 08:22:05 iMac-225-C16-2 kernel[0]: Resetting IOCatalogue.

Any input is highly appreciated! Thanks in advance.

[bpavlov]

What OS? Any recent software deployed? Any A/V? Any policies that require a restart?

[tobiaslinder]

Hi bpavlov
Thanks for your time.

Till now it hit mostly 10.9.5 machines.
We have no AV on this machines
There are policies that sometimes trigger a restart (Apple Software Update)

With best regards, Tobias Linder

[tobiaslinder]

By now we already have aprox. 70 devices (mostly 10.9.5 but also a few 10.10.5) that are hit by this weird error.

When I look at the hard drive the System folder (/System) is missing completely so no wonder the machine can't startup anymore.

We had now with one client a whole room of 18 iMacs not starting up anymore. Two where still running when we came on site.

iMac 1
Restart -> start into recovery disk with "System" Folder missing on Macintosh HD

iMac 2
removing jamf binary (sudo jamf removeFramework)
Restart -> no problem

So I'm a bit worried that something goes wrong during the logout hook but I have not created any new policies to run at logout lately.

Here the system log from machine 1:

Feb 10 11:50:15 os-computerraum-17.local sessionlogoutd[14604]: sessionlogoutd Launched Feb 10 11:50:15 os-computerraum-17.local sessionlogoutd[14604]: DEAD_PROCESS: 93 console Feb 10 11:50:15 os-computerraum-17.local sharingd[14603]: 11:50:15.632 : No delegate parameters from account, account exists = NO Feb 10 11:50:15 os-computerraum-17.local sharingd[14603]: 11:50:15.633 : Tethering: Identifier needs fixing Feb 10 11:50:15 os-computerraum-17.local sharingd[14603]: 11:50:15.638 : No delegate parameters from account, account exists = NO Feb 10 11:50:15 os-computerraum-17.local sharingd[14603]: 11:50:15.651 : No delegate parameters from account, account exists = NO Feb 10 11:50:17 os-computerraum-17.local defaults[14669]: The domain/default pair of (/Library/Preferences/com.jamfsoftware.jamf, use_legacy_policy) does not exist Feb 10 11:50:20 os-computerraum-17.local jamf[14666]: open on /Library/Application Support/JAMF/tmp/anykey IT Client Logout: File exists Feb 10 11:50:21 os-computerraum-17.local AddressBookManager[14744]: Could not statfs file at path: /Users/schueler/Library/Application Support/AddressBook: No such file or directory Feb 10 11:50:21 os-computerraum-17.local AddressBookManager[14744]: CoreData: error: -addPersistentStoreWithType:SQLite configuration:(null) URL:file:///Users/schueler/Library/Application%20Support/AddressBook/AddressBook-v22.abcddb options:{ NSReadOnlyPersistentStoreOption = 0; } ... returned error Error Domain=NSCocoaErrorDomain Code=512 "The file couldn’t be saved." UserInfo=0x7fd5dac70b00 {reason=Failed to create file; code = 2} with userInfo dictionary { reason = "Failed to create file; code = 2"; } Feb 10 11:50:21 os-computerraum-17.local AddressBookManager[14744]: 0x7fd5dac69f90: ManagedObjectContext (ABAddressBook) could not create the store at path: </Users/schueler/Library/Application Support/AddressBook/AddressBook-v22.abcddb> - creation error: Error Domain=NSCocoaErrorDomain Code=512 "The file couldn’t be saved." UserInfo=0x7fd5dac70b00 {reason=Failed to create file; code = 2} { reason = "Failed to create file; code = 2"; } Feb 10 11:50:33 os-computerraum-17.local secinitd[14072]: UID[503]: cache changed on disk, exiting. Feb 10 11:50:33 os-computerraum-17.local secinitd[163]: UID[0]: cache changed on disk, exiting. Feb 10 11:50:45 os-computerraum-17.local com.apple.kextd[45]: Rescanning kernel extensions. Feb 10 11:50:51 os-computerraum-17 kernel[0]: Resetting IOCatalogue. Feb 10 11:50:52 os-computerraum-17.local com.apple.kextcache[14746]: rebuilding //System/Library/Caches/com.apple.kext.caches/Startup/kernelcache Feb 10 11:50:58 os-computerraum-17.local com.apple.kextcache[14748]: / locked; waiting for lock. Feb 10 11:51:03 os-computerraum-17.local com.apple.kextcache[14749]: / locked; waiting for lock. Feb 10 11:51:08 os-computerraum-17 com.apple.xpc.launchd[1] (com.apple.ReportCrash[14751]): Could not find and/or execute program specified by service: 2: No such file or directory: /System/Library/CoreServices/ReportCrash Feb 10 11:51:08 os-computerraum-17 com.apple.xpc.launchd[1] (com.apple.ReportCrash[14751]): Service setup event to handle failure and will not launch until it fires. Feb 10 11:51:10 os-computerraum-17.local com.apple.kextcache[14747]: Memory allocation failure. Feb 10 11:51:10 os-computerraum-17.local com.apple.kextcache[14747]: AppleKextExcludeList.kext has invalid signature; Trust cache is disabled. Feb 10 11:51:11 os-computerraum-17.local com.apple.kextd[45]: Warning: /System/Library/Extensions: No such file or directory Feb 10 11:51:11 os-computerraum-17.local com.apple.kextd[45]: Can't stat source folders for cache file /System/Library/Caches/com.apple.kext.caches/Directories/System/Library/Extensions/KextIdentifiers.plist.gz. Feb 10 11:51:11 os-computerraum-17.local com.apple.kextd[45]: Rescanning kernel extensions. Feb 10 11:51:11 os-computerraum-17 kernel[0]: Resetting IOCatalogue. Feb 10 11:51:12 os-computerraum-17.local com.apple.kextd[45]: Can't stat /System/Library/Extensions - No such file or directory. Feb 10 11:51:13 os-computerraum-17 com.apple.xpc.launchd[1] (com.apple.ReportCrash.Root[14754]): Could not find and/or execute program specified by service: 2: No such file or directory: /System/Library/CoreServices/ReportCrash Feb 10 11:51:13 os-computerraum-17 com.apple.xpc.launchd[1] (com.apple.ReportCrash.Root[14754]): Service setup event to handle failure and will not launch until it fires. Feb 10 11:51:37 os-computerraum-17.local com.apple.kextcache[14747]: IOStorageFamily.kext is invalid; can't resolve dependencies. Feb 10 11:51:37 os-computerraum-17.local com.apple.kextcache[14747]: SoftRAID.kext is missing dependencies (including anyway; dependencies may be available from elsewhere) Feb 10 11:52:20 os-computerraum-17.local com.apple.kextd[45]: Warning: /System/Library/Extensions: No such file or directory Feb 10 11:52:20 os-computerraum-17.local com.apple.kextd[45]: Note: /System/Library/Kernels/kernel: No such file or directory Feb 10 11:52:20 os-computerraum-17.local com.apple.kextd[45]: Can't stat source folders for cache file /System/Library/Caches/com.apple.kext.caches/Directories/System/Library/Extensions/KextIdentifiers.plist.gz. Feb 10 11:52:20 os-computerraum-17.local com.apple.kextd[45]: Rescanning kernel extensions. Feb 10 11:52:20 os-computerraum-17 kernel[0]: Resetting IOCatalogue. Feb 10 11:52:20 os-computerraum-17.local com.apple.kextd[45]: Can't stat /System/Library/Extensions - No such file or directory. Feb 10 11:52:59 os-computerraum-17.local com.apple.kextd[45]: Warning: /System/Library/Extensions: No such file or directory Feb 10 11:52:59 os-computerraum-17.local com.apple.kextd[45]: Note: /System/Library/Kernels/kernel: No such file or directory Feb 10 11:52:59 os-computerraum-17.local com.apple.kextd[45]: Can't stat source folders for cache file /System/Library/Caches/com.apple.kext.caches/Directories/System/Library/Extensions/KextIdentifiers.plist.gz. Feb 10 11:52:59 os-computerraum-17.local com.apple.kextd[45]: Rescanning kernel extensions. Feb 10 11:52:59 os-computerraum-17 kernel[0]: Resetting IOCatalogue. Feb 10 11:52:59 os-computerraum-17.local com.apple.kextd[45]: Can't stat /System/Library/Extensions - No such file or directory. Feb 10 11:53:05 os-computerraum-17.local com.apple.kextcache[14747]: IOSCSIParallelFamily.kext is invalid; can't resolve dependencies. Feb 10 11:53:05 os-computerraum-17.local com.apple.kextcache[14747]: IOPCIFamily.kext is invalid; can't resolve dependencies. Feb 10 11:53:05 os-computerraum-17.local com.apple.kextcache[14747]: PromiseSTEX.kext is missing dependencies (including anyway; dependencies may be available from elsewhere) Feb 10 11:53:44 os-computerraum-17.local cfprefsd[14477]: CFPreferences: error creating file /Users/schueler/Library/Preferences/ByHost/com.apple.QuickLookDaemon.E22FF058-2795-53D2-8B35-60D83246FFA3.plist.u9HEi4z: 2 Feb 10 11:53:44 os-computerraum-17.local com.apple.kextcache[14747]: IOSCSIParallelFamily.kext is invalid; can't resolve dependencies. Feb 10 11:53:44 os-computerraum-17.local com.apple.kextcache[14747]: IOPCIFamily.kext is invalid; can't resolve dependencies. Feb 10 11:53:44 os-computerraum-17.local com.apple.kextcache[14747]: HighPointRR.kext is missing dependencies (including anyway; dependencies may be available from elsewhere) Feb 10 11:53:54 os-computerraum-17.local com.apple.kextcache[14747]: IOSCSIParallelFamily.kext is invalid; can't resolve dependencies. Feb 10 11:53:54 os-computerraum-17.local com.apple.kextcache[14747]: IOPCIFamily.kext is invalid; can't resolve dependencies. Feb 10 11:53:54 os-computerraum-17.local com.apple.kextcache[14747]: HighPointIOP.kext is missing dependencies (including anyway; dependencies may be available from elsewhere) Feb 10 11:53:59 os-computerraum-17.local com.apple.kextcache[14747]: IOSCSIParallelFamily.kext is invalid; can't resolve dependencies. Feb 10 11:53:59 os-computerraum-17.local com.apple.kextcache[14747]: IOPCIFamily.kext is invalid; can't resolve dependencies. Feb 10 11:53:59 os-computerraum-17.local com.apple.kextcache[14747]: CalDigitHDProDrv.kext is missing dependencies (including anyway; dependencies may be available from elsewhere) Feb 10 11:54:08 os-computerraum-17 com.apple.xpc.launchd[1] (com.apple.mdworker.bundles[14761]): Could not find and/or execute program specified by service: 2: No such file or directory: /System/Library/Frameworks/CoreServices.framework/Frameworks/Metadata.framework/Versions/A/Support/mdworker Feb 10 11:54:08 os-computerraum-17 com.apple.xpc.launchd[1] (com.apple.mdworker.bundles[14761]): Service setup event to handle failure and will not launch until it fires. Feb 10 11:54:08 os-computerraum-17 com.apple.xpc.launchd[1] (com.apple.mdworker.bundles[14762]): Could not find and/or execute program specified by service: 2: No such file or directory: /System/Library/Frameworks/CoreServices.framework/Frameworks/Metadata.framework/Versions/A/Support/mdworker Feb 10 11:54:08 os-computerraum-17 com.apple.xpc.launchd[1] (com.apple.mdworker.bundles[14762]): Service setup event to handle failure and will not launch until it fires. Feb 10 11:54:08 os-computerraum-17 com.apple.xpc.launchd[1] (com.apple.mdworker.bundles[14763]): Could not find and/or execute program specified by service: 2: No such file or directory: /System/Library/Frameworks/CoreServices.framework/Frameworks/Metadata.framework/Versions/A/Support/mdworker Feb 10 11:54:08 os-computerraum-17 com.apple.xpc.launchd[1] (com.apple.mdworker.bundles[14763]): Service setup event to handle failure and will not launch until it fires. Feb 10 11:54:08 os-computerraum-17 com.apple.xpc.launchd[1] (com.apple.mdworker.bundles[14764]): Could not find and/or execute program specified by service: 2: No such file or directory: /System/Library/Frameworks/CoreServices.framework/Frameworks/Metadata.framework/Versions/A/Support/mdworker Feb 10 11:54:08 os-computerraum-17 com.apple.xpc.launchd[1] (com.apple.mdworker.bundles[14764]): Service setup event to handle failure and will not launch until it fires. Feb 10 11:54:11 os-computerraum-17.local com.apple.kextcache[14747]: IOSCSIParallelFamily.kext is invalid; can't resolve dependencies. Feb 10 11:54:11 os-computerraum-17.local com.apple.kextcache[14747]: IOPCIFamily.kext is invalid; can't resolve dependencies. Feb 10 11:54:11 os-computerraum-17.local com.apple.kextcache[14747]: ATTOExpressSASRAID2.kext is missing dependencies (including anyway; dependencies may be available from elsewhere) Feb 10 11:54:41 os-computerraum-17.local com.apple.kextcache[14747]: IOSCSIParallelFamily.kext is invalid; can't resolve dependencies. Feb 10 11:54:41 os-computerraum-17.local com.apple.kextcache[14747]: IOPCIFamily.kext is invalid; can't resolve dependencies.

[Josh_Smith]

I'd look at the Policy history for iMac 1, as well as the Policies in Scope (in case an offending policy wasn't able to report back that it ran). My first guess would be a typo in a script that is deleting /System .....don't know if that is possible....never tried it myself!

[tobiaslinder]

Hi Josh
That was also my first idea and I checked all the scripts but I can't find any mistakes. Also when I enroll a machine newly into the instance and let run all policies then the machine doesn't break :-(

[bentoms]

@tobiaslinder Restricted software?