NetBoot not working on current hardware models

[friedelj]

We've received new iMacs and MacBook Pros (which have not undergone a hardware refresh and can still run OS 10.6) and are unable to Netboot them using our existing 10.6-based NetBoot or a new 10.7-based NetBoot image created using that hardware.

The computers download the Netboot image from the server and complete the boot process (verbose boot completes) but then the system just sits there with the background image showing (10.6 shows the Casper Suite background, 10.7 shows Apple startup white screen) and a movable mouse cursor. The computers have an IP address at that point and respond to pings, but cannot be connected to via SSH.

Older hardware models in our environment boot both NetBoot images without any problem. Our NetBoot server is running 10.6.8 Server.

Anybody have any suggestions or even seen this before?

[jhbush]

Did you build your images with the JAMF auto mater action or just a basic net boot without the action? I skipped the JAMF action for now. I can image MBP and MPA just fine so far. I've seen the MBA take a long time to get going, but it works in the end. I would also make sure your NB images are fully patched before creating them.

[andrewseago]

We have a similar issue already with Apple. It has to do with the latest EFI that supports netbooting to Apples servers over the web when the Hard Drive dies.

[friedelj]

@jhbush I originally tried the Automater action, but decided to manually perform those actions so I could change the desktop background and add additional dock icons. I have ensured the images are fully patched.

@andrew So, you have an open bug with Apple for this problem? Perhaps I should contact that to increment the number of people experiencing the problem.

[jyoung]

Here is what we've observed:

1 - Our Netboot server requires dhcp-trust to be enabled on its port in the switch, allowing the switch to pass through dhcp-like messages that are normally blocked to prevent rogue dhcp on client ports. No objections here and we've had it configured this way all along. By default, our client ports are all dhcp UNtrusted (blocks all but DHCP requests and client ACKs, etc). We also have DHCP snooping in our switches that monitors the conversations and provides us with a binding table showing client leases, etc.

2 - Our DHCP server is standalone and outside of our Netboot server. It's a Linux server running standard ISC dhcpd. We've run this for years.

3 - With the previously described conditions in place, any Macs we have that have NOT received (or are not, by age, eligible) for the recent thunderbolt firmware update appear to netboot just fine. Plug them into any client port in the network and they locate and boot from the server almost immediately.

4 - Plug a Mac that has received the recent thunderbolt firmware upgrade into the same (or any other standard) client port and it receives DHCP properly, but never netboots. There are sometimes (?) a series of INFORM messages in the server log... but the process ultimately times out and the local OS X is booted.

5 - Enable dhcp-trust on a thunderbolt Mac's client port (making it the same as the server and, by our definition, very dangerous to our network) and it suddenly will netboot just fine. Enabling dhcp-trust on a port is actually a broad set of behaviors related to dhcp snooping, some arp anti-spoofing tools, etc. At a lower level, if I disable dhcp snooping (which builds the dhcp binding table and implements some related protections) in the VLAN these systems all share, it also works.

Long story short: our pre-thunderbolt-firmware-updated Macs all seem to work perfectly well on the secure network configuration we've had in place for years. Some machines that were net booting prior to the thunderbolt firmware update (so I'm told) stopped being able to boot under normal conditions once they were updated. The only way to get them to work is by giving them the same level of dhcp-trust on our switches that the server gets (or disabling our dhcp snooping and safeties vlan-wide). This is not a safe configuration for us and definitely suggests a change in behavior on the Mac side. Were it not for the pre/post-thunderbolt firmware update difference, we'd be more focused on our network configuration.

Our switches are Juniper EX series 3200 and 4200's running JUNOS 10.x releases (10.4R5.5 will be unified on them all soon as we do rolling upgrades).

[andrewseago]

jyoung - That is exactly the symptoms we are seeing. I really thing we all need to file bugs with Apple. This is an apparent flaw in the method they are using in the new EFI that enabled Netboot over the web with the Hard Drive is dead or erased on latest generation hardware.

We have been trying for over a month now and are getting the run around. Which reminds me I need to check back up on this.

[friedelj]

jyoung - I appreciate the detailed response. I was just working with our networking team and verified this is the same issue we're having.

After doing some packet capturing, they determined that the client originally requests an IP address with one client ID, then later in the boot process requests an IP address with a slightly different client ID, then tries to go back and use the original IP it was issued for the first client ID which is causing the problem.

We're going to verify that older clients don't behave this way and then I'm not sure what next, probably contacting Apple.

[mclemens]

We are seeing the same thing with our "late 2011" macbook pro's. Keep us posted on what you hear from Apple.

[andrewseago]

Has anyone had any success with this yet?

[jyoung]

I have been in touch with Apple's Engineering Group on several occasions. They have requested numerous WireShark traces among other things. Still in progress.... So far our only solution is to disable DHCP Snooping/Trust on the switches in our repair area so we can do imaging. I am hoping for a fix soon so we can just image in the labs.

[friedelj]

My last update from the Apple Engineer we're working with is that the Engineering team is aware that this is a problem with the current firmware for Thunderbolt-capable Macs and that they will be developing a fix that will eventually be released in the form of new firmware updates for those models. I was not provided with a timetable.

As a workaround, we also have disabled our DHCP security on select ports in our work areas and also have a bootable external disk with Casper Imaging for use in the field.

[rtrouton]

Out of curiosity, has anyone seeing the problem tried booting from a NetBoot volume other than the Casper NetBoot set?

[friedelj]

We first had the problem with our Snow Leopard Casper NetInstall. We made a Lion NetBoot image which was created without using the Casper Automator Action from the Resource Kit, so that really is just OS X Lion with the Casper Imaging app installed on it. We also made a OS X Lion NetInstall just to test and it experienced the same problem.

[Kumarasinghe]

Latest EFI Firmware update released on 23 February 2012 fixed my NetBoot issue. Thanks Apple.

From Apple;

This update improves the reliability of booting from the network, addresses an issue that can prevent HDCP authentication after a reboot, and resolves an issue with boot device selection when a USB storage device is hot-plugged.

[johnnasset]

No luck for us on the latest 2.7 firmware update. All of our older macs netboot fine but no Thunderbolt or Lion mac will netboot. We get the flashing globe but then the process stops and boots to the internal drive. We have also created a closed network with the server, a dummy switch and client and not had any change.

[jyoung]

Just tested with the firmware update released last week. Mac Mini v1.6 and MacBook Pro 2.7. Did not fix any issue for us.

[johnnasset]

We've been doing some further testing prior to calling AppleCare again and here's what we've found:

If I setup DHCP on our NetBoot server and hook a client machine directly using a cross-over cable, it actually netboots. That tells me it's something in our local infrastructure.

If I hook both the server and client machine to a switch (not on the LAN) with DHCP on the server, netboots fine.

Once I turn off DHCP on the server with the above setup, no netboot (makes sense, netboot requires DHCP)

With both machines hooked up to the local network, no DHCP on the apple server (using our regular DHCP server), no netboot for Lion clients.

Something seems to have changed in the way netboot works in Lion regarding DHCP. Our DHCP sever is on a different VLAN/subnet, which is where we are focusing our testing. Older Mac's running Snow Leopard boot fine w/o issue. I'm not the network guy so if anybody has any tips or thoughts on this, please let me know.

[sean]

This sounds like you don't have bootp relay, iphelper address or whatever you would like to call them turned on, unless you are saying that you can netboot on your infrastructure using 10.6, but not if you netboot 10.7.

[johnnasset]

Yep, we can netboot fine (even to 10.7 images created on machines we can't netboot with) with older Mac's running 10.6 on the normal network infrastructure, not with any Lion-shipped Mac.

[bentoms]

John, have these troublesome macs had all the very latest firmware updates installed?

[johnnasset]

Yes, all of the most recent firmware updates (2.7 for the MBP). The white MacBook didn't get a release with the rest of the updates. It's version is 2.1.

[friedelj]

The latest EFI firmware updates fixed netbooting on both our Thunderbolt iMacs and Macbook Pros.

[Joseph_Morris]

Have you built a new NetBoot Image using the latest build for that particular machine and tested it? That is where I would start first. I have a 15" MacBook Pro that shipped with Lion and I have no problems netbooting to my Casper NB servers. The first thing that I would do is rebuild a brand new NetBoot Image for the machine you're using before suspecting something else.

[wangl2]

Hi everyone,
I am having similar problem with Joe. The NetBoot suddenly stopped working on the new iMacs we just received. I have recreated NBI on the newest hardware and it didn't fix the problem. All the existing iMacs are netbooting fine. My NetBoot server sits on different VLAN as the clients. Updating to the latest firmware didn't fix my problem.
Any advice are welcome!
Thanks all!

[ehorn99]

Created a BaseImage of 10.6.8 with all updates except Thunderbolt Software Update on a MBP bought Summer 2011. Netboot diskless works. Updated with Thunderbolt, Netboot diskless fails. I get the circle with a slash through it.

Does anyone know what this updates does to change the Netboot Sequence? I thought this update was for the Thunderbolt display.

[kevinstcroix]

I can verify that all of my mid 2012 MacBook Pro's (shipped with Lion) are having the net boot issue. The models are base 15" mbp's and Retina Display w/512 gb.