Mavericks Dropping Network connection on test Mac Mini server

[dkagle]

Hi JamfNation.

I am working on setting up a Casper 9.22 test server. Mid 2012 Mac Mini, 10.9.1, Server app 3.0.2. I have been round and round with my networking team and for some unknown reason my ethernet connection keeps dropping every 15-20 minutes. I've rebuilt the server from scratch once but the issue still exists. I made sure all of my energy saver settings were setup to never and even went as far as making ethernet the only network interface. I ruled out it being a hardware issue by hooking up a new Macbook pro with 10.9.1 to my port and it even dropped the connection. I check Apple forums and a lot of people seem to be having issues with Mavericks and staying connected. I was just curious is anyone else out there have run into any issues like this.

[bentoms]

I have 11 Mac Mini servers globally deployed & none have this issue.

Neither do my 10.9 clients.

Does sound a bit like an issue with that port/switch.

[dkagle]

Thanks bentoms. Sadly, my internal network admins refuse to believe it's a networking side issue since a PC plugged into the same port with the same static IP and DNS name doesn't have a problem. Oh the woes of being a Mac admin is a windows world.

[evarona]

Can't say I've seen this issue but the only other thing that comes to mind is to check that the network service order sets your wired connection to the highest ordered connection. For some stupid reason, Apple sets Bluetooth DUN highest.

You can check this with

/usr/sbin/networksetup -listnetworkserviceorder

and set with

sudo /usr/sbin/networksetup -ordernetworkservices <service1> <service2> <service3> <...>

or from the main Network SysPref pane (the gear on the lower left side)

Good luck and hope this helps

[dlondon]

As it's a test machine, can't you plug it in somewhere else to see if it works anywhere? If it does work then you have your evidence of some weird interaction with that particular switch. However if it still acts up you can send it back to Apple :)

[Mauricio]

We have found network issues from our Macs that were upgraded to 10.9.
There is a new security feature in Mavericks, 'ARP Validation'.
The blog from Citrix explaining it in more details.
http://blogs.citrix.com/2013/10/31/citrix-on-osx-10-9-mavericks/
Hope this can help you.
Regards
Mauricio

[dkagle]

Thanks everyone for the replies. I changed the service order and I confirmed it was fine plugged in elsewhere. The article Mauricio linked me is almost identical to my issue, a Mac mini running 10.9 plugged into a Cisco core and it's dropping connection. I will try the fix in the article and reply if it fixes it. Thanks again fellow Jamf Nation members for the help!!!!

[dkagle]

Unfortunately disabling the ARP didn't help my issue. Off to the Apple Store I go.

[jmercier]

hi

im having the same issue on mac pro server 10.9.1...

AND now on my other mac mini server that i just upgraded to 10.9.1...

none of my other 10.8 or 10.7 server doing that... looks really problem with 10.9.1 !!!

[tanderson]

I just started seeing this late last week on my 10.9 Mac mini server. It had been running fine for weeks and just started up. I see network disconnect errors in the logs and have noticed that user sessions aren't getting disconnected when it happens (they hang around in the Connected Users tab). Threads on Apple support discussions about it too. Upgraded to 10.9.2 and no change.

[dkagle]

I had to put my Mac mini server back on 10.8.5 for now. I setup a NetBoot/SUS appliance to handle managing updates for 10.9 clients. I am waiting for the 10.9.3 GM seed and then crossing my fingers.

[taugust04]

Just wanted to add some more information here:

I'm also seeing this connectivity issue on a Mac Mini Server running 10.9.2, where I'm seeing periodic dropped packets from just standard pinging from the Network Utility. Tried running the script provided by Citrix, and I'm still seeing the same issue. However, I have a feeling this is not just a software issue. I have an Xserve also running 10.9.2 and I'm not seeing any connectivity issues at all, so this might be related to the NIC and or the NIC driver for the Ethernet hardware built into the Mac Mini and or Mac Pro. Both the Xserve and the Mini I tested on are connected directly to Cisco cores.

[nick1313]

I am also having this issue with a couple of mac mini servers. I've tried the fixes mentioned in this thread, none have worked.

I have contacted AppleCare support but this issue is too high level for them so I will be contacted enterprise level support as well, as soon as we have an agreement in place.

In the meantime has anyone found anymore information on this? We have HP procurve core switches that the apple servers are plugged into and are on their own vlan (same one as about 150 other servers, various windows/linux, that aren't having this issue). We are also attacking this from an HP switch angle, we have an open case with them in an effort to see if they can find something.

A couple oddities that we have found.

1. If the servers are statically assigned to our local subnet (not in a vlan) they don't have this issue, which I think might rule out the hardware issue.
2. The servers still seem to be transmitting data when they drop pings/packets.

We are looking to, potentially, buy a bunch more of these servers to help manage our ipads/macbooks, so hopefully this issue will get resolved.

I will report back anything I get from enterprise support.

[cbrewer]

Seeing very similar issues with a couple of Mac Minis connected to Cisco 3560X switches running 12.2(53)SE2. I've reproduced the issue on both 10.9.2 and 10.9.3(Build 13D45a) with a current gen Mac Mini as well as a mid 2012 MacBook Pro.

[jmercier]

hi

same here... 2 different machines... mac mini and mac pro... all on 10.9.2. They all work everywhere else on our VLAN but not in this VLAN

everything is pointing out to the version our Cisco switches are running their OS... Its the only place using old version of OS on cisco...

All our other servers running 10.8 and before is working fine...

10.9.3 did not solve the issue for us with beta testing...

[tanderson]

Cisco switches here as well but I'd have to check with our network team on models and software versions. I've been working on another project that will wrap up next week, after that I plan to submit a ticket to AppleCare Enterprise support as well.

[jmercier]

as far as my network team told me... the only place in our huge network we have problems is equipped with OS i think version 12...

all other places is recent version... we might get into upgrading our stack this summer

[nick1313]

The 1st level support (applecare) guy I talked to said he never heard of any of this, I even linked him to this thread. Maybe if more people contact them it will become a little higher priority for them, because it's obviously happening to more people.

He basically said there wasn't much he could do, since it was more high level, but also because I was the first person that he heard it from. Also since it's happening on multiple brands of switches/networks/etc it's gotta be something with the new OS.

[jmercier]

I know it has to do with ARP Inspection and redundancy protocols used on managed switches environment....

He can't say he doesn't know... many threads only reports this problem... weird...

[nick1313]

Yea, maybe that was just his way of deflecting the issue to enterprise support or maybe at the enterprise level they have a fix but want you to pay for it, I don't know.

I would think if it were a bug in the software, which it definitely seems to be, they should send out an update to fix the issue and not make you pay for enterprise support.

I'll be honest though, I don't know next to nothing about how switches are setup and what ARP Inspection and redundancy protocols are, we have two network guys that do that stuff. However they aren't here today.

[mapurcel]

we are also seeing this issue with Cisco gear using Gateway Load Balancing Protocol (with Mavericks clients), this Apple article describes the issue and the script has worked for us so far in a few tests.

Article: https://discussions.apple.com/thread/5483424?start=0&tstart=0
Script: https://github.com/MacMiniVault/Mac-Scripts/tree/master/unicastarp

[jmercier]

Tried this patch multiple times and multiple systems 10.9.... not working at all for us... very bad situation

[nick1313]

I wanted to throw out an update on this and I think taugust04 is on to something. One of our network guys had the bright idea of trying out a usb to ethernet adapter and we haven't had any time outs for the last 2 hours. I'm going to keep monitoring it for the next day.

[jmercier]

Hi

Se tried thunder to ethernet and we had same problems

[mpermann]

@nick1313, did you use an Apple USB to Ethernet adapter or a third party adapter? If it's a third party adapter, do you mind sharing the specifics of the adapter you are testing?

[jmercier]

hi

same comment here... i tried thunderbolt to ethernet... same problem... do you think usb to ethernet would be different ?

[Sonic84]

I'm also seeing a similar issue. I have 4 Mac Minis (out of ~15 globally) which go "offline" every now and again. I've escalated this issue to both Apple enterprise support and McAfee enterprise support. No resolution so far. Two of the servers are plugged into Cisco Nexus core switches (US) and a third is plugged into a catalyst 3750 (Europe). I'm not sure what the 4th is plugged into since this server is a special case... All servers are running 10.9.2. The issue started when the servers still had 10.8 loaded on them. The only difference is when they went "offline" they also stopped recurring any logs. Now that we are on 10.9, the logs continue, but fill up with errors by the kernel.

[nick1313]

One adapter I am using is an Apple one and the other is a Cisco adapter. Only problem with these two particular adapters is that they are 100mb max, but for troubleshooting they'll work for now.

Those of you that tried Tbolt adapters, maybe try a USB one.

[cbrewer]

No problem so far for me while using an Apple USB ethernet adapter.

[jmercier]

hi

situation a lot more stable with usb adapter.... but god its too slow to be used in production...

our solution will be to update the OS of our cisco core... or move the server to other ip range on updated switches....

[sburrows]

We are seeing this issue with one of our brand new Mac Mini's (10.9.3) connected to a Cisco 2960G switch. Has anyone narrowed down the issue?

[nick1313]

I'm working with enterprise support from Apple and have been sending them logs and what not but still no solution. I just updated the servers to 10.9.3 and it's still happening.

One odd thing is the tech wanted me to grab some packet captures of the server while it was doing it and then run a sysdiagnose, however it won't ever drop pings while I'm running the packet capture. As soon as I stop capturing packets it will drop connection again. This is getting very frustrating.

[jmercier]

hi

i have 2 situations...

situation 1 : cisco switch stack with 2 server with 10.9.X, with old Cisco OS : i have tons of network drops...

Situation 2 : cisco switch stack (same model as situation 1) with 2 same server 10.9.X as situation 1 with updated Cisco OS : No more drops...

So we are planning of doing the upgrade of the cisco os on the defective Stack this summer...

edit : on other thing... apple specialist told us that disabling the ARP inspection scan on the cisco switches would fix the problem... but we prefer doing the update...

[cbrewer]

@jmercier Can you find out exactly what 2 Cisco OS versions you did your tests with?

[Sonic84]

By any chance, are you seeing any of these errors in your System.log?

May 15 11:58:11 mac-01 kernel[0]: AppleBCM5701Ethernet [en0]:        8      5ea LimitMinFragSize: Unable to realloc packet

or

Mar 18 19:41:05 mac-01 kernel[0]: ctl_enqueuedata: m_allocpacket_internal(1072) failed

[jmercier]

@cbrewer :

Cisco version not working : 12.2(55) SE1

Cisco version working fine : 15.0(2)SE5

[sburrows]

Hey jmercier do you know what model Cisco switches and the OS that they are running where you aren't haven't any issues? I checked with Cisco and they have a newer OS for my 2960G switch, but they are now recommending the use of version 12 instead of 15. Seems a bit weird.

After doing some testing this morning I plugged our Mini in to an un-managed D-link switch that is plugged in to our Cisco 2960G and we had no network loss. I currently have it plugged in to a 2960S and have noticed a few pings timeout. I confirmed that neither switch has ARP inspection on.

[jmercier]

Hi

switch not working ; WS-C3750E : version 12.2

switch working : WS-C3750X - version 15.0

[ryan_w]

We are seeing these issues too on a couple of Mac Minis connected to our Dell switches. Glad to see I'm not the only one. I hope they release a fix soon, but will start testing a workaround using USB adapters or unmanaged go between switches.

[ryan_w]

On the Mac Mini's ethernet port I set the hardware settings to Manual > 1000 Base-T > Full Duplex, Flow Control > Standard MTU > Unchecked AVB/EAV Mode and things seem to be much better so far.

Never mind on this one, it only worked good for about an hour and then started doing the same thing.