Link aggregation, lom & casper

[bentoms]

Hi guys,

Wondering if any of you with x-serves have setup link aggregation of ethernet ports with lom working on an x-serve being used to host jss or just a distribution point.

Have you noticed any signifcant gain? is there any caveats?

Regards,

Ben Toms

[Bukira]

Hi,

I have all my XServes with Link Aggregation but i dont use LOM and use the servers for DP's and JSS as well as OD and Netbooting,

There is a significant improvement, but only if your network can support it,

i.e.

I tested a xserve on a gigabite switch on an isolated lab network and could get 1.9gb on a 2bg bonded nics, and that was using my DP's

However on our live network i find it hard to get that speed, but thats coz i think our network infrastructure is poor,

The XServes can handle up to 25gb/sec on the BUS so you could bond multiple nics together and make a very high speed network which means you could image large numbers of macs at once,

My XServes have 32gb of RAM and top spec. and my casper complied image is over 100gb.

I always recommended bonding the two nics, i used to do it on a G$ Xserves on a small network and it improved performance.

The only tricky bit is the switch configuration, making sure the ports are setup right, especially if your switch guy isnt used to it.

Criss

Criss Myers
Senior IT Analyst (Mac Services)
iPhone / iPad Developer
Apple Certified Technical Coordinator v10.5
LIS Development Team
Adelphi Building AB28
University of Central Lancashire
Preston PR1 2HE
Ex 5054
01772 895054

[#3]

I have both my xserves on Link Aggregation. I have an extra 2-port gigabit card in both and uses 1 internal and the 2 extra ports for a total of 3GBit/server. The second internal port is used for LOM. If you use Link Aggregation, the LOM will be disabled, it cannot use a linked port, so I have to save the last port for LOM, but since my experience is pretty bad with the LOM anyways, I'm thinking about going 4-port LA.

One thing to remember when running LA is that you will under no circumstances see higher rates than what one single port can deliver between 2 computers. Sorry if that sounded strange. In my case: I have three 1GBit ports aggregated. Every computer on the Campus have access to gigabit ports, so, between any given computer and the server, the transfer rate will be 1GBit (max). Between my 2 xserves with 3GBit each, located on the same Extreme switch will give me a transfer rate of... tadaa: 1GBit. That is because the way LA works. The only use for LA is when you have lots of clients accessing one server. LA is a "function" that splits the traffic on several ports, but it cannot split packets between ports, only send them the same way, always. The way the switch does this is by looking at the MAC address and then run a modulus on the last 4 digits and the number of ports in the trunk. So, any one computer will always communicate with the server on the same port. The more computers that access the server, the better the distribution of the ports will be.
Example:
Computer 1 access the server, the switch decides on the first port in the trunk, max speed 1GBit.
Computer 2 access the server, the switch decides on the second port of the trunk, max speed 1GBit.
Computer 3 access the server, the switch decides on the second port of the trunk, max speed half a gigabit and the speed of computer 2 decreases to half a giga bit too and port 3 is idle.
Computer 4 access the server, the switch decides on the third port of the trunk, max speed 1GBit.

And so on... So, you see, the more clients access the server, the better the distribution in the trunk. If you have a low number of clients, the gain can be quite low.
I have made tests with a lot of clients and I can achieve 3GBits across the 3 ports if the distribution is good. One funny thing is that Activity Monitor on the server goes bonkers on aggregated ports and usually tells me I have around 5GBits tops, on 3 ports, it's always way too high. The server admin graphs are right though. Very weird.

//Patrik

---------------------------------------------------------------
Patrik Sonestad
Avdelningsansvarig
Avdelningen för ABM och Bokhistoria
Institutionen för kulturvetenskaper
Lunds universitet
Telefon: 046-2223141, 070-3219074
E-post: Patrik.Sonestad at kultur.lu.se

12 jan 2011 kl. 08.57 skrev Ben.Toms at grey.com:

[bentoms]

Thanks that's exactly the kindof setup I'm looking at, our nw is all 1GB.

I'll be on to our supplier to get a quote for an additional dual gigabit ethernet card..

Regards,
Ben Toms
IT Support Analyst GREY Group
The Johnson Building, 77 Hatton Garden, London, EC1N 8JS
T: +44 (0) 20-3037-3819 |
Main: +44 (0) 20 3037 3000 | IT Helpdesk: +44 (0) 20 3037 3883

[Bukira]

You also have to have a good network backbone bandwidth, any bottleneck
will reduce performance,

As well as the link between switches

But its nice to hear that someone out there is getitng good bandwidth
in a live setup

Can you tell me what your backbone bandwidth is? What your link speed between switches it? How many switches in the average chain? Is this across buildings? Is it a routed network?

Cheers

I like to have proof i can go to my network guys with to show them im
not dreaming

Criss

Criss Myers
Senior IT Analyst (Mac Services)
iPhone / iPad Developer
Apple Certified Technical Coordinator v10.5
LIS Development Team
Adelphi Building AB28
University of Central Lancashire
Preston PR1 2HE
Ex 5054
01772 895054

[#6]

The network here at Lund university is quite good ;-) We have a big campus stretching at most 40km, but for long distances we use the Swedish uni network SUNET, which is at least 10GBit to every uni in Sweden:
http://www.sunet.se/download/18.5980f13b12211bd97fd80002041/OptosunetDesignv10.pdf

The servers are located at the "Computer Central" which is one of 2 main hubs of the uni net. The backbone of the data centre is 10GBits but as the 10GBit-ports for the Extreme switches are too expensive, I couldn't get 10GBits which was my main idea for the servers but had to settle for LA.
To every building on campus is so far only 1GBit (but dual, so there is backup in case one link goes down, but they are not used together to get 2GBits), which sux, but they are looking into upgrading that to 10GBits very soon, there are lots of fibre to every building. A selected few buildings (big ones) have 2GBit and one or 2 actually already have 10.

All buildings connect to the data centre directly and every building has a backbone of switches connected with 1GBit :(
So, my computer here in my room have 4 switches between it and the server, it can vary all the way down to 2 switches depending on building. I can easily max out the 1GBit. One thing we had to do was to lower the priority of the traffic to the servers. Before we did that I could actually down 3 buildings networks by doing reimaging of a few computers in every house. Now, the imaging process (actually all traffic to and from the JSS/distribution point) only uses whatever is left on the network. IP phones have highest prio and then "ordinary" traffic and all the way at the bottom, my poor JSS :)
The network is not really routed, but is divided into subnets. We have the entire 130.235.x.x-net (B-net) but it's divided into smaller portions and the local switches masquerade as a gateway, but in reality there is only one. This of course to get smaller networks in case things go haywire. So, the network is sort of routed, but not really.

The network is 100% Extreme Networks (except a few resistive departments still running their own equipment).

Link Aggregation is no dream, or vaporware, it's here and now ;-) but it's use is somewhat limited. I would have gone for 10GBits any day instead, the network cards for xserve is not that expensive, I think the cheapest was around €500 when I last checked. The 10Giga ports for the extreme switches was about €2000 or more, so...

If you need a better explanation or your network guys want more "proof" I can get you in touch with the network guy I have been working with, he's also a die hard mac geek and the Apple guy of Lund University so he really understands the mac point of view. ;-)

//Patrik

---------------------------------------------------------------
Patrik Sonestad
Avdelningsansvarig
Avdelningen för ABM och Bokhistoria
Institutionen för kulturvetenskaper
Lunds universitet
Telefon: 046-2223141, 070-3219074
E-post: Patrik.Sonestad at kultur.lu.se

12 jan 2011 kl. 10.00 skrev Criss Myers:

[#7]

Our network switch is a Cisco 6509, and it is remotely managed. If I want to
aggregate ports on an Xserve, I have to know which physical ports on the
switch are involved, and have the manager of the 6509 modify the switch to
allow aggregation on these ports.

--
Karl Schoenefeld | IT Department
SGS St Louis | 1035 Hanley Industrial Court | St Louis, MO 63144
Direct: 314-918-3126 | Cell: 314-680-0359