AMBER Archive (2004)

Subject: RE: AMBER: Gigabit Ethernet Switch/Opteron 246

From: Ross Walker (ross_at_rosswalker.co.uk)
Date: Tue Dec 07 2004 - 12:11:24 CST

Dear Jian
 
This link is a bit old now but may be useful:
<http://amber.ch.ic.ac.uk/archive/200304/0179.html>
http://amber.ch.ic.ac.uk/archive/200304/0179.html
 
I am shoping for a 64-bit AMD opteron cluster for MD simulations using
Amber. This cluster will be around 14 nodes, plus a server. I would like to
have your advice/experience before I spend the money.
 
1) Which brand of Gigabit ethernet switch is better?
 
The switch is very important. Often people are tempted by very cheap gigabit
switches. I would be very very cautious here. Ask the vendor what speed the
backplane is. If it is a 24 port full duplex gigabit switch then unless the
backplane is at least 24GBps the switch will not be "non-blocking" over all
ports. In otherwords pairs of ports cannot communicate with each other at
full speed. Often the cheap switches only have backplanes of around 10GBps
or so.
 
Currently, I have a 32-bit cluster using NetGear GS516T gigabit swtich
(copper). It is fairly stable. To do a very simple test, I ping node1 from
the server. The time is around 0.08ms if the job load in the cluster is
zero. Is the NetGear GS516T a good choice for the 64-bit opteron cluster?
If you are using an AMD opteron cluster, what kind of ethernet swtich do you
use?
 
This test doesn't really tell you a lot. What you need to know is how the
switch performs under heavy load. For example if you have 12 ports (6 pairs)
all talking to each other at full speed (12GBps total) and then connect up
two other ports that are not under load and ping between them is the time
the same as if the other 12 ports were not in use? Also if you max out the
switch on all ports what is the effective throughput? Is it 16GBps (for a 16
port switch?). I have not tried the NetGear switch you mention so can't
really comment but 0.08ms seems a little on the high side. But if it is
stable and performs well on your 32bit cluster then there is no reason why
it won't work well for the opteron cluster. Except that scaling might not be
quite a good as the processors are faster and so do more communication in a
given time slot. What you should consider, however, is obtaining some 64bit
66MHz gigabit network cards to take advantage of the wider bus in the
opteron machines.
 
You may also want to consider setting up two networks, a cheap GB one (cheap
switch, cards etc) for doing NFS and system monitoring over and then an
expensive one (good cards and good switch) for doing the MPI communication.
 
2) To run amber, do you expect significant performance gain of Opteron 248
($690, AMD homepage) over Opteron 246 ($455)? I was told Opteron 246 is a
good choice in term of price/performance ratio. If you have experience with
various opteron models, would you share your experience?
 
I can't comment specifically here as I haven't tried this out myself with
these processors but in general I find that if two processors have the same
size cache you can take the ratio of the processor speeds (2.2/2.0 = 1.1).
This gives you the "raw" difference in processor speed. In this case 10%.
However, normally just increasing the cpu speed does not increase things
like the memory bandwidth or network bandwidth. Hence you typically need to
knock around 15 to 20% off this figure (this will get worse as you go to
bigger clusters). Hence your speed up on going to the faster processor will
be somewhere around 8% for a price increase of 50%. Your choice....
 
I hope this helps.
All the best
Ross

/\
\/
|\oss Walker

| Department of Molecular Biology TPC15 |
| The Scripps Research Institute |
| Tel:- +1 858 784 8889 | EMail:- ross_at_rosswalker.co.uk |
| <http://www.rosswalker.co.uk/> http://www.rosswalker.co.uk/ | PGP Key
available on request |

 

 

-----------------------------------------------------------------------
The AMBER Mail Reflector
To post, send mail to amber_at_scripps.edu
To unsubscribe, send "unsubscribe amber" to majordomo_at_scripps.edu