AMBER Archive (2003)

Subject: RE: anyone have AMBER benchmarks with gigabit ethernet?

From: Ross Walker (r.c.walker_at_ic.ac.uk)
Date: Thu Apr 24 2003 - 08:20:07 CDT

Dear Carlos,

The hardware we are using is as follows:

Redhat 7.3, kernel 2.4.20
Dual Athlon 2200MP chips
8 machines (16 cpus)
512MB DDR PC2100 ECC ram in each machine
Head node disk = Maxtor 80GB 7200rpm UDMA133 IDE
Motherboard = MSI K7D Master Dual AMD Athlon Mobo DDR

Network Cards = Scan Etan 64 bit gigabit adaptors -
http://web6.scan.co.uk/Products/Info.asp?WPID=15143 (cheap :-))
These use the ns83820 module that is part of kernel 2.4.20.

Also, make sure you are using Cat5E cables and not just Cat5 since if
the network card detects a Cat5 cable it will default to 100MBps instead
of gigabit.

The switch is a 24 port 3COM gigabit switch - SuperStack 3 - 4924
http://www.3com.com/prod/en_UK_EMEA/detail.jsp?tab=features&pathtype=pur
chase&sku=3C17701-US
The important thing with this switch is that it is non-blocking across
all ports. A lot of the cheaper gigabit switches on the market only have
a 10GBps back plane and so not all ports can talk to each other at full
speed at the same time.

Some things you might want to check / try.

1) Make sure you compile Amber 7 with -DRLE and -DROWAT

2) Recompile LAM MPI from source using the intel compilers

3) Optimise Head node harddrive options with hdparm (you can test the
speed with "hdparm -tT /dev/hda")
   I use the following in rc.local:
        #c1 = 32bit
        #d1 = DMA
        #S0 = No Power Down
        #W1 = Write Cache
        #X70 = UDMA 133
        #u1 = multiple IRQ mode
        #m16 = 16Kb prefetch
        hdparm -c1 -d1 -S0 -W1 -u1 -m16 -X70 /dev/hda >/dev/null

4) Increase the network send and receive buffers (in rc.local):
        echo "262144" >/proc/sys/net/core/wmem_max
        echo "262144" >/proc/sys/net/core/rmem_max
        echo "131072" >/proc/sys/net/core/wmem_default
        echo "131072" >/proc/sys/net/core/rmem_default

I hope this helps,

All the best
Ross

> -----Original Message-----
> From: Carlos Simmerling [mailto:carlos_at_csb.sunysb.edu]
> Sent: 24 April 2003 12:18
> To: Ross Walker
> Subject: Re: anyone have AMBER benchmarks with gigabit ethernet?
>
>
> Ross,
> thanks for the info. Can you tell me what gigabit
> hardware you used? What card and switch...
> thanks
> Carlos
>
> ----- Original Message -----
> From: "Ross Walker" <r.c.walker_at_ic.ac.uk>
> To: "'Carlos Simmerling'" <carlos.simmerling_at_stonybrook.edu>
> Sent: Thursday, April 24, 2003 4:22 AM
> Subject: RE: anyone have AMBER benchmarks with gigabit ethernet?
>
>
> > Dear Carlos,
> >
> > If you go to
> http://www.rosswalker.co.uk/files/amber7_bench.xls there is
> > an excel spreadsheet of a range of amber 7 tests I have
> conducted using
> > dual processor athlon 2200MP's with gigabit and scali.
> >
> > Using lam 6.5.9 with the intel v7.0 ifc compilers with -D_REENTRANT,
> > -O3, -tpp6 options and including -DMPI, -DROWAT and -DRLE
> in the system
> > flags at compilation I can get (for LADH ca 76000 atoms)
> 9.58x speed up
> > over 16 cpus with scali and 8.69x speedup for gigabit.
> >
> > For the JAC benchmark I get 7.68x speedup over scali (16
> cpus) and 6.14x
> > over gigabit which is essentially the same as those quoted
> for redfish
> > on the amber website.
> >
> > I hope this helps
> >
> > All the best
> > Ross
> >
> > /\
> > \/
> > |\oss Walker
> >
> > | Imperial College London |
> > | Department of Biological and Biophysical Chemistry |
> > | Tel:- +44 20 759(45851) |
> > | EMail:- ross_at_rosswalker.co.uk | http://www.rosswalker.co.uk/ |
> > | PGP Key available on request |
> >
> > Please note:
> > The above message is not necessarily the views of the
> author, Imperial
> > College or The Department of Chemistry. The author of this message
> > accepts no responsibility for any offence caused. If this
> message is
> > forwarded to anyone not on the original contact list then the
> > responsibility lies solely with the person forwarding the
> email and not
> > with the author.
> >
> > > -----Original Message-----
> > > From: Carlos Simmerling [mailto:carlos_at_csb.sunysb.edu]
> > > Sent: 23 April 2003 19:37
> > > To: amber_at_heimdal.compchem.ucsf.edu
> > > Subject: anyone have AMBER benchmarks with gigabit ethernet?
> > >
> > >
> > > does anyone have benchmarks for AMBER7 using
> > > gigabit ethernet? There are numbers on the AMBER
> > > benchmark page but we are unable to reproduce the
> > > scaling values (ours are much worse). I'd like to hear
> > > from anyone else what they have tried.
> > > Note that this is for gigabit ethernet, not Myrinet or Giganet.
> > > thanks
> > > Carlos
> > >
> ===================================================================
> > > Carlos L. Simmerling, Ph.D.
> > > Assistant Professor Phone: (631) 632-1336
> > > Center for Structural Biology Fax: (631) 632-1555
> > > Stony Brook University Web:
> > > http://comp.chem.sunysb.edu/carlos
> > > Stony Brook, NY 11794-5115
> > > E-mail: carlos.simmerling_at_stonybrook.edu
> > >
> ===================================================================
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
>
>
>
>