AMBER Archive (2007)

Subject: AMBER: PMEMD configuration and scaling

From: Lars.Skjarven_at_biomed.uib.no
Date: Tue Oct 09 2007 - 05:05:37 CDT

Again, thank you Bob and Ross for your replies. It provided what I
needed to get this running.. at least with scali mpi. I learned
yesterday that Mvapich2 was not ready on the cluster yet.. Therefor I
stick with scampi for now.. So, PMEMD now runs over the infinband with
scampi and intel compilers on the opteron cluster. Maybe this can be
usefull for someone else as well, and I therefore post the
configuration file below. It serves as a complement to the following
post:
http://structbio.vanderbilt.edu/archives/amber-archive/2007/1569.php

As Bob suggested I provide the JAC benchmark that ships with pmemd:

With infiniband:
#CPU - ps/day - scaling %
2 - 587 - 100
4 - 1093 - 93
8 - 1878 - 79
12 - 2541 - 72
16 - 2979 - 63
20 - 3756 - 63
24 - 4320 - 31
28 - 4800 - 58
32 - 5082 - 54

Over GB ethernet (obviously very unstable):
#CPU - ps/day - scaling %
2 - 587 - 100
4 - 1107 - 94
8 - 1694 - 72
12 - 2009 - 57
16 - 1093 - 23
20 - 970 - 16
24 - 2215 - 31
28 - 2880 - 35
32 - 939 - 10

### config.h for linux64_opteron, ifort, scampi, bintraj ###
MATH_DEFINES =
MATH_LIBS =
IFORT_RPATH =
/site/intel/fce/9.1/lib:/site/intel/cce/9.1/lib:/opt/scali/lib64:/opt/scali/lib:/opt/gridengine/lib/lx26-amd64:/site/pathscale/lib/3.0/32:/sit
e/pathscale/lib/3.0:/opt/gridengine/lib/lx26-amd64:/opt/globus/lib:/opt/lam/gnu/lib
MATH_DEFINES = -DMKL
MATH_LIBS = -L/site/intel/cmkl/8.1/lib/em64t -lmkl_em64t -lpthread
FFT_DEFINES = -DPUBFFT
FFT_INCLUDE =
FFT_LIBS =
NETCDF_HOME = /site/NetCDF
NETCDF_DEFINES = -DBINTRAJ
NETCDF_MOD = netcdf.mod
NETCDF_LIBS = $(NETCDF_HOME)/lib/libnetcdf.a
DIRFRC_DEFINES = -DDIRFRC_EFS -DDIRFRC_NOVEC
CPP = /lib/cpp
CPPFLAGS = -traditional -P
F90_DEFINES = -DFFTLOADBAL_2PROC

F90 = ifort
MODULE_SUFFIX = mod
F90FLAGS = -c -auto
F90_OPT_DBG = -g -traceback
F90_OPT_LO = -tpp7 -O0
F90_OPT_MED = -tpp7 -O2
F90_OPT_HI = -tpp7 -xW -ip -O3
F90_OPT_DFLT = $(F90_OPT_HI)

CC = gcc
CFLAGS =

LOAD = ifort
LOADFLAGS =
LOADLIBS = -limf -lsvml -Wl,-rpath=$(IFORT_RPATH)

MPI_HOME = /opt/scali
MPI_DEFINES = -DMPI
MPI_INCLUDE = -I$(MPI_HOME)/include64
MPI_LIBDIR = $(MPI_HOME)/lib64
MPI_LIBS = -L$(MPI_LIBDIR) -lmpi -lfmpi
#####

On 10/7/07, Robert Duke <rduke_at_email.unc.edu> wrote:

     Hi Lars,
     Okay, a library you are specifying in the link line is not being
found where
     you said it was by the linker. So you need to be sure that 1)
you actually
     are linking to the files needed by the current version of mvapich, and 2)
     your have that location as the value for MPI_LIBDIR2. You can
get that info
     for your mvapich by doing the following:
     1) first enter the command 'which mpif77' to see where the mpif77 command
     currently in your path is. If that looks like a likely location for an
     mvapich install, move to step 2; if not, you may want to talk to whoever
     installed mpi s/w on the machine (probably really want to do this anyway).
     2) Once you are sure you have the right mpif77, enter a 'mpif77
-link_info'.
     This should give you the location and names of all the library files you
     need for mvapich, as installed on your machine, to run.
     I cover this, and a variety of other useful issues in the README
file under
     amber9/src/pmemd (this stuff is specifically in the section
entitled "Beware
     of Nonstandard MPI Library Installations". The problem we
encounter here is
     that the other products we link to are not configuration-static,
and library
     requirements may change, or in many instances, the folks that
installed mpi
     either did not put it in the recommended location, and/or worse, actually
     changed the names of things to get around problems with multiple
things with
     the same name (the classic case - changing the mpi compile script names to
     specify the compiler in use). In an ideal world, I would do more
     autoconfigure. In the real world, pmemd runs on everything from
the biggest
     supercomputers around down to single cpu workstations (unlike a number of
     the other amber programs, I don't attempt to also do windows
laptops, enough
     is enough), and a lot of these configurations are nonstandard,
and there are
     even big headaches with the compute nodes being configured
differently than
     the compile nodes. So the bottom line is that you know nothing about the
     hardware and system software, then the probability pmemd will install
     correctly and run well is small in any case (ie., I want the
installs being
     done by folks who do know the machines).

     One final problem. Once you have the right mpi implementation,
it needs to
     have been built with knowledge about which fortran compiler will
be used for
     building your application. This is the case because the name
mangling done
     by different fortran compilers is different (and even
configurable), so when
     the compiler encounters a statement like 'call mpi_foo()' in the code, it
     may assume it is actually supposed to go looking for mpi_foo, or maybe
     mpi_foo_, or mpi_foo__, or several other patterns that are generated by
     different name mangling schemes (time to read the mpi
documentation and the
     fortran compiler documentation).

     All these issues have been discussed by both Ross and myself at
some length
     on the amber.scripps.edu webpage. Ross is probably more up-to-date on
     current changes; I typically am evolving the pmemd algorithms and doing
     other things that relate to electrostatics methods, so I
generally only look
     in detail at how everything has changed in the software and hardware
     environments in the six months or so in front of an amber release.

     Hope this helps you to get the job done, and also helps to
explain why it is
     not drop-dead easy. Hopefully Russ Brown will have some specific info for
     you about Sun machines, and your sys admin person can make sure that you
     have the best mpi implementation for the job and it is configured
to use the
     correct interface (ie., infiniband between the multicore nodes, and shared
     memory between the cores themselves).

     Best Regards - Bob Duke

     ----- Original Message -----
     From: <Lars.Skjarven_at_biomed.uib.no>
     To: <amber_at_scripps.edu>
     Sent: Sunday, October 07, 2007 6:40 AM
     Subject: AMBER: PMEMD configuration and scaling

> Bob, Ross, Thank you for your helpful replies. I will definitively get
> back here with the jac benchmark results as Bob propose. This is amber9
> yes.. Whether or not the scali mpi is setup to use the infiniband or
> not, I have no idea, and will definitively check that with the tech on
> Monday.
>
> After your reply yesterday I used the day to try and compile it with
> ifort and mvapich2 as you suggest. However, it results in the following
> error:
>
> IPO link: can not find -lmtl_common
> ifort: error: problem during multi-file optimization
compilation (code 1)
> make[1]: *** [pmemd] Error 1
>
> From the config.h file, the following is defined which may cause some
> trouble?
> MPI_LIBS
> = -L$(MPI_LIBDIR) -lmpich -L$(MPI_LIBDIR2) -lmtl_common -lvapi
-lmosal -lmpga
> -lpthread
>
> Using
> - "Intel ifort compiler found; version information: Version 9.1"
> - Intel MKL (under /site/intel/cmkl/8.1)
> - NetCDF
> - mvapich2 (/site/mvapich2)
> - Inifinband libraries (/usr/lib64/infiniband)
>
> Hoping you see anything that can help me out.. Thanks again..
>
> Lars
>
> On 10/6/07, Ross Walker < ross_at_rosswalker.co.uk> wrote:
>
> Hi Lars,
>
> I have never used scali MPI - first question - are you certain it is
> setup
> to use the infiniband interconnect and not going over gigabit
> ethernet? -
> Those numbers look to me like it's going over ethernet.
>
> For infiniband I would recommend using MVAPICH / MVAPICH2 or VMI2 -
> both
> compiled using the Intel compiler (yes I know they are Opteron chips
> but
> surprise surprise the Intel compiler produces the fastest code on
> opterons
> in my experience) and then compile PMEMD with the same compiler.
>
> Make sure you run the MPI benchmarks with the mpi installation and
> check
> that you are getting ping-pong and random-ring latencies and
> bandwidths that
> match the specs of the infiniband - All to All tests etc will also
> check you
> don't have a flakey cable connection which can be common with
> infiniband.
>
> Good luck.
> Ross
>
> /\
> \/
> |\oss Walker
>
> | HPC Consultant and Staff Scientist |
> | San Diego Supercomputer Center |
> | Tel: +1 858 822 0854 | EMail:- ross_at_rosswalker.co.uk |
> | http://www.rosswalker.co.uk | PGP Key available on request |
>
> Note: Electronic Mail is not secure, has no guarantee of
delivery, may
> not
> be read every day, and should not be used for urgent or sensitive
> issues.
>
> > -----Original Message-----
> > From: owner-amber_at_scripps.edu
> > [mailto: owner-amber_at_scripps.edu] On Behalf Of
> > Lars.Skjarven_at_biomed.uib.no
> > Sent: Saturday, October 06, 2007 04:35
> > To: amber_at_scripps.edu
> > Subject: AMBER: PMEMD configuration and scaling
> >
> >
> > Dear Amber Users,
> >
> > We recently got access to a cluster consisting of Opteron
> > dual-cpu-dual-core (4
> > cores) SUN nodes with InfiniBand interconnects. After what I
> > have read about
> > pmemd and scaling, this hardware should be good enough to
> > achieve relatively
> > good scaling up to at least 16-32 cpu's (correct?). However,
> > my small benchmark
> > test yields a peak at 8 cpu's (two nodes):
> >
> > 2 cpus: 85 ps/day - 100%
> > 4 cpus: 140 ps/day - 81%
> > 8 cpus: 215 ps/day - 62%
> > 12 cpus: 164 ps/day - 31%
> > 16 cpus: 166 ps/day - 24%
> > 32 cpus: 111 ps/day - 8%
> >
> > This test is done using 400.000 atoms and with a simulation of 20
> ps.
> >
> > Is it possible that our configuration of pmemd can cause this
> > problem? If so, do
> > you see any apparent flaws in the config.h file below?
> >
> > In the config.h below we use ScaliMPI and ifort (./configure
> > linux64_opteron
> > ifort mpi). We also have pathscale and portland as available
> > compilers. however,
> > I never managed to build pmemd using these..
> >
> > Any hints and tips will be highly appreciated.
> >
> > Best regards,
> > Lars Skjærven
> > University of Bergen, Norway
> >
> > ## config.h file ##
> > MATH_DEFINES =
> > MATH_LIBS =
> > IFORT_RPATH =
> > /site/intel/fce/9.1/lib:/site/intel/cce/9.1/lib:/opt/scali/lib
> > 64:/opt/scali/lib:/opt/gridengine/lib/lx26-amd64:/site/pathsca
> le/lib/3.0/32:/site/pathscale/lib/3.0:/op
> > t/gridengine/lib/lx26-amd64:/opt/globus/lib:/opt/lam/gnu/lib
> > MATH_DEFINES = -DMKL
> > MATH_LIBS = -L/site/intel/cmkl/8.1/lib/em64t -lmkl_em64t -lpthread
> > FFT_DEFINES = -DPUBFFT
> > FFT_INCLUDE =
> > FFT_LIBS =
> > NETCDF_HOME = /site/NetCDF
> > NETCDF_DEFINES = -DBINTRAJ
> > NETCDF_MOD = netcdf.mod
> > NETCDF_LIBS = $(NETCDF_HOME)/lib/libnetcdf.a
> > DIRFRC_DEFINES = -DDIRFRC_EFS -DDIRFRC_NOVEC
> > CPP = /lib/cpp
> > CPPFLAGS = -traditional -P
> > F90_DEFINES = -DFFTLOADBAL_2PROC
> >
> > F90 = ifort
> > MODULE_SUFFIX = mod
> > F90FLAGS = -c -auto
> > F90_OPT_DBG = -g -traceback
> > F90_OPT_LO = -tpp7 -O0
> > F90_OPT_MED = -tpp7 -O2
> > F90_OPT_HI = -tpp7 -xW -ip -O3
> > F90_OPT_DFLT = $(F90_OPT_HI)
> >
> > CC = gcc
> > CFLAGS =
> >
> > LOAD = ifort
> > LOADFLAGS = -L/opt/scali/lib64 -lmpi -lfmpi
> > LOADLIBS = -limf -lsvml -Wl,-rpath=$(IFORT_RPATH)
> > ## config.h ends ##
> >
> >
> >
> >
> > --------------------------------------------------------------
> > ---------
> > The AMBER Mail Reflector
> > To post, send mail to amber_at_scripps.edu
> > To unsubscribe, send "unsubscribe amber" to majordomo_at_scripps.edu
> >
>
>
>
-----------------------------------------------------------------------
> The AMBER Mail Reflector
> To post, send mail to amber_at_scripps.edu
> To unsubscribe, send "unsubscribe amber" to majordomo_at_scripps.edu
>
>
>
> -----------------------------------------------------------------------
> The AMBER Mail Reflector
> To post, send mail to amber_at_scripps.edu
> To unsubscribe, send "unsubscribe amber" to majordomo_at_scripps.edu
>

     -----------------------------------------------------------------------
     The AMBER Mail Reflector
     To post, send mail to amber_at_scripps.edu
     To unsubscribe, send "unsubscribe amber" to majordomo_at_scripps.edu

-----------------------------------------------------------------------
The AMBER Mail Reflector
To post, send mail to amber_at_scripps.edu
To unsubscribe, send "unsubscribe amber" to majordomo_at_scripps.edu