AMBER Archive (2007)

Subject: Re: AMBER: PMEMD configuration and scaling

From: Robert Duke (rduke_at_email.unc.edu)
Date: Sun Oct 07 2007 - 10:04:03 CDT

Hi Lars,
Okay, a library you are specifying in the link line is not being found where
you said it was by the linker. So you need to be sure that 1) you actually
are linking to the files needed by the current version of mvapich, and 2)
your have that location as the value for MPI_LIBDIR2. You can get that info
for your mvapich by doing the following:
1) first enter the command 'which mpif77' to see where the mpif77 command
currently in your path is. If that looks like a likely location for an
mvapich install, move to step 2; if not, you may want to talk to whoever
installed mpi s/w on the machine (probably really want to do this anyway).
2) Once you are sure you have the right mpif77, enter a 'mpif77 -link_info'.
This should give you the location and names of all the library files you
need for mvapich, as installed on your machine, to run.
I cover this, and a variety of other useful issues in the README file under
amber9/src/pmemd (this stuff is specifically in the section entitled "Beware
of Nonstandard MPI Library Installations". The problem we encounter here is
that the other products we link to are not configuration-static, and library
requirements may change, or in many instances, the folks that installed mpi
either did not put it in the recommended location, and/or worse, actually
changed the names of things to get around problems with multiple things with
the same name (the classic case - changing the mpi compile script names to
specify the compiler in use). In an ideal world, I would do more
autoconfigure. In the real world, pmemd runs on everything from the biggest
supercomputers around down to single cpu workstations (unlike a number of
the other amber programs, I don't attempt to also do windows laptops, enough
is enough), and a lot of these configurations are nonstandard, and there are
even big headaches with the compute nodes being configured differently than
the compile nodes. So the bottom line is that you know nothing about the
hardware and system software, then the probability pmemd will install
correctly and run well is small in any case (ie., I want the installs being
done by folks who do know the machines).

One final problem. Once you have the right mpi implementation, it needs to
have been built with knowledge about which fortran compiler will be used for
building your application. This is the case because the name mangling done
by different fortran compilers is different (and even configurable), so when
the compiler encounters a statement like 'call mpi_foo()' in the code, it
may assume it is actually supposed to go looking for mpi_foo, or maybe
mpi_foo_, or mpi_foo__, or several other patterns that are generated by
different name mangling schemes (time to read the mpi documentation and the
fortran compiler documentation).

All these issues have been discussed by both Ross and myself at some length
on the amber.scripps.edu webpage. Ross is probably more up-to-date on
current changes; I typically am evolving the pmemd algorithms and doing
other things that relate to electrostatics methods, so I generally only look
in detail at how everything has changed in the software and hardware
environments in the six months or so in front of an amber release.

Hope this helps you to get the job done, and also helps to explain why it is
not drop-dead easy. Hopefully Russ Brown will have some specific info for
you about Sun machines, and your sys admin person can make sure that you
have the best mpi implementation for the job and it is configured to use the
correct interface (ie., infiniband between the multicore nodes, and shared
memory between the cores themselves).

Best Regards - Bob Duke

----- Original Message -----
From: <Lars.Skjarven_at_biomed.uib.no>
To: <amber_at_scripps.edu>
Sent: Sunday, October 07, 2007 6:40 AM
Subject: AMBER: PMEMD configuration and scaling

> Bob, Ross, Thank you for your helpful replies. I will definitively get
> back here with the jac benchmark results as Bob propose. This is amber9
> yes.. Whether or not the scali mpi is setup to use the infiniband or
> not, I have no idea, and will definitively check that with the tech on
> Monday.
>
> After your reply yesterday I used the day to try and compile it with
> ifort and mvapich2 as you suggest. However, it results in the following
> error:
>
> IPO link: can not find -lmtl_common
> ifort: error: problem during multi-file optimization compilation (code 1)
> make[1]: *** [pmemd] Error 1
>
> From the config.h file, the following is defined which may cause some
> trouble?
> MPI_LIBS
> = -L$(MPI_LIBDIR) -lmpich -L$(MPI_LIBDIR2) -lmtl_common -lvapi -lmosal -lmpga
> -lpthread
>
> Using
> - "Intel ifort compiler found; version information: Version 9.1"
> - Intel MKL (under /site/intel/cmkl/8.1)
> - NetCDF
> - mvapich2 (/site/mvapich2)
> - Inifinband libraries (/usr/lib64/infiniband)
>
> Hoping you see anything that can help me out.. Thanks again..
>
> Lars
>
> On 10/6/07, Ross Walker < ross_at_rosswalker.co.uk> wrote:
>
> Hi Lars,
>
> I have never used scali MPI - first question - are you certain it is
> setup
> to use the infiniband interconnect and not going over gigabit
> ethernet? -
> Those numbers look to me like it's going over ethernet.
>
> For infiniband I would recommend using MVAPICH / MVAPICH2 or VMI2 -
> both
> compiled using the Intel compiler (yes I know they are Opteron chips
> but
> surprise surprise the Intel compiler produces the fastest code on
> opterons
> in my experience) and then compile PMEMD with the same compiler.
>
> Make sure you run the MPI benchmarks with the mpi installation and
> check
> that you are getting ping-pong and random-ring latencies and
> bandwidths that
> match the specs of the infiniband - All to All tests etc will also
> check you
> don't have a flakey cable connection which can be common with
> infiniband.
>
> Good luck.
> Ross
>
> /\
> \/
> |\oss Walker
>
> | HPC Consultant and Staff Scientist |
> | San Diego Supercomputer Center |
> | Tel: +1 858 822 0854 | EMail:- ross_at_rosswalker.co.uk |
> | http://www.rosswalker.co.uk | PGP Key available on request |
>
> Note: Electronic Mail is not secure, has no guarantee of delivery, may
> not
> be read every day, and should not be used for urgent or sensitive
> issues.
>
> > -----Original Message-----
> > From: owner-amber_at_scripps.edu
> > [mailto: owner-amber_at_scripps.edu] On Behalf Of
> > Lars.Skjarven_at_biomed.uib.no
> > Sent: Saturday, October 06, 2007 04:35
> > To: amber_at_scripps.edu
> > Subject: AMBER: PMEMD configuration and scaling
> >
> >
> > Dear Amber Users,
> >
> > We recently got access to a cluster consisting of Opteron
> > dual-cpu-dual-core (4
> > cores) SUN nodes with InfiniBand interconnects. After what I
> > have read about
> > pmemd and scaling, this hardware should be good enough to
> > achieve relatively
> > good scaling up to at least 16-32 cpu's (correct?). However,
> > my small benchmark
> > test yields a peak at 8 cpu's (two nodes):
> >
> > 2 cpus: 85 ps/day - 100%
> > 4 cpus: 140 ps/day - 81%
> > 8 cpus: 215 ps/day - 62%
> > 12 cpus: 164 ps/day - 31%
> > 16 cpus: 166 ps/day - 24%
> > 32 cpus: 111 ps/day - 8%
> >
> > This test is done using 400.000 atoms and with a simulation of 20
> ps.
> >
> > Is it possible that our configuration of pmemd can cause this
> > problem? If so, do
> > you see any apparent flaws in the config.h file below?
> >
> > In the config.h below we use ScaliMPI and ifort (./configure
> > linux64_opteron
> > ifort mpi). We also have pathscale and portland as available
> > compilers. however,
> > I never managed to build pmemd using these..
> >
> > Any hints and tips will be highly appreciated.
> >
> > Best regards,
> > Lars Skjærven
> > University of Bergen, Norway
> >
> > ## config.h file ##
> > MATH_DEFINES =
> > MATH_LIBS =
> > IFORT_RPATH =
> > /site/intel/fce/9.1/lib:/site/intel/cce/9.1/lib:/opt/scali/lib
> > 64:/opt/scali/lib:/opt/gridengine/lib/lx26-amd64:/site/pathsca
> le/lib/3.0/32:/site/pathscale/lib/3.0:/op
> > t/gridengine/lib/lx26-amd64:/opt/globus/lib:/opt/lam/gnu/lib
> > MATH_DEFINES = -DMKL
> > MATH_LIBS = -L/site/intel/cmkl/8.1/lib/em64t -lmkl_em64t -lpthread
> > FFT_DEFINES = -DPUBFFT
> > FFT_INCLUDE =
> > FFT_LIBS =
> > NETCDF_HOME = /site/NetCDF
> > NETCDF_DEFINES = -DBINTRAJ
> > NETCDF_MOD = netcdf.mod
> > NETCDF_LIBS = $(NETCDF_HOME)/lib/libnetcdf.a
> > DIRFRC_DEFINES = -DDIRFRC_EFS -DDIRFRC_NOVEC
> > CPP = /lib/cpp
> > CPPFLAGS = -traditional -P
> > F90_DEFINES = -DFFTLOADBAL_2PROC
> >
> > F90 = ifort
> > MODULE_SUFFIX = mod
> > F90FLAGS = -c -auto
> > F90_OPT_DBG = -g -traceback
> > F90_OPT_LO = -tpp7 -O0
> > F90_OPT_MED = -tpp7 -O2
> > F90_OPT_HI = -tpp7 -xW -ip -O3
> > F90_OPT_DFLT = $(F90_OPT_HI)
> >
> > CC = gcc
> > CFLAGS =
> >
> > LOAD = ifort
> > LOADFLAGS = -L/opt/scali/lib64 -lmpi -lfmpi
> > LOADLIBS = -limf -lsvml -Wl,-rpath=$(IFORT_RPATH)
> > ## config.h ends ##
> >
> >
> >
> >
> > --------------------------------------------------------------
> > ---------
> > The AMBER Mail Reflector
> > To post, send mail to amber_at_scripps.edu
> > To unsubscribe, send "unsubscribe amber" to majordomo_at_scripps.edu
> >
>
>
> -----------------------------------------------------------------------
> The AMBER Mail Reflector
> To post, send mail to amber_at_scripps.edu
> To unsubscribe, send "unsubscribe amber" to majordomo_at_scripps.edu
>
>
>
> -----------------------------------------------------------------------
> The AMBER Mail Reflector
> To post, send mail to amber_at_scripps.edu
> To unsubscribe, send "unsubscribe amber" to majordomo_at_scripps.edu
>

-----------------------------------------------------------------------
The AMBER Mail Reflector
To post, send mail to amber_at_scripps.edu
To unsubscribe, send "unsubscribe amber" to majordomo_at_scripps.edu