AMBER Archive (2008)

Subject: RE: AMBER: Compile AMBER 9 on TACC Ranger super computer

From: Ross Walker (ross_at_rosswalker.co.uk)
Date: Wed Aug 13 2008 - 22:39:47 CDT

Hi Lei,

 

I think you are getting caught way down in the weeds here. You almost
certainly have clashes between 32 bit and 64 bit compilations and I think
you are just making things over complicated for yourself. I'm not sure why
you want to compile your own mpich2. The only option you have for anything
close to decent performance on Ranger is to use mvapich2 or mvapich. Both
versions of mvapich leak like a sieve right now with respect to memory so in
some cases it is possible to have mvapich segfault after several hours of
running with buffer allocation errors but this tends not to be a problem in
pmemd - it is more a problem with sander.MPI which uses a lot of collectives
but you probably shouldn't be running sander on this machine, if you do
(i.e. for REMD) just be prepared to restart it every few hours if need be.
I.e multiple jobs in a single submission script. This will of course depend
on the simulation being run so you will have to test.

 

I would suggest you just try and use mvapich2 to begin with. Here is what I
did, all be it for AMBER 10 - I haven't actually tried AMBER 9 on Ranger.
The performance should better with PMEMD 10.

 

which mpif90

>/opt/apps/pgi7_1/mvapich2/1.0/bin/mpif90

 

tar xvjf AmberTools.tar.bz2

tar xvjf Amber10.tar.bz2

cd amber10

wget http://www.ambermd.org/bugfixes/10.0/bugfix.all

patch -p0 -N -r patch-rejects <bugfix.all

rm -f bugfix.all

wget http://www.ambermd.org/bugfixes/AmberTools/1.2/bugfix.all

patch -p0 -N -r patch-rejects <bugfix.all

cd src

./configure_at

make -f Makefile_at

setenv MPI_HOME /opt/apps/pgi7_1/mvapich2/1.0/

./configure_amber -mpich2 -nosanderidc pgf90

 

 

Note I had to edit evb_matrix.f here and change:

 

   use evb_pimd, only: nbead

 

to

 

#if defined(LES)

   use evb_pimd, only: nbead

#endif

 

to avoid a problem with an undefined reference to evb_pimd when linking
sander.MPI. This is probably an actual bug but I haven't looked into it
further yet.

 

make parallel

 

cd pmemd

./configure linux64_opteron pgf90 mvapich pubfft bintraj

 

>Please enter name of directory where Infiniband libraries are installed:

>/opt/apps/pgi7_1/mvapich2/1.0/lib/

 

edit config.h and change all pgf90's to mpif90

 

change MPI_LIBS line to be MPI_LIBS =

 

make

make install

 

>Run test cases for PMEMD

 

1) WORK filesystem is down AGAIN on Ranger - this seems to never work - so
for testing I do:

mkdir ~/work

setenv WORK ~/work

 

Job submission script (test_amber10_pmemd.x)...

 

#!/bin/bash

#$ -V # Inherit the submission environment

#$ -cwd # Start job in submission directory

#$ -N testPMEMD # Job Name

#$ -j y # combine stderr & stdout into stdout

#$ -o $JOB_NAME.o$JOB_ID # Name of the output file (eg. myMPI.oJobID)

#$ -pe 4way 32 # Requests 4 cores/node, 32/16 = 2 nodes total (8
cpu)

#$ -q development # Queue name

#$ -l h_rt=00:30:00 # Run time (hh:mm:ss) - 0.5 hours

 set -x #{echo cmds, use "set echo" in csh}

 setenv AMBRHOME ~/amber10

 cd $AMBERHOME/test

 setenv DO_PARALLEL ibrun

 make test.parallel

 make test.pmemd

 

qsub test_amber10_pmemd.x

 

You can also do this on a single login node with:

 

setenv DO_PARALLEL 'mpirun -np 4'

cd $AMBERHOME/test/

make clean

make test.parallel

make test.pmemd

 

This works fine for amber10 - tweaking can help with performance somewhat
although the degrees of freedom are huge... Note it can be 'very' beneficial
to leave cores idle on Ranger - It is 4 cpus x 4 cores - so running 4 cpus
per node will generally give you the most optimum scaling although you get
charged for all 16 cores so there is a tradeoff to be had - you can also try
8 cores per node - this will normally give performance far superior to 16
cores per node so works out cheaper in SUs per ns even though you get
charged for all 16. Note though that the performance you get is VERY problem
specific.

 

As for AMBER 9 you should be able to compile pmemd using pgf90 by following
the above tips for PMEMD with AMBER 10 - AMBER 9 sander might need some more
hacking due to problems with pgf90 64 bit at the time amber9 was released -
you can probably just remove the -tp p7 flag from the pgf90 lines - make
sure you test things though since the compiler bug may still be present.

 

Good luck,

Ross

 

From: owner-amber_at_scripps.edu [mailto:owner-amber_at_scripps.edu] On Behalf Of
jialei
Sent: Wednesday, August 13, 2008 3:26 PM
To: amber_at_scripps.edu
Subject: AMBER: Compile AMBER 9 on TACC Ranger super computer

 

Dear AMBER Users,

 

I am trying to compile AMBER9 on the Texas Advance Computing Center (TACC)'s
Ranger machine (ranger.tacc.utexas.edu). I cannot complete compiling the
parallel version. Could anyone please help me? Thank you very much. Here is
the details of my problems:

 

Ranger has AMD Opteron processes. So I set the configuration to
“./configure -mpich2 -opteron pgf90”.

 

Due to problems of using native MPI programs on ranger (mvapich2 and
openmpi), I have compiled a version of Mpich2 in my local directory with the
pgi compilers. And I used this Mpich2 to compile AMBER9.

 

The AMBER9 parallel compilation process stopped with the follow error
message:

 

“ /usr/bin/ld: skipping incompatible
/share/home/00654/tg458141/local/mpich2-1.0.7/lib/libmpichf90.a when
searching for -lmpichf90

/usr/bin/ld: cannot find -lmpichf90

make[1]: *** [sander.MPI] Error 2”

 

On the AMBER reflector, Dr. Ross Walker suggested that compiling mpich2 and
AMBER9 in the 32bit setting may solve the problem. So I tried to compile
mpich2 and AMBER9 again by setting '-tp p7' to force 32bit compilation.
However, same error messages were obtained when compiling AMBER9.

 

When I tried to use intel 10.1 compiler on Ranger to compile Mpich2 and
AMBER9, I got the following error message:

 

“ checking for C compiler default output file name... a.out

checking whether the C compiler works... configure: error: cannot run C
compiled programs.

If you meant to cross compile, use `--host'.

See `config.log' for more details.”

 

Any suggestions are appreciated.

 

Sincerely,

 

Lei Jia

 

  _____

用 Windows Live Spaces 展示个性自我，与好友分享生活！
<http://spaces.live.com/?page=HP> 了解更多信息！

-----------------------------------------------------------------------
The AMBER Mail Reflector
To post, send mail to amber_at_scripps.edu
To unsubscribe, send "unsubscribe amber" (in the *body* of the email)
      to majordomo_at_scripps.edu