AMBER Archive (2009)

Subject: Re: [AMBER] Error in PMEMD run

From: Robert Duke (rduke_at_email.unc.edu)
Date: Fri May 08 2009 - 13:14:12 CDT

Hi Marek,
Best approach, to me, seems to be go to:
https://registrationcenter.intel.com/RegCenter/FileSearch.aspx
Here I get rerouted due to a cookie, but there has got to be a way you have
a login dialog to get to the list of your own products. Then if you select
one of the products (it will have a limited number of versions shown), you
will get another screen that will have in the lower half the ability to
select a wide variety of different versions (all the way back to 8 for ifort
I believe). This was not that difficult in the past; more "improvements"
from intel.
Regards - Bob

----- Original Message -----
From: "Marek Malý" <maly_at_sci.ujep.cz>
To: "AMBER Mailing List" <amber_at_ambermd.org>
Sent: Friday, May 08, 2009 2:02 PM
Subject: Re: [AMBER] Error in PMEMD run

Hi Bob,

If I am not wrong I have finally found the right download link for your
recommended version of ifort + MKL.

bruno_at_phas.ubc.ca&Sequence=722173">https://registrationcenter.intel.com/RegCenter/RegisterSNInfo.aspx?sn=N3KT-C5SGT7FD&EmailID=bruno_at_phas.ubc.ca&Sequence=722173

Am I right ?

   Best,

       Marek

Dne Fri, 08 May 2009 15:29:48 +0200 Robert Duke <rduke_at_email.unc.edu>
napsal/-a:

> Okay, what is the interconnect? IF it is gigabit ethernet, you are not
> going to do well with pmemd or even sander. This particular quad core
> configuration (the 5300 series is clovertown, so I believe this is what
> you have) is widely known to not scale very well at all, even with the
> best interconnects, and it is hard to efficiently use all the cpu's on
> one node (so the scaling on the node itself is bad - insufficient memory
> bandwidth basically). One thing folks do is to not use all the cores on
> each node. That way they actually do get better performance and scaling.
> The good news is that the next generation chips (nehalem) should be
> better; I don't yet have my hands on one though. So first critical
> question - do you have a gigabit ethernet interconnect? If so, you all
> have ignored all kinds of advice to not expect this to work well on the
> list. If you do have infiniband, then there are possible configuration
> issues (lots of different bandwidth options there), possible issues with
> the bandwidth on the system bus, etc. There are many many ways to not
> get optimal performance out of this sort of hardware, and it has not been
> getting easier to get it right. Okay, the 4 node results. What you want
> from a system under increasing load is called "graceful degradation".
> This basically means that things never get worse in terms of the work
> throughput as the load increases. But unbalanced hardware systems don't
> do that, and probably what you are seeing with pmemd is because it can
> generate so much more interconnect data traffic per core, it is just
> driving the system to a more completely degraded state - ethernet would
> be really bad about this due to the fact it is a CSMA/CD protocol
> (carrier sense multiple access with collision detection; the result of
> two interconnect nics trying to transmit at the same time (a collision)
> is they both do a random exponential "backoff" (wait and try again
> later); if a collision occurs in the next time they wait a random but
> longer time, and so on - so the effect of too many nodes trying to
> communicate over an ethernet at the same time is that the bandwidth drops
> way down). Anyway, look at the amber 10 benchmarks that Ross put out on
> www. ambermd.org - this is what happens, even with really GOOD
> interconnects, with these darn 8 cpu nodes you have. Then look at my
> amber 9 benchmarks for things like the ibm sp5, for the cray xt3 (when it
> had 1 cpu/node), for topsail before it was "upgraded" to quad core 8
> cpu/node, etc. We are in the valley of the shadow now with the current
> generation of supercomputers, and should be crawling back out hopefully
> over the next two years as folks realize what they have done (you can get
> near or to a petaflop with these machines, but it is a meaningless
> petaflop - the systems cannot communicate adequately, even at the level
> of the node, to be able to solve a real world parallel problem). Anyway,
> 1) more info about your system is required to understand whether you can
> do any better than this and 2) you may be able to get more throughput
> (effective work per unit time) by not using all the cpu's in each node
> (currently I think the best results for kraken, a monster xt4 at ORNL are
> obtained using only 1 of the 4 cpu's in each node - what a mess).
> Regards - Bob
>
> ----- Original Message ----- From: "Marek Malý" <maly_at_sci.ujep.cz>
> To: "AMBER Mailing List" <amber_at_ambermd.org>
> Sent: Friday, May 08, 2009 8:54 AM
> Subject: Re: [AMBER] Error in PMEMD run
>
>
> Hi Bob,
>
> I don't know what to say ...
>
> Today I made a test to investigate SANDER/PMEMD scaling on our cluster
> after reinstaling SANDER (using ifort 11) and with old compilation of
> PMEMD (10.1.012)
> which just uses new cc/MKL dynamic libraries.
> I have obtained this results:
>
> SANDER
>
> 1node 197,79 s
> 2nodes 186,47 s
> 4nodes 218 s
>
> PMEMD
>
> 1node 145 s
> 2nodes 85 s
> 4nodes 422 s
>
> 1 node = 2 x Xeon Quad-core 5365 (3,00 GHz ) = 8 CPU cores
>
> Molecular test system has 60 000 atoms (explicit wat) with cut-off 10A,
> 1000 time steps.
>
>
> As you can see regarding to SANDER scaling is bad and is really wasting
> of
> CPUs to use more than 1 node per job.
>
> Regarding to PMEMD situation is different. As you see there are two
> pretty
> strange opposite extreems.
> Extreemly nice result for 2 nodes and extreemly bad for 4 nodes. I
> repeated and check twice ...
>
> All tests was performed using the same 4 empty nodes.
>
> This behaviour is pretty strange but if I use for whole job = SANDER
> (minimisation + heating + density), PMEMD (NPT equil, NVT production)
> 2 nodes I could have pretty fine results at least from the time point of
> view. But there is another issue regarding to reliability
> of computed data, not just here but also later when I will compute
> binding
> energ. MM_PBSA/NAB.
>
> So I do prefer your suggestion to recompile whole Amber (including PMEMD)
> with ifort v. 10.1.021 as you suggested.
>
> Could you please advise me where is possible to download this old version
> (if possible with relevant MKL and cc libraries) ?
>
> Thank you very much in advance !
>
> Best,
>
> Marek
>
>
>
>
>
>
>
>
>
>
> Dne Fri, 08 May 2009 02:11:25 +0200 Robert Duke <rduke_at_email.unc.edu>
> napsal/-a:
>
>> Marek,
>> There has been a whole other thread running about how ifort 11, various
>> versions, will hang if you try to use it to compile pmemd (actual mails
>> on the reflector right around yours...). I have recommended using ifort
>> 10.1.021 because I know it works fine. As far as ifort 11.*, I have no
>> experience, but there are reports of it hanging (this is a compiler
>> ug - the compiler is defective). I also have coworkers that have tried
>> to build gaussian 03 with ifort 11.*, and it compiles, but the
>> executables don't pass tests. I think German Ecklenberg (I am guessing
>> at the name - I unfortunately cleaned up some mail and the may amber
>> reflector postings are not available yet) did get some version of 11 to
>> work (might have been 11.0.084, but we are dealing with a very dim
>> recollection here), but I would still prefer to just trust 10.1.021...
>> Boy, you are getting to hit all the speed bumps... These days I would
>> not trust any software intel releases for about 6 months after it is
>> released - let other guys do the bleeding on the bleeding edge... Ross
>> concurs with me on this one.
>> Best Regards - Bob
>> ----- Original Message ----- From: "Marek Malý" <maly_at_sci.ujep.cz>
>> To: "AMBER Mailing List" <amber_at_ambermd.org>
>> Sent: Thursday, May 07, 2009 7:59 PM
>> Subject: Re: [AMBER] Error in PMEMD run
>>
>>
>> Dear Ross and Bob,
>>
>> first of all thank you very much for your time and effort
>> which really brought good result however some problem
>> is still present ...
>>
>> OK,
>>
>>
>> Our admin installed today ifort version 11 including corresponding cc,
>> MKL.
>>
>> Here is actual LD_LIBRARY_PATH :
>>
>> LD_LIBRARY_PATH=/opt/intel/impi/3.2.0.011/lib64:/opt/intel/mkl/10.1.0.015/lib/em64t:/opt/intel/cc/11.0.074/lib/intel64:/opt/intel/fc/11.0.074/lib/intel64::/opt/intel/impi/3.2/lib64
>>
>> Of course first of all I tried just to compile pmemd with this new
>> settings but I didn't succeeded :((
>>
>> Here is my configuration statement:
>>
>> ./configure linux_em64t ifort intelmpi
>>
>>
>> Compilation started fine but after some time it "stopped" it means just
>> progress stopped but not
>> the compilation process it means after cca 1 hour the process is still
>> alive see this part
>> of the "top" list:
>>
>> 30599 mmaly 20 0 61496 16m 7012 R 50 0.1 58:51.64 fortcom
>>
>>
>> But almost whole hour it was got jammed here:
>>
>> .....
>>
>> runmin.f90(465): (col. 11) remark: LOOP WAS VECTORIZED.
>> runmin.f90(482): (col. 3) remark: LOOP WAS VECTORIZED.
>> runmin.f90(486): (col. 3) remark: LOOP WAS VECTORIZED.
>>
>> ib/cpp -traditional -P -I/opt/intel/impi/3.2/include -DPUBFFT -DBINTRAJ
>> -DMPI -DDIRFRC_EFS -DDIRFRC_COMTRANS -DDIRFRC_NOVEC -DMKL
>> -DFFTLOADBAL_2PROC veclib.fpp veclib.f90
>> ifort -c -auto -tpp7 -xP -ip -O3 veclib.f90
>> ifort: command line remark #10148: option '-tp' not supported
>> gcc -c pmemd_clib.c
>>
>> ib/cpp -traditional -P -I/opt/intel/impi/3.2/include -DPUBFFT -DBINTRAJ
>> -DMPI -DDIRFRC_EFS -DDIRFRC_COMTRANS -DDIRFRC_NOVEC -DMKL
>> -DFFTLOADBAL_2PROC gb_alltasks_setup.fpp gb_alltasks_setup.f90
>> ifort -c -auto -tpp7 -xP -ip -O3 gb_alltasks_setup.f90
>> ifort: command line remark #10148: option '-tp' not supported
>>
>> ib/cpp -traditional -P -I/opt/intel/impi/3.2/include -DPUBFFT -DBINTRAJ
>> -DMPI -DDIRFRC_EFS -DDIRFRC_COMTRANS -DDIRFRC_NOVEC -DMKL
>> -DFFTLOADBAL_2PROC pme_alltasks_setup.fpp pme_alltasks_setup.f90
>> ifort -c -auto -tpp7 -xP -ip -O3 pme_alltasks_setup.f90
>> ifort: command line remark #10148: option '-tp' not supported
>>
>> ib/cpp -traditional -P -I/opt/intel/impi/3.2/include -DPUBFFT -DBINTRAJ
>> -DMPI -DDIRFRC_EFS -DDIRFRC_COMTRANS -DDIRFRC_NOVEC -DMKL
>> -DFFTLOADBAL_2PROC pme_setup.fpp pme_setup.f90
>> ifort -c -auto -tpp7 -xP -ip -O3 pme_setup.f90
>> ifort: command line remark #10148: option '-tp' not supported
>> pme_setup.f90(145): (col. 17) remark: LOOP WAS VECTORIZED.
>> pme_setup.f90(159): (col. 22) remark: LOOP WAS VECTORIZED.
>> pme_setup.f90(80): (col. 8) remark: LOOP WAS VECTORIZED.
>> pme_setup.f90(80): (col. 8) remark: LOOP WAS VECTORIZED.
>>
>> ib/cpp -traditional -P -I/opt/intel/impi/3.2/include -DPUBFFT -DBINTRAJ
>> -DMPI -DDIRFRC_EFS -DDIRFRC_COMTRANS -DDIRFRC_NOVEC -DMKL
>> -DFFTLOADBAL_2PROC get_cmdline.fpp get_cmdline.f90
>> ifort -c -auto -tpp7 -xP -ip -O3 get_cmdline.f90
>> ifort: command line remark #10148: option '-tp' not supported
>>
>> ib/cpp -traditional -P -I/opt/intel/impi/3.2/include -DPUBFFT -DBINTRAJ
>> -DMPI -DDIRFRC_EFS -DDIRFRC_COMTRANS -DDIRFRC_NOVEC -DMKL
>> -DFFTLOADBAL_2PROC master_setup.fpp master_setup.f90
>> ifort -c -auto -tpp7 -xP -ip -O3 master_setup.f90
>> ifort: command line remark #10148: option '-tp' not supported
>>
>> ib/cpp -traditional -P -I/opt/intel/impi/3.2/include -DPUBFFT -DBINTRAJ
>> -DMPI -DDIRFRC_EFS -DDIRFRC_COMTRANS -DDIRFRC_NOVEC -DMKL
>> -DFFTLOADBAL_2PROC pmemd.fpp pmemd.f90
>> ifort -c -auto -tpp7 -xP -ip -O3 pmemd.f90
>> ifort: command line remark #10148: option '-tp' not supported
>> <<<<-HERE IS THE LAST LINE OF THE COMPILATION PROCESS
>>
>> after this one hour I killed compilation and obtained this typical
>> messages:
>>
>> make[1]: *** Deleting file `pmemd.o'
>> make[1]: *** [pmemd.o] Error 2
>> make: *** [install] Interrupt
>>
>> I really do not understand how it is possible that compiler is using 50%
>> of CPU for one hour and get jammed in one line ...
>>
>> I have to say that compilation with old version of ifort package was
>> question of some minutes.
>>
>> It seems to me as an typical case of "infinity" loop ...
>>
>> But nevertheless then I got the idea just use the old pmemd compilation
>> with the new installed libraries (cc ...) and it works :)) !!!
>>
>> Another situation was with SANDER but after compleet recompilation of
>> Amber Tools and Amber, everything is OK ( at least now :)) ).
>>
>> So I think that my problem is solved but still is here some strange
>> question about impossiblity to finish in real time compilation
>> of the PMEMD with ifort11 package. Just to be complex I have to say,
>> that
>> I tried pmemd instalation with original "configure" file
>> but also with Bob's late night one. The result is the same. Fortunately
>> it
>> is not crucial problem for me now ...
>>
>> So thank you both again !!!
>>
>> Best,
>>
>> Marek
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Dne Thu, 07 May 2009 01:04:10 +0200 Robert Duke <rduke_at_email.unc.edu>
>> napsal/-a:
>>
>>> Oh, very good find Ross; I have not had the experience of mixing these,
>>> but I bet you are right! - Bob
>>> ----- Original Message ----- From: "Ross Walker"
>>> <ross_at_rosswalker.co.uk>
>>> To: "'AMBER Mailing List'" <amber_at_ambermd.org>
>>> Sent: Wednesday, May 06, 2009 5:53 PM
>>> Subject: RE: [AMBER] Error in PMEMD run
>>>
>>>
>>>> Hi Marek,
>>>>
>>>>> here is the content of the LD_LIBRARY_PATH variable:
>>>>>
>>>>> LD_LIBRARY_PATH=/opt/intel/impi/3.1/lib64:/opt/intel/mkl/10.0.011/lib/e
>>>>> m64t:/opt
>>>>> /intel/cce/9.1.043/lib:/opt/intel/fce/10.1.012/lib::/opt/intel/impi/3.1
>>>>> /lib64
>>>>
>>>> I suspect this is the origin of your problems... You have cce v9.1.043
>>>> defined and fce v10.1.012 defined. I bet these are not compatible.
>>>> Note
>>>> there is a libsvml.so in /intel/cce/9.1.043/lib/ and this comes first
>>>> in
>>>> your LD path so will get picked up before the Fortran one. This is
>>>> probably
>>>> leading to all sorts of problems.
>>>>
>>>> My advice would be to remove the old cce library spec from the path so
>>>> it
>>>> picks up the correct libsvml. Or upgrade your cce to match the fce
>>>> compiler
>>>> version - this should probably always be done and I am surprised Intel
>>>> let
>>>> you have mixed versions this way but alas..... <sigh>
>>>>
>>>> All the best
>>>> Ross
>>>>
>>>>
>>>> /\
>>>> \/
>>>> |\oss Walker
>>>>
>>>> | Assistant Research Professor |
>>>> | San Diego Supercomputer Center |
>>>> | Tel: +1 858 822 0854 | EMail:- ross_at_rosswalker.co.uk |
>>>> | http://www.rosswalker.co.uk | PGP Key available on request |
>>>>
>>>> Note: Electronic Mail is not secure, has no guarantee of delivery, may
>>>> not
>>>> be read every day, and should not be used for urgent or sensitive
>>>> issues.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> AMBER mailing list
>>>> AMBER_at_ambermd.org
>>>> http://lists.ambermd.org/mailman/listinfo/amber
>>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> AMBER mailing list
>>> AMBER_at_ambermd.org
>>> http://lists.ambermd.org/mailman/listinfo/amber
>>>
>>> __________ Informace od NOD32 4051 (20090504) __________
>>>
>>> Tato zprava byla proverena antivirovym systemem NOD32.
>>> http://www.nod32.cz
>>>
>>>
>>
> 

-- 
Tato zpráva byla vytvořena převratným poštovním klientem Opery:
http://www.opera.com/mail/

_______________________________________________
AMBER mailing list
AMBER_at_ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber

_______________________________________________
AMBER mailing list
AMBER_at_ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber