AMBER Archive (2008)

Subject: Re: AMBER: comparison of MD trajectories recorded with pmemd and sander

From: Robert Duke (rduke_at_email.unc.edu)
Date: Wed Dec 17 2008 - 10:42:07 CST

Hi Therese,
I understand your concern. But I think the general experience has been as Carlos states below - mixing runs with sander and pmemd should effectively be equivalent, and I have worked hard to insure that is the case. One does get slightly different results over some short trajectory segment by mixing sander vs. pmemd, by mixing machine architecture a vs machine architecture b, even by mixing a run on n processors vs a run on m processors. Any runs done in parallel, if done in replicate, are only identical in the mdout for somewhere between 200 and 500 steps (some of my pmemd mods make duplication out to 500 steps more likely, comparing two pmemd runs - this has to do with effectively better force and energy splining in direct force calcs). But still, network indeterminacy will cause even two identical runs on more than one processor to diverge after a while (it is that rounding error thing, and due to unpredictability of the order of completion of network events, you get different orders of addition with different rounding errors). Heck, if you use fftw, you can get slightly different results on two identical single processor runs. Why? FFTW (the library itself) has this cool feature whereby on startup it does some testing to pick the fastest fft algorithms. Sometimes, due to other events on a machine, it will get a different "best" algorithm, and while the algorithms are "equivalent", in a few hundred steps you will see a difference (note this is a "feature" of any numerical methods code that uses fftw). The ugly truth about md - it is statistical in nature, and most runs are probably done for too short of a time to get really good statistics. So using pmemd, you at least have a shot at getting better statistics. I would counsel that you use pmemd as much as you can so you get the better efficiency, and use sander for the functionality you need that pmemd does not currently support.
Best Regards - Bob Duke
  ----- Original Message -----
  From: Carlos Simmerling
  To: amber_at_scripps.edu
  Sent: Wednesday, December 17, 2008 10:12 AM
  Subject: Re: AMBER: comparison of MD trajectories recorded with pmemd and sander

  I often use sander for equilibration and pmemd for production, and consider this homogeneous. I think that is fine. the differences between the programs are comparable to differences one would expect between different architectures or running on different levels of parallelism. the same would be true for running the same force field with another program, under the assumption that all other things match, such as thermostat/barostat, and that the energies/forces have been validated to match sander, etc.

  On Wed, Dec 17, 2008 at 9:04 AM, Thérèse Malliavin <terez_at_pasteur.fr> wrote:

    Dear Prof. Duke,

    I am sorry for having forced you to answer again to questions you already discussed in the past in the AMBER discussion list.
    The purpose of my mail is not to question the enormous work which was put into the pmemd development, and I am perfectly convinced that this program does bring certainly a lot to the AMBER package.

    But, I am concerned by the following problem. Molecular modeling studies are often based on the comparison of MD trajectories run with several conditions. In that way, two sander trajectories are recorded
    with different conditions and compared. If one trajectory is recorded with
    pmemd, and the other with sander, is the comparison still meaningful?

    Also, if one uses an additional trajectory recorded by CHARMM, GROMACS or NAMD with the AMBER force-field, will the pmemd trajectory be "closer" to the sander trajectory than the CHARMM, GROMACS or NAMD trajectory?

    If two trajectories are recorded with pmemd and sander starting from the same input, should we consider that they are no more different than
    two trajectories recorded with the same program (sander or pmemd) but using different initial velocities?

    Another question is: let one suppose that a trajectory was recorded using alternatively sander and pmemd for different time intervals, in the following way: some ns with pmemd, then restart with keeping velocities
    and then additional ns with sander. Should the complete trajectory obtained with these different interval be considered as an "homogeneous" trajectory which can be analyzed as a whole?

    I am sorry for insisting on these questions, but they are important for me in order to plan future calculations. I hope that I do not waste too much your time. Also, I realize that it is probably difficult to answer these questions, except by doing tests on each studied system, but I am just interested to read your opinion about these points.

    Best regards,

    Therese Malliavin
    Unite de Bioinformatique Structurale
    Institut Pasteur, Paris
    France

    On Tue, 16 Dec 2008, Robert Duke wrote:

      Okay, this has been discussed a lot. PMEMD should replicate sander results for a couple of hundred steps at least, unless you have an unbelievably bad starting configuration with a couple of atoms on top of each other (in which case some of the force gradients are huge and the simulation is bad anyway). However, the thing with MD is that there are on the order of millions, if not billions, of calculations per step, including additions, and the thing about addition of floating point numbers on computers is that it is not truly associative - the order in which the additions are performed DOES matter, due to truncation in the floating point representation of the number. So what this means is that if you have an algorithm that is different AT ALL, even in logically insignificant ways, there will be a rounding error, and due to the nature of MD, this rounding error will rather quickly grow. The main sources of difference between pmemd and sander are probably the following: 1) a different splining function for the erf() function in pmemd for some implementations (there is an optimization, and pmemd is actually more accurate than sander), 2) workload distribution differences running in parallel (which effect which force additions will occur with net-limited precision of a 64 bit floating point number), and 3) differences in the order of force additions arising from differences in calculation and communication order. The thing to note about rounding error - we are talking about a loss in precision down around 1e-17 I believe - rather small. Now, the erf() splining errors are probably closer to 1e-11 - probably the lowest precision transcendental we have, but the other transcendental functions are probably between these two numbers in precision (rough guess, have not looked recently, and it will be machine-dependent). Now all this junk does not really matter, because your calculation is probably off by at least 1e-5 (actually much worse) based on precision of forcefield parameterization, the fact that coulomb's law does not really get electrostatics just right, the fact that (substitute here the next force term generator) just right, ... And the standard justification for not being disturbed by all this - the different errors just mean that you sample different parts of phase space, and if you run long enough, you will get it all (this last point is why I have labored so long to make pmemd fast). Run your system on some other software and you will see some more dramatic differences in phase space sampling... Heck, just change the cutoffs a bit, the fft grid densities, etc. etc. etc. I have gone on-and-on about this stuff for the last several years on the amber reflector (see ambermd.org for links), probably hitting different high and low points - perhaps worth going back to look over, if you want the complete discussion. I always jump on these questions, but am sort-of answering for Ross here because I am 3 hrs closer to Europe and he is hopefully still asleep ;-)
      Regards - Bob Duke

      ----- Original Message ----- From: "Thérèse Malliavin" <terez_at_pasteur.fr>
      To: <amber_at_scripps.edu>
      Sent: Tuesday, December 16, 2008 7:57 AM
      Subject: RE: AMBER: launching a job works with sander.MPI and fail with pmemd.MPI

      Hi Ross,

      Thank you for your mail. Finally, I tried to use AMBER 10 in place of
      AMBER 9, and pmemd runs without any problem. Now, I have another naive
      question. I already realized that pmemd runs significantly faster than
      sander even on 4 processors. But, if I compare the results obtained
      by sander and pmemd starting from the same system, as for example the
      total energy, the two runs seem not to be so much correlated. So, I would
      like to know whether we have to expect that pmemd or sander should produce
      the same numbers if the runs start from the same system. The
      differences observed come probably from a different architecture of the
      two programs, could you please tell me little bit more about that?

      Thank you for your help,

      Best regards,

      Therese

      On Mon, 15 Dec 2008, Ross Walker wrote:

        Hi Therese,

        First thing to check. PMEMD when built in parallel (which I assume you did)
        is called pmemd, not pmemd.MPI. Hence you should be getting a file not found
        error - which in parallel may be masking itself as a lamboot failure.

        Also I would make sure you do the following to run cleanly in your script:

        export AMBERHOME=/foo/bar/amber10
        lamboot
        mpirun -np 4 $AMBERHOME/exe/pmemd -O -i ...
        lamhalt

        Then you can nohup the entire script. You should probably make sure you kill
        any existing lambood or lamd instances on your machine first though since
        some will probably be left over from earlier runs. You should also make sure
        that pmemd was built with the same version of lam as your mpirun refers to.
        Makes sure you run the test cases:

        export DO_PARALLEL='mpirun -np 4'
        lamboot
        cd $AMBERHOME/test/
        make test.pmemd
        lamhalt

        Good luck,
        Ross

          -----Original Message-----
          From: owner-amber_at_scripps.edu [mailto:owner-amber_at_scripps.edu] On Behalf
          Of Thérèse Malliavin
          Sent: Monday, December 15, 2008 6:42 AM
          To: amber_at_scripps.edu
          Cc: terez_at_pasteur.fr
          Subject: AMBER: launching a job works with sander.MPI and fail with
          pmemd.MPI

          Dear AMBER Netters,

          I have a question about the use of PMEMD. It is probably a trivial
          question, but, as I did not find an answer neither on the Web pages
          neither in the manuals, I am asking it to you.

          I am doing the parallel calculations with sander.MPI using a lamd deamon
          and the command nohup to launch the job, so I am doing:

          . /Bis/shared/centos-3_x86_64/etc/custom.d/amber9_intel8.1_lam-
          7.1.2_intel-8.1.sh
          lamboot

          before starting the AMBER calculations. The typical command line for
          sander.MPI is then:

          mpirun -np 4 ${AMBERHOME}/exe/sander.MPI -O -i mdr1.in -o mdr1.out -inf
          mdr1.inf -x mdr1.crd -c eq7.rst -p prmtop -r mdr1.rst

          But, if I replace in the command line sander.MPI by pmemd.MPI:

          mpirun -np 4 ${AMBERHOME}/exe/pmemd.MPI -O -i mdr1.in -o mdr1.out -inf
          mdr1.inf -x mdr1.crd -c eq7.rst -p prmtop -r mdr1.rst

          I get an error saying that lamboot was not started.

          I am trying to do these calculation on an 64 bits 8-proc Linux machine,
          running under centos-3. The lam used is the version 7.1.2_intel-8.1.

          Also, I am only using features which should exist in PMEMD according to
          the AMBER manual.

          Do you have any idea what I could check or what to find information to fix
          this problem?

          Thank you in abvance for your help,

          Therese Malliavin
          Unite de Bioinformatique Structurale
          Institut Pasteur, Paris
          France

          -----------------------------------------------------------------------
          The AMBER Mail Reflector
          To post, send mail to amber_at_scripps.edu
          To unsubscribe, send "unsubscribe amber" (in the *body* of the email)
               to majordomo_at_scripps.edu

        -----------------------------------------------------------------------
        The AMBER Mail Reflector
        To post, send mail to amber_at_scripps.edu
        To unsubscribe, send "unsubscribe amber" (in the *body* of the email)
            to majordomo_at_scripps.edu

      -----------------------------------------------------------------------
      The AMBER Mail Reflector
      To post, send mail to amber_at_scripps.edu
      To unsubscribe, send "unsubscribe amber" (in the *body* of the email)
         to majordomo_at_scripps.edu

-----------------------------------------------------------------------
The AMBER Mail Reflector
To post, send mail to amber_at_scripps.edu
To unsubscribe, send "unsubscribe amber" (in the *body* of the email)
      to majordomo_at_scripps.edu