AMBER Archive (2007)

Subject: Re: AMBER: PMEMD internals !!

From: Robert Duke (rduke_at_email.unc.edu)
Date: Fri Dec 14 2007 - 07:41:43 CST

Well, if I had time, and if the whole issue of how to do this stuff best was
not a moving target, I guess I would have written a paper. The PMEMD
architecture is driven mostly by computational considerations, and is fairly
complex, so there is no way I would attempt to describe it completely in an
email. Also it is changing with each release. Versions 8 and 9 aim at low
to intermediate scaling (out to several hundred processors) and life is made
fairly complex by 1) the fact that there are multiple implementations of
most key algorithms in order to optimize for different hardware, and 2)
there is a general philosophy of dynamic adjustment of parallelization
parameters to both optimize the distribution of the workload at the
beginning of a run as well as track changes in the workload itself or in the
performance of the nodes. Thus, within limits, if the compute or
communications capability of an individual cpu changes, the loading on that
will be adjusted accordingly. In the current architecture, both fft slabs
and nonbonded atom "images" are loadbalanced. I don't want to go into
details on images, but it is basically an abstraction of atom that
establishes spatial locality for purposes of achieving significantly better
memory cache utilization as well as significantly cutting interconnect
communications overhead (mpi). When workload distribution (loadbalancing)
occurs, it is in terms of contiguous, spatially colocated images. When the
code initializes, the images are assigned to different processors, and if an
image is assigned to a given task, the probability that the corresponding
atom will be assigned to that task is high but not 1 due to
bond/angle/dihedral considerations. The task that "owns" an atom is
responsible for collecting all force terms on that atom, computing the
coordinate change in a step, and communicating the changed coordinates ot
any other task that "uses" that atom (basically sources of force terms);
there is a wrinkle on bonds/angles/dihedrals that I won't go into, but these
and some other forces are handled a bit differently for good reasons. But
here's the rub. At each list build, DIFFERENT image id's are associated
with the atoms, depending on their location in space. So the atoms are
moving. So with time (ie., significant movement of atoms), or if fft slabs
are reassigned (an operation that causes massive changes in the direct force
image assignments), the atom distribution among processors becomes
nonoptimal. So if 1) we reassign fft slabs, or 2) we detect nonoptimal
atom-task assignments using a communications workload sensing heuristic,
then we redistribute the atoms among the processors. Sorry, but you will
spend a lot of time trying to figure out the logic of what I have done, even
though there are comments, and this is just the tip of the iceberg. There
IS a logic for everything, assuming low to intermediate scaling. I am
working on stuff that makes sense in a truly highscaling environment at the
moment.
Regards - Bob Duke

----- Original Message -----
From: "Sampath Koppole" <sampathkoppole_at_yahoo.com>
To: <amber_at_scripps.edu>
Sent: Friday, December 14, 2007 4:13 AM
Subject: AMBER: PMEMD internals !!

> Dear All,
> I am going through the PMEMD code to understand how
> scaling is achieved.
>
> I guess I understand that : all atoms are divided
> among the processors (using
> do_initial_pme_atom_division routine) and each
> processor owns a subset of atoms. I want to understand
> how the long-range non-bonded and electrostatic
> interactions (most expensive, I guess !!) taken into
> account during force calculations and how do you
> minimize communication between the processors.
>
> Its interesting to see that: If required the atoms are
> redistributed using do_atm_redistribution during
> pressure calculations (since its molecule based). But
> I still don't understand why and when do you have to
> redistribute atoms !!
>
> Could anyone please give "highlights" of how PMEMD
> achieves reasonably good scaling.
>
> This information is not available anywhere and it
> might be a good idea to put it in the Amber mailing
> list for a curious users !!
>
> Thanks a lot,
> Cheers,
> Sampath.
>
>
>
>
>
> ____________________________________________________________________________________
> Looking for last minute shopping deals?
> Find them fast with Yahoo! Search.
> http://tools.search.yahoo.com/newsearch/category.php?category=shopping
> -----------------------------------------------------------------------
> The AMBER Mail Reflector
> To post, send mail to amber_at_scripps.edu
> To unsubscribe, send "unsubscribe amber" to majordomo_at_scripps.edu
>

-----------------------------------------------------------------------
The AMBER Mail Reflector
To post, send mail to amber_at_scripps.edu
To unsubscribe, send "unsubscribe amber" to majordomo_at_scripps.edu