AMBER Archive (2007)

Subject: AMBER: workaround for ibm xlf90 powerpc FORTRAN compiler: -q64 and -O3 yield Run.dhfr.min failure

From: Chris Moth (chris.moth_at_vanderbilt.edu)
Date: Thu Feb 01 2007 - 12:33:33 CST

I'm writing not to ask for help or to request a bugfix - but to leave a
record of a workaround for a failure of the Run.dhfr.min serial sander
test on
our JS20 IBM PowerPC cluster architecture and the IBM XLF90 Fortran
compiler (flags -O3 -qstrict -q64) for Linux (bug is in both versions
9.1 and 10.1)

Without the workaround, the failure appears as a large, positive, EEL
energy value in the Run.dhfr.min test, starting with step 2.

I hope that these notes might save someone else some time if they
encounter similar problems. And, if anyone at Scripps would like me to
do further detailed debugging, I'd be happy to work with them off-line.

The workaround is to simply compile $AMBERHOME/src/sander/runmin.f with
optimizations turned off. Though not elegant, a brute force approach is
to hand-edit $AMBERHOME/src/sander/depend so that the FOPTFLAG variable
for runmin.o is replaced with an explicit list of flags that turn off
optimization. i.e. "-O0" in place of "-O3 -qstrict" and all other
FOPTFLAGS unchanged.

-----------------------------------------------------
More information/details for those of you interested:
-----------------------------------------------------

Excerpted from config.h, our Fortran compiler options are:

#------------------------------------------------------------------------------
# Fortran preprocessing and compiler.
# FPPFLAGS holds the main Fortran options, such as whether MPI is used.
#------------------------------------------------------------------------------
FPPFLAGS= -P -traditional-cpp -DCLINK_PLAIN -DXLF90 $(AMBERBUILDFLAGS)
-DF90_TIMER
FPP= cpp $(FPPFLAGS)
FC= xlf90
FFLAGS= -qfixed -c $(LOCALFLAGS) $(AMBERBUILDFLAGS) -q64
FOPTFLAGS= -qfixed -O3 -qstrict -q64 -qmaxmem=-1 -qarch=auto -qtune=auto
-c $(LOCALFLAGS) $(AMBERBUILDFLAGS)
FREEFORMAT_FLAG= -qfree=f90 -q64

The first few lines of mdout.dhfr.min.dif after ./Run.dhfr.min fails
show clear miscaluclation.

Note that step 1 has reasonable EEL and is not a problem. But, then EEL
goes very positive in step 2 and subsequently.

$ head -18 mdout.dhfr.min.dif
83c83
< 2 -7.1873E+4 1.7336E+1 1.0535E+2 C 1855 

---
 >       2       9.1283E+5     8.1856E+1     1.8334E+2     O       13223
85c85
<  VDWAALS =     8106.9482  EEL     =   -89882.6659  HBOND      =        0.
---
 >  VDWAALS =     8106.9482  EEL     =   894818.7954  HBOND      
=        0.  <<< OUCH BAD EEL in step 2
88,91c88,91
<       3      -7.1885E+4     1.7318E+1     1.0448E+2     C        1855
<  BOND    =      443.9987  ANGLE   =     1271.5621  DIHED      =      
967.6424
<  VDWAALS =     8104.0210  EEL     =   -89883.8930  HBOND      =        
0.  <<< GOOD/EXPECTED RESULT
<  1-4 VDW =      545.7307  1-4 EEL =     6666.3248  RESTRAINT  =        0.
---
 >       3       9.1284E+5     8.1855E+1     1.8333E+2     O       13223
 >  BOND    =      447.9747  ANGLE   =     1274.3896  DIHED      =      
968.0028
 >  VDWAALS =     8106.9279  EEL     =   894831.2130  HBOND      
=        0.  <<< OUCH BAD EEL
 >  1-4 VDW =      545.8298  1-4 EEL =     6666.3730  RESTRAINT  =        0.

More about our setup
----------------------

A Summary of our architecture is here:

http://www.accre.vanderbilt.edu/mission/researcher_text/pubstext.php

To be compatible with libraries on our system, libraries that are not 
under our control, we must compile sander (at least the mpi version) 
with the -q64 flag.

( Compiling all modules with -O0 or -q32 eliminated the xlf90 problem in 
serial sander.   )

Using the newest 10.1 compiler had no impact.  -q64 with -O3/-qstrict 
continued to fail.  So, we don't think there is a "compiler patch" yet 
to fix this.

Using a binary search approach - we quickly homed in on runmin.f as the 
"problem" and, with optimization turned off for this one module, all 
appears well.

It would be nice to report this to IBM in a short sample program...  But 
without assistance, I don't think I can produce that very quickly.

I am not a FORTRAN guru - but I am very experienced in C/C++ and gdb 
debugging.  If anyone would like to coach me through a little debugging 
under xlf90, I'd be happy to try to home in on exactly what lines of 
code are so distasteful to xlf90 for ppc/-q64. 

Chris

-----------------------------------------------------------------------
The AMBER Mail Reflector
To post, send mail to amber_at_scripps.edu
To unsubscribe, send "unsubscribe amber" to majordomo_at_scripps.edu