AMBER Archive (2009)

Subject: RE: [AMBER] ptraj limit on trajectory size

From: Ross Walker (ross_at_rosswalker.co.uk)
Date: Mon Aug 24 2009 - 17:07:00 CDT

Hi Sasha,

Looking at the code in clusterLib.c one can find the source of the warning:

CLUSTER_API SymmetricMatrix* AllocateSymmetricMatrix(int Size)
{
    SymmetricMatrix* pMatrix;
    /* */
    if (Size >= 32768) {
        fprintf(stderr, "Error in AllocateSymmetricMatrix, Size (%d) of
SymmetricMatrix is too big. Please limit the size to be less
 than 32768.\n", Size);
        exit(1);
    }
    unsigned int DataBlockSizeInBytes;
    pMatrix = (SymmetricMatrix*)SafeMalloc(__FILE__, __LINE__,
sizeof(Matrix));
    pMatrix->Size = Size;
    DataBlockSizeInBytes = (sizeof(float) * Size * (Size+1)) / 2;
    pMatrix->Data = (float*)SafeMalloc(__FILE__, __LINE__,
DataBlockSizeInBytes);
    memset(pMatrix->Data,0,DataBlockSizeInBytes);
    return pMatrix;
}

The issue at hand is the size of DataBlockSizeInBytes. If you calculate this
with size >= 32768 you find that it exceeds 2GB in size and this is the
origin of the limit. Note in principle since DataBlockSizeInBytes is
unsigned this could be increased to 4GB which would be 46341. However, this
might break the code elsewhere if this value ends up in a signed int. In
principle one could try recompiling with 64 bit ints which if the unsigned
works then the limit could be 3037000500 assuming of course that you have 16
exa bytes of memory.

I have no idea though if a 64 bit integer compile will break anything else
in the code. Often problems occur if integer offsets are assumed or it links
to some external library that expects 32 bit ints. I think ptraj does not do
this though so you could try forcing a 64 bit int compile and see if it 1)
does not segfault and 2) passes some test cases. Obviously commenting out
the check above.

This of course does not help you with the huge memory requirement of
clustering. Probably the algorithm should be updated to not require N^2
memory if such memory is not available. This is unfortunately WORK. :-(.

Good luck,
Ross

/\
\/
|\oss Walker

| Assistant Research Professor |
| San Diego Supercomputer Center |
| Tel: +1 858 822 0854 | EMail:- ross_at_rosswalker.co.uk |
| http://www.rosswalker.co.uk | PGP Key available on request |

Note: Electronic Mail is not secure, has no guarantee of delivery, may not
be read every day, and should not be used for urgent or sensitive issues.

> -----Original Message-----
> From: amber-bounces_at_ambermd.org [mailto:amber-bounces_at_ambermd.org] On
> Behalf Of Sasha Buzko
> Sent: Monday, August 24, 2009 10:00 AM
> To: AMBER Mailing List
> Subject: [AMBER] ptraj limit on trajectory size
>
> Hi all,
> ptraj appears to have a hard-coded limit on the number of trajectory
> frames that can be used as the input for clustering. I have a rather
> long trajectory, and I keep hitting an AllocateSymmetricMatrix error
> with a complaint that the size of the matrix should be less than 32768.
> Is there a way to change this value in the source code and recompile
> ptraj to handle bigger frame sets? Or is it an inherent algorithmic
> restriction?
>
> Thanks for any suggestions
>
> Sasha
>
> _______________________________________________
> AMBER mailing list
> AMBER_at_ambermd.org
> http://lists.ambermd.org/mailman/listinfo/amber

_______________________________________________
AMBER mailing list
AMBER_at_ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber