AMBER Archive (2008)

Subject: Re: RE: RE: AMBER: memory usage

From: David A. Case (case_at_scripps.edu)
Date: Mon Apr 28 2008 - 12:26:22 CDT

On Mon, Apr 28, 2008, Ye Mei wrote:
>
> Even larger memory is required for pmemd. So I attach config.h for pmemd in
> this mail. Maybe it can help.
>

I can (sort of) confirm what Ye Mei is seeing. Here are memory stats on our
altix (1.6Ghz itanium2) while running the factor_ix benchmark using pmemd 10:

One node:

Mem: 15936736k total, 8689712k used, 7247024k free, 0k buffers
Swap: 10239968k total, 42208k used, 10197760k free, 7722784k cached

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24046 case 25 0 150m 137m 3072 R 100 0.9 0:14.83 pmemd.nopar

Two nodes:

Mem: 15936736k total, 8756944k used, 7179792k free, 0k buffers
Swap: 10239968k total, 42208k used, 10197760k free, 7722784k cached

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23467 case 25 0 4090m 116m 4464 R 100 0.7 0:14.49 pmemd
23466 case 25 0 4089m 105m 5328 R 99 0.7 0:14.43 pmemd

Four nodes:

Mem: 15936736k total, 8836128k used, 7100608k free, 0k buffers
Swap: 10239968k total, 42208k used, 10197760k free, 7722784k cached

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23245 case 25 0 7970m 88m 4784 R 100 0.6 1:02.49 pmemd
23246 case 25 0 7970m 88m 4800 R 100 0.6 1:02.48 pmemd
23247 case 25 0 7970m 89m 4768 R 100 0.6 1:02.47 pmemd
23244 case 25 0 7970m 78m 6496 R 99 0.5 1:02.26 pmemd

Eight nodes:

Mem: 15936736k total, 9013200k used, 6923536k free, 0k buffers
Swap: 10239968k total, 42208k used, 10197760k free, 7722784k cached

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23372 case 25 0 15.4g 75m 5296 R 100 0.5 0:24.44 pmemd
23373 case 25 0 15.4g 75m 5312 R 100 0.5 0:24.45 pmemd
23374 case 25 0 15.4g 75m 5280 R 100 0.5 0:24.44 pmemd
23376 case 25 0 15.4g 75m 5280 R 100 0.5 0:24.45 pmemd
23377 case 25 0 15.4g 75m 5296 R 100 0.5 0:24.45 pmemd
23378 case 25 0 15.4g 75m 5280 R 100 0.5 0:24.45 pmemd
23375 case 25 0 15.4g 76m 5344 R 100 0.5 0:24.34 pmemd
23371 case 25 0 15.4g 66m 7824 R 99 0.4 0:24.38 pmemd

Notes:

1. The "VIRT" memory stat looks like it is certainly caused by MPI buffers,
which appear to be using about 2 Gbyte per node. The nonparallel run has
only 150 MByte of virtual memory.

2. However, the total memory used (8.6 to 9.0 Gbytes) seems to be independent
of these buffers. Even when the virtual memory on 8 nodes is 15.4g, the
reported memory using is almost the same as for the nopar run, which reports
almost no virtual memory. My particular altix shows 8.5 Gbytes of memory used
even when there are no user jobs (amber or otherwise) running, the memory used
is no higher with 8 cpus running Amber than with no cpus running Amber. So,
please double check that the large reported virtual memory is actually hurting
your machine's performance.

I'm also cc-ing these to Roberto at SGI. If it is really a problem, my guess
is that there is someway to configure SGI's mpi to allocate less virtual
memory.

...regards...dac

-----------------------------------------------------------------------
The AMBER Mail Reflector
To post, send mail to amber_at_scripps.edu
To unsubscribe, send "unsubscribe amber" to majordomo_at_scripps.edu