[aspect-devel] update memory

[Glerum, A.C. (Anne)]

Hi Wolfgang,

Thank you for helping with the memory problem.

I have updated Aspect and deal.ii yesterday to the newest versions. I use version 10.4.2 of Trilinos.

We have some problems getting valgrind to work, but I did test the same parameter file with 8 levels of global refinement on different numbers of processors using Ganglia.

For 1processor I get a flat memory usage of about 2 Gb.
For 8 processors I get a increasing memory usage now at 4 Gb after running 212 units of model time.
For 16 processors I get a increasing memory usage now at 7 Gb after running 289 units of model time.
For 32 processors I get a increasing memory usage now at 9 Gb after running 422 units of model time.

I will let them run to time=2000.

Could you tell me how much memory you need for a similar run?

Thanks!

Anne

Anne Glerum, MSc | PhD candidate | Department of Earth Sciences | Utrecht University|  Budapestlaan 4, 3584 CD Utrecht | Room Z.204 | A.C.Glerum at uu.nl<mailto:D.J.J.vanHinsbergen at uu.nl> |



On Feb 5, 2013, at 8:49 PM, Wolfgang Bangerth wrote:


Cedric & Anne,

I'll resubmit a job tomorrow morning on 1 core and see how the memory is evolving.
Right now, I have a 1024x1024 (no grid ref , i.e. the input file I sent you) running, claiming 5Gb more by the hour.

I've played with this a bit but I can't really reproduce it. Here's what
I tried:

1/ Single processor, 4 and 6 levels of global refinement: I'm running
this under valgrind's massif tool which gives me a graphical
representation of memory consumption as a function of time (or, rather,
CPU instructions). This turns out to be completely flat -- no increase
in memory over time.

2/ 8-processor MPI job, 6 levels, running under massif: also, no
increase in memory consumption for any of the 8 MPI jobs. The command I
used is
  mpirun -np 8 valgrind --tool=massif ./lib/aspect vkk.prm
which yields 8 output files that I can annotate via ms_print (part of
valgrind's massif).

3/ 32-processors, 8 levels of refinement, no massif: massif is too slow
(runs ~20 times slower than the original program) to run very large
jobs, but I'm just running this job just so and monitor memory
consumption with 'top'. I started the job 3 hours ago, it's at
approximately time=1500 now (so 3/4 done) and the memory consumption is
+- the same as right after starting it.

That leaves me with only three hypotheses:
1/ It's something specific to your environment.
2/ It's got something to do with the Trilinos/deal.II/Aspect version you
are using. Can you send the versions/svn revisions of each?
3/ Or, maybe, there is a problem with refining 10 instead of 8 times. Do
you see the same effect when you refine only 8 times? 6 times? 4 times?

It would be interesting to see if there is a difference between your
tools and mine. I don't have ganglia (which requires significant
infrastructure to install on a cluster), so can't test this. But could
you possibly run your jobs as I did above with valgrind/massif? It's
going to be very slow, so it would be useful to know up front whether
the same problem exists on coarser meshes as well.

Best & sorry for not being more help here...
 W.
--
------------------------------------------------------------------------
Wolfgang Bangerth               email:            bangerth at math.tamu.edu<mailto:bangerth at math.tamu.edu>
                                www: http://www.math.tamu.edu/~bangerth/

_______________________________________________
Aspect-devel mailing list
Aspect-devel at geodynamics.org<mailto:Aspect-devel at geodynamics.org>
http://geodynamics.org/cgi-bin/mailman/listinfo/aspect-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://geodynamics.org/pipermail/aspect-devel/attachments/20130208/8ffc8e43/attachment.htm