[aspect-devel] update memory

[Timo Heister]

Hey,

is only the virtual memory consumption increasing, or also the resident size?

On Fri, Feb 8, 2013 at 11:06 AM, Glerum, A.C. (Anne) <A.C.Glerum at uu.nl> wrote:
> Hi Wolfgang,
>
> Thank you for helping with the memory problem.
>
> I have updated Aspect and deal.ii yesterday to the newest versions. I use
> version 10.4.2 of Trilinos.
>
> We have some problems getting valgrind to work, but I did test the same
> parameter file with 8 levels of global refinement on different numbers of
> processors using Ganglia.
>
> For 1processor I get a flat memory usage of about 2 Gb.
> For 8 processors I get a increasing memory usage now at 4 Gb after running
> 212 units of model time.
> For 16 processors I get a increasing memory usage now at 7 Gb after running
> 289 units of model time.
> For 32 processors I get a increasing memory usage now at 9 Gb after running
> 422 units of model time.
>
> I will let them run to time=2000.
>
> Could you tell me how much memory you need for a similar run?
>
> Thanks!
>
> Anne
>
> Anne Glerum, MSc | PhD candidate | Department of Earth Sciences | Utrecht
> University|  Budapestlaan 4, 3584 CD Utrecht | Room Z.204 | A.C.Glerum at uu.nl
> |
>
>
>
> On Feb 5, 2013, at 8:49 PM, Wolfgang Bangerth wrote:
>
>
> Cedric & Anne,
>
> I'll resubmit a job tomorrow morning on 1 core and see how the memory is
> evolving.
>
> Right now, I have a 1024x1024 (no grid ref , i.e. the input file I sent you)
> running, claiming 5Gb more by the hour.
>
>
> I've played with this a bit but I can't really reproduce it. Here's what
> I tried:
>
> 1/ Single processor, 4 and 6 levels of global refinement: I'm running
> this under valgrind's massif tool which gives me a graphical
> representation of memory consumption as a function of time (or, rather,
> CPU instructions). This turns out to be completely flat -- no increase
> in memory over time.
>
> 2/ 8-processor MPI job, 6 levels, running under massif: also, no
> increase in memory consumption for any of the 8 MPI jobs. The command I
> used is
>   mpirun -np 8 valgrind --tool=massif ./lib/aspect vkk.prm
> which yields 8 output files that I can annotate via ms_print (part of
> valgrind's massif).
>
> 3/ 32-processors, 8 levels of refinement, no massif: massif is too slow
> (runs ~20 times slower than the original program) to run very large
> jobs, but I'm just running this job just so and monitor memory
> consumption with 'top'. I started the job 3 hours ago, it's at
> approximately time=1500 now (so 3/4 done) and the memory consumption is
> +- the same as right after starting it.
>
> That leaves me with only three hypotheses:
> 1/ It's something specific to your environment.
> 2/ It's got something to do with the Trilinos/deal.II/Aspect version you
> are using. Can you send the versions/svn revisions of each?
> 3/ Or, maybe, there is a problem with refining 10 instead of 8 times. Do
> you see the same effect when you refine only 8 times? 6 times? 4 times?
>
> It would be interesting to see if there is a difference between your
> tools and mine. I don't have ganglia (which requires significant
> infrastructure to install on a cluster), so can't test this. But could
> you possibly run your jobs as I did above with valgrind/massif? It's
> going to be very slow, so it would be useful to know up front whether
> the same problem exists on coarser meshes as well.
>
> Best & sorry for not being more help here...
>  W.
> --
> ------------------------------------------------------------------------
> Wolfgang Bangerth               email:            bangerth at math.tamu.edu
>                                 www: http://www.math.tamu.edu/~bangerth/
>
> _______________________________________________
> Aspect-devel mailing list
> Aspect-devel at geodynamics.org
> http://geodynamics.org/cgi-bin/mailman/listinfo/aspect-devel
>
>
>
> _______________________________________________
> Aspect-devel mailing list
> Aspect-devel at geodynamics.org
> http://geodynamics.org/cgi-bin/mailman/listinfo/aspect-devel



-- 
Timo Heister
http://www.math.tamu.edu/~heister/