<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
Hi Wolfgang,
<div><br>
</div>
<div>Thank you for helping with the memory problem.</div>
<div><br>
</div>
<div>I have updated Aspect and deal.ii yesterday to the newest versions. I use version 10.4.2 of Trilinos.</div>
<div><br>
</div>
<div>We have some problems getting valgrind to work, but I did test the same parameter file with 8 levels of global refinement on different numbers of processors using Ganglia.</div>
<div><br>
</div>
<div>For 1processor I get a flat memory usage of about 2 Gb.</div>
<div>For 8 processors I get a increasing memory usage now at 4 Gb after running 212 units of model time.</div>
<div>For 16 processors I get a increasing memory usage now at 7 Gb after running 289 units of model time.</div>
<div>For 32 processors I get a increasing memory usage now at 9 Gb after running 422 units of model time.</div>
<div><br>
</div>
<div>I will let them run to time=2000.</div>
<div><br>
</div>
<div>Could you tell me how much memory you need for a similar run? </div>
<div><br>
</div>
<div>Thanks!</div>
<div><br>
</div>
<div>Anne</div>
<div><br>
<div apple-content-edited="true">
<div><b style="color: rgb(51, 0, 153); "><span lang="NL" style="font-size: 7.5pt; "><font face="Verdana">Anne Glerum, MSc </font></span></b><a name="x_SafeHtmlFilter_13a4ab32afe5707b_OLE_LINK2" style="color: rgb(153, 0, 0); "><span lang="NL" style="font-size: 8pt; "><font face="Verdana">|</font></span></a><font face="Verdana" style="color: rgb(51, 0, 153); "><span><span lang="NL" style="font-size: 8pt; "> PhD
candidate</span></span><span><span lang="NL" style="font-size: 8pt; "> </span></span></font><a name="x_SafeHtmlFilter_13a4ab32afe5707b_OLE_LINK2" style="color: rgb(153, 0, 0); "><span lang="NL" style="font-size: 8pt; "><font face="Verdana">|</font></span></a><font face="Verdana" style="color: rgb(51, 0, 153); "><span><span lang="NL" style="font-size: 8pt; "></span></span></font><font face="Verdana" style="color: rgb(51, 0, 153); "><span><span lang="NL" style="font-size: 8pt; "></span></span><span><span lang="NL" style="font-size: 8pt; "></span></span><span><span lang="NL" style="font-size: 8pt; "> Department</span></span><span><span lang="NL" style="font-size: 8pt; "> of
Earth Sciences</span></span><span><span lang="NL" style="font-size: 8pt; "> </span></span></font><a name="x_SafeHtmlFilter_13a4ab32afe5707b_OLE_LINK2" style="color: rgb(153, 0, 0); "><span lang="NL" style="font-size: 8pt; "><font face="Verdana">|</font></span></a><font face="Verdana" style="color: rgb(51, 0, 153); "><span><span lang="NL" style="font-size: 8pt; "></span></span></font><font face="Verdana" style="color: rgb(51, 0, 153); "><span><span lang="NL" style="font-size: 8pt; "></span></span><span><span lang="NL" style="font-size: 8pt; "> </span></span><span><span lang="NL" style="font-size: 8pt; ">Utrecht
University</span></span><span><span lang="NL" style="font-size: 8pt; "></span></span></font><a name="x_SafeHtmlFilter_13a4ab32afe5707b_OLE_LINK2" style="color: rgb(153, 0, 0); "><span lang="NL" style="font-size: 8pt; "><font face="Verdana">|</font></span></a><font face="Verdana" style="color: rgb(51, 0, 153); "><span><span lang="NL" style="font-size: 8pt; "></span></span></font><font face="Verdana" style="color: rgb(51, 0, 153); "><span><span lang="NL" style="font-size: 8pt; "> </span></span><span><span lang="NL" style="font-size: 8pt; "> Budapestlaan
4</span></span><span><span lang="NL" style="font-size: 8pt; "></span></span><span><span lang="NL" style="font-size: 8pt; ">, 3584 CD Utrecht</span></span><span><span lang="NL" style="font-size: 8pt; "> </span></span></font><a name="x_SafeHtmlFilter_13a4ab32afe5707b_OLE_LINK2" style="color: rgb(153, 0, 0); "><span lang="NL" style="font-size: 8pt; "><font face="Verdana">|</font></span></a><font face="Verdana" style="color: rgb(51, 0, 153); "><span><span lang="NL" style="font-size: 8pt; "></span></span></font><font face="Verdana" style="color: rgb(51, 0, 153); "><span><span lang="NL" style="font-size: 8pt; "></span></span><span><span lang="NL" style="font-size: 8pt; "> </span></span><span><span lang="NL" style="font-size: 8pt; ">Room
Z.204</span></span><span><span lang="NL" style="font-size: 8pt; "> </span></span><span><span lang="NL" style="font-size: 8pt; "></span></span></font><a name="x_SafeHtmlFilter_13a4ab32afe5707b_OLE_LINK2" style="color: rgb(153, 0, 0); "><span lang="NL" style="font-size: 8pt; "><font face="Verdana">| </font></span></a><a href="mailto:D.J.J.vanHinsbergen@uu.nl" name="x_SafeHtmlFilter_13a4ab32afe5707b_OLE_LINK2" target="_blank" style="color: rgb(51, 0, 153); "><span lang="NL" style="font-size: 8pt; "><font face="Verdana">A.C.Glerum@uu.nl</font></span></a><font face="Verdana"><span style="color: rgb(51, 0, 153); "><span style="font-size: 8pt; "> </span></span></font><a name="x_SafeHtmlFilter_13a4ab32afe5707b_OLE_LINK2" style="color: rgb(153, 0, 0); "><span lang="NL" style="font-size: 8pt; "><font face="Verdana">|</font></span></a></div>
<div><br>
</div>
<br class="Apple-interchange-newline">
</div>
<br>
<div>
<div>On Feb 5, 2013, at 8:49 PM, Wolfgang Bangerth wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<div><br>
Cedric & Anne,<br>
<br>
<blockquote type="cite">I'll resubmit a job tomorrow morning on 1 core and see how the memory is evolving.<br>
</blockquote>
<blockquote type="cite">Right now, I have a 1024x1024 (no grid ref , i.e. the input file I sent you) running, claiming 5Gb more by the hour.<br>
</blockquote>
<br>
I've played with this a bit but I can't really reproduce it. Here's what <br>
I tried:<br>
<br>
1/ Single processor, 4 and 6 levels of global refinement: I'm running <br>
this under valgrind's massif tool which gives me a graphical <br>
representation of memory consumption as a function of time (or, rather, <br>
CPU instructions). This turns out to be completely flat -- no increase <br>
in memory over time.<br>
<br>
2/ 8-processor MPI job, 6 levels, running under massif: also, no <br>
increase in memory consumption for any of the 8 MPI jobs. The command I <br>
used is<br>
mpirun -np 8 valgrind --tool=massif ./lib/aspect vkk.prm<br>
which yields 8 output files that I can annotate via ms_print (part of <br>
valgrind's massif).<br>
<br>
3/ 32-processors, 8 levels of refinement, no massif: massif is too slow <br>
(runs ~20 times slower than the original program) to run very large <br>
jobs, but I'm just running this job just so and monitor memory <br>
consumption with 'top'. I started the job 3 hours ago, it's at <br>
approximately time=1500 now (so 3/4 done) and the memory consumption is <br>
+- the same as right after starting it.<br>
<br>
That leaves me with only three hypotheses:<br>
1/ It's something specific to your environment.<br>
2/ It's got something to do with the Trilinos/deal.II/Aspect version you <br>
are using. Can you send the versions/svn revisions of each?<br>
3/ Or, maybe, there is a problem with refining 10 instead of 8 times. Do <br>
you see the same effect when you refine only 8 times? 6 times? 4 times?<br>
<br>
It would be interesting to see if there is a difference between your <br>
tools and mine. I don't have ganglia (which requires significant <br>
infrastructure to install on a cluster), so can't test this. But could <br>
you possibly run your jobs as I did above with valgrind/massif? It's <br>
going to be very slow, so it would be useful to know up front whether <br>
the same problem exists on coarser meshes as well.<br>
<br>
Best & sorry for not being more help here...<br>
W.<br>
-- <br>
------------------------------------------------------------------------<br>
Wolfgang Bangerth email: <a href="mailto:bangerth@math.tamu.edu">bangerth@math.tamu.edu</a><br>
www: <a href="http://www.math.tamu.edu/~bangerth/">
http://www.math.tamu.edu/~bangerth/</a><br>
<br>
_______________________________________________<br>
Aspect-devel mailing list<br>
<a href="mailto:Aspect-devel@geodynamics.org">Aspect-devel@geodynamics.org</a><br>
http://geodynamics.org/cgi-bin/mailman/listinfo/aspect-devel<br>
</div>
</blockquote>
</div>
<br>
</div>
</body>
</html>