<div dir="ltr"><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Jan 17, 2014 at 7:55 AM, Timo Heister <span dir="ltr"><<a href="mailto:heister@clemson.edu" target="_blank">heister@clemson.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class="im">> Figured that I should give this a shot, since I was the one interested in<br>
> PETSc/Trilinos comparisons. I am using petsc 3.4.2 optimized and Trilinos<br>
> 11.2.3. For the most part here I am following suit by testing with<br>
> composition-passive.prm.<br>
<br>
</div>Thanks for looking into this!<br>
<div class="im"><br>
> 1) I don't see any assertions being hit in debug mode (but see later).<br>
<br>
</div>good.<br>
<div class="im"><br></div></blockquote><div style="font-family:arial,helvetica,sans-serif" class="gmail_default">I take it back, I did find an Assertion being tripped, but in the Trilinos debug mode. In the temperature statistics postprocessor, calls to local_range() are failing the assertion </div>
<div style="font-family:arial,helvetica,sans-serif" class="gmail_default"><br></div><div style="font-family:arial,helvetica,sans-serif" class="gmail_default">end-begin == vector->Map().NumMyElements()</div><div style="font-family:arial,helvetica,sans-serif" class="gmail_default">
<br></div><div style="font-family:arial,helvetica,sans-serif" class="gmail_default">in trilinos_vector_base.h. Seems like probably an easy fix somewhere?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div class="im">
> 2) Visually, the results seem to be pretty much the same for what I have<br>
> looked at. I am, however, seeing surpisingly large differences in iteration<br>
> counts. Specifically, for the temperature/composition solves with PETSc,<br>
> iteration counts are 30-50, as compared to 10-15 for Trilinos.<br>
<br>
</div>This might be related to different handling of residuals (do we use<br>
Trilinos, PETSc, or deal.II inner solvers?) but most likely this is<br>
just an artifact of different preconditioners (Hyper vs ML). They were<br>
similar for me in the past, but only for scalar Poisson type problems.<br>
<div class="im"><br></div></blockquote><div style="font-family:arial,helvetica,sans-serif" class="gmail_default">Sure, especially for the AMG preconditioners. I'm a little surprised that there is such a difference for the ILU preconditioners.</div>
<div style="font-family:arial,helvetica,sans-serif" class="gmail_default"></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div class="im">
> 3) The overall speed of the two versions is not wildly different for what I<br>
> have looked at. Certainly nothing like 50x difference. Perhaps it comes<br>
> down to optimized vs debug PETSc?<br>
<br>
</div>This is aspect running in debug or optimized mode? My 50x was optimized.<br>
<div class="im"><br></div></blockquote><div style="font-family:arial,helvetica,sans-serif" class="gmail_default">Optimized mode, but only on one process.</div><div style="font-family:arial,helvetica,sans-serif" class="gmail_default">
</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class="im">
> Okay, I have run into a big problem, though. Something is up with running<br>
> on several processors with PETSc in optimized mode. Basically, aspect never<br>
> gets started. It looks like not all the processors are returning from the<br>
> MPI initialization step, but I'm still trying to track it down. It runs<br>
> fine in debug mode, or with one process in optimized mode.<br>
<br>
</div>PETSc optimized or Aspect optimized?<br>
<br></blockquote><div style="font-family:arial,helvetica,sans-serif" class="gmail_default">Both are optimized.</div><div style="font-family:arial,helvetica,sans-serif" class="gmail_default"></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Do you get any output if you do<br>
mpirun -n 2 ./aspect -log_summary<br>
or "-log_trace out" and look at the out.* files?<br>
<div class=""><div class="h5"><br>
<br></div></div></blockquote><div style="font-family:arial,helvetica,sans-serif" class="gmail_default">Okay, due to the hanging, log_summary does not produce any output. log_trace, however, seems to have useful information. I was wrong about it not getting past the MPI_Init stage. </div>
<div style="font-family:arial,helvetica,sans-serif" class="gmail_default"><br></div><div style="font-family:arial,helvetica,sans-serif" class="gmail_default">In general, there are several thousand petsc calls per processor, and then the processors appear to get out of sync and hang. That is to say, each process is reporting the same set of calls for a while, but then start reporting different calls. Very strange. Here is some output of `paste out.0 out.1 | awk '$5 != $10'` for a two process run:</div>
<div style="font-family:arial,helvetica,sans-serif" class="gmail_default"><br></div><div class="gmail_default"><div class="gmail_default"><font face="arial, helvetica, sans-serif">[0] 1.06899 Event begin: MatMult<span class="" style="white-space:pre"> </span>[1] 1.06912 Event begin: VecCopy</font></div>
<div class="gmail_default"><font face="arial, helvetica, sans-serif"> [0] 1.069 Event begin: VecScatterBegin<span class="" style="white-space:pre"> </span>[1] 1.06913 Event end: VecCopy</font></div><div class="gmail_default">
<font face="arial, helvetica, sans-serif"> [0] 1.06901 Event end: VecScatterBegin<span class="" style="white-space:pre"> </span>[1] 1.06914 Event begin: VecScale</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif"> [0] 1.06904 Event begin: VecScatterEnd<span class="" style="white-space:pre"> </span>[1] 1.06914 Event end: VecScale</font></div>
<div class="gmail_default"><font face="arial, helvetica, sans-serif"><span class="" style="white-space:pre"> </span>[1] 1.06918 Event begin: PCApply</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif"><span class="" style="white-space:pre"> </span> [1] 1.06919 Event begin: VecPointwiseMult</font></div>
<div class="gmail_default"><font face="arial, helvetica, sans-serif"><span class="" style="white-space:pre"> </span> [1] 1.06919 Event end: VecPointwiseMult</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif"><span class="" style="white-space:pre"> </span>[1] 1.0692 Event end: PCApply</font></div>
<div class="gmail_default"><font face="arial, helvetica, sans-serif"><span class="" style="white-space:pre"> </span>[1] 1.0692 Event begin: VecCopy</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif"><span class="" style="white-space:pre"> </span>[1] 1.06921 Event end: VecCopy</font></div>
<div class="gmail_default"><font face="arial, helvetica, sans-serif"><span class="" style="white-space:pre"> </span>[1] 1.06921 Event begin: VecScale</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif"><span class="" style="white-space:pre"> </span>[1] 1.06922 Event end: VecScale</font></div>
<div class="gmail_default"><font face="arial, helvetica, sans-serif"><span class="" style="white-space:pre"> </span>[1] 1.06922 Event begin: VecDot</font></div><div style="font-family:arial,helvetica,sans-serif"><br></div>
<div style="font-family:arial,helvetica,sans-serif">This is where it hangs, after ~15k calls that are matched. For more than two processes, the same general thing seems to happen: after a few thousand PETSc calls, one or more of the processors starts doing something different and the whole thing hangs.</div>
</div><div style="font-family:arial,helvetica,sans-serif" class="gmail_default"><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div class=""><div class="h5">
--<br>
Timo Heister<br>
<a href="http://www.math.clemson.edu/~heister/" target="_blank">http://www.math.clemson.edu/~heister/</a><br>
_______________________________________________<br>
Aspect-devel mailing list<br>
<a href="mailto:Aspect-devel@geodynamics.org">Aspect-devel@geodynamics.org</a><br>
<a href="http://geodynamics.org/cgi-bin/mailman/listinfo/aspect-devel" target="_blank">http://geodynamics.org/cgi-bin/mailman/listinfo/aspect-devel</a><br>
</div></div></blockquote></div><br></div></div>