<div dir="ltr"><div class="gmail_default"><div class="gmail_default" style><font face="arial, helvetica, sans-serif">Alright, the problem seems to be in one of the inner CG solves. I can make it disappear by replacing </font></div>
<div class="gmail_default" style><font face="arial, helvetica, sans-serif"> </font></div><div class="gmail_default" style><font face="arial, helvetica, sans-serif">SolverCG<LinearAlgebra::Vector> solver(solver_control);</font></div>
<div style>with </div><div style>PETScWrappers::SolverCG solver(solver_control);<br></div><div style><br></div><div style><br></div><div style>Some timing information follows, composition_passive.prm run in optimized mode with four processors--</div>
<div style><br></div><div class="gmail_default" style><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default" style><font face="arial, helvetica, sans-serif">Aspect optimized, PETSc optimized:</font></div>
<div class="gmail_default"><font face="arial, helvetica, sans-serif">+---------------------------------------------+------------+------------+</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">| Total wallclock time elapsed since start | 69.9s | |</font></div>
<div class="gmail_default"><font face="arial, helvetica, sans-serif">| | | |</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">| Section | no. calls | wall time | % of total |</font></div>
<div class="gmail_default"><font face="arial, helvetica, sans-serif">+---------------------------------+-----------+------------+------------+</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">| Assemble Stokes system | 101 | 9.88s | 14% |</font></div>
<div class="gmail_default"><font face="arial, helvetica, sans-serif">| Assemble composition system | 202 | 12.6s | 18% |</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">| Assemble temperature system | 101 | 8.66s | 12% |</font></div>
<div class="gmail_default"><font face="arial, helvetica, sans-serif">| Build Stokes preconditioner | 101 | 7.66s | 11% |</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">| Build composition preconditioner| 202 | 0.0144s | 0.021% |</font></div>
<div class="gmail_default"><font face="arial, helvetica, sans-serif">| Build temperature preconditioner| 101 | 0.00868s | 0.012% |</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">| Solve Stokes system | 101 | 15.2s | 22% |</font></div>
<div class="gmail_default"><font face="arial, helvetica, sans-serif">| Solve composition system | 202 | 8.11s | 12% |</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">| Solve temperature system | 101 | 4.12s | 5.9% |</font></div>
<div class="gmail_default"><font face="arial, helvetica, sans-serif">| Initialization | 2 | 0.0451s | 0.065% |</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">| Postprocessing | 101 | 1.27s | 1.8% |</font></div>
<div class="gmail_default"><font face="arial, helvetica, sans-serif">| Setup dof systems | 1 | 0.157s | 0.22% |</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">+---------------------------------+-----------+------------+------------+</font></div>
<div style="font-family:arial,helvetica,sans-serif"><br></div><div style="font-family:arial,helvetica,sans-serif">Aspect optimized, Trilinos: </div><div><div><font face="arial, helvetica, sans-serif">+---------------------------------------------+------------+------------+</font></div>
<div><font face="arial, helvetica, sans-serif">| Total wallclock time elapsed since start | 49.1s | |</font></div><div><font face="arial, helvetica, sans-serif">| | | |</font></div>
<div><font face="arial, helvetica, sans-serif">| Section | no. calls | wall time | % of total |</font></div><div><font face="arial, helvetica, sans-serif">+---------------------------------+-----------+------------+------------+</font></div>
<div><font face="arial, helvetica, sans-serif">| Assemble Stokes system | 101 | 9.16s | 19% |</font></div><div><font face="arial, helvetica, sans-serif">| Assemble composition system | 202 | 9.31s | 19% |</font></div>
<div><font face="arial, helvetica, sans-serif">| Assemble temperature system | 101 | 6.03s | 12% |</font></div><div><font face="arial, helvetica, sans-serif">| Build Stokes preconditioner | 101 | 8.51s | 17% |</font></div>
<div><font face="arial, helvetica, sans-serif">| Build composition preconditioner| 202 | 1.32s | 2.7% |</font></div><div><font face="arial, helvetica, sans-serif">| Build temperature preconditioner| 101 | 0.652s | 1.3% |</font></div>
<div><font face="arial, helvetica, sans-serif">| Solve Stokes system | 101 | 8.24s | 17% |</font></div><div><font face="arial, helvetica, sans-serif">| Solve composition system | 202 | 1.57s | 3.2% |</font></div>
<div><font face="arial, helvetica, sans-serif">| Solve temperature system | 101 | 0.836s | 1.7% |</font></div><div><font face="arial, helvetica, sans-serif">| Initialization | 2 | 0.0537s | 0.11% |</font></div>
<div><font face="arial, helvetica, sans-serif">| Postprocessing | 101 | 1.04s | 2.1% |</font></div><div><font face="arial, helvetica, sans-serif">| Setup dof systems | 1 | 0.157s | 0.32% |</font></div>
<div><font face="arial, helvetica, sans-serif">+---------------------------------+-----------+------------+------------+</font></div><div style="font-family:arial,helvetica,sans-serif"><br></div></div></div></div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Fri, Jan 17, 2014 at 11:32 AM, Ian Rose <span dir="ltr"><<a href="mailto:ian.rose@berkeley.edu" target="_blank">ian.rose@berkeley.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Okay, I can reproduce the 50x slowdown by linking to the debug PETSc. So there's that. However, I still get unsynchronized calls to PETSc when running on several processors with the debug version... It's certainly possible that something is up with my install, but the step-40 tutorial does seem to run fine.</div>
</div>
</blockquote></div><br></div>