[aspect-devel] PETSc support

[Ian Rose]

Hi all,

Figured that I should give this a shot, since I was the one interested in
PETSc/Trilinos comparisons.  I am using petsc 3.4.2 optimized and Trilinos
11.2.3.  For the most part here I am following suit by testing with
composition-passive.prm.

1) I don't see any assertions being hit in debug mode (but see later).

2) Visually, the results seem to be pretty much the same for what I have
looked at.  I am, however, seeing surpisingly large differences in
iteration counts.  Specifically, for the temperature/composition solves
with PETSc, iteration counts are 30-50, as compared to 10-15 for Trilinos.
 For the Stokes solve, PETSc seems to be pretty consistent with ~20
iterations, but Trilinos varies much more widely, frequently going to the
more expensive solver after 30 iterations.  I don't really understand these
results.

3) The overall speed of the two versions is not wildly different for what
I have looked at.  Certainly nothing like 50x difference.  Perhaps it comes
down to optimized vs debug PETSc?


Okay, I have run into a big problem, though.  Something is up with running
on several processors with PETSc in optimized mode.  Basically, aspect
never gets started.  It looks like not all the processors are returning
from the MPI initialization step, but I'm still trying to track it down.
 It runs fine in debug mode, or with one process in optimized mode.

I have no idea what is going on with this hanging, and it is possible that
it is specific to my machine... on the other hand, the step-40 tutorial
seems to work fine.

Any thoughts or tips?

Best,
Ian


On Sat, Dec 28, 2013 at 11:18 AM, Timo Heister <heister at clemson.edu> wrote:

> Hey everyone,
>
> I just commited a couple of patches to enable PETSc support in Aspect.
> Just go into global.h and remove the comment in front of the line
> //#define USE_PETSC
> to try it out.
>
> It seems to work (cookbooks/composition-passive.prm produces pictures
> that look reasonable), but if anybody wants to help:
> 1. Run a couple of tests in debug mode(!) in parallel(at least 2 CPUs)
> to make sure we are not hitting any Assertion
> 2. Compare the results between Trilinos and PETSc. Values and
> especially iteration numbers can be slightly different, of course. We
> need to validate that everything is correct though!
> 3. Find out why it is 50x slower than Trilinos across the board (at
> least on my machine, see below). Maybe compile PETSc in optimized
> mode? Track allocations? Just my machine?
> 4. Take a look at the linear solvers, check iteration numbers and
> residuals, check that the AMG settings are reasonable, etc.
>
> PETSc (Aspect in optimized; mpirun -n 4 ./aspect
> cookbooks/composition-passive.prm ; PETSc in debug mode):
> +---------------------------------------------+------------+------------+
> | Total wallclock time elapsed since start    |  1.19e+03s |            |
> |                                             |            |            |
> | Section                         | no. calls |  wall time | % of total |
> +---------------------------------+-----------+------------+------------+
> | Assemble Stokes system          |       101 |      68.2s |       5.7% |
> | Assemble composition system     |       202 |       400s |        33% |
> | Assemble temperature system     |       101 |       309s |        26% |
> | Build Stokes preconditioner     |       101 |        58s |       4.9% |
> | Build composition preconditioner|       202 |    0.0205s |    0.0017% |
> | Build temperature preconditioner|       101 |    0.0107s |    0.0009% |
> | Solve Stokes system             |       101 |       153s |        13% |
> | Solve composition system        |       202 |       105s |       8.8% |
> | Solve temperature system        |       101 |      51.8s |       4.3% |
> | Initialization                  |         2 |    0.0682s |    0.0057% |
> | Postprocessing                  |       101 |      37.4s |       3.1% |
> | Setup dof systems               |         1 |     0.107s |     0.009% |
> +---------------------------------+-----------+------------+------------+
>
> Trilinos:
> +---------------------------------------------+------------+------------+
> | Total wallclock time elapsed since start    |      19.5s |            |
> |                                             |            |            |
> | Section                         | no. calls |  wall time | % of total |
> +---------------------------------+-----------+------------+------------+
> | Assemble Stokes system          |       101 |      3.72s |        19% |
> | Assemble composition system     |       202 |      3.46s |        18% |
> | Assemble temperature system     |       101 |      2.08s |        11% |
> | Build Stokes preconditioner     |       101 |      3.09s |        16% |
> | Build composition preconditioner|       202 |      0.56s |       2.9% |
> | Build temperature preconditioner|       101 |     0.275s |       1.4% |
> | Solve Stokes system             |       101 |      4.26s |        22% |
> | Solve composition system        |       202 |     0.767s |       3.9% |
> | Solve temperature system        |       101 |     0.392s |         2% |
> | Initialization                  |         2 |    0.0218s |      0.11% |
> | Postprocessing                  |       101 |     0.496s |       2.5% |
> | Setup dof systems               |         1 |    0.0487s |      0.25% |
> +---------------------------------+-----------+------------+------------+
>
>
> --
> Timo Heister
> http://www.math.clemson.edu/~heister/
> _______________________________________________
> Aspect-devel mailing list
> Aspect-devel at geodynamics.org
> http://geodynamics.org/cgi-bin/mailman/listinfo/aspect-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://geodynamics.org/pipermail/aspect-devel/attachments/20140116/11a83ffb/attachment.html>