[aspect-devel] PETSc support

[Timo Heister]

Hey everyone,

I just commited a couple of patches to enable PETSc support in Aspect.
Just go into global.h and remove the comment in front of the line
//#define USE_PETSC
to try it out.

It seems to work (cookbooks/composition-passive.prm produces pictures
that look reasonable), but if anybody wants to help:
1. Run a couple of tests in debug mode(!) in parallel(at least 2 CPUs)
to make sure we are not hitting any Assertion
2. Compare the results between Trilinos and PETSc. Values and
especially iteration numbers can be slightly different, of course. We
need to validate that everything is correct though!
3. Find out why it is 50x slower than Trilinos across the board (at
least on my machine, see below). Maybe compile PETSc in optimized
mode? Track allocations? Just my machine?
4. Take a look at the linear solvers, check iteration numbers and
residuals, check that the AMG settings are reasonable, etc.

PETSc (Aspect in optimized; mpirun -n 4 ./aspect
cookbooks/composition-passive.prm ; PETSc in debug mode):
+---------------------------------------------+------------+------------+
| Total wallclock time elapsed since start    |  1.19e+03s |            |
|                                             |            |            |
| Section                         | no. calls |  wall time | % of total |
+---------------------------------+-----------+------------+------------+
| Assemble Stokes system          |       101 |      68.2s |       5.7% |
| Assemble composition system     |       202 |       400s |        33% |
| Assemble temperature system     |       101 |       309s |        26% |
| Build Stokes preconditioner     |       101 |        58s |       4.9% |
| Build composition preconditioner|       202 |    0.0205s |    0.0017% |
| Build temperature preconditioner|       101 |    0.0107s |    0.0009% |
| Solve Stokes system             |       101 |       153s |        13% |
| Solve composition system        |       202 |       105s |       8.8% |
| Solve temperature system        |       101 |      51.8s |       4.3% |
| Initialization                  |         2 |    0.0682s |    0.0057% |
| Postprocessing                  |       101 |      37.4s |       3.1% |
| Setup dof systems               |         1 |     0.107s |     0.009% |
+---------------------------------+-----------+------------+------------+

Trilinos:
+---------------------------------------------+------------+------------+
| Total wallclock time elapsed since start    |      19.5s |            |
|                                             |            |            |
| Section                         | no. calls |  wall time | % of total |
+---------------------------------+-----------+------------+------------+
| Assemble Stokes system          |       101 |      3.72s |        19% |
| Assemble composition system     |       202 |      3.46s |        18% |
| Assemble temperature system     |       101 |      2.08s |        11% |
| Build Stokes preconditioner     |       101 |      3.09s |        16% |
| Build composition preconditioner|       202 |      0.56s |       2.9% |
| Build temperature preconditioner|       101 |     0.275s |       1.4% |
| Solve Stokes system             |       101 |      4.26s |        22% |
| Solve composition system        |       202 |     0.767s |       3.9% |
| Solve temperature system        |       101 |     0.392s |         2% |
| Initialization                  |         2 |    0.0218s |      0.11% |
| Postprocessing                  |       101 |     0.496s |       2.5% |
| Setup dof systems               |         1 |    0.0487s |      0.25% |
+---------------------------------+-----------+------------+------------+


-- 
Timo Heister
http://www.math.clemson.edu/~heister/