[aspect-devel] ASPECT scaling on a Cray XC30

[Timo Heister]

> Oh thanks for the hint, I did not consider this. I can repeat that small
> model with pinned processes.

This website might help to figure out what you are doing right now:
http://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager#Process-core_Binding

Otherwise you need to check the docs for the cluster you are on. If
you run with y < max_cores you want each rank to have max_cores/y
cores to use (in the same socket!).

>> - the Stokes solver is not scaling well weakly but looks okay in
>> strong scaling (I wonder why that is happening because the setup is
>> not included)
>
> Actually I used the same setup as in my earlier mail on the topic. It is
> somewhat resolution dependent because of a spherical anomaly and uses a
> loose solver tolerance to keep the computing time low.

Well, the number of iterations is constant, so that can't be the problem.

>> - the Stokes solver always needs something like 30+6 iterations, which
>> is not ideal. Can you try switching the cheap iterations off and
>> compare the runtime (a single data point would be enough)?
>
> A single data point (Global resolution 5, 96 cores, 50kDoFs/core)
> "30+2" Iterations: 4,39 s Stokes solve
> "0+9" Iterations: 7,42 s Stokes solve
> The rest of the timing is completely equal (within errors). I will redo
> some other computations with 50 kDoFs/core after my vacation next week
> to see whether the number of iterations is now more constant for fixed
> #DoFs/core.

Interesting how much more expensive the expensive iterations are. It
might be worth it to set the number of cheap iterations to 50, so that
you get something like 40+0.

>> - I wonder if it would be worthwhile to look at the second timestep to
>> ignore effects like MPI buffer creation.
>
> Yes I thought about that as well. Also a thing on the to-do list. Maybe
> we can combine all this to an example section on ASPECT scaling in the
> manual? Essentially it is just an extension of the part in the paper,
> but it might be useful for others to estimate whether ASPECT is suitable
> for their model sizes. In this case, I would maybe remove the parameter
> file of all specific options for our model (getting rid of this
> resolution dependency) and redo the models once more.

This would be quite useful to have, yes.

-- 
Timo Heister
http://www.math.clemson.edu/~heister/