[aspect-devel] Performance issue with Trilinos solver on Aspect

[Wolfgang Bangerth]

Siqi,
let me move this discussion to the aspect developer mailing list.


 > Currently we try to put a parameterized core evolution code as the bottom
> thermal boundary condition to couple core evolution with mantle flow, we may
> considering contribute this to aspect later when we make it good looking and
> bug free.

That would of course be fantastic. One of the things that ultimately we as a 
community need to do is couple the different parts of the earth description, 
and coupling a mantle convection solver with a core evolution model would be 
an excellent step in this direction!


> However at the moment I have a weird performance issue using both
> multi-threads and MPI. The Deal.II and Aspect doing well with both of them,
> but the Trilinos solver they use doesn't. When I configure Trilinos without
> "Trilinos_ENABLE_OpenMP=ON" it will only work with single thread (check with
> top, only one CPU is used).

That's expected, right? (I.e., if you configure *without* OpenMP, then you 
expect Trilinos to only use one thread.) Or do you mean that *Aspect* only 
uses one thread in this case?


> While when I enable it, the solver do use more
> cores (also checked with top, multiple CPU used) but the solver works even
> slower than the single thread build !!! I wonder if did something wrong in the
> configuration and you may give me some hint. Thanks.

That's a good question. I have never tried to use OpenMP with Trilinos. Do you 
have a link that would describe what Trilinos actually does in this case, 
i.e., where it uses threads? Also: I assume that you are using MPI at the same 
time -- how many cores do you have per machine, and how many MPI processes do 
you have running?

My best guess is that Trilinos only uses OpenMP for relatively trivial things, 
like vector-vector products. One typically gets only rather moderate speedups 
this way -- for example a speedup of 2 on 8 cores -- but you now use 8 cores 
completely. If at the same time you have 8 MPI jobs running on this machine, 
you will get contention for these processors since now every MPI process is 
trying to run on 8 cores, and the end result is that everything is slower. On 
the other hand, if you just started 8 single-threaded MPI jobs, then you would 
get a speedup (compared to one core) of around 8.


> By the way, I met your friend Luc in Korea last month, he is a very nice guy.
> After knowing I am using deal.II and aspect he pushed me to send you an email.

Yes, please continue to do so. This is what the mailing lists are there for. 
(Luc was my geophysics postdoc mentor. I learned an incredible lot from him.)

Best
  W.

-- 
------------------------------------------------------------------------
Wolfgang Bangerth               email:            bangerth at math.tamu.edu
                                 www: http://www.math.tamu.edu/~bangerth/