[aspect-devel] Performance issue with Trilinos solver on Aspect
bangerth at math.tamu.edu
Fri Sep 13 19:12:03 PDT 2013
let me move this discussion to the aspect developer mailing list.
> Currently we try to put a parameterized core evolution code as the bottom
> thermal boundary condition to couple core evolution with mantle flow, we may
> considering contribute this to aspect later when we make it good looking and
> bug free.
That would of course be fantastic. One of the things that ultimately we as a
community need to do is couple the different parts of the earth description,
and coupling a mantle convection solver with a core evolution model would be
an excellent step in this direction!
> However at the moment I have a weird performance issue using both
> multi-threads and MPI. The Deal.II and Aspect doing well with both of them,
> but the Trilinos solver they use doesn't. When I configure Trilinos without
> "Trilinos_ENABLE_OpenMP=ON" it will only work with single thread (check with
> top, only one CPU is used).
That's expected, right? (I.e., if you configure *without* OpenMP, then you
expect Trilinos to only use one thread.) Or do you mean that *Aspect* only
uses one thread in this case?
> While when I enable it, the solver do use more
> cores (also checked with top, multiple CPU used) but the solver works even
> slower than the single thread build !!! I wonder if did something wrong in the
> configuration and you may give me some hint. Thanks.
That's a good question. I have never tried to use OpenMP with Trilinos. Do you
have a link that would describe what Trilinos actually does in this case,
i.e., where it uses threads? Also: I assume that you are using MPI at the same
time -- how many cores do you have per machine, and how many MPI processes do
you have running?
My best guess is that Trilinos only uses OpenMP for relatively trivial things,
like vector-vector products. One typically gets only rather moderate speedups
this way -- for example a speedup of 2 on 8 cores -- but you now use 8 cores
completely. If at the same time you have 8 MPI jobs running on this machine,
you will get contention for these processors since now every MPI process is
trying to run on 8 cores, and the end result is that everything is slower. On
the other hand, if you just started 8 single-threaded MPI jobs, then you would
get a speedup (compared to one core) of around 8.
> By the way, I met your friend Luc in Korea last month, he is a very nice guy.
> After knowing I am using deal.II and aspect he pushed me to send you an email.
Yes, please continue to do so. This is what the mailing lists are there for.
(Luc was my geophysics postdoc mentor. I learned an incredible lot from him.)
Wolfgang Bangerth email: bangerth at math.tamu.edu
More information about the Aspect-devel