[aspect-devel] Performance issue with Trilinos solver on Aspect

[Siqi Zhang]

Hi Timo and Wolfgang,

Thanks for your replies.
I only launch one process on each node with MPI, and expect TBB/OpenMP to
make use of all the CPUs on the nodes.
It seem aspect is using TBB+MPI and Epetra of Trilinos uses OpenMP+MPI. As
Timo suggested TBB is not getting along well with OpenMP, I think that why
I have the performance issue.
The TBB+MPI combination of aspect works quite well here, only the solver
takes longer time when I enable OpenMP on Trilinos.
Right now I am using Aspect with OpenMP disabled on Trilinos, I think I
will stay that way. I will launch more MPI processes to make use of all the
CPUs. Just feel sad that this great two layer parallelism doesn't work out
together.

Regards,

Siqi

On Sunday, September 15, 2013, Timo Heister wrote:

> >> While when I enable it, the solver do use more
> >> cores (also checked with top, multiple CPU used) but the solver works
> even
> >> slower than the single thread build !!! I wonder if did something wrong
> in
> >> the
> >> configuration and you may give me some hint. Thanks.
>
> We have not experimented with multithreading inside Trilinos and the
> combination with deal.II. The big problem is that mixing Intel TBB
> (what we are using) with OpenMP in a single program won't work as far
> as I know. Both systems will mess with thread affinities and create
> threads that are busy-waiting and hogging cores.
> This is a big problem in the scientific computing community. For now I
> recommend disabling OpenMP (export OMP_NUM_THREADS=1) and using MPI
> only.
>
>
> --
> Timo Heister
> http://www.math.clemson.edu/~heister/
>


-- 
Siqi Zhang
Research Associate
Department of Earth and Planetary Sciences
Faculty of Science
Macquarie University

E6A, Rm 248
Macquarie University
Macquarie Park, NSW 2113
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://geodynamics.org/pipermail/aspect-devel/attachments/20130916/90513823/attachment-0001.html>