[aspect-devel] Performance issue with Trilinos solver on Aspect

[Wolfgang Bangerth]

Siqi,

> I only launch one process on each node with MPI, and expect TBB/OpenMP to make
> use of all the CPUs on the nodes.
> It seem aspect is using TBB+MPI and Epetra of Trilinos uses OpenMP+MPI. As
> Timo suggested TBB is not getting along well with OpenMP, I think that why I
> have the performance issue.
> The TBB+MPI combination of aspect works quite well here, only the solver takes
> longer time when I enable OpenMP on Trilinos.
> Right now I am using Aspect with OpenMP disabled on Trilinos, I think I will
> stay that way. I will launch more MPI processes to make use of all the CPUs.
> Just feel sad that this great two layer parallelism doesn't work out together.

Yes, we feel the same. But, like I said, you don't usually get particularly 
good speedup from just OpenMP anyway. It has usually been our strategy to 
disable both deal.II multithreading (=TBB) and OpenMP multithreading and just 
use as many MPI processes as there are cores per node. This usually works 
pretty well.

Best
  W.

-- 
------------------------------------------------------------------------
Wolfgang Bangerth               email:            bangerth at math.tamu.edu
                                 www: http://www.math.tamu.edu/~bangerth/