[aspect-devel] Writing output in parallel
Wolfgang Bangerth
bangerth at math.tamu.edu
Mon Feb 27 09:11:29 PST 2012
Howdy,
one of the things we had long wanted to do was to write graphical output
in the background because I/O is slow and because other things can
profitably be done at the same time. I implemented that a couple of
weeks back but Thomas noticed that it crashes the simulation with a
segmentation fault and I can confirm that this happens on my machine
here as well.
After some digging this is what I observe: in the background_writer()
function source/postprocess/visualization.cc that is spawned on a new
thread and from which we do the writing, we also have code that makes
sure that only a certain number of MPI processes write at the same time,
using a periodic MPI_Barrier. The problem now appears to be that we have
MPI communication going on from the main thread as well as occasionally
from the background_writer() thread, and MPI doesn't deal with this
gracefully. I have confirmed that the problem disappears if I remove the
call to the MPI_Barrier.
Now for a couple of questions:
- I don't recall who actually put the throttling of output in there.
Martin? Timo? Do you recall the reasons?
- What does one do in this case short of requiring the use of a
thread-safe MPI library?
- I could work around this by making the maximal number of concurrent
writers a run-time parameter, and if it is larger than the number
of current MPI processes (i.e., no throttling would happen) do the
background writes; if throttling would happen, writing would have to
happen in the foreground. That would be a shame, but at least it
should work.
- Any other ideas?
Thanks in advance
W.
--
------------------------------------------------------------------------
Wolfgang Bangerth email: bangerth at math.tamu.edu
www: http://www.math.tamu.edu/~bangerth/
More information about the Aspect-devel
mailing list