[aspect-devel] Writing output in parallel

Wolfgang Bangerth bangerth at math.tamu.edu
Mon Feb 27 09:11:29 PST 2012


Howdy,
one of the things we had long wanted to do was to write graphical output 
in the background because I/O is slow and because other things can 
profitably be done at the same time. I implemented that a couple of 
weeks back but Thomas noticed that it crashes the simulation with a 
segmentation fault and I can confirm that this happens on my machine 
here as well.

After some digging this is what I observe: in the background_writer() 
function source/postprocess/visualization.cc that is spawned on a new 
thread and from which we do the writing, we also have code that makes 
sure that only a certain number of MPI processes write at the same time, 
using a periodic MPI_Barrier. The problem now appears to be that we have 
MPI communication going on from the main thread as well as occasionally 
from the background_writer() thread, and MPI doesn't deal with this 
gracefully. I have confirmed that the problem disappears if I remove the 
call to the MPI_Barrier.

Now for a couple of questions:
- I don't recall who actually put the throttling of output in there.
   Martin? Timo? Do you recall the reasons?
- What does one do in this case short of requiring the use of a
   thread-safe MPI library?
- I could work around this by making the maximal number of concurrent
   writers a run-time parameter, and if it is larger than the number
   of current MPI processes (i.e., no throttling would happen) do the
   background writes; if throttling would happen, writing would have to
   happen in the foreground. That would be a shame, but at least it
   should work.
- Any other ideas?

Thanks in advance
  W.

-- 
------------------------------------------------------------------------
Wolfgang Bangerth               email:            bangerth at math.tamu.edu
                                 www: http://www.math.tamu.edu/~bangerth/



More information about the Aspect-devel mailing list