[aspect-devel] Writing output in parallel

[Timo Heister]

> I have no idea how MPI would implement this sort of thing. I would
> imagine that they give asynchronous sends lower priority, but then you
> never know.

I doubt it. You typically want to finish those writes as fast as
possible too, so that it can be hidden behind local computation.

> I'm reluctant to implement one scheme over another without any evidence
> that this improves the situation at all. Do you have tools that can
> determine where the bottleneck is?

In the case Thomas has, the safest bet is to take anything regarding
MPI out of the equation and do the file transfer using system mv.

> Timo: Do you think you could try using the async MPI I/O routines, store
> the token you get back in the same way I currently store the token for
> the thread I create (or, wanted to create) and simply check whether the
> I/O has terminated next time we come to the same place (again in the
> same way I currently join the thread)?

Yes, that would be possible. It is a bit uglier, though. Right now I
have a simple DataOut::write_vtu_in_parallel("filename",
communicator); call, that does all the magic. Nonblocking would mean
that I need to put that code into aspect and expose more internals of
the vtu format (separate header/footer/main block), I fear.
I am not sure it is a good idea to wait until the next I/O to close
the file. I hope the file is valid and flushed before that. But that
could work.

> PS: Looking at our code, the next thing after postprocessing is mesh
> refinement or some linear algebra before assembly. The former has some
> lead time without communication (estimating errors) but the latter needs
> to communicate again right away.

Hm, is communication the bottleneck here? I doubt that.


-- 
Timo Heister
http://www.math.tamu.edu/~heister/