[aspect-devel] Writing output in parallel

[Timo Heister]

> When you
> merge it using (synchronous) MPI I/O, you still have at least one node
> that has to wait till the writing is finished. This will take a long
> time no matter what you do (writing from each node to the central file
> system or writing via MPI I/O) and that the only way to avoid this is
> doing it in the background.

For a cluster with a decent parallel file system (like the gluster in
our hurr cluster) we can write 500MB/s or more. That means writing 180
Million DoFs in 3d takes a second or two compared to solving the
linear system which takes a minute or more. I don't think we need to
do tricks with background i/o, especially because we don't want to
output every time step anyway.

> Is there an asynchronous MPI I/O interface where you hand MPI the data
> and then go off doing something else while MPI decides what to do with
> the data?

Yes, but the default MPI implementation is not thread-safe (you
discovered that too). So one can only start a non-blocking
MPI_file_write* and then complete it at a later point in time. But
where in the code are we going to complete the call (I don't see a
clean way to do that)? Additionally, MPI I/O will use the fast
interconnect and not the slow ethernet in the case Thomas has.
I think Thomas is best of writing to the local disk only. There is no
danger in symlinking the output folder to local scratch space or a
different filesystem (I am doing that all the time).

I think we have two cases:
1. big cluster with fast interconnect: use MPI I/O and hope we have a
fast parallel filesystem (not some NFS)
2. slow small cluster or desktop: we just use the method we have right
now and you have the option to write to local disk space.

-- 
Timo Heister
http://www.math.tamu.edu/~heister/