[aspect-devel] Writing output in parallel

[Timo Heister]

> Now for a couple of questions:
> - I don't recall who actually put the throttling of output in there.
>   Martin? Timo? Do you recall the reasons?

I originally added the throttling, because on Hurr with more than
about 100 cores some of the writes to nfs would fail, if all were done
at the same time.

> - Any other ideas?

Yes. I have a solution I worked on some over the weekend that will
simplify and speed up the I/O. I can easily merge several or all of
the .vtu files into one big file (a vtu file can contain more than one
<piece>, so I can basically just copy the files together and add a
header). This is preferable for several reasons, but most importantly
filesystems do not deal well with a ton of files in the same
directory.
I wrote a small test application that uses MPI I/O to write test data
into one big file. Internally MPI I/O will group the writes to a few
nodes (so effectively only a small number of processes will access the
file system).
The performance is impressive: Now I can write with over 500MB/s (a
large 10GB file) to the gluster filesystem. Without the grouping (so
it is like posix I/O) I get about 50MB/s. This is without throttling
(as it is just one file). With the additional throttling, I assume we
are getting even less than 50MB/s right now.

So:
- we rip out all the background writing because it will be fast enough
without it
- I will finish the write_vtu_in_parallel() routine, that organizes
writing into larger vtu files with MPI I/O.

-- 
Timo Heister
http://www.math.tamu.edu/~heister/