ok <div>that sounds good to me</div><div>i would like to do this io in the background so i will re implement that and put it in as an option for the poor mans cluster</div><div><br></div><div>thomas</div><div><br></div><div>
<div class="gmail_quote">On Tue, Feb 28, 2012 at 5:12 PM, Timo Heister <span dir="ltr"><<a href="mailto:heister@math.tamu.edu" target="_blank">heister@math.tamu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>> When you<br>
> merge it using (synchronous) MPI I/O, you still have at least one node<br>
> that has to wait till the writing is finished. This will take a long<br>
> time no matter what you do (writing from each node to the central file<br>
> system or writing via MPI I/O) and that the only way to avoid this is<br>
> doing it in the background.<br>
<br>
</div>For a cluster with a decent parallel file system (like the gluster in<br>
our hurr cluster) we can write 500MB/s or more. That means writing 180<br>
Million DoFs in 3d takes a second or two compared to solving the<br>
linear system which takes a minute or more. I don't think we need to<br>
do tricks with background i/o, especially because we don't want to<br>
output every time step anyway.<br>
<div><br>
> Is there an asynchronous MPI I/O interface where you hand MPI the data<br>
> and then go off doing something else while MPI decides what to do with<br>
> the data?<br>
<br>
</div>Yes, but the default MPI implementation is not thread-safe (you<br>
discovered that too). So one can only start a non-blocking<br>
MPI_file_write* and then complete it at a later point in time. But<br>
where in the code are we going to complete the call (I don't see a<br>
clean way to do that)? Additionally, MPI I/O will use the fast<br>
interconnect and not the slow ethernet in the case Thomas has.<br>
I think Thomas is best of writing to the local disk only. There is no<br>
danger in symlinking the output folder to local scratch space or a<br>
different filesystem (I am doing that all the time).<br>
<br>
I think we have two cases:<br>
1. big cluster with fast interconnect: use MPI I/O and hope we have a<br>
fast parallel filesystem (not some NFS)<br>
2. slow small cluster or desktop: we just use the method we have right<br>
now and you have the option to write to local disk space.<br>
<div><br>
--<br>
Timo Heister<br>
<a href="http://www.math.tamu.edu/~heister/" target="_blank">http://www.math.tamu.edu/~heister/</a><br>
</div><div><div>_______________________________________________<br>
Aspect-devel mailing list<br>
<a href="mailto:Aspect-devel@geodynamics.org" target="_blank">Aspect-devel@geodynamics.org</a><br>
<a href="http://geodynamics.org/cgi-bin/mailman/listinfo/aspect-devel" target="_blank">http://geodynamics.org/cgi-bin/mailman/listinfo/aspect-devel</a><br>
</div></div></blockquote></div><br></div>