[aspect-devel] [cse.ucdavis.edu #13562] Fwd: Fwd: error when writing checkpoint files

[Magali Billen]

Hi everyone, 

My cluster is really old (GigE…), and is perhaps of a dying breed of “individual PI” clusters.
 So, that problem is not fixable until I write a grant to add nodes to a larger, newer cluster with a more modern set-up.

Bill - is MPI-IO enabled on Peleton? This cluster, or one like it, is what I would be buying nodes to add to.

Wolfgang  -  your responses help to solve another mystery of why VTU works and not checkpointing. 
I started my PRM files from a file that John Naliboff gave me (he was using my cluster with a visiting student),
 and in it the parameter “Number of grouped files” is set to zero (see below).  I had not dug into what that meant, 
but now its clear. 

Maybe the only related question, is whether it is possible to create a similar variable for Checkpointing?
If not, I guess that's just really strong motivation for me to write an IFR proposal quickly ;-) (and a proposal for time
on a national lab machine).

-Magali

# INFORMATION ON OUTPUT TO BE CREATED
subsection Postprocess
  set List of postprocessors = visualization, velocity statistics, temperature statistics

  subsection Visualization
    set List of output variables      = density, viscosity, strain rate
    set Output format                 = vtu
    set Time between graphical output = 0.10e6
    set Number of grouped files       = 0
  end
end


> On Jul 8, 2018, at 5:22 AM, Wolfgang Bangerth <bangerth at colostate.edu> wrote:
> 
> On 07/07/2018 06:45 AM, Magali Billen wrote:
>> I’ll be at CIDER the last two weeks of  July and I’ll try to talk to Rene in person about this issue and try to understand
>> more about what options might exists.  Since this is all handled by other libraries (p4est), there may be no real option.   I
>>  don’t feel like I have the expertise or experience with Aspect to wade into this on my own. Maybe after talking with Rene,
>> we can see about trying to compiling p4est with mpi-io and see what happens.
> 
> p4est can be configured to disable MPI-IO. So that's a problem that can be solved. But deal.II also uses MPI-IO, here:
> 
> https://github.com/dealii/dealii/blob/master/source/base/data_out_base.cc#L7286
> 
> This deal.II function is called from essentially all ASPECT runs:
> 
> https://github.com/geodynamics/aspect/blob/master/source/postprocess/visualization.cc#L594
> 
> The default for the number of grouped files is 16, and I suspect that most people leave it as is -- so basically everyone ends up in the `else` branch in line 598.
> 
> In other words, while I don't know whether people use checkpoint/restart frequently, pretty much everyone I know uses VTU output, and that uses MPI-IO. I can't really reconcile this, but it seems to suggest that MPI-IO must work for most of our users.
> 
> Best
> Wolfgang
> 
> 
> -- 
> ------------------------------------------------------------------------
> Wolfgang Bangerth          email:                 bangerth at colostate.edu
>                           www: http://www.math.colostate.edu/~bangerth/
> 

____________________________________________________________
Professor of Geophysics 
Earth & Planetary Sciences Dept., UC Davis
Davis, CA 95616
2129 Earth & Physical Sciences Bldg.
Office Phone: (530) 752-4169
http://magalibillen.faculty.ucdavis.edu

Currently on Sabbatical at Munich University (LMU)
Department of Geophysics (PST + 9 hr)

Avoid implicit bias - check before you submit: 
http://www.tomforth.co.uk/genderbias/
___________________________________________________________