[aspect-devel] error when writing checkpoint file
Magali Billen
mibillen at ucdavis.edu
Mon Jul 9 00:58:02 PDT 2018
Hi Julianne,
Thanks, if I had gotten this far with aspect earlier in my sabbatical, this would have been a great resource, but my sabbatical
is officially over and I’ll be heading back to the US during the fall quarter.
So, I think I need to look at resources in the US for now.
Magali
> On Jul 8, 2018, at 6:28 PM, Juliane Dannberg <judannberg at gmail.com> wrote:
>
> Magali,
> one thing I can really recommend is making use of the High-Performance Computing Centers in Germany that are built to support research.
> I only have experience with the HLRN (which is for universities in Northern Germany) and you would have to get an account at the HLRS (four Southern Germany). You can get a free test account <https://www.hlrs.de/solutions-services/academic-users/test-access/> with 50.000 core hours right away if you plan to apply for computing time, and from my experience with the HLRN, they are usually pretty quick with reviewing applications (for us, it took about a month to get the project, but they had quarterly application deadlines).
> If you are affiliated with a university in Southern Germany, I think you will get the computing time for free, and it is usually not much effort to write a proposal. I also felt the support staff were very professional and helpful, and we made a really good experience with running ASPECT models on their machines (on up to 1500 cores for time-dependent models).
> Cheers,
> Juliane
>
> On 07/08/2018 07:44 AM, Timo Heister wrote:
>> Magali,
>>
>> can you try to set "number of grouped files" to 1 and see if this
>> works when you run with 2 or more cores?
>>
>> Note that it was just a guess on my end that MPI IO is the problem. It
>> might be something else we are triggering inside p4est.
>>
>> On Sun, Jul 8, 2018 at 4:19 AM Magali Billen <mibillen at ucdavis.edu> <mailto:mibillen at ucdavis.edu> wrote:
>>> Hi everyone,
>>>
>>> My cluster is really old (GigE…), and is perhaps of a dying breed of “individual PI” clusters.
>>> So, that problem is not fixable until I write a grant to add nodes to a larger, newer cluster with a more modern set-up.
>>>
>>> Bill - is MPI-IO enabled on Peleton? This cluster, or one like it, is what I would be buying nodes to add to.
>>>
>>> Wolfgang - your responses help to solve another mystery of why VTU works and not checkpointing.
>>> I started my PRM files from a file that John Naliboff gave me (he was using my cluster with a visiting student),
>>> and in it the parameter “Number of grouped files” is set to zero (see below). I had not dug into what that meant,
>>> but now its clear.
>>>
>>> Maybe the only related question, is whether it is possible to create a similar variable for Checkpointing?
>>> If not, I guess that's just really strong motivation for me to write an IFR proposal quickly ;-) (and a proposal for time
>>> on a national lab machine).
>>>
>>> -Magali
>>>
>>> # INFORMATION ON OUTPUT TO BE CREATED
>>> subsection Postprocess
>>> set List of postprocessors = visualization, velocity statistics, temperature statistics
>>>
>>> subsection Visualization
>>> set List of output variables = density, viscosity, strain rate
>>> set Output format = vtu
>>> set Time between graphical output = 0.10e6
>>> set Number of grouped files = 0
>>> end
>>> end
>>>
>>>
>>>> On Jul 8, 2018, at 5:22 AM, Wolfgang Bangerth <bangerth at colostate.edu> <mailto:bangerth at colostate.edu> wrote:
>>>>
>>>> On 07/07/2018 06:45 AM, Magali Billen wrote:
>>>>> I’ll be at CIDER the last two weeks of July and I’ll try to talk to Rene in person about this issue and try to understand
>>>>> more about what options might exists. Since this is all handled by other libraries (p4est), there may be no real option. I
>>>>> don’t feel like I have the expertise or experience with Aspect to wade into this on my own. Maybe after talking with Rene,
>>>>> we can see about trying to compiling p4est with mpi-io and see what happens.
>>>> p4est can be configured to disable MPI-IO. So that's a problem that can be solved. But deal.II also uses MPI-IO, here:
>>>>
>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dealii_dealii_blob_master_source_base_data-5Fout-5Fbase.cc-23L7286&d=DwIFaQ&c=Ngd-ta5yRYsqeUsEDgxhcqsYYY1Xs5ogLxWPA_2Wlc4&r=c08Btfq4m9QEScXN3ZQwLZzzWQE7S8CYq1IYuzKV_Zk&m=y9C9vb_vomIN-3tVapS2IIRVdYUvMB4xfn9pOkNxI68&s=vJUTaxRsIyqCAsdOOH8xWMlRmOK5PJN_RU7OMVVIiYg&e= <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dealii_dealii_blob_master_source_base_data-5Fout-5Fbase.cc-23L7286&d=DwIFaQ&c=Ngd-ta5yRYsqeUsEDgxhcqsYYY1Xs5ogLxWPA_2Wlc4&r=c08Btfq4m9QEScXN3ZQwLZzzWQE7S8CYq1IYuzKV_Zk&m=y9C9vb_vomIN-3tVapS2IIRVdYUvMB4xfn9pOkNxI68&s=vJUTaxRsIyqCAsdOOH8xWMlRmOK5PJN_RU7OMVVIiYg&e=>
>>>>
>>>> This deal.II function is called from essentially all ASPECT runs:
>>>>
>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_geodynamics_aspect_blob_master_source_postprocess_visualization.cc-23L594&d=DwIFaQ&c=Ngd-ta5yRYsqeUsEDgxhcqsYYY1Xs5ogLxWPA_2Wlc4&r=c08Btfq4m9QEScXN3ZQwLZzzWQE7S8CYq1IYuzKV_Zk&m=y9C9vb_vomIN-3tVapS2IIRVdYUvMB4xfn9pOkNxI68&s=5YWmXmPeTwHWBTODyC3uNUM8HgGXShapMXyso39CVJQ&e= <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_geodynamics_aspect_blob_master_source_postprocess_visualization.cc-23L594&d=DwIFaQ&c=Ngd-ta5yRYsqeUsEDgxhcqsYYY1Xs5ogLxWPA_2Wlc4&r=c08Btfq4m9QEScXN3ZQwLZzzWQE7S8CYq1IYuzKV_Zk&m=y9C9vb_vomIN-3tVapS2IIRVdYUvMB4xfn9pOkNxI68&s=5YWmXmPeTwHWBTODyC3uNUM8HgGXShapMXyso39CVJQ&e=>
>>>>
>>>> The default for the number of grouped files is 16, and I suspect that most people leave it as is -- so basically everyone ends up in the `else` branch in line 598.
>>>>
>>>> In other words, while I don't know whether people use checkpoint/restart frequently, pretty much everyone I know uses VTU output, and that uses MPI-IO. I can't really reconcile this, but it seems to suggest that MPI-IO must work for most of our users.
>>>>
>>>> Best
>>>> Wolfgang
>>>>
>>>>
>>>> --
>>>> ------------------------------------------------------------------------
>>>> Wolfgang Bangerth email: bangerth at colostate.edu <mailto:bangerth at colostate.edu>
>>>> www: https://urldefense.proofpoint.com/v2/url?u=http-3A__www.math.colostate.edu_-7Ebangerth_&d=DwIFaQ&c=Ngd-ta5yRYsqeUsEDgxhcqsYYY1Xs5ogLxWPA_2Wlc4&r=c08Btfq4m9QEScXN3ZQwLZzzWQE7S8CYq1IYuzKV_Zk&m=y9C9vb_vomIN-3tVapS2IIRVdYUvMB4xfn9pOkNxI68&s=TUx1aZuf9zhb89hGAhffqpCu2xRVQJRbYRgoGPCkN9s&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.math.colostate.edu_-7Ebangerth_&d=DwIFaQ&c=Ngd-ta5yRYsqeUsEDgxhcqsYYY1Xs5ogLxWPA_2Wlc4&r=c08Btfq4m9QEScXN3ZQwLZzzWQE7S8CYq1IYuzKV_Zk&m=y9C9vb_vomIN-3tVapS2IIRVdYUvMB4xfn9pOkNxI68&s=TUx1aZuf9zhb89hGAhffqpCu2xRVQJRbYRgoGPCkN9s&e=>
>>>>
>>> ____________________________________________________________
>>> Professor of Geophysics
>>> Earth & Planetary Sciences Dept., UC Davis
>>> Davis, CA 95616
>>> 2129 Earth & Physical Sciences Bldg.
>>> Office Phone: (530) 752-4169
>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__magalibillen.faculty.ucdavis.edu&d=DwIFaQ&c=Ngd-ta5yRYsqeUsEDgxhcqsYYY1Xs5ogLxWPA_2Wlc4&r=c08Btfq4m9QEScXN3ZQwLZzzWQE7S8CYq1IYuzKV_Zk&m=y9C9vb_vomIN-3tVapS2IIRVdYUvMB4xfn9pOkNxI68&s=ST9ZpC9d-huOnaBrBTuHiLv5_iYcrhexJMoBXuns99g&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__magalibillen.faculty.ucdavis.edu&d=DwIFaQ&c=Ngd-ta5yRYsqeUsEDgxhcqsYYY1Xs5ogLxWPA_2Wlc4&r=c08Btfq4m9QEScXN3ZQwLZzzWQE7S8CYq1IYuzKV_Zk&m=y9C9vb_vomIN-3tVapS2IIRVdYUvMB4xfn9pOkNxI68&s=ST9ZpC9d-huOnaBrBTuHiLv5_iYcrhexJMoBXuns99g&e=>
>>>
>>> Currently on Sabbatical at Munich University (LMU)
>>> Department of Geophysics (PST + 9 hr)
>>>
>>> Avoid implicit bias - check before you submit:
>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.tomforth.co.uk_genderbias_&d=DwIFaQ&c=Ngd-ta5yRYsqeUsEDgxhcqsYYY1Xs5ogLxWPA_2Wlc4&r=c08Btfq4m9QEScXN3ZQwLZzzWQE7S8CYq1IYuzKV_Zk&m=y9C9vb_vomIN-3tVapS2IIRVdYUvMB4xfn9pOkNxI68&s=pcYkjkVZe6Fm65h3jTIbtSFYXMKC7KFrmHFOT3-STvM&e= <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.tomforth.co.uk_genderbias_&d=DwIFaQ&c=Ngd-ta5yRYsqeUsEDgxhcqsYYY1Xs5ogLxWPA_2Wlc4&r=c08Btfq4m9QEScXN3ZQwLZzzWQE7S8CYq1IYuzKV_Zk&m=y9C9vb_vomIN-3tVapS2IIRVdYUvMB4xfn9pOkNxI68&s=pcYkjkVZe6Fm65h3jTIbtSFYXMKC7KFrmHFOT3-STvM&e=>
>>> ___________________________________________________________
>>>
>>
>
> _______________________________________________
> Aspect-devel mailing list
> Aspect-devel at geodynamics.org
> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/aspect-devel
____________________________________________________________
Professor of Geophysics
Earth & Planetary Sciences Dept., UC Davis
Davis, CA 95616
2129 Earth & Physical Sciences Bldg.
Office Phone: (530) 752-4169
http://magalibillen.faculty.ucdavis.edu
Currently on Sabbatical at Munich University (LMU)
Department of Geophysics (PST + 9 hr)
Avoid implicit bias - check before you submit:
http://www.tomforth.co.uk/genderbias/
___________________________________________________________
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.geodynamics.org/pipermail/aspect-devel/attachments/20180709/a6c214ab/attachment.html>
More information about the Aspect-devel
mailing list