[aspect-devel] Aspect hangs after several time steps

[Lev Karatun]

Hi,

the simulation that was run on 1 core resumed after a while (which never
happened before - sorry about the confusion), and produced an error on time
step 103 (attached). The same simulation on 8 cores is still hanging. The
simulation on 8 cores without free surface ran without problems.

Best regards,
Lev Karatun.

2016-01-26 14:04 GMT-05:00 Lev Karatun <lev.karatun at gmail.com>:

> Hi Timo,
>
> I have found a good setup that reproduces the problem (after fixing the
> error that I had, that is). On both 8 cores and 1 core the simulation stops
> on time step 59. I attached the necessary files.
>
> Best regards,
> Lev Karatun.
>
> 2016-01-26 5:15 GMT-05:00 Lev Karatun <lev.karatun at gmail.com>:
>
>> Hi Timo,
>>
>> I composed a lengthy email with answers to your questions, but then I
>> realized I have a mistake in boundary conditions, so now I need a bit more
>> time to explore if the fixed ones work correctly.
>>
>> Thank you for you help.
>>
>> Best regards,
>> Lev Karatun.
>>
>> 2016-01-25 7:09 GMT-05:00 Timo Heister <timo.heister at gmail.com>:
>>
>>> Lev,
>>>
>>> I'm not sure what could cause a hang like you observe. Your first goal
>>> should be to make this problem reproducible as quickly as possible
>>> with the smallest number of processors.
>>>
>>> > Disabling free surface (changing the top boundary to free-slip) makes
>>> the simulation hang after 1 time step
>>>
>>> This is great. How many cores do you need to have this happen? If you
>>> can reduce it further? Does this happen with any of the included
>>> cookbooks? If not, please share your input file and all details
>>> (number of cores, when it hangs, etc.).
>>>
>>>
>>> On Mon, Jan 25, 2016 at 12:01 PM, Lev Karatun <lev.karatun at gmail.com>
>>> wrote:
>>> > Hi,
>>> >
>>> > (I already reported the problem last year
>>> > (
>>> http://lists.geodynamics.org/pipermail/aspect-devel/2015-June/000919.html
>>> ),
>>> > but at that moment it wasn't as severe. Now I moved to the higher
>>> resolution
>>> > models and it has gotten much worse)
>>> >
>>> > The problem is - Aspect hangs after a (seemingly) random timestep
>>> without
>>> > producing any error. I ran several tests in attempts to narrow down the
>>> > problem, and this is what I found:
>>> > - Neither assigning a value to $TMP/$TMPDIR or adding a "Number of
>>> grouped
>>> > files" parameter helps
>>> > - Disabling visualization postprocessor doesn't help
>>> > - Running Apsect in development mode produced the same results: hanging
>>> > without any error messages
>>> > - Restarting the hanging simulation from a checkpoint helps partially,
>>> it
>>> > goes past the timestep at which in hung, but then after it hangs again
>>> > - Disabling free surface (changing the top boundary to free-slip)
>>> makes the
>>> > simulation hang after 1 time step
>>> > - Changing CFL number affects how long the simulation runs before
>>> hanging:
>>> > at 0.05 is hangs at timestep 13, at 0.1 - 56, at 0.2 it ran without
>>> hanging
>>> > up to time step 153 (with disabled visualization though)
>>> >
>>> > Could you please help me solve this problem? It's really hurting the
>>> > progress of my research.
>>> > Thanks in advance!
>>> >
>>> > Best regards,
>>> > Lev Karatun.
>>> >
>>> > _______________________________________________
>>> > Aspect-devel mailing list
>>> > Aspect-devel at geodynamics.org
>>> > http://lists.geodynamics.org/cgi-bin/mailman/listinfo/aspect-devel
>>> _______________________________________________
>>> Aspect-devel mailing list
>>> Aspect-devel at geodynamics.org
>>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/aspect-devel
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.geodynamics.org/pipermail/aspect-devel/attachments/20160126/a45adec4/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 124a.e33786755
Type: application/octet-stream
Size: 2884 bytes
Desc: not available
URL: <http://lists.geodynamics.org/pipermail/aspect-devel/attachments/20160126/a45adec4/attachment.obj>