[aspect-devel] Aspect hangs after several time steps
Rene Gassmoeller
rene.gassmoeller at mailbox.org
Tue Feb 23 14:58:33 PST 2016
Hi Lev,
sorry for the compilation trouble. The issues should be fixed now
(courtesy of Ian, https://github.com/geodynamics/aspect/pull/768).
Your model actually discovered something like a bug, I do not exactly
know how it ran on Timo's machine. Did you activate the compositional
fields Timo?
The problem is that the 'Simpler' material model is not using
compositional fields, and does not consider that the model might still
be solved with passive compositional fields. Therefore the
'reaction_terms' variable, which describes the change of the
compositional fields over one timestep remains unfilled and contains the
uninitialized value 'NaN'.
There are two solutions to this problem: We can either forbid the use of
compositional fields with the 'Simpler' material model and crash with a
better error message, or (my preferred solution) just ignore the
compositional fields in the material model, but make sure the model runs.
You can get the second solution by adding the line
out.reaction_terms[i] = 0.0;
to source/material_model/simpler.cc:68, and recompiling.
Could you check if that works? In case it does: Feel free to pack it
into a pull request and submit it to github if you know how to do this
(we want to write a section about this in the manual for a long time),
but otherwise I can do that for you.
Let me know if it works,
Rene
On 02/23/2016 01:19 PM, Lev Karatun wrote:
> Hi Timo,
>
> make test on the version that I was using (1.4.0-pre) produces the
> following output:
>
>> Running tests...
>> Test project /home/lev/aspect/aspect_debug_old
>> Start 1: quick_mpi
>> 1/1 Test #1: quick_mpi ........................***Not Run 0.00 sec
>> 0% tests passed, 1 tests failed out of 1
>> Total Test time (real) = 0.00 sec
>> The following tests FAILED:
>> 1 - quick_mpi (BAD_COMMAND)
>
> Not sure what you mean by other examples.. other cookbooks? I didn't try
> all of them but the couple that I tried worked fine.
>
> I'm trying to install the recent version with the bug fix, but it doesn't
> compile =(
>
>> /home/lev/aspect/aspect_debug/source/simulator/assembly.cc:1767: error:
>> ‘struct aspect::internal::Assembly::Scratch::AdvectionSystem<3>’ has no
>> member named ‘face_finite_element_values’
>> /home/lev/aspect/aspect_debug/source/simulator/assembly.cc:163: error:
>> class ‘aspect::internal::Assembly::Scratch::AdvectionSystem<dim>’ does not
>> have any field named ‘face_finite_element_values’
> and other errors, I attached the full log. Could you have a look please?
>
>
> Best regards,
> Lev Karatun.
>
> 2016-02-23 8:32 GMT-05:00 Timo Heister <heister at clemson.edu>:
>
>> Hey Lev,
>>
>> running your test 135.prm I discovered a bug introduced over the last
>> few days, see https://github.com/geodynamics/aspect/pull/767
>>
>> But with that fix, your tests runs fine here. What version of ASPECT
>> are you using? What does "make test" report? Do other examples work
>> for you?
>>
>>
>> On Mon, Feb 22, 2016 at 5:59 PM, Lev Karatun <lev.karatun at gmail.com>
>> wrote:
>>> Hi again,
>>>
>>> so I simplified the setup (prm file attached) to the point when I have a
>> box
>>> with a single compositional field, tangential velocity b.c. for all 6
>> faces,
>>> constant temperature throughout the entire box - so I'm literally
>> modelling
>>> a constant state of material with nothing happening. And the simulation
>>> fails with the error " The residual in the last step was nan". However,
>> if I
>>> delete the compositional fields from the input file, the simulation runs
>>> fine.
>>> What can be causing this?..
>>>
>>> Best regards,
>>> Lev Karatun.
>>>
>>> 2016-02-22 0:07 GMT-05:00 Lev Karatun <lev.karatun at gmail.com>:
>>>> Hi Timo,
>>>>
>>>> I've simplified the model to the extreme, and still the solver doesn't
>>>> converge (The residual in the last step was nan). It's not even using
>> any of
>>>> the plugins that I used, so I'm really confused. The cookbooks,
>> however, run
>>>> without problems. (and my model in release mode runs successfully too)
>>>>
>>>> I'm trying to run the models on a lab computer, single core. I attached
>>>> the prm file that I'm using, could you have a look please?
>>>> Thanks in advance!
>>>>
>>>> Best regards,
>>>> Lev Karatun.
>>>>
>>>> 2016-02-12 9:10 GMT-05:00 Timo Heister <heister at clemson.edu>:
>>>>> Lev,
>>>>>
>>>>> your system setup (compiler, operating system, standard libraries)
>>>>> doesn't allow for the debugging trick to detect floating point
>>>>> exceptions. You could try a recent version of clang (or gcc if you
>>>>> were using clang) and see. Sorry, I don't know a sure way to get this
>>>>> enabled (I have no problems with it on ubuntu 14.04 using gcc or
>>>>> clang).
>>>>>
>>>>> That said, convergence failures can have other reasons not detectable
>>>>> using this technique. It is hard to tell, but you should probably
>>>>> simplify your test problem and if it works, increase the complexity
>>>>> one by one. Do the included files like convection-box or shell 2d/3d
>>>>> work correctly on your machine (is it a cluster?)?
>>>>>
>>>>> Best,
>>>>> Timo
>>>>>
>>>>> On Fri, Feb 12, 2016 at 5:21 AM, Lev Karatun <lev.karatun at gmail.com>
>>>>> wrote:
>>>>>> Hi Wolfgang,
>>>>>>
>>>>>> I fixed the issues that Timo pointed out, but I'm still getting the
>>>>>> "did not
>>>>>> converge" error. Is there a way for me to debug it the same way Timo
>>>>>> did?
>>>>>> What am I missing? A compiler with C++11 support? Something else?
>>>>>>
>>>>>> Best regards,
>>>>>> Lev Karatun.
>>>>>>
>>>>>> 2016-02-03 9:20 GMT-05:00 Wolfgang Bangerth <bangerth at tamu.edu>:
>>>>>>>
>>>>>>> Lev,
>>>>>>> this is complicated.
>>>>>>>
>>>>>>> You switched on floating point signals by hand, but your compiler
>>>>>>> support
>>>>>>> library has functions that trip this, so your program gets stopped
>> in
>>>>>>> a
>>>>>>> place that is outside anyone's control (and where it doesn't
>> actually
>>>>>>> matter
>>>>>>> much). This is why floating point exceptions were disabled
>>>>>>> automatically
>>>>>>> when you ran the ASPECT cmake script. In other words, to make
>>>>>>> progress, you
>>>>>>> need to disable floating point exceptions again.
>>>>>>>
>>>>>>> Of course, without being able to run FPE on your system, you won't
>> be
>>>>>>> able
>>>>>>> to reproduce the issues Timo found on his system. But you don't have
>>>>>>> to find
>>>>>>> these issues -- he already did that for you. Just fix these places
>> and
>>>>>>> you
>>>>>>> should be able to make progress.
>>>>>>>
>>>>>>> Best
>>>>>>> W.
>>>>>>>
>>>>>>>
>>>>>>> On 02/01/2016 11:00 PM, Lev Karatun wrote:
>>>>>>>> Hi Wolfgang,
>>>>>>>>
>>>>>>>> I've tried this, the last stack frame is:
>>>>>>>>
>>>>>>>> #13 0x00000000012e50d5 in main (argc=2, argv=0x7fffffffe128) at
>>>>>>>> /home/lev/aspect/aspect_debug_new/source/main.cc:513
>>>>>>>>
>>>>>>>>
>>>>>>>> The 513 line in main.cc is:
>>>>>>>>
>>>>>>>> aspect::Simulator<3>::declare_parameters(prm);
>>>>>>>>
>>>>>>>>
>>>>>>>> Looks like it stops at the attempt of reading the Start time line:
>>>>>>>>
>>>>>>>> #10 0x00007ffff537a569 in
>> dealii::ParameterHandler::declare_entry
>>>>>>>> (this=0x7fffffffdc90, entry="Start time", default_value="0",
>>>>>>>> pattern=...,
>>>>>>>> documentation=
>>>>>>>> "The start time of the simulation. Units: Years if the
>> 'Use
>>>>>>>> years in
>>>>>>>> output instead of seconds' parameter is set; seconds
>> otherwise.")
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>> /home/lev/distrib/dealii_debug_new/source/base/parameter_handler.cc:1628
>>>>>>>>
>>>>>>>> but it's just "set Start time = 0", I never changed it.
>>>>>>>>
>>>>>>>> Full backtrace:
>>>>>>>>
>>>>>>>> Program received signal SIGFPE, Arithmetic exception.
>>>>>>>> __mpn_lshift () at ../sysdeps/x86_64/lshift.S:26
>>>>>>>> 26 movq -8(%rsi,%rdx,8), %mm7
>>>>>>>> (gdb) bt
>>>>>>>> #0 __mpn_lshift () at ../sysdeps/x86_64/lshift.S:26
>>>>>>>> #1 0x0000003db304a53e in ___printf_fp (fp=0x7fffffff71b0,
>>>>>>>> info=0x7fffffff70b0, args=<value optimized out>) at
>>>>>>>> printf_fp.c:483
>>>>>>>> #2 0x0000003db30458a0 in _IO_vfprintf_internal (s=<value
>>>>>>>> optimized
>>>>>>>> out>,
>>>>>>>> format=<value optimized out>, ap=<value optimized out>) at
>>>>>>>> vfprintf.c:1640
>>>>>>>> #3 0x0000003db306f752 in _IO_vsnprintf (string=0x7fffffff7420
>>>>>>>> "",
>>>>>>>> maxlen=<value optimized out>, format=0x7fffffff74f0 "%.*g",
>>>>>>>> args=0x7fffffff7310)
>>>>>>>> at vsnprintf.c:120
>>>>>>>> #4 0x0000003dbd87eb4f in std::__convert_from_v (__cloc=<value
>>>>>>>> optimized
>>>>>>>> out>, __out=0x7fffffff7420 "", __size=45, __fmt=0x7fffffff74f0
>>>>>>>> "%.*g")
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>> /usr/src/debug/gcc-4.4.7-20120601/obj-x86_64-redhat-linux/x86_64-redhat-linux/libstdc++-v3/include/x86_64-redhat-linux/bits/c++locale.h:89
>>>>>>>> #5 0x0000003dbd880f23 in std::num_put<char,
>>>>>>>> std::ostreambuf_iterator<char, std::char_traits<char> >
>>>>>>>> >::_M_insert_float<double> (this=0x3dbdaf22e0, __s=..., __io=
>>>>>>>> ..., __fill=32 ' ', __mod=<value optimized out>,
>>>>>>>> __v=-1.7976931348623157e+308)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>> /usr/src/debug/gcc-4.4.7-20120601/obj-x86_64-redhat-linux/x86_64-redhat-linux/libstdc++-v3/include/bits/locale_facets.tcc:980
>>>>>>>> #6 0x0000003dbd881249 in std::num_put<char,
>>>>>>>> std::ostreambuf_iterator<char, std::char_traits<char> >
>>> ::do_put
>>>>>>>> (this=<value optimized out>, __s=...,
>>>>>>>> __io=<value optimized out>, __fill=<value optimized out>,
>>>>>>>> __v=<value
>>>>>>>> optimized out>)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>> /usr/src/debug/gcc-4.4.7-20120601/obj-x86_64-redhat-linux/x86_64-redhat-linux/libstdc++-v3/include/bits/locale_facets.tcc:1127
>>>>>>>> #7 0x0000003dbd89487f in put (this=0x7fffffff7630,
>>>>>>>> __v=-1.7976931348623157e+308)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>> /usr/src/debug/gcc-4.4.7-20120601/obj-x86_64-redhat-linux/x86_64-redhat-linux/libstdc++-v3/include/bits/locale_facets.h:2390
>>>>>>>> #8 std::basic_ostream<char, std::char_traits<char>
>>>>>>>>> ::_M_insert<double>
>>>>>>>> (this=0x7fffffff7630, __v=-1.7976931348623157e+308)
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>> /usr/src/debug/gcc-4.4.7-20120601/obj-x86_64-redhat-linux/x86_64-redhat-linux/libstdc++-v3/include/bits/ostream.tcc:73
>>>>>>>> #9 0x00007ffff5373468 in dealii::Patterns::Double::description
>>>>>>>> (this=0x1f92620) at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>> /home/lev/distrib/dealii_debug_new/source/base/parameter_handler.cc:290
>>>>>>>> #10 0x00007ffff537a569 in
>> dealii::ParameterHandler::declare_entry
>>>>>>>> (this=0x7fffffffdc90, entry="Start time", default_value="0",
>>>>>>>> pattern=...,
>>>>>>>> documentation=
>>>>>>>> "The start time of the simulation. Units: Years if the
>> 'Use
>>>>>>>> years in
>>>>>>>> output instead of seconds' parameter is set; seconds
>> otherwise.")
>>>>>>>> at
>>>>>>>>
>>>>>>>>
>>>>>>>>
>> /home/lev/distrib/dealii_debug_new/source/base/parameter_handler.cc:1628
>>>>>>>> #11 0x000000000103fef2 in
>>>>>>>> aspect::Parameters<3>::declare_parameters
>>>>>>>> (prm=...) at
>>>>>>>>
>>>>>>>> /home/lev/aspect/aspect_debug_new/source/simulator/parameters.cc:88
>>>>>>>> #12 0x000000000104e2e8 in
>>>>>>>> aspect::Simulator<3>::declare_parameters
>>>>>>>> (prm=...) at
>>>>>>>>
>>>>>>>>
>> /home/lev/aspect/aspect_debug_new/source/simulator/parameters.cc:1283
>>>>>>>> #13 0x00000000012e50d5 in main (argc=2, argv=0x7fffffffe128) at
>>>>>>>> /home/lev/aspect/aspect_debug_new/source/main.cc:513
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>> Lev Karatun.
>>>>>>>>
>>>>>>>> 2016-02-01 21:34 GMT-05:00 Wolfgang Bangerth <bangerth at tamu.edu
>>>>>>>> <mailto:bangerth at tamu.edu>>:
>>>>>>>>
>>>>>>>> On 02/01/2016 06:42 PM, Lev Karatun wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> Could you please tell me what I'm doing wrong?..
>>>>>>>> Thanks in advance.
>>>>>>>>
>>>>>>>>
>>>>>>>> I don't think you *need* any of these packages. You may not get
>>>>>>>> to
>>>>>>>> see the
>>>>>>>> exact location where the error happened in BLAS or some other
>>>>>>>> low-level
>>>>>>>> library, but ultimately you only want to know the place where
>> you
>>>>>>>> were in
>>>>>>>> your own code. In other words, if the debugger stops at the
>>>>>>>> location
>>>>>>>> where
>>>>>>>> the error happens, get a backtrace and go to the last stack
>> frame
>>>>>>>> in
>>>>>>>> your
>>>>>>>> code. Your task is then to find out what is wrong in that line.
>>>>>>>>
>>>>>>>> Best
>>>>>>>> W.
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>>
>>>>>>>>
>> ------------------------------------------------------------------------
>>>>>>>> Wolfgang Bangerth email: bangerth at math.tamu.edu
>>>>>>>> <mailto:bangerth at math.tamu.edu>
>>>>>>>> www:
>>>>>>>> http://www.math.tamu.edu/~bangerth/
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Aspect-devel mailing list
>>>>>>>> Aspect-devel at geodynamics.org
>>>>>>>> <mailto:Aspect-devel at geodynamics.org>
>>>>>>>>
>>>>>>>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/aspect-devel
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Aspect-devel mailing list
>>>>>>>> Aspect-devel at geodynamics.org
>>>>>>>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/aspect-devel
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>>
>> ------------------------------------------------------------------------
>>>>>>> Wolfgang Bangerth email:
>>>>>>> bangerth at math.tamu.edu
>>>>>>> www:
>>>>>>> http://www.math.tamu.edu/~bangerth/
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Aspect-devel mailing list
>>>>>>> Aspect-devel at geodynamics.org
>>>>>>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/aspect-devel
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Aspect-devel mailing list
>>>>>> Aspect-devel at geodynamics.org
>>>>>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/aspect-devel
>>>>>
>>>>>
>>>>> --
>>>>> Timo Heister
>>>>> http://www.math.clemson.edu/~heister/
>>>>> _______________________________________________
>>>>> Aspect-devel mailing list
>>>>> Aspect-devel at geodynamics.org
>>>>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/aspect-devel
>>>>
>>>
>>> _______________________________________________
>>> Aspect-devel mailing list
>>> Aspect-devel at geodynamics.org
>>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/aspect-devel
>>
>>
>> --
>> Timo Heister
>> http://www.math.clemson.edu/~heister/
>> _______________________________________________
>> Aspect-devel mailing list
>> Aspect-devel at geodynamics.org
>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/aspect-devel
>
>
> _______________________________________________
> Aspect-devel mailing list
> Aspect-devel at geodynamics.org
> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/aspect-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.geodynamics.org/pipermail/aspect-devel/attachments/20160223/0b4d9455/attachment-0001.html>
More information about the Aspect-devel
mailing list