[CIG-SHORT] Rate and State and Initial Traction
Brad Aagaard
baagaard at usgs.gov
Thu Aug 30 14:04:44 PDT 2012
Romain,
The stacktrace shows which solve is encountering inf or nan. The routine
is _sensitivitySolve(). This is the solve associated with determining
how much to increment the slip to match the friction criterion.
However, it looks like the problem is in the KSP solver tolerances. Note
that the solver stops with a residual of 2e-10 due to ksp_rtol being
met. However, the friction_tolerance is (by default) 1.0e-10. The error
in the solution is on the order of 2.0e-10 but the tolerance for
detecting whether a point is slipping is 1.0e-10. You want the error in
the linear solution to be smaller than the tolerance for detecting
sliding. This means you should reduce your ksp_rtol to something much
smaller, like 1.0e-20, to force ksp_atol to control convergence.
Usually, the starting residual is a few orders of magnitude smaller than
what you have so the tolerances I suggested work.
Bottom line: Quasi-static friction solves generally require the linear
solves to converge due to the absolute tolerance, which *must* be
smaller than the zero_tolerance for friction.
We will look into adding more consistency checking into the code to try
and catch these types of errors.
Regards,
Brad
On 08/30/2012 01:38 PM, Romain Jolivet wrote:
> Thanks for the advice. I changed the preconditioner settings to what you suggested. I don't see that jump in the linear norm anymore.
> However, I still have the floating point exception problem. This arise after the first round of ksp.
>
> In the log file (attached), I see to messages saying the ksp solver for friction does converge (twice), then the SNES residual norm is printed, then KSP starts, and when it has to compute the new SNES residual norm, it says that VecNorm() encoutered a Inf or NAN value:
>
> 141 KSP Residual norm 1.185626397065e-09
> 142 KSP Residual norm 8.558244459183e-10
> 143 KSP Residual norm 4.537120666370e-10
> 144 KSP Residual norm 2.695590025796e-10
> 145 KSP Residual norm 2.263153111449e-10
> 146 KSP Residual norm 2.156010427246e-10
> Linear solve converged due to CONVERGED_RTOL iterations 146
> [0]PETSC ERROR: --------------------- Error Message ------------------------------------
> [0]PETSC ERROR: Floating point exception!
> [0]PETSC ERROR: Infinite or not-a-number generated in norm!
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Development HG revision: 3bede28f1195e406d4257c9fbeebefe827793801 HG Date: Mon Jun 11 22:14:12 2012 -0400
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: /Users/jolivetinsar/MYBIN/pylith/pylith-1.7.1-darwin-10.6.8/bin/mpinemesis on a pylith-da named dhcp-74-229.caltech.edu by jolivetinsar Thu Aug 30 13:34:55 2012
> [0]PETSC ERROR: Libraries linked from /Users/buildbot/install/pylith_darwin_10.6_binbot/lib
> [0]PETSC ERROR: Configure run at Tue Jun 12 06:09:00 2012
> [0]PETSC ERROR: Configure options --prefix=/Users/buildbot/install/pylith_darwin_10.6_binbot --download-f-blas-lapack=1 --with-sieve=1 --with-c2html=0 --CXXFLAGS=-DMPICH_IGNORE_CXX_SEEK --with-clanguage=C++ --with-mpicompilers=1 --with-debugging=0 --download-chaco=1 --download-boost=1 --download-ml=1 --with-hdf5=1 --with-hdf5-dir=/Users/buildbot/install/pylith_darwin_10.6_binbot --with-x=0 --with-shared-libraries=1
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: VecNorm() line 169 in src/vec/vec/interface/rvector.c
> [0]PETSC ERROR: VecNormalize() line 263 in src/vec/vec/interface/rvector.c
> [0]PETSC ERROR: KSPGMRESCycle() line 127 in src/ksp/ksp/impls/gmres/gmres.c
> [0]PETSC ERROR: KSPSolve_GMRES() line 231 in src/ksp/ksp/impls/gmres/gmres.c
> [0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: _sensitivitySolve() line 2172 in "unknowndirectory/"faults/FaultCohesiveDyn.cc
> Traceback (most recent call last):
> File "/Users/jolivetinsar/MYBIN/pylith/pylith-1.7.1-darwin-10.6.8/lib/python2.7/site-packages/pylith/apps/PetscApplication.py", line 65, in onComputeNodes
> Fatal error. Calling MPI_Abort() to abort PyLith application.
> self.main(*args, **kwds)
> File "/Users/jolivetinsar/MYBIN/pylith/pylith-1.7.1-darwin-10.6.8/lib/python2.7/site-packages/pylith/apps/PyLithApp.py", line 119, in main
> self.problem.run(self)
> File "/Users/jolivetinsar/MYBIN/pylith/pylith-1.7.1-darwin-10.6.8/lib/python2.7/site-packages/pylith/problems/TimeDependent.py", line 154, in run
> self.formulation.step(t, dt)
> File "/Users/jolivetinsar/MYBIN/pylith/pylith-1.7.1-darwin-10.6.8/lib/python2.7/site-packages/pylith/problems/Implicit.py", line 219, in step
> self.solver.solve(dispIncr, self.jacobian, residual)
> File "/Users/jolivetinsar/MYBIN/pylith/pylith-1.7.1-darwin-10.6.8/lib/python2.7/site-packages/pylith/problems/problems.py", line 180, in solve
> def solve(self, *args): return _problems.SolverNonlinear_solve(self, *args)
> RuntimeError: std::exception
> application called MPI_Abort(MPI_COMM_WORLD, -1) - process 0
> /Users/jolivetinsar/MYBIN/pylith/pylith-1.7.1-darwin-10.6.8/bin/nemesis: mpirun: exit 255
> /Users/jolivetinsar/MYBIN/pylith/pylith-1.7.1-darwin-10.6.8/bin/pylith: /Users/jolivetinsar/MYBIN/pylith/pylith-1.7.1-darwin-10.6.8/bin/nemesis: exit 1
>
> Any idea?
> Romain
>
>
>
> Romain Jolivet
> Postdoctoral Scholar
> Geological and Planetary Sciences
> California Institute of Technology, Pasadena, CA
> rjolivet at caltech.edu
> +1 (626) 560 6356
>
> On Aug 30, 2012, at 12:00 PM, cig-short-request at geodynamics.org wrote:
>
>> Send CIG-SHORT mailing list submissions to
>> cig-short at geodynamics.org
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>> http://geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>> or, via email, send a message with subject or body 'help' to
>> cig-short-request at geodynamics.org
>>
>> You can reach the person managing the list at
>> cig-short-owner at geodynamics.org
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of CIG-SHORT digest..."
>>
>>
>> Today's Topics:
>>
>> 1. Re: Rate and State and Initial Traction (Brad Aagaard)
>> 2. Fwd: Rate and State and Initial Traction (Matthew Knepley)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Wed, 29 Aug 2012 12:36:08 -0700
>> From: Brad Aagaard <baagaard at usgs.gov>
>> Subject: Re: [CIG-SHORT] Rate and State and Initial Traction
>> To: cig-short at geodynamics.org
>> Message-ID: <503E6F28.6050001 at usgs.gov>
>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>
>> Romain,
>>
>> It looks like you are using Schur complement preconditioner settings.
>> The jump in the KSP residual at iteration 10 is odd and suggests
>> something is not quite right in the linear solve (probably related to
>> the preconditioner). Try using the AMG preconditioner. It may not be
>> quite as fast but it is more robust. Matt can probably help you tune the
>> Schur complement solver conditions some more if necessary.
>>
>> Your SNES tolerances are quite large. I suggest using tolerances similar
>> to those in examples/3d/hex8/step14.cfg.
>>
>> SOLVER TOLERANCES
>>
>> ksp_rtol = 1.0e-16
>> ksp_atol = 1.0e-12
>>
>> snes_rtol = 1.0e-14
>> snes_atol = 1.0e-10
>>
>>
>> AMG PRECONDITIONER SETTINGS
>>
>> [pylithapp.timedependent.formulation]
>> split_fields = True
>> matrix_type = aij
>>
>> [pylithapp.petsc]
>> ksp_gmres_restart = 100
>> fs_pc_type = fieldsplit
>> fs_pc_fieldsplit_real_diagonal = true
>> fs_pc_fieldsplit_type = multiplicative
>> fs_fieldsplit_0_pc_type = ml
>> fs_fieldsplit_1_pc_type = jacobi
>> fs_fieldsplit_0_ksp_type = preonly
>> fs_fieldsplit_1_ksp_type = preonly
>>
>> Brad
>>
>>
>> On 08/29/2012 11:42 AM, Romain Jolivet wrote:
>>> Hello Brad, Matt, Charles and the pylith team,
>>>
>>> Following your advice from the other day, I am experimenting how to create earthquakes on a frictional interface by applying a traction change history. It works fine with the SlipWeakening law in 2D. I'm trying to move on to the RateStateAgeing behavior and I have problems applying the initial traction conditions:
>>>
>>> I use a SimpleDB to apply the frictional properties on the fault: one domain is rate-strengthening, the other is rate-weakening with a linear transition between the two domains. If I do not try to turn on any traction_perturbation option, the code runs great, and I have a slow slip all the along the fault at any time.
>>>
>>> As I want the rate-weakening domain to be stuck sometimes, I want to turn on traction_perturbation.db_initial. I use a SimpleDB to put a negative traction all along the fault, and a shear traction compatible with the friction coefficients I impose. However, when I run this, I have the following error message from PETSc (I attach the log):
>>>
>>>>> /Users/jolivetinsar/MYBIN/pylith/pylith-1.7.1-darwin-10.6.8/lib/python2.7/site-packages/pylith/problems/Implicit.py:215:step
>>> -- implicit(info)
>>> -- Solving equations.
>>> 0 SNES Function norm 2.679589636063e+01
>>> 0 KSP Residual norm 9.637565795900e-01
>>> 1 KSP Residual norm 3.564937135471e-08
>>> 2 KSP Residual norm 4.313963401911e-16
>>> 3 KSP Residual norm 3.050427683646e-16
>>> 4 KSP Residual norm 2.490663081181e-16
>>> 5 KSP Residual norm 2.156976950523e-16
>>> 6 KSP Residual norm 1.929258577738e-16
>>> 7 KSP Residual norm 1.761163907914e-16
>>> 8 KSP Residual norm 1.630520838230e-16
>>> 9 KSP Residual norm 1.525212509706e-16
>>> 10 KSP Residual norm 1.833325155789e-11
>>> 11 KSP Residual norm 7.039784754868e-23
>>> Linear solve converged due to CONVERGED_ATOL iterations 11
>>> [0]PETSC ERROR: --------------------- Error Message ------------------------------------
>>> [0]PETSC ERROR: Floating point exception!
>>> [0]PETSC ERROR: Infinite or not-a-number generated in norm!
>>>
>>> If anyone has an idea about that error message....
>>>
>>> I think it has to do with some slip vector. If I set friction.min_slip_rate=1e-1, this problem does not happen (the nonlinear solver does not converge, but at least, no crash). If I set the normal traction to be positive, it does not crash as well (and it does not converge as well). I join to this message the cfg file I am using for that experiment.
>>>
>>> Thank you
>>> Romain
>>>
>>>
>>>
>>>
>>> Romain Jolivet
>>> Postdoctoral Scholar
>>> Geological and Planetary Sciences
>>> California Institute of Technology, Pasadena, CA
>>> rjolivet at caltech.edu
>>> +1 (626) 560 6356
>>>
>>>
>>>
>>> _______________________________________________
>>> CIG-SHORT mailing list
>>> CIG-SHORT at geodynamics.org
>>> http://geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>>>
>>
>>
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Wed, 29 Aug 2012 22:09:25 -0500
>> From: Matthew Knepley <knepley at mcs.anl.gov>
>> Subject: [CIG-SHORT] Fwd: Rate and State and Initial Traction
>> To: cig-short at geodynamics.org
>> Message-ID:
>> <CAMYG4GkMcUqRjhMaYZ8Ooj-x=06QKFrc7fiwW1dStmDSDO1qWA at mail.gmail.com>
>> Content-Type: text/plain; charset="iso-8859-1"
>>
>> On Wed, Aug 29, 2012 at 2:36 PM, Brad Aagaard <baagaard at usgs.gov> wrote:
>>
>>> Romain,
>>>
>>> It looks like you are using Schur complement preconditioner settings.
>>> The jump in the KSP residual at iteration 10 is odd and suggests
>>> something is not quite right in the linear solve (probably related to
>>>
>>
>> Brad is right that the tolerances here will produce some garbage. Going
>> below
>> machine epsilon (about 10^-16) means you get randomness and ultimately
>> underflow which I think you got in a norm calculation since things get
>> squared.
>> The solver should have converged on iterate 2, and will with the tolerances
>> that
>> Brad suggested.
>>
>> Thanks,
>>
>> Matt
>>
>>
>>> the preconditioner). Try using the AMG preconditioner. It may not be
>>> quite as fast but it is more robust. Matt can probably help you tune the
>>> Schur complement solver conditions some more if necessary.
>>>
>>> Your SNES tolerances are quite large. I suggest using tolerances similar
>>> to those in examples/3d/hex8/step14.cfg.
>>>
>>> SOLVER TOLERANCES
>>>
>>> ksp_rtol = 1.0e-16
>>> ksp_atol = 1.0e-12
>>>
>>> snes_rtol = 1.0e-14
>>> snes_atol = 1.0e-10
>>>
>>>
>>> AMG PRECONDITIONER SETTINGS
>>>
>>> [pylithapp.timedependent.formulation]
>>> split_fields = True
>>> matrix_type = aij
>>>
>>> [pylithapp.petsc]
>>> ksp_gmres_restart = 100
>>> fs_pc_type = fieldsplit
>>> fs_pc_fieldsplit_real_diagonal = true
>>> fs_pc_fieldsplit_type = multiplicative
>>> fs_fieldsplit_0_pc_type = ml
>>> fs_fieldsplit_1_pc_type = jacobi
>>> fs_fieldsplit_0_ksp_type = preonly
>>> fs_fieldsplit_1_ksp_type = preonly
>>>
>>> Brad
>>>
>>>
>>> On 08/29/2012 11:42 AM, Romain Jolivet wrote:
>>>> Hello Brad, Matt, Charles and the pylith team,
>>>>
>>>> Following your advice from the other day, I am experimenting how to
>>> create earthquakes on a frictional interface by applying a traction change
>>> history. It works fine with the SlipWeakening law in 2D. I'm trying to move
>>> on to the RateStateAgeing behavior and I have problems applying the initial
>>> traction conditions:
>>>>
>>>> I use a SimpleDB to apply the frictional properties on the fault: one
>>> domain is rate-strengthening, the other is rate-weakening with a linear
>>> transition between the two domains. If I do not try to turn on any
>>> traction_perturbation option, the code runs great, and I have a slow slip
>>> all the along the fault at any time.
>>>>
>>>> As I want the rate-weakening domain to be stuck sometimes, I want to
>>> turn on traction_perturbation.db_initial. I use a SimpleDB to put a
>>> negative traction all along the fault, and a shear traction compatible with
>>> the friction coefficients I impose. However, when I run this, I have the
>>> following error message from PETSc (I attach the log):
>>>>
>>>>>>
>>> /Users/jolivetinsar/MYBIN/pylith/pylith-1.7.1-darwin-10.6.8/lib/python2.7/site-packages/pylith/problems/Implicit.py:215:step
>>>> -- implicit(info)
>>>> -- Solving equations.
>>>> 0 SNES Function norm 2.679589636063e+01
>>>> 0 KSP Residual norm 9.637565795900e-01
>>>> 1 KSP Residual norm 3.564937135471e-08
>>>> 2 KSP Residual norm 4.313963401911e-16
>>>> 3 KSP Residual norm 3.050427683646e-16
>>>> 4 KSP Residual norm 2.490663081181e-16
>>>> 5 KSP Residual norm 2.156976950523e-16
>>>> 6 KSP Residual norm 1.929258577738e-16
>>>> 7 KSP Residual norm 1.761163907914e-16
>>>> 8 KSP Residual norm 1.630520838230e-16
>>>> 9 KSP Residual norm 1.525212509706e-16
>>>> 10 KSP Residual norm 1.833325155789e-11
>>>> 11 KSP Residual norm 7.039784754868e-23
>>>> Linear solve converged due to CONVERGED_ATOL iterations 11
>>>> [0]PETSC ERROR: --------------------- Error Message
>>> ------------------------------------
>>>> [0]PETSC ERROR: Floating point exception!
>>>> [0]PETSC ERROR: Infinite or not-a-number generated in norm!
>>>>
>>>> If anyone has an idea about that error message....
>>>>
>>>> I think it has to do with some slip vector. If I set
>>> friction.min_slip_rate=1e-1, this problem does not happen (the nonlinear
>>> solver does not converge, but at least, no crash). If I set the normal
>>> traction to be positive, it does not crash as well (and it does not
>>> converge as well). I join to this message the cfg file I am using for that
>>> experiment.
>>>>
>>>> Thank you
>>>> Romain
>>>>
>>>>
>>>>
>>>>
>>>> Romain Jolivet
>>>> Postdoctoral Scholar
>>>> Geological and Planetary Sciences
>>>> California Institute of Technology, Pasadena, CA
>>>> rjolivet at caltech.edu
>>>> +1 (626) 560 6356
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> CIG-SHORT mailing list
>>>> CIG-SHORT at geodynamics.org
>>>> http://geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>>>>
>>>
>>> _______________________________________________
>>> CIG-SHORT mailing list
>>> CIG-SHORT at geodynamics.org
>>> http://geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>>>
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: http://geodynamics.org/pipermail/cig-short/attachments/20120829/9e91899c/attachment.html
>>
>> ------------------------------
>>
>> _______________________________________________
>> CIG-SHORT mailing list
>> CIG-SHORT at geodynamics.org
>> http://geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>>
>>
>> End of CIG-SHORT Digest, Vol 72, Issue 21
>> *****************************************
>
> _______________________________________________
> CIG-SHORT mailing list
> CIG-SHORT at geodynamics.org
> http://geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>
More information about the CIG-SHORT
mailing list