[CIG-SHORT] Rate and State and Initial Traction
Brad Aagaard
baagaard at usgs.gov
Thu Aug 30 14:14:31 PDT 2012
Matt,
I suggested that Romain change ksp_rtol only, not ksp_atol. The relative
tolerance will be small enough so that the absolute tolerance (1.0e-12)
will be what stops the KSP solve.
Brad
On 08/30/2012 02:08 PM, Matthew Knepley wrote:
> On Thu, Aug 30, 2012 at 4:04 PM, Brad Aagaard <baagaard at usgs.gov> wrote:
>
>> Romain,
>>
>> The stacktrace shows which solve is encountering inf or nan. The routine
>> is _sensitivitySolve(). This is the solve associated with determining
>> how much to increment the slip to match the friction criterion.
>>
>> However, it looks like the problem is in the KSP solver tolerances. Note
>> that the solver stops with a residual of 2e-10 due to ksp_rtol being
>> met. However, the friction_tolerance is (by default) 1.0e-10. The error
>> in the solution is on the order of 2.0e-10 but the tolerance for
>> detecting whether a point is slipping is 1.0e-10. You want the error in
>> the linear solution to be smaller than the tolerance for detecting
>> sliding. This means you should reduce your ksp_rtol to something much
>> smaller, like 1.0e-20, to force ksp_atol to control convergence.
>> Usually, the starting residual is a few orders of magnitude smaller than
>> what you have so the tolerances I suggested work.
>>
>
> Okay, last time I told him not to use anything less than 1e-16 since there
> are funny effects below roundoff. Could you try 1e-15?
>
> We do need consistency checks in these nested iterations.
>
> Thanks,
>
> Matt
>
>
>> Bottom line: Quasi-static friction solves generally require the linear
>> solves to converge due to the absolute tolerance, which *must* be
>> smaller than the zero_tolerance for friction.
>>
>> We will look into adding more consistency checking into the code to try
>> and catch these types of errors.
>>
>> Regards,
>> Brad
>>
>>
>> On 08/30/2012 01:38 PM, Romain Jolivet wrote:
>>> Thanks for the advice. I changed the preconditioner settings to what you
>> suggested. I don't see that jump in the linear norm anymore.
>>> However, I still have the floating point exception problem. This arise
>> after the first round of ksp.
>>>
>>> In the log file (attached), I see to messages saying the ksp solver for
>> friction does converge (twice), then the SNES residual norm is printed,
>> then KSP starts, and when it has to compute the new SNES residual norm, it
>> says that VecNorm() encoutered a Inf or NAN value:
>>>
>>> 141 KSP Residual norm 1.185626397065e-09
>>> 142 KSP Residual norm 8.558244459183e-10
>>> 143 KSP Residual norm 4.537120666370e-10
>>> 144 KSP Residual norm 2.695590025796e-10
>>> 145 KSP Residual norm 2.263153111449e-10
>>> 146 KSP Residual norm 2.156010427246e-10
>>> Linear solve converged due to CONVERGED_RTOL iterations 146
>>> [0]PETSC ERROR: --------------------- Error Message
>> ------------------------------------
>>> [0]PETSC ERROR: Floating point exception!
>>> [0]PETSC ERROR: Infinite or not-a-number generated in norm!
>>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>>> [0]PETSC ERROR: Petsc Development HG revision:
>> 3bede28f1195e406d4257c9fbeebefe827793801 HG Date: Mon Jun 11 22:14:12 2012
>> -0400
>>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>>> [0]PETSC ERROR: See docs/index.html for manual pages.
>>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>>> [0]PETSC ERROR:
>> /Users/jolivetinsar/MYBIN/pylith/pylith-1.7.1-darwin-10.6.8/bin/mpinemesis
>> on a pylith-da named dhcp-74-229.caltech.edu by jolivetinsar Thu Aug 30
>> 13:34:55 2012
>>> [0]PETSC ERROR: Libraries linked from
>> /Users/buildbot/install/pylith_darwin_10.6_binbot/lib
>>> [0]PETSC ERROR: Configure run at Tue Jun 12 06:09:00 2012
>>> [0]PETSC ERROR: Configure options
>> --prefix=/Users/buildbot/install/pylith_darwin_10.6_binbot
>> --download-f-blas-lapack=1 --with-sieve=1 --with-c2html=0
>> --CXXFLAGS=-DMPICH_IGNORE_CXX_SEEK --with-clanguage=C++
>> --with-mpicompilers=1 --with-debugging=0 --download-chaco=1
>> --download-boost=1 --download-ml=1 --with-hdf5=1
>> --with-hdf5-dir=/Users/buildbot/install/pylith_darwin_10.6_binbot
>> --with-x=0 --with-shared-libraries=1
>>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>>> [0]PETSC ERROR: VecNorm() line 169 in src/vec/vec/interface/rvector.c
>>> [0]PETSC ERROR: VecNormalize() line 263 in
>> src/vec/vec/interface/rvector.c
>>> [0]PETSC ERROR: KSPGMRESCycle() line 127 in
>> src/ksp/ksp/impls/gmres/gmres.c
>>> [0]PETSC ERROR: KSPSolve_GMRES() line 231 in
>> src/ksp/ksp/impls/gmres/gmres.c
>>> [0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c
>>> [0]PETSC ERROR: _sensitivitySolve() line 2172 in
>> "unknowndirectory/"faults/FaultCohesiveDyn.cc
>>> Traceback (most recent call last):
>>> File
>> "/Users/jolivetinsar/MYBIN/pylith/pylith-1.7.1-darwin-10.6.8/lib/python2.7/site-packages/pylith/apps/PetscApplication.py",
>> line 65, in onComputeNodes
>>> Fatal error. Calling MPI_Abort() to abort PyLith application.
>>> self.main(*args, **kwds)
>>> File
>> "/Users/jolivetinsar/MYBIN/pylith/pylith-1.7.1-darwin-10.6.8/lib/python2.7/site-packages/pylith/apps/PyLithApp.py",
>> line 119, in main
>>> self.problem.run(self)
>>> File
>> "/Users/jolivetinsar/MYBIN/pylith/pylith-1.7.1-darwin-10.6.8/lib/python2.7/site-packages/pylith/problems/TimeDependent.py",
>> line 154, in run
>>> self.formulation.step(t, dt)
>>> File
>> "/Users/jolivetinsar/MYBIN/pylith/pylith-1.7.1-darwin-10.6.8/lib/python2.7/site-packages/pylith/problems/Implicit.py",
>> line 219, in step
>>> self.solver.solve(dispIncr, self.jacobian, residual)
>>> File
>> "/Users/jolivetinsar/MYBIN/pylith/pylith-1.7.1-darwin-10.6.8/lib/python2.7/site-packages/pylith/problems/problems.py",
>> line 180, in solve
>>> def solve(self, *args): return
>> _problems.SolverNonlinear_solve(self, *args)
>>> RuntimeError: std::exception
>>> application called MPI_Abort(MPI_COMM_WORLD, -1) - process 0
>>> /Users/jolivetinsar/MYBIN/pylith/pylith-1.7.1-darwin-10.6.8/bin/nemesis:
>> mpirun: exit 255
>>> /Users/jolivetinsar/MYBIN/pylith/pylith-1.7.1-darwin-10.6.8/bin/pylith:
>> /Users/jolivetinsar/MYBIN/pylith/pylith-1.7.1-darwin-10.6.8/bin/nemesis:
>> exit 1
>>>
>>> Any idea?
>>> Romain
>>>
>>>
>>>
>>> Romain Jolivet
>>> Postdoctoral Scholar
>>> Geological and Planetary Sciences
>>> California Institute of Technology, Pasadena, CA
>>> rjolivet at caltech.edu
>>> +1 (626) 560 6356
>>>
>>> On Aug 30, 2012, at 12:00 PM, cig-short-request at geodynamics.org wrote:
>>>
>>>> Send CIG-SHORT mailing list submissions to
>>>> cig-short at geodynamics.org
>>>>
>>>> To subscribe or unsubscribe via the World Wide Web, visit
>>>> http://geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>>>> or, via email, send a message with subject or body 'help' to
>>>> cig-short-request at geodynamics.org
>>>>
>>>> You can reach the person managing the list at
>>>> cig-short-owner at geodynamics.org
>>>>
>>>> When replying, please edit your Subject line so it is more specific
>>>> than "Re: Contents of CIG-SHORT digest..."
>>>>
>>>>
>>>> Today's Topics:
>>>>
>>>> 1. Re: Rate and State and Initial Traction (Brad Aagaard)
>>>> 2. Fwd: Rate and State and Initial Traction (Matthew Knepley)
>>>>
>>>>
>>>> ----------------------------------------------------------------------
>>>>
>>>> Message: 1
>>>> Date: Wed, 29 Aug 2012 12:36:08 -0700
>>>> From: Brad Aagaard <baagaard at usgs.gov>
>>>> Subject: Re: [CIG-SHORT] Rate and State and Initial Traction
>>>> To: cig-short at geodynamics.org
>>>> Message-ID: <503E6F28.6050001 at usgs.gov>
>>>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>>>
>>>> Romain,
>>>>
>>>> It looks like you are using Schur complement preconditioner settings.
>>>> The jump in the KSP residual at iteration 10 is odd and suggests
>>>> something is not quite right in the linear solve (probably related to
>>>> the preconditioner). Try using the AMG preconditioner. It may not be
>>>> quite as fast but it is more robust. Matt can probably help you tune the
>>>> Schur complement solver conditions some more if necessary.
>>>>
>>>> Your SNES tolerances are quite large. I suggest using tolerances similar
>>>> to those in examples/3d/hex8/step14.cfg.
>>>>
>>>> SOLVER TOLERANCES
>>>>
>>>> ksp_rtol = 1.0e-16
>>>> ksp_atol = 1.0e-12
>>>>
>>>> snes_rtol = 1.0e-14
>>>> snes_atol = 1.0e-10
>>>>
>>>>
>>>> AMG PRECONDITIONER SETTINGS
>>>>
>>>> [pylithapp.timedependent.formulation]
>>>> split_fields = True
>>>> matrix_type = aij
>>>>
>>>> [pylithapp.petsc]
>>>> ksp_gmres_restart = 100
>>>> fs_pc_type = fieldsplit
>>>> fs_pc_fieldsplit_real_diagonal = true
>>>> fs_pc_fieldsplit_type = multiplicative
>>>> fs_fieldsplit_0_pc_type = ml
>>>> fs_fieldsplit_1_pc_type = jacobi
>>>> fs_fieldsplit_0_ksp_type = preonly
>>>> fs_fieldsplit_1_ksp_type = preonly
>>>>
>>>> Brad
>>>>
>>>>
>>>> On 08/29/2012 11:42 AM, Romain Jolivet wrote:
>>>>> Hello Brad, Matt, Charles and the pylith team,
>>>>>
>>>>> Following your advice from the other day, I am experimenting how to
>> create earthquakes on a frictional interface by applying a traction change
>> history. It works fine with the SlipWeakening law in 2D. I'm trying to move
>> on to the RateStateAgeing behavior and I have problems applying the initial
>> traction conditions:
>>>>>
>>>>> I use a SimpleDB to apply the frictional properties on the fault: one
>> domain is rate-strengthening, the other is rate-weakening with a linear
>> transition between the two domains. If I do not try to turn on any
>> traction_perturbation option, the code runs great, and I have a slow slip
>> all the along the fault at any time.
>>>>>
>>>>> As I want the rate-weakening domain to be stuck sometimes, I want to
>> turn on traction_perturbation.db_initial. I use a SimpleDB to put a
>> negative traction all along the fault, and a shear traction compatible with
>> the friction coefficients I impose. However, when I run this, I have the
>> following error message from PETSc (I attach the log):
>>>>>
>>>>>>>
>> /Users/jolivetinsar/MYBIN/pylith/pylith-1.7.1-darwin-10.6.8/lib/python2.7/site-packages/pylith/problems/Implicit.py:215:step
>>>>> -- implicit(info)
>>>>> -- Solving equations.
>>>>> 0 SNES Function norm 2.679589636063e+01
>>>>> 0 KSP Residual norm 9.637565795900e-01
>>>>> 1 KSP Residual norm 3.564937135471e-08
>>>>> 2 KSP Residual norm 4.313963401911e-16
>>>>> 3 KSP Residual norm 3.050427683646e-16
>>>>> 4 KSP Residual norm 2.490663081181e-16
>>>>> 5 KSP Residual norm 2.156976950523e-16
>>>>> 6 KSP Residual norm 1.929258577738e-16
>>>>> 7 KSP Residual norm 1.761163907914e-16
>>>>> 8 KSP Residual norm 1.630520838230e-16
>>>>> 9 KSP Residual norm 1.525212509706e-16
>>>>> 10 KSP Residual norm 1.833325155789e-11
>>>>> 11 KSP Residual norm 7.039784754868e-23
>>>>> Linear solve converged due to CONVERGED_ATOL iterations 11
>>>>> [0]PETSC ERROR: --------------------- Error Message
>> ------------------------------------
>>>>> [0]PETSC ERROR: Floating point exception!
>>>>> [0]PETSC ERROR: Infinite or not-a-number generated in norm!
>>>>>
>>>>> If anyone has an idea about that error message....
>>>>>
>>>>> I think it has to do with some slip vector. If I set
>> friction.min_slip_rate=1e-1, this problem does not happen (the nonlinear
>> solver does not converge, but at least, no crash). If I set the normal
>> traction to be positive, it does not crash as well (and it does not
>> converge as well). I join to this message the cfg file I am using for that
>> experiment.
>>>>>
>>>>> Thank you
>>>>> Romain
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Romain Jolivet
>>>>> Postdoctoral Scholar
>>>>> Geological and Planetary Sciences
>>>>> California Institute of Technology, Pasadena, CA
>>>>> rjolivet at caltech.edu
>>>>> +1 (626) 560 6356
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> CIG-SHORT mailing list
>>>>> CIG-SHORT at geodynamics.org
>>>>> http://geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>>>>>
>>>>
>>>>
>>>>
>>>> ------------------------------
>>>>
>>>> Message: 2
>>>> Date: Wed, 29 Aug 2012 22:09:25 -0500
>>>> From: Matthew Knepley <knepley at mcs.anl.gov>
>>>> Subject: [CIG-SHORT] Fwd: Rate and State and Initial Traction
>>>> To: cig-short at geodynamics.org
>>>> Message-ID:
>>>> <CAMYG4GkMcUqRjhMaYZ8Ooj-x=
>> 06QKFrc7fiwW1dStmDSDO1qWA at mail.gmail.com>
>>>> Content-Type: text/plain; charset="iso-8859-1"
>>>>
>>>> On Wed, Aug 29, 2012 at 2:36 PM, Brad Aagaard <baagaard at usgs.gov>
>> wrote:
>>>>
>>>>> Romain,
>>>>>
>>>>> It looks like you are using Schur complement preconditioner settings.
>>>>> The jump in the KSP residual at iteration 10 is odd and suggests
>>>>> something is not quite right in the linear solve (probably related to
>>>>>
>>>>
>>>> Brad is right that the tolerances here will produce some garbage. Going
>>>> below
>>>> machine epsilon (about 10^-16) means you get randomness and ultimately
>>>> underflow which I think you got in a norm calculation since things get
>>>> squared.
>>>> The solver should have converged on iterate 2, and will with the
>> tolerances
>>>> that
>>>> Brad suggested.
>>>>
>>>> Thanks,
>>>>
>>>> Matt
>>>>
>>>>
>>>>> the preconditioner). Try using the AMG preconditioner. It may not be
>>>>> quite as fast but it is more robust. Matt can probably help you tune
>> the
>>>>> Schur complement solver conditions some more if necessary.
>>>>>
>>>>> Your SNES tolerances are quite large. I suggest using tolerances
>> similar
>>>>> to those in examples/3d/hex8/step14.cfg.
>>>>>
>>>>> SOLVER TOLERANCES
>>>>>
>>>>> ksp_rtol = 1.0e-16
>>>>> ksp_atol = 1.0e-12
>>>>>
>>>>> snes_rtol = 1.0e-14
>>>>> snes_atol = 1.0e-10
>>>>>
>>>>>
>>>>> AMG PRECONDITIONER SETTINGS
>>>>>
>>>>> [pylithapp.timedependent.formulation]
>>>>> split_fields = True
>>>>> matrix_type = aij
>>>>>
>>>>> [pylithapp.petsc]
>>>>> ksp_gmres_restart = 100
>>>>> fs_pc_type = fieldsplit
>>>>> fs_pc_fieldsplit_real_diagonal = true
>>>>> fs_pc_fieldsplit_type = multiplicative
>>>>> fs_fieldsplit_0_pc_type = ml
>>>>> fs_fieldsplit_1_pc_type = jacobi
>>>>> fs_fieldsplit_0_ksp_type = preonly
>>>>> fs_fieldsplit_1_ksp_type = preonly
>>>>>
>>>>> Brad
>>>>>
>>>>>
>>>>> On 08/29/2012 11:42 AM, Romain Jolivet wrote:
>>>>>> Hello Brad, Matt, Charles and the pylith team,
>>>>>>
>>>>>> Following your advice from the other day, I am experimenting how to
>>>>> create earthquakes on a frictional interface by applying a traction
>> change
>>>>> history. It works fine with the SlipWeakening law in 2D. I'm trying to
>> move
>>>>> on to the RateStateAgeing behavior and I have problems applying the
>> initial
>>>>> traction conditions:
>>>>>>
>>>>>> I use a SimpleDB to apply the frictional properties on the fault: one
>>>>> domain is rate-strengthening, the other is rate-weakening with a linear
>>>>> transition between the two domains. If I do not try to turn on any
>>>>> traction_perturbation option, the code runs great, and I have a slow
>> slip
>>>>> all the along the fault at any time.
>>>>>>
>>>>>> As I want the rate-weakening domain to be stuck sometimes, I want to
>>>>> turn on traction_perturbation.db_initial. I use a SimpleDB to put a
>>>>> negative traction all along the fault, and a shear traction compatible
>> with
>>>>> the friction coefficients I impose. However, when I run this, I have
>> the
>>>>> following error message from PETSc (I attach the log):
>>>>>>
>>>>>>>>
>>>>>
>> /Users/jolivetinsar/MYBIN/pylith/pylith-1.7.1-darwin-10.6.8/lib/python2.7/site-packages/pylith/problems/Implicit.py:215:step
>>>>>> -- implicit(info)
>>>>>> -- Solving equations.
>>>>>> 0 SNES Function norm 2.679589636063e+01
>>>>>> 0 KSP Residual norm 9.637565795900e-01
>>>>>> 1 KSP Residual norm 3.564937135471e-08
>>>>>> 2 KSP Residual norm 4.313963401911e-16
>>>>>> 3 KSP Residual norm 3.050427683646e-16
>>>>>> 4 KSP Residual norm 2.490663081181e-16
>>>>>> 5 KSP Residual norm 2.156976950523e-16
>>>>>> 6 KSP Residual norm 1.929258577738e-16
>>>>>> 7 KSP Residual norm 1.761163907914e-16
>>>>>> 8 KSP Residual norm 1.630520838230e-16
>>>>>> 9 KSP Residual norm 1.525212509706e-16
>>>>>> 10 KSP Residual norm 1.833325155789e-11
>>>>>> 11 KSP Residual norm 7.039784754868e-23
>>>>>> Linear solve converged due to CONVERGED_ATOL iterations 11
>>>>>> [0]PETSC ERROR: --------------------- Error Message
>>>>> ------------------------------------
>>>>>> [0]PETSC ERROR: Floating point exception!
>>>>>> [0]PETSC ERROR: Infinite or not-a-number generated in norm!
>>>>>>
>>>>>> If anyone has an idea about that error message....
>>>>>>
>>>>>> I think it has to do with some slip vector. If I set
>>>>> friction.min_slip_rate=1e-1, this problem does not happen (the
>> nonlinear
>>>>> solver does not converge, but at least, no crash). If I set the normal
>>>>> traction to be positive, it does not crash as well (and it does not
>>>>> converge as well). I join to this message the cfg file I am using for
>> that
>>>>> experiment.
>>>>>>
>>>>>> Thank you
>>>>>> Romain
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Romain Jolivet
>>>>>> Postdoctoral Scholar
>>>>>> Geological and Planetary Sciences
>>>>>> California Institute of Technology, Pasadena, CA
>>>>>> rjolivet at caltech.edu
>>>>>> +1 (626) 560 6356
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> CIG-SHORT mailing list
>>>>>> CIG-SHORT at geodynamics.org
>>>>>> http://geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> CIG-SHORT mailing list
>>>>> CIG-SHORT at geodynamics.org
>>>>> http://geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which
>> their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which
>> their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>> -------------- next part --------------
>>>> An HTML attachment was scrubbed...
>>>> URL:
>> http://geodynamics.org/pipermail/cig-short/attachments/20120829/9e91899c/attachment.html
>>>>
>>>> ------------------------------
>>>>
>>>> _______________________________________________
>>>> CIG-SHORT mailing list
>>>> CIG-SHORT at geodynamics.org
>>>> http://geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>>>>
>>>>
>>>> End of CIG-SHORT Digest, Vol 72, Issue 21
>>>> *****************************************
>>>
>>> _______________________________________________
>>> CIG-SHORT mailing list
>>> CIG-SHORT at geodynamics.org
>>> http://geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>>>
>>
>> _______________________________________________
>> CIG-SHORT mailing list
>> CIG-SHORT at geodynamics.org
>> http://geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>>
>
>
>
More information about the CIG-SHORT
mailing list