[CIG-SHORT] Rate and State and Initial Traction
Matthew Knepley
knepley at mcs.anl.gov
Thu Aug 30 14:08:04 PDT 2012
On Thu, Aug 30, 2012 at 4:04 PM, Brad Aagaard <baagaard at usgs.gov> wrote:
> Romain,
>
> The stacktrace shows which solve is encountering inf or nan. The routine
> is _sensitivitySolve(). This is the solve associated with determining
> how much to increment the slip to match the friction criterion.
>
> However, it looks like the problem is in the KSP solver tolerances. Note
> that the solver stops with a residual of 2e-10 due to ksp_rtol being
> met. However, the friction_tolerance is (by default) 1.0e-10. The error
> in the solution is on the order of 2.0e-10 but the tolerance for
> detecting whether a point is slipping is 1.0e-10. You want the error in
> the linear solution to be smaller than the tolerance for detecting
> sliding. This means you should reduce your ksp_rtol to something much
> smaller, like 1.0e-20, to force ksp_atol to control convergence.
> Usually, the starting residual is a few orders of magnitude smaller than
> what you have so the tolerances I suggested work.
>
Okay, last time I told him not to use anything less than 1e-16 since there
are funny effects below roundoff. Could you try 1e-15?
We do need consistency checks in these nested iterations.
Thanks,
Matt
> Bottom line: Quasi-static friction solves generally require the linear
> solves to converge due to the absolute tolerance, which *must* be
> smaller than the zero_tolerance for friction.
>
> We will look into adding more consistency checking into the code to try
> and catch these types of errors.
>
> Regards,
> Brad
>
>
> On 08/30/2012 01:38 PM, Romain Jolivet wrote:
> > Thanks for the advice. I changed the preconditioner settings to what you
> suggested. I don't see that jump in the linear norm anymore.
> > However, I still have the floating point exception problem. This arise
> after the first round of ksp.
> >
> > In the log file (attached), I see to messages saying the ksp solver for
> friction does converge (twice), then the SNES residual norm is printed,
> then KSP starts, and when it has to compute the new SNES residual norm, it
> says that VecNorm() encoutered a Inf or NAN value:
> >
> > 141 KSP Residual norm 1.185626397065e-09
> > 142 KSP Residual norm 8.558244459183e-10
> > 143 KSP Residual norm 4.537120666370e-10
> > 144 KSP Residual norm 2.695590025796e-10
> > 145 KSP Residual norm 2.263153111449e-10
> > 146 KSP Residual norm 2.156010427246e-10
> > Linear solve converged due to CONVERGED_RTOL iterations 146
> > [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> > [0]PETSC ERROR: Floating point exception!
> > [0]PETSC ERROR: Infinite or not-a-number generated in norm!
> > [0]PETSC ERROR:
> ------------------------------------------------------------------------
> > [0]PETSC ERROR: Petsc Development HG revision:
> 3bede28f1195e406d4257c9fbeebefe827793801 HG Date: Mon Jun 11 22:14:12 2012
> -0400
> > [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> > [0]PETSC ERROR: See docs/index.html for manual pages.
> > [0]PETSC ERROR:
> ------------------------------------------------------------------------
> > [0]PETSC ERROR:
> /Users/jolivetinsar/MYBIN/pylith/pylith-1.7.1-darwin-10.6.8/bin/mpinemesis
> on a pylith-da named dhcp-74-229.caltech.edu by jolivetinsar Thu Aug 30
> 13:34:55 2012
> > [0]PETSC ERROR: Libraries linked from
> /Users/buildbot/install/pylith_darwin_10.6_binbot/lib
> > [0]PETSC ERROR: Configure run at Tue Jun 12 06:09:00 2012
> > [0]PETSC ERROR: Configure options
> --prefix=/Users/buildbot/install/pylith_darwin_10.6_binbot
> --download-f-blas-lapack=1 --with-sieve=1 --with-c2html=0
> --CXXFLAGS=-DMPICH_IGNORE_CXX_SEEK --with-clanguage=C++
> --with-mpicompilers=1 --with-debugging=0 --download-chaco=1
> --download-boost=1 --download-ml=1 --with-hdf5=1
> --with-hdf5-dir=/Users/buildbot/install/pylith_darwin_10.6_binbot
> --with-x=0 --with-shared-libraries=1
> > [0]PETSC ERROR:
> ------------------------------------------------------------------------
> > [0]PETSC ERROR: VecNorm() line 169 in src/vec/vec/interface/rvector.c
> > [0]PETSC ERROR: VecNormalize() line 263 in
> src/vec/vec/interface/rvector.c
> > [0]PETSC ERROR: KSPGMRESCycle() line 127 in
> src/ksp/ksp/impls/gmres/gmres.c
> > [0]PETSC ERROR: KSPSolve_GMRES() line 231 in
> src/ksp/ksp/impls/gmres/gmres.c
> > [0]PETSC ERROR: KSPSolve() line 446 in src/ksp/ksp/interface/itfunc.c
> > [0]PETSC ERROR: _sensitivitySolve() line 2172 in
> "unknowndirectory/"faults/FaultCohesiveDyn.cc
> > Traceback (most recent call last):
> > File
> "/Users/jolivetinsar/MYBIN/pylith/pylith-1.7.1-darwin-10.6.8/lib/python2.7/site-packages/pylith/apps/PetscApplication.py",
> line 65, in onComputeNodes
> > Fatal error. Calling MPI_Abort() to abort PyLith application.
> > self.main(*args, **kwds)
> > File
> "/Users/jolivetinsar/MYBIN/pylith/pylith-1.7.1-darwin-10.6.8/lib/python2.7/site-packages/pylith/apps/PyLithApp.py",
> line 119, in main
> > self.problem.run(self)
> > File
> "/Users/jolivetinsar/MYBIN/pylith/pylith-1.7.1-darwin-10.6.8/lib/python2.7/site-packages/pylith/problems/TimeDependent.py",
> line 154, in run
> > self.formulation.step(t, dt)
> > File
> "/Users/jolivetinsar/MYBIN/pylith/pylith-1.7.1-darwin-10.6.8/lib/python2.7/site-packages/pylith/problems/Implicit.py",
> line 219, in step
> > self.solver.solve(dispIncr, self.jacobian, residual)
> > File
> "/Users/jolivetinsar/MYBIN/pylith/pylith-1.7.1-darwin-10.6.8/lib/python2.7/site-packages/pylith/problems/problems.py",
> line 180, in solve
> > def solve(self, *args): return
> _problems.SolverNonlinear_solve(self, *args)
> > RuntimeError: std::exception
> > application called MPI_Abort(MPI_COMM_WORLD, -1) - process 0
> > /Users/jolivetinsar/MYBIN/pylith/pylith-1.7.1-darwin-10.6.8/bin/nemesis:
> mpirun: exit 255
> > /Users/jolivetinsar/MYBIN/pylith/pylith-1.7.1-darwin-10.6.8/bin/pylith:
> /Users/jolivetinsar/MYBIN/pylith/pylith-1.7.1-darwin-10.6.8/bin/nemesis:
> exit 1
> >
> > Any idea?
> > Romain
> >
> >
> >
> > Romain Jolivet
> > Postdoctoral Scholar
> > Geological and Planetary Sciences
> > California Institute of Technology, Pasadena, CA
> > rjolivet at caltech.edu
> > +1 (626) 560 6356
> >
> > On Aug 30, 2012, at 12:00 PM, cig-short-request at geodynamics.org wrote:
> >
> >> Send CIG-SHORT mailing list submissions to
> >> cig-short at geodynamics.org
> >>
> >> To subscribe or unsubscribe via the World Wide Web, visit
> >> http://geodynamics.org/cgi-bin/mailman/listinfo/cig-short
> >> or, via email, send a message with subject or body 'help' to
> >> cig-short-request at geodynamics.org
> >>
> >> You can reach the person managing the list at
> >> cig-short-owner at geodynamics.org
> >>
> >> When replying, please edit your Subject line so it is more specific
> >> than "Re: Contents of CIG-SHORT digest..."
> >>
> >>
> >> Today's Topics:
> >>
> >> 1. Re: Rate and State and Initial Traction (Brad Aagaard)
> >> 2. Fwd: Rate and State and Initial Traction (Matthew Knepley)
> >>
> >>
> >> ----------------------------------------------------------------------
> >>
> >> Message: 1
> >> Date: Wed, 29 Aug 2012 12:36:08 -0700
> >> From: Brad Aagaard <baagaard at usgs.gov>
> >> Subject: Re: [CIG-SHORT] Rate and State and Initial Traction
> >> To: cig-short at geodynamics.org
> >> Message-ID: <503E6F28.6050001 at usgs.gov>
> >> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
> >>
> >> Romain,
> >>
> >> It looks like you are using Schur complement preconditioner settings.
> >> The jump in the KSP residual at iteration 10 is odd and suggests
> >> something is not quite right in the linear solve (probably related to
> >> the preconditioner). Try using the AMG preconditioner. It may not be
> >> quite as fast but it is more robust. Matt can probably help you tune the
> >> Schur complement solver conditions some more if necessary.
> >>
> >> Your SNES tolerances are quite large. I suggest using tolerances similar
> >> to those in examples/3d/hex8/step14.cfg.
> >>
> >> SOLVER TOLERANCES
> >>
> >> ksp_rtol = 1.0e-16
> >> ksp_atol = 1.0e-12
> >>
> >> snes_rtol = 1.0e-14
> >> snes_atol = 1.0e-10
> >>
> >>
> >> AMG PRECONDITIONER SETTINGS
> >>
> >> [pylithapp.timedependent.formulation]
> >> split_fields = True
> >> matrix_type = aij
> >>
> >> [pylithapp.petsc]
> >> ksp_gmres_restart = 100
> >> fs_pc_type = fieldsplit
> >> fs_pc_fieldsplit_real_diagonal = true
> >> fs_pc_fieldsplit_type = multiplicative
> >> fs_fieldsplit_0_pc_type = ml
> >> fs_fieldsplit_1_pc_type = jacobi
> >> fs_fieldsplit_0_ksp_type = preonly
> >> fs_fieldsplit_1_ksp_type = preonly
> >>
> >> Brad
> >>
> >>
> >> On 08/29/2012 11:42 AM, Romain Jolivet wrote:
> >>> Hello Brad, Matt, Charles and the pylith team,
> >>>
> >>> Following your advice from the other day, I am experimenting how to
> create earthquakes on a frictional interface by applying a traction change
> history. It works fine with the SlipWeakening law in 2D. I'm trying to move
> on to the RateStateAgeing behavior and I have problems applying the initial
> traction conditions:
> >>>
> >>> I use a SimpleDB to apply the frictional properties on the fault: one
> domain is rate-strengthening, the other is rate-weakening with a linear
> transition between the two domains. If I do not try to turn on any
> traction_perturbation option, the code runs great, and I have a slow slip
> all the along the fault at any time.
> >>>
> >>> As I want the rate-weakening domain to be stuck sometimes, I want to
> turn on traction_perturbation.db_initial. I use a SimpleDB to put a
> negative traction all along the fault, and a shear traction compatible with
> the friction coefficients I impose. However, when I run this, I have the
> following error message from PETSc (I attach the log):
> >>>
> >>>>>
> /Users/jolivetinsar/MYBIN/pylith/pylith-1.7.1-darwin-10.6.8/lib/python2.7/site-packages/pylith/problems/Implicit.py:215:step
> >>> -- implicit(info)
> >>> -- Solving equations.
> >>> 0 SNES Function norm 2.679589636063e+01
> >>> 0 KSP Residual norm 9.637565795900e-01
> >>> 1 KSP Residual norm 3.564937135471e-08
> >>> 2 KSP Residual norm 4.313963401911e-16
> >>> 3 KSP Residual norm 3.050427683646e-16
> >>> 4 KSP Residual norm 2.490663081181e-16
> >>> 5 KSP Residual norm 2.156976950523e-16
> >>> 6 KSP Residual norm 1.929258577738e-16
> >>> 7 KSP Residual norm 1.761163907914e-16
> >>> 8 KSP Residual norm 1.630520838230e-16
> >>> 9 KSP Residual norm 1.525212509706e-16
> >>> 10 KSP Residual norm 1.833325155789e-11
> >>> 11 KSP Residual norm 7.039784754868e-23
> >>> Linear solve converged due to CONVERGED_ATOL iterations 11
> >>> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> >>> [0]PETSC ERROR: Floating point exception!
> >>> [0]PETSC ERROR: Infinite or not-a-number generated in norm!
> >>>
> >>> If anyone has an idea about that error message....
> >>>
> >>> I think it has to do with some slip vector. If I set
> friction.min_slip_rate=1e-1, this problem does not happen (the nonlinear
> solver does not converge, but at least, no crash). If I set the normal
> traction to be positive, it does not crash as well (and it does not
> converge as well). I join to this message the cfg file I am using for that
> experiment.
> >>>
> >>> Thank you
> >>> Romain
> >>>
> >>>
> >>>
> >>>
> >>> Romain Jolivet
> >>> Postdoctoral Scholar
> >>> Geological and Planetary Sciences
> >>> California Institute of Technology, Pasadena, CA
> >>> rjolivet at caltech.edu
> >>> +1 (626) 560 6356
> >>>
> >>>
> >>>
> >>> _______________________________________________
> >>> CIG-SHORT mailing list
> >>> CIG-SHORT at geodynamics.org
> >>> http://geodynamics.org/cgi-bin/mailman/listinfo/cig-short
> >>>
> >>
> >>
> >>
> >> ------------------------------
> >>
> >> Message: 2
> >> Date: Wed, 29 Aug 2012 22:09:25 -0500
> >> From: Matthew Knepley <knepley at mcs.anl.gov>
> >> Subject: [CIG-SHORT] Fwd: Rate and State and Initial Traction
> >> To: cig-short at geodynamics.org
> >> Message-ID:
> >> <CAMYG4GkMcUqRjhMaYZ8Ooj-x=
> 06QKFrc7fiwW1dStmDSDO1qWA at mail.gmail.com>
> >> Content-Type: text/plain; charset="iso-8859-1"
> >>
> >> On Wed, Aug 29, 2012 at 2:36 PM, Brad Aagaard <baagaard at usgs.gov>
> wrote:
> >>
> >>> Romain,
> >>>
> >>> It looks like you are using Schur complement preconditioner settings.
> >>> The jump in the KSP residual at iteration 10 is odd and suggests
> >>> something is not quite right in the linear solve (probably related to
> >>>
> >>
> >> Brad is right that the tolerances here will produce some garbage. Going
> >> below
> >> machine epsilon (about 10^-16) means you get randomness and ultimately
> >> underflow which I think you got in a norm calculation since things get
> >> squared.
> >> The solver should have converged on iterate 2, and will with the
> tolerances
> >> that
> >> Brad suggested.
> >>
> >> Thanks,
> >>
> >> Matt
> >>
> >>
> >>> the preconditioner). Try using the AMG preconditioner. It may not be
> >>> quite as fast but it is more robust. Matt can probably help you tune
> the
> >>> Schur complement solver conditions some more if necessary.
> >>>
> >>> Your SNES tolerances are quite large. I suggest using tolerances
> similar
> >>> to those in examples/3d/hex8/step14.cfg.
> >>>
> >>> SOLVER TOLERANCES
> >>>
> >>> ksp_rtol = 1.0e-16
> >>> ksp_atol = 1.0e-12
> >>>
> >>> snes_rtol = 1.0e-14
> >>> snes_atol = 1.0e-10
> >>>
> >>>
> >>> AMG PRECONDITIONER SETTINGS
> >>>
> >>> [pylithapp.timedependent.formulation]
> >>> split_fields = True
> >>> matrix_type = aij
> >>>
> >>> [pylithapp.petsc]
> >>> ksp_gmres_restart = 100
> >>> fs_pc_type = fieldsplit
> >>> fs_pc_fieldsplit_real_diagonal = true
> >>> fs_pc_fieldsplit_type = multiplicative
> >>> fs_fieldsplit_0_pc_type = ml
> >>> fs_fieldsplit_1_pc_type = jacobi
> >>> fs_fieldsplit_0_ksp_type = preonly
> >>> fs_fieldsplit_1_ksp_type = preonly
> >>>
> >>> Brad
> >>>
> >>>
> >>> On 08/29/2012 11:42 AM, Romain Jolivet wrote:
> >>>> Hello Brad, Matt, Charles and the pylith team,
> >>>>
> >>>> Following your advice from the other day, I am experimenting how to
> >>> create earthquakes on a frictional interface by applying a traction
> change
> >>> history. It works fine with the SlipWeakening law in 2D. I'm trying to
> move
> >>> on to the RateStateAgeing behavior and I have problems applying the
> initial
> >>> traction conditions:
> >>>>
> >>>> I use a SimpleDB to apply the frictional properties on the fault: one
> >>> domain is rate-strengthening, the other is rate-weakening with a linear
> >>> transition between the two domains. If I do not try to turn on any
> >>> traction_perturbation option, the code runs great, and I have a slow
> slip
> >>> all the along the fault at any time.
> >>>>
> >>>> As I want the rate-weakening domain to be stuck sometimes, I want to
> >>> turn on traction_perturbation.db_initial. I use a SimpleDB to put a
> >>> negative traction all along the fault, and a shear traction compatible
> with
> >>> the friction coefficients I impose. However, when I run this, I have
> the
> >>> following error message from PETSc (I attach the log):
> >>>>
> >>>>>>
> >>>
> /Users/jolivetinsar/MYBIN/pylith/pylith-1.7.1-darwin-10.6.8/lib/python2.7/site-packages/pylith/problems/Implicit.py:215:step
> >>>> -- implicit(info)
> >>>> -- Solving equations.
> >>>> 0 SNES Function norm 2.679589636063e+01
> >>>> 0 KSP Residual norm 9.637565795900e-01
> >>>> 1 KSP Residual norm 3.564937135471e-08
> >>>> 2 KSP Residual norm 4.313963401911e-16
> >>>> 3 KSP Residual norm 3.050427683646e-16
> >>>> 4 KSP Residual norm 2.490663081181e-16
> >>>> 5 KSP Residual norm 2.156976950523e-16
> >>>> 6 KSP Residual norm 1.929258577738e-16
> >>>> 7 KSP Residual norm 1.761163907914e-16
> >>>> 8 KSP Residual norm 1.630520838230e-16
> >>>> 9 KSP Residual norm 1.525212509706e-16
> >>>> 10 KSP Residual norm 1.833325155789e-11
> >>>> 11 KSP Residual norm 7.039784754868e-23
> >>>> Linear solve converged due to CONVERGED_ATOL iterations 11
> >>>> [0]PETSC ERROR: --------------------- Error Message
> >>> ------------------------------------
> >>>> [0]PETSC ERROR: Floating point exception!
> >>>> [0]PETSC ERROR: Infinite or not-a-number generated in norm!
> >>>>
> >>>> If anyone has an idea about that error message....
> >>>>
> >>>> I think it has to do with some slip vector. If I set
> >>> friction.min_slip_rate=1e-1, this problem does not happen (the
> nonlinear
> >>> solver does not converge, but at least, no crash). If I set the normal
> >>> traction to be positive, it does not crash as well (and it does not
> >>> converge as well). I join to this message the cfg file I am using for
> that
> >>> experiment.
> >>>>
> >>>> Thank you
> >>>> Romain
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> Romain Jolivet
> >>>> Postdoctoral Scholar
> >>>> Geological and Planetary Sciences
> >>>> California Institute of Technology, Pasadena, CA
> >>>> rjolivet at caltech.edu
> >>>> +1 (626) 560 6356
> >>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> CIG-SHORT mailing list
> >>>> CIG-SHORT at geodynamics.org
> >>>> http://geodynamics.org/cgi-bin/mailman/listinfo/cig-short
> >>>>
> >>>
> >>> _______________________________________________
> >>> CIG-SHORT mailing list
> >>> CIG-SHORT at geodynamics.org
> >>> http://geodynamics.org/cgi-bin/mailman/listinfo/cig-short
> >>>
> >>
> >>
> >>
> >> --
> >> What most experimenters take for granted before they begin their
> >> experiments is infinitely more interesting than any results to which
> their
> >> experiments lead.
> >> -- Norbert Wiener
> >>
> >>
> >>
> >> --
> >> What most experimenters take for granted before they begin their
> >> experiments is infinitely more interesting than any results to which
> their
> >> experiments lead.
> >> -- Norbert Wiener
> >> -------------- next part --------------
> >> An HTML attachment was scrubbed...
> >> URL:
> http://geodynamics.org/pipermail/cig-short/attachments/20120829/9e91899c/attachment.html
> >>
> >> ------------------------------
> >>
> >> _______________________________________________
> >> CIG-SHORT mailing list
> >> CIG-SHORT at geodynamics.org
> >> http://geodynamics.org/cgi-bin/mailman/listinfo/cig-short
> >>
> >>
> >> End of CIG-SHORT Digest, Vol 72, Issue 21
> >> *****************************************
> >
> > _______________________________________________
> > CIG-SHORT mailing list
> > CIG-SHORT at geodynamics.org
> > http://geodynamics.org/cgi-bin/mailman/listinfo/cig-short
> >
>
> _______________________________________________
> CIG-SHORT mailing list
> CIG-SHORT at geodynamics.org
> http://geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://geodynamics.org/pipermail/cig-short/attachments/20120830/9196bda7/attachment-0001.htm
More information about the CIG-SHORT
mailing list