[CIG-LONG] [0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file

Wed Aug 24 11:30:45 PDT 2011

I am running a simple 3D model with flow induced by velocities on the boundary of the box. I have run the relevant input xml file before without problems.
Now when I run using the same input, I get the following error (see tail of my log file, which shows both the stdout and std err output to the screen).
I get this error when running using gale v. 1.6.0 and v. 1.6.1. I am running in single cpu mode, using the 64 bit Linux binary from the Gale website.
The final line of the error message is:
[0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
I have noticed a couple of other messages indicating a similar error. I suspect that this is a simple problem but made cryptic by an opaque error message.
Any help would be much appreciated.
Best,
Mark Brandon

TimeStep = 0, Time = 0
In SystemLinearEquations_NonLinearExecute

Non linear solver - iteration 0
Linear solver (stokesEqn-execute)
Relative residual < 1e-05 for Uzawa stopping condition
Initial l2Norm( Div u ) / l2Norm( u ) = 5.381885
Loop = 0, absResidual = 2.78125025e-01, relResidual = 5.18772831e-01
Loop = 1, absResidual = 1.99162241e-01, relResidual = 3.71487462e-01
Loop = 2, absResidual = 1.35208723e-01, relResidual = 2.52198133e-01
Loop = 3, absResidual = 9.79933687e-02, relResidual = 1.82782176e-01
Loop = 4, absResidual = 6.65633422e-02, relResidual = 1.24157305e-01
Loop = 5, absResidual = 4.14099989e-02, relResidual = 7.72400197e-02
Loop = 6, absResidual = 2.76821797e-02, relResidual = 5.16341986e-02
Loop = 7, absResidual = 1.91184083e-02, relResidual = 3.56606199e-02
Loop = 8, absResidual = 1.23066717e-02, relResidual = 2.29550250e-02
Loop = 9, absResidual = 7.84549777e-03, relResidual = 1.46338183e-02
Loop = 10, absResidual = 4.89534033e-03, relResidual = 9.13103582e-03
Loop = 11, absResidual = 2.84408435e-03, relResidual = 5.30492965e-03
Loop = 12, absResidual = 1.70103300e-03, relResidual = 3.17285259e-03
Loop = 13, absResidual = 1.14637760e-03, relResidual = 2.13828136e-03
Loop = 14, absResidual = 7.73706821e-04, relResidual = 1.44315701e-03
Loop = 15, absResidual = 5.02272790e-04, relResidual = 9.36864553e-04
Loop = 16, absResidual = 3.43380675e-04, relResidual = 6.40490962e-04
Loop = 17, absResidual = 2.43255829e-04, relResidual = 4.53733047e-04
Loop = 18, absResidual = 1.64946148e-04, relResidual = 3.07665879e-04
Loop = 19, absResidual = 1.09683318e-04, relResidual = 2.04586859e-04
Loop = 20, absResidual = 7.24288105e-05, relResidual = 1.35097872e-04
Loop = 21, absResidual = 4.66939042e-05, relResidual = 8.70958263e-05
Loop = 22, absResidual = 2.92023039e-05, relResidual = 5.44696108e-05
Loop = 23, absResidual = 1.83228182e-05, relResidual = 3.41766451e-05
Loop = 24, absResidual = 1.18370684e-05, relResidual = 2.20790973e-05
Loop = 25, absResidual = 8.09870206e-06, relResidual = 1.51061078e-05
Loop = 26, absResidual = 5.62546108e-06, relResidual = 1.04928939e-05
Loop = 27, absResidual = 3.67960467e-06, relResidual = 6.86338432e-06
Summary:
  Uzawa its. = 0028 , Uzawa residual = 6.8633843208976e-06
  |G^T u|/|u|               = 4.70259774e+00
  |f - K u - G p|/|f|       = 3.31124582e-07
  |f - K u - G p|_w/|f|_w   = 1.00000000e+00
  |u|_{\infty} = 1.15951032e+00 , u_rms = 5.12308691e-01
  |p|_{\infty} = 4.05888682e-01 , p_rms = 6.97875337e-03
  min/max(u) = -1.15951032e+00 [1554] / 2.16587215e-03 [1691346]
  min/max(p) = -6.02931304e-02 [706617] / 4.05888682e-01 [699335]
  \sum_i p_i = 2.83726962e+03
Linear solver (stokesEqn-execute), solution time 6.956696e+03 (secs)
Non linear solver - iteration 1
Linear solver (stokesEqn-execute)
Relative residual < 1e-05 for Uzawa stopping condition
Initial l2Norm( Div u ) / l2Norm( u ) = 4.702592
Loop = 0, absResidual = 5.30216454e-06, relResidual = 9.88986485e-06
Summary:
  Uzawa its. = 0001 , Uzawa residual = 9.8898648524070e-06
  |G^T u|/|u|               = 4.70259467e+00
  |f - K u - G p|/|f|       = 3.09184251e-07
  |f - K u - G p|_w/|f|_w   = 1.00000000e+00
  |u|_{\infty} = 1.15950894e+00 , u_rms = 5.12309181e-01
  |p|_{\infty} = 4.05888174e-01 , p_rms = 6.97874921e-03
  min/max(u) = -1.15950894e+00 [1554] / 2.16479288e-03 [1691346]
  min/max(p) = -6.02941979e-02 [706617] / 4.05888174e-01 [699335]
  \sum_i p_i = 2.83726890e+03
Linear solver (stokesEqn-execute), solution time 4.649267e+03 (secs)
In func SystemLinearEquations_NonLinearExecute: Iteration 1 of 500 - Residual 5.2166e-06 - Tolerance = 0.01
Non linear solver - Residual 5.21656101e-06; Tolerance 1.0000e-02 - Converged - 1.526749e+04 (secs)

In func SystemLinearEquations_NonLinearExecute: Converged after 1 iterations.
Time Integration
Time Integration
      EP:        EulerDeform_IntegrationSetup -    0.7536 [min] /    0.7536 [max] (secs)
      EP:        SwarmAdvector_AdvectionSetup -    0.0014 [min] /    0.0014 [max] (secs)
1st order:               materialSwarmAdvector -  553.2917 [min] /  553.2917 [max] (secs)
1st order:                EulerDeform_Velocity -   13.7820 [min] /   13.7820 [max] (secs)
      EP:                 EulerDeform_Execute -   43.1226 (secs)
      EP:       SwarmAdvector_AdvectionFinish -  516.6078 (secs)
      EP:          MaterialPointsSwarm_Update -   16.5889 (secs)
In func WeightsCalculator_CalculateAll(): for swarm "picIntegrationPoints"
Calculating weights for the particles in the 684000 global cells
done 10% (68400 cells)...
done 20% (136800 cells)...
done 30% (205200 cells)...
done 40% (273600 cells)...
done 50% (342000 cells)...
done 60% (410400 cells)...
done 70% (478800 cells)...
done 80% (547200 cells)...
done 90% (615600 cells)...
done 100% (684000 cells)...
WeightsCalculator_CalculateAll(): finished update of weights for swarm "picIntegrationPoints"
      EP:       IntegrationPointsSwarm_Update -  380.5335 (secs)
      EP:       IntegrationPointsSwarm_Update -    0.0113 (secs)
Time Integration - 1524.73 [min] / 1524.73 [max] (secs)
Time Integration - 1525 [min] / 1525 [max] (secs)
TimeStep = 1, Time = 3.22549
In SystemLinearEquations_NonLinearExecute

Non linear solver - iteration 0
Linear solver (stokesEqn-execute)
Relative residual < 1e-05 for Uzawa stopping condition
Initial l2Norm( Div u ) / l2Norm( u ) = 4.702603
Loop = 0, absResidual = 3.76515653e-04, relResidual = 7.02343854e-04
Loop = 1, absResidual = 2.44464158e-04, relResidual = 4.56017958e-04
Loop = 2, absResidual = 1.76789503e-04, relResidual = 3.29779174e-04
Loop = 3, absResidual = 1.24251444e-04, relResidual = 2.31775857e-04
Loop = 4, absResidual = 8.40585794e-05, relResidual = 1.56800989e-04
Loop = 5, absResidual = 5.66290386e-05, relResidual = 1.05634538e-04
Loop = 6, absResidual = 3.77533298e-05, relResidual = 7.04242147e-05
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 1 Hang up: Some other process (or the batch system) has told this process to end
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/petsc-as/documentation/troubleshooting.html#Signal[0]PETSC ERROR: or try http://valgrind.org on linux or man libgmalloc on Apple to find memory corruption errors
[0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run
[0]PETSC ERROR: to get more information on the crash.
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Signal received!
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.0.0, Patch 8, Fri Aug 21 14:02:12 CDT 2009
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: ./Gale-1_6_1 on a linux-gnu named pele.geology.yale.edu by markb Wed Aug 24 01:37:18 2011
[0]PETSC ERROR: Libraries linked from /home/walter/src/petsc-3.0.0-p8/linux-gnu-c-opt/lib
[0]PETSC ERROR: Configure run at Sat Jul 24 18:51:39 2010
[0]PETSC ERROR: Configure options --with-fortran=0 --with-x=0 --download-c-blas-lapack=1 --with-shared=0 --with-debugging=0
[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: User provided function() line 0 in unknown directory unknown file
[unset]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0