[CIG-LONG] Crash – Error in TimeIntegrand

[María Helga Guðmundsdóttir]

Hi Walter,

I've been running the restraining bend models that George Hilley has been troubleshooting with you lately, with mixed results. Two of them have run fine upto the amount of displacement we want, but another four models have crashed in an identical fashion at completely different points during the run (timesteps 8, 11, 90, and 289 respectively, with total displacements of 4, 10, 17, and 37km). The errors look like the following:

In .out file, crash occurs right at the start of timestep X (in this example, 90). The last lines of output are

> TimeStep = 90, Time = 3.15022e-09
> Time Integration
> 	       EP:        EulerDeform_IntegrationSetup -    0.0462 [min] /    0.0668 [max] (secs)
> 	       EP:        SwarmAdvector_AdvectionSetup -    0.0005 [min] /    0.0210 [max] (secs)

In .err file, the following:

> 15: Error - in TimeIntegrand_FirstOrder(), for TimeIntegrand "materialSwarmAdvector" of type SwarmAdvector: When trying to find time deriv for item 6073 in step 1, *failed*.
> --------------------------------------------------------------------------
> MPI_ABORT was invoked on rank 15 in communicator MPI_COMM_WORLD 
> with errorcode 0.
> 
> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
> You may or may not see output from other processes, depending on
> exactly when Open MPI kills them.
> --------------------------------------------------------------------------
> --------------------------------------------------------------------------
> mpirun has exited due to process rank 15 with PID 15877 on
> node cees007 exiting without calling "finalize". This may
> have caused other processes in the application to be
> terminated by signals sent by mpirun (as reported here).
> --------------------------------------------------------------------------


Any insights? I've looked at the mesh for timestep 89, and it doesn't look so drastically distorted to me that it should be causing problems, but I'm not sure I'm the one to judge that (see attached png, zoomed in on the edge of the model by the 'tear'). I've uploaded the input and the last two timesteps' worth of output for this particular model at the following link:

http://dl.dropbox.com/u/46861316/S0_27_A8.zip

Many thanks,
Maria







María Helga Guðmundsdóttir
MS Student, Stanford University
Geological and Environmental Sciences
mariahg at stanford.edu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://geodynamics.org/pipermail/cig-long/attachments/20120507/33f31dda/attachment-0001.htm 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: visit0001.png
Type: image/png
Size: 489707 bytes
Desc: not available
Url : http://geodynamics.org/pipermail/cig-long/attachments/20120507/33f31dda/attachment-0001.png