[CIG-SHORT] Pylith stuck in the final step "Formulation finalize."

[Brad Aagaard]

Kejing Huang,

When PyLith appears to be stuck, it usually means that at least one 
process aborted but MPI didn't exit cleanly, so one or more processes 
are left hanging. It sounds like there is a chance you are running out 
of disk space.

If you are not using HDF5 output, the first thing to do is to switch 
from VTK output (default) to HDF5 output. Secondly, write out only the 
data fields that you are interested in; if you do not care about volume 
output, turn it off or only write the final values. See the examples 
(such as examples/3d/subduction) in the PyLith manual for illustrations 
of how to do this.

Regards,
Brad

On 09/28/2018 09:28 PM, hkj at mail.ustc.edu.cn wrote:
> Dear Sir,
> I'm now running a .exo model with 55000 nodes. In the pylith simulation 
> I used total time = 50 yr with dt = 0.01yr(since I need viscosity to be 
> small, 0.01yr is the largest time step as pylith suggested).Thus the 
> output file is about 80GB.
> 
> The simulation and iteration run correctly, but it has about 20% 
> probability to stuck at the final step after it shows "Formulation 
> finalize".
> 
> I waited for about 10 hours(even longer than the whole simulation time), 
> but it still stuck at the final step. The last few lines of output is 
> showed below
> 
> /
> /
> /``` [OUTPUT]/
> /......./
> /......./
> /......./
> /Linear solve converged due to CONVERGED_ATOL iterations 50/
> / >> 
> /home/collyking/Software/Pylith/lib/python2.7/site-packages/pylith/problems/TimeDependent.py:208:run/
> / -- timedependent(info)/
> / -- Finishing advancing solution from t=1.57725e+09*s to t=1.57788e+09*s./
> / >> 
> /home/collyking/Software/Pylith/lib/python2.7/site-packages/pylith/problems/Implicit.py:242:poststep/
> / -- implicit(info)/
> / -- Writing solution fields./
> / >> 
> /home/collyking/Software/Pylith/lib/python2.7/site-packages/pylith/faults/Fault.py:191:writeData/
> / -- faultcohesivekin(info)/
> / -- Writing fault data./
> / >> 
> /home/collyking/Software/Pylith/lib/python2.7/site-packages/pylith/faults/Fault.py:191:writeData/
> / -- faultcohesivekin(info)/
> / -- Writing fault data./
> / >> 
> /home/collyking/Software/Pylith/lib/python2.7/site-packages/pylith/problems/Formulation.py:273:finalize/
> / -- implicit(info)/
> / -- Formulation finalize./
> ```
> 
> Is it the normal case since the output file is too large?
> 
> Sincerely
> Kejing Huang