[CIG-SHORT] Pylith Segmentation Fault when running a dynamic rupture simulation

[Charles Williams]

Dear Ge,

How much memory per node does the cluster have?  Your mesh is fairly large (~13M vertices, ~78M cells).  Also, the fact that there was a SEGV while distributing the mesh makes it seem likely that you have run out of memory, since this is where PyLith uses the most memory.

There was some memory logging facility within PyLith, but I”m not sure if it still works.  The alternative would be to login to the first node on which PyLith is running, and run ‘top’ as your code is running.  Keep an eye on the memory usage and determine when it maxes out.  If it approaches the amount of RAM for that node, I would suspect that is the problem.

Cheers,
Charles

p.s.  The other question is whether you are sharing nodes with any other jobs, which could also limit your available memory.


> On 28/06/2018, at 10:58 AM, Ge Li <ge.li2 at mail.mcgill.ca> wrote:
> 
> Hi there,
> I’m using Pylith to simulate a dynamic rupture on a Cluster.
> However, the program was aborted due a segmentation fault error  when 
> distributing the mesh:
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range.
>  
> As suggested in the manual, I rerun Pylith under the debugger mode and with valgrind turned on, respectively, which 
> Generated some log files. These files are attached here.
> I also tried:
> export OMPI_MCA_mpi_warn_on_fork=0
> 
> export OMPI_MCA_btl_openib_want_fork_support=0
> 
> But it didn’t work.
>  
> This error was eliminated when :
> Run with a coarser mesh;
> Run with a single processor.
>  
> Can you help me to target any potential issues that may cause this problem?
> Many thanks!
>  
> -- 
> Ge Li
> Ph.D. Candidate
> Department of Earth & Planetary Science, 
> McGill University
> 3450 University Street
> Montreal, QC, Canada 
> H3A 0E8
> ge.li2 at mail.mcgill.ca <mailto:ge.li2 at mail.mcgill.ca>
>  
>  
> <valgrind-log><debugger_error.zip>_______________________________________________
> CIG-SHORT mailing list
> CIG-SHORT at geodynamics.org <mailto:CIG-SHORT at geodynamics.org>
> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
Charles Williams I Geodynamic Modeler
GNS Science I Te Pῡ Ao
1 Fairway Drive, Avalon 5010, PO Box 30368, Lower Hutt 5040, New Zealand
Ph 0064-4-570-4566 I Mob 0064-22-350-7326 I Fax 0064-4-570-4600
http://www.gns.cri.nz/ <http://www.gns.cri.nz/> I Email: C.Williams at gns.cri.nz <mailto:your.email at gns.cri.nz>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.geodynamics.org/pipermail/cig-short/attachments/20180628/fd73b23f/attachment.html>