<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Matt<div><br><div><div><blockquote type="cite"><div class="gmail_quote"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div style=""><div><div>Yes it fails at the same point on both machines. However if I use a different material property then it fails much later, but again the time step (at which it fails) is same on the two machines.</div> </div></div></blockquote><div><br><br>Not just same time step. Same iterate. Everything.</div></div></blockquote><div><br></div><div>I deleted my output file on one machine but will run it again</div><div><br></div><div><br></div><blockquote type="cite"><div class="gmail_quote"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div style=""><div><div><div class="im"><blockquote type="cite"><div class="gmail_quote"><div>3) Are the error messages identical on the two machines?</div></div></blockquote><div><br></div></div>Yes</div> </div></div></blockquote><div><br>I need the entire error message. I mean EXACTLY the same. Letter for letter</div></div></blockquote><div><br></div><div><br></div><div>Here's the exact message. You can see out.txt at <a href="http://stali.freeshell.org/out.txt.gz">http://stali.freeshell.org/out.txt.gz</a></div><div><br></div><div>$ pylith --nodes=4 --petsc.ksp_type=cg > out.txt</div><div><br></div><div><div>[cli_0]: aborting job:</div><div>Fatal error in MPI_Wait: Error message texts are not available</div><div>[cli_1]: aborting job:</div><div>Fatal error in MPI_Wait: Error message texts are not available</div><div>[cli_3]: aborting job:</div><div>Fatal error in MPI_Wait: Error message texts are not available</div><div>[cli_2]: aborting job:</div><div>Fatal error in MPI_Wait: Error message texts are not available</div><div>mpiexec: Warning: tasks 0-3 exited with status 1.</div><div>--pyre-start: mpiexec: exit 1</div><div>/usr/rmt_share/scratch96/s/stali/pylith/bin/pylith: /usr/rmt_share/scratch96/s/stali/pylith/bin/nemesis: exit 1</div><div><br></div></div>Tabrez</div><div><br></div><div><br><blockquote type="cite"><div class="gmail_quote"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div style=""><div><div><div class="h5"><div><blockquote type="cite"><div class="gmail_quote"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"> <div><div><div><div><div><div><div>On Apr 30, 2009, at 9:15 AM, Matthew Knepley wrote:</div><br><blockquote type="cite">On Thu, Apr 30, 2009 at 8:11 AM, Tabrez Ali <span dir="ltr"><<a href="mailto:stali@purdue.edu" target="_blank">stali@purdue.edu</a>></span> wrote:<br> <div class="gmail_quote"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"> Brad<br> <br> The solution at the last working step does converge and looks okay but<br> then nothing happens and it dies. I am however experimenting with<br> time_step and will also try to use the debugger.<br> <br> Btw do you know if I can use --petsc.on_error_attach_debugger when the<br> job is submitted via PBS or should I just run it interactively?</blockquote> <div><br>I do not understand why this is labeled a convergence issue. Unless I miss what<br>you mean by "die". Non-convergence will result in a bad ConvergenceReason<br> from the solver, but nothing else. The code will continue to run.<br> <br>This looks like death from a signal. With the very little information in front of<br>me, this looks like a bug in the MPI on this machine. If it was doing Sieve stuff,<br> I would put the blame on me. But with PETSc stuff (10+ years old and used by<br> thousands of people), I put the blame on MPI or hardware for this computer.<br><br> Matt<br> </div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"> <br> ...<br> ...<br> 87 KSP Residual norm 3.579491816101e-07<br> 88 KSP Residual norm 3.241876854223e-07<br> 89 KSP Residual norm 2.836307394788e-07<br> <br> [cli_0]: aborting job:<br> Fatal error in MPI_Wait: Error message texts are not available<br> [cli_1]: aborting job:<br> Fatal error in MPI_Wait: Error message texts are not available<br> [cli_3]: aborting job:<br> Fatal error in MPI_Wait: Error message texts are not available<br> [cli_2]: aborting job:<br> Fatal error in MPI_Wait: Error message texts are not available<br> mpiexec: Warning: tasks 0-3 exited with status 1.<br> --pyre-start: mpiexec: exit 1<br> /usr/rmt_share/scratch96/s/stali/pylith/bin/pylith: /usr/rmt_share/<br> scratch96/s/stali/pylith/bin/nemesis: exit 1<br> <br> Tabrez<br> <br> On Apr 29, 2009, at 4:26 PM, Brad Aagaard wrote:<br> <br> > Tabrez-<br> ><br> > You may want to set ksp_monitor=true so that you can see the<br> > residual. If the<br> > residual increases significantly, the solution is losing<br> > convergence. This<br> > can be alleviated a bit by using an absolute convergence tolerance<br> > (ksp_atol). You probably need a slightly smaller time step or<br> > slightly higher<br> > quality mesh (improve the aspect ratio of the most distorted cells).<br> ><br> > Brad<br> ><br> ><br> > On Wednesday 29 April 2009 1:13:21 pm Tabrez Ali wrote:<br> >> Brad<br> >><br> >> I think you were right. The elastic problem worked out fine. I will<br> >> now try to play with time step (for the viscous runs)<br> >><br> >> Tabrez<br> >><br> >> On Apr 29, 2009, at 1:19 PM, Brad Aagaard wrote:<br> >>> On Wednesday 29 April 2009 10:09:26 am Tabrez Ali wrote:<br> >>>> Also I dont see the error until ~9000 time steps with one set of<br> >>>> material properties but get the error at around 4000th time step<br> >>>> with<br> >>>> a different set of material properties (on the same mesh).<br> >>><br> >>> This seems to indicate a time-integration stability issue. Does the<br> >>> one that<br> >>> has an error after 4000 time steps have a smaller Maxwell time? You<br> >>> might try<br> >>> running with purely elastic properties. If that works, then you may<br> >>> need to<br> >>> reduce your time step.<br> ><br> ><br> <br> _______________________________________________<br> CIG-SHORT mailing list<br> <a href="mailto:CIG-SHORT@geodynamics.org" target="_blank">CIG-SHORT@geodynamics.org</a><br> <a href="http://geodynamics.org/cgi-bin/mailman/listinfo/cig-short" target="_blank">http://geodynamics.org/cgi-bin/mailman/listinfo/cig-short</a><br> </blockquote></div></blockquote></div></div></div></div></div></div></blockquote></div></blockquote></div></div></div></div></div></blockquote></div></blockquote></div><br></div></div></body></html>