On Thu, Apr 30, 2009 at 8:42 AM, Tabrez Ali <span dir="ltr"><<a href="mailto:stali@purdue.edu">stali@purdue.edu</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div style="">Matt<div><br><div><div class="im"><blockquote type="cite"><div class="gmail_quote"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div> </div></blockquote><div>1) Are you positive that it fails inside KSP? That is the most bulletproof part of the code.<br><br>2) Does it always fail in the same place? If not, I can believe it fails in KSP due to<br><br>
a) Memory corruption somewhere else<br><br> b) Mismatched MPI calls hanging around<br><br> c) Someone holding on to MPI resources somewhere else</div></div></blockquote><div><br></div></div>Yes it fails at the same point on both machines. However if I use a different material property then it fails much later, but again the time step (at which it fails) is same on the two machines.</div>
</div></div></blockquote><div><br><br>Not just same time step. Same iterate. Everything. <br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div style=""><div><div></div><div><div class="im"><blockquote type="cite"><div class="gmail_quote"><div><br><br>3) Are the error messages identical on the two machines?</div></div></blockquote><div><br></div></div>Yes</div>
</div></div></blockquote><div><br>I need the entire error message. I mean EXACTLY the same. Letter for letter.<br><br> Matt<br> </div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div style=""><div><div></div><div><div></div><div class="h5"><div>Tabrez</div><div><br></div><div><br></div><div><blockquote type="cite"><div class="gmail_quote"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div><div><div><div><div><div><div>On Apr 30, 2009, at 9:15 AM, Matthew Knepley wrote:</div><br><blockquote type="cite">On Thu, Apr 30, 2009 at 8:11 AM, Tabrez Ali <span dir="ltr"><<a href="mailto:stali@purdue.edu" target="_blank">stali@purdue.edu</a>></span> wrote:<br>
<div class="gmail_quote"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"> Brad<br> <br> The solution at the last working step does converge and looks okay but<br>
then nothing happens and it dies. I am however experimenting with<br> time_step and will also try to use the debugger.<br> <br> Btw do you know if I can use --petsc.on_error_attach_debugger when the<br> job is submitted via PBS or should I just run it interactively?</blockquote>
<div><br>I do not understand why this is labeled a convergence issue. Unless I miss what<br>you mean by "die". Non-convergence will result in a bad ConvergenceReason<br> from the solver, but nothing else. The code will continue to run.<br>
<br>This looks like death from a signal. With the very little information in front of<br>me, this looks like a bug in the MPI on this machine. If it was doing Sieve stuff,<br> I would put the blame on me. But with PETSc stuff (10+ years old and used by<br>
thousands of people), I put the blame on MPI or hardware for this computer.<br><br> Matt<br> </div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br> ...<br> ...<br> 87 KSP Residual norm 3.579491816101e-07<br> 88 KSP Residual norm 3.241876854223e-07<br> 89 KSP Residual norm 2.836307394788e-07<br> <br> [cli_0]: aborting job:<br> Fatal error in MPI_Wait: Error message texts are not available<br>
[cli_1]: aborting job:<br> Fatal error in MPI_Wait: Error message texts are not available<br> [cli_3]: aborting job:<br> Fatal error in MPI_Wait: Error message texts are not available<br> [cli_2]: aborting job:<br> Fatal error in MPI_Wait: Error message texts are not available<br>
mpiexec: Warning: tasks 0-3 exited with status 1.<br> --pyre-start: mpiexec: exit 1<br> /usr/rmt_share/scratch96/s/stali/pylith/bin/pylith: /usr/rmt_share/<br> scratch96/s/stali/pylith/bin/nemesis: exit 1<br> <br> Tabrez<br>
<br> On Apr 29, 2009, at 4:26 PM, Brad Aagaard wrote:<br> <br> > Tabrez-<br> ><br> > You may want to set ksp_monitor=true so that you can see the<br> > residual. If the<br> > residual increases significantly, the solution is losing<br>
> convergence. This<br> > can be alleviated a bit by using an absolute convergence tolerance<br> > (ksp_atol). You probably need a slightly smaller time step or<br> > slightly higher<br> > quality mesh (improve the aspect ratio of the most distorted cells).<br>
><br> > Brad<br> ><br> ><br> > On Wednesday 29 April 2009 1:13:21 pm Tabrez Ali wrote:<br> >> Brad<br> >><br> >> I think you were right. The elastic problem worked out fine. I will<br> >> now try to play with time step (for the viscous runs)<br>
>><br> >> Tabrez<br> >><br> >> On Apr 29, 2009, at 1:19 PM, Brad Aagaard wrote:<br> >>> On Wednesday 29 April 2009 10:09:26 am Tabrez Ali wrote:<br> >>>> Also I dont see the error until ~9000 time steps with one set of<br>
>>>> material properties but get the error at around 4000th time step<br> >>>> with<br> >>>> a different set of material properties (on the same mesh).<br> >>><br> >>> This seems to indicate a time-integration stability issue. Does the<br>
>>> one that<br> >>> has an error after 4000 time steps have a smaller Maxwell time? You<br> >>> might try<br> >>> running with purely elastic properties. If that works, then you may<br>
>>> need to<br> >>> reduce your time step.<br> ><br> ><br> <br> _______________________________________________<br> CIG-SHORT mailing list<br> <a href="mailto:CIG-SHORT@geodynamics.org" target="_blank">CIG-SHORT@geodynamics.org</a><br>
<a href="http://geodynamics.org/cgi-bin/mailman/listinfo/cig-short" target="_blank">http://geodynamics.org/cgi-bin/mailman/listinfo/cig-short</a><br> </blockquote></div><br><br clear="all"><br>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<br> </blockquote></div><br></div></div></div></div></div></blockquote></div><br><br clear="all"><br>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<br></blockquote></div><br></div></div></div></div></blockquote></div><br><br clear="all"><br>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<br>