<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><div><div></div><div>Luckily, there are commented fprintf statements in just that part of the code... we'll continue to dig...</div><div><br></div><div>Oh, and just to eliminate the new cluster from suspicion, we downloaded, compiled and ran CitcomS</div><div>example1.cfg on the same cluster with the same compilers, and their was not problem.</div><div><br></div><div>Maybe this is the sign that I'm suppose to finally switch from CitcomCU to CitcomS... :-(</div><div>Magali</div><div><br></div><div>On Nov 17, 2009, at 5:02 PM, Eh Tan wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>Hi Magali,<br><br>How many processors are you using? If more than 100 processors are used,<br>you are seeing this bug:<br><a href="http://www.geodynamics.org/pipermail/cig-mc/2008-March/000080.html">http://www.geodynamics.org/pipermail/cig-mc/2008-March/000080.html</a><br><br><br>Eh<br><br><br><br>Magali Billen wrote:<br><blockquote type="cite">One correction to the e-mail below, we've been compiling CitcomCU<br></blockquote><blockquote type="cite">using openmpi on our old<br></blockquote><blockquote type="cite">cluster, so the compiler on the new cluster is the same. The big<br></blockquote><blockquote type="cite">difference is that the cluster<br></blockquote><blockquote type="cite">is about twice as fast as the 5-year old cluster. This suggests that<br></blockquote><blockquote type="cite">this change to a much faster<br></blockquote><blockquote type="cite">clsuter may have exposed an existing race condition in CitcomCU??<br></blockquote><blockquote type="cite">Magali<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Begin forwarded message:<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><blockquote type="cite">*From: *Magali Billen <mibillen@ucdavis.edu<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><mailto:mibillen@ucdavis.edu>><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">*Date: *November 17, 2009 4:23:45 PM PST<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">*To: *cig-mc@geodynamics.org <mailto:cig-mc@geodynamics.org><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">*Subject: **[CIG-MC] MPI_Isend error*<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Hello,<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">I'm using CitcomCU and am having a strange problem with problem<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">either hanging (no error, just doesn't <br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">go anywhere) or it dies with an MPI_Isend error (see below). I seem<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">to recall having problems with the MPI_Isend <br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">command and the lam-mpi version of mpi, but I've not had any problems<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">with mpich-2.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">On the new cluster we are compling with openmpi instead of MPICH-2.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">The MPI_Isend error seems to occur during Initialization in the call<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">to the function mass_matrix, which then <br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">calls exchange_node_f20, which is where the call to MPI_Isend is.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">--snip--<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">ok14: parallel shuffle element and id arrays<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">ok15: construct shape functions<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">[farm.caes.ucdavis.edu:27041] *** An error occurred in MPI_Isend<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">[farm.caes.ucdavis.edu:27041] *** on communicator MPI_COMM_WORLD<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">[farm.caes.ucdavis.edu:27041] *** MPI_ERR_RANK: invalid rank<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">[farm.caes.ucdavis.edu:27041] *** MPI_ERRORS_ARE_FATAL (your MPI job<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">will now abort)<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Has this (or these) types of error occurred for other versions of<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Citcom using MPI_Isend (it seems that CitcomS uses<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">this command also). I'm not sure how to debug this error,<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">especially since sometimes it just hangs with no error.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Any advice you have would be hepful,<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Magali<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">-----------------------------<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Associate Professor, U.C. Davis<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Department of Geology/KeckCAVEs<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Physical & Earth Sciences Bldg, rm 2129<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Davis, CA 95616<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">-----------------<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">mibillen@ucdavis.edu <mailto:mibillen@ucdavis.edu><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">(530) 754-5696<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">*-----------------------------*<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">*** Note new e-mail, building, office*<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">* information as of Sept. 2009 ***<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">-----------------------------<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">_______________________________________________<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">CIG-MC mailing list<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">CIG-MC@geodynamics.org <mailto:CIG-MC@geodynamics.org><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">http://geodynamics.org/cgi-bin/mailman/listinfo/cig-mc<br></blockquote></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">-----------------------------<br></blockquote><blockquote type="cite">Associate Professor, U.C. Davis<br></blockquote><blockquote type="cite">Department of Geology/KeckCAVEs<br></blockquote><blockquote type="cite">Physical & Earth Sciences Bldg, rm 2129<br></blockquote><blockquote type="cite">Davis, CA 95616<br></blockquote><blockquote type="cite">-----------------<br></blockquote><blockquote type="cite">mibillen@ucdavis.edu <mailto:mibillen@ucdavis.edu><br></blockquote><blockquote type="cite">(530) 754-5696<br></blockquote><blockquote type="cite">*-----------------------------*<br></blockquote><blockquote type="cite">*** Note new e-mail, building, office*<br></blockquote><blockquote type="cite">* information as of Sept. 2009 ***<br></blockquote><blockquote type="cite">-----------------------------<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">------------------------------------------------------------------------<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">_______________________________________________<br></blockquote><blockquote type="cite">CIG-MC mailing list<br></blockquote><blockquote type="cite">CIG-MC@geodynamics.org<br></blockquote><blockquote type="cite">http://geodynamics.org/cgi-bin/mailman/listinfo/cig-mc<br></blockquote><blockquote type="cite"><br></blockquote><br>-- <br>Eh Tan<br>Staff Scientist<br>Computational Infrastructure for Geodynamics<br>California Institute of Technology, 158-79<br>Pasadena, CA 91125<br>(626) 395-1693<br>http://www.geodynamics.org<br><br>_______________________________________________<br>CIG-MC mailing list<br>CIG-MC@geodynamics.org<br>http://geodynamics.org/cgi-bin/mailman/listinfo/cig-mc<br></div></blockquote></div><br><div apple-content-edited="true"> <div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: 'Lucida Grande'; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><div>-----------------------------</div><div>Associate Professor, U.C. Davis</div><div>Department of Geology/KeckCAVEs</div><div>Physical & Earth Sciences Bldg, rm 2129</div><div>Davis, CA 95616</div><div>-----------------</div><div><a href="mailto:mibillen@ucdavis.edu">mibillen@ucdavis.edu</a></div><div>(530) 754-5696</div><div><b>-----------------------------</b></div><div><b>** Note new e-mail, building, office</b></div><div><b> information as of Sept. 2009 **</b></div><div>-----------------------------</div></div></div></span></div> </div><br></div></body></html>