<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">One correction to the e-mail below, we've been compiling CitcomCU using openmpi on our old<div>cluster, so the compiler on the new cluster is the same. The big difference is that the cluster</div><div>is about twice as fast as the 5-year old cluster. This suggests that this change to a much faster</div><div>clsuter may have exposed an existing race condition in CitcomCU??</div><div>Magali</div><div><br></div><div><div><div><br><div>Begin forwarded message:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="4" color="#000000" style="font: 13.0px Helvetica; color: #000000"><b>From: </b></font><font face="Helvetica" size="4" style="font: 13.0px Helvetica">Magali Billen <<a href="mailto:mibillen@ucdavis.edu">mibillen@ucdavis.edu</a>></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="4" color="#000000" style="font: 13.0px Helvetica; color: #000000"><b>Date: </b></font><font face="Helvetica" size="4" style="font: 13.0px Helvetica">November 17, 2009 4:23:45 PM PST</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="4" color="#000000" style="font: 13.0px Helvetica; color: #000000"><b>To: </b></font><font face="Helvetica" size="4" style="font: 13.0px Helvetica"><a href="mailto:cig-mc@geodynamics.org">cig-mc@geodynamics.org</a></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="4" color="#000000" style="font: 13.0px Helvetica; color: #000000"><b>Subject: </b></font><font face="Helvetica" size="4" style="font: 13.0px Helvetica"><b>[CIG-MC] MPI_Isend error</b></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div> </div><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hello,<div><br></div><div>I'm using CitcomCU and am having a strange problem with problem either hanging (no error, just doesn't </div><div>go anywhere) or it dies with an MPI_Isend error (see below). I seem to recall having problems with the MPI_Isend </div><div>command and the lam-mpi version of mpi, but I've not had any problems with mpich-2.</div><div>On the new cluster we are compling with openmpi instead of MPICH-2.</div><div><br></div><div>The MPI_Isend error seems to occur during Initialization in the call to the function mass_matrix, which then </div><div>calls exchange_node_f20, which is where the call to MPI_Isend is.</div><div><br></div><div><div>--snip--</div><div>ok14: parallel shuffle element and id arrays</div><div>ok15: construct shape functions</div></div><div><div>[farm.caes.ucdavis.edu:27041] *** An error occurred in MPI_Isend</div><div>[farm.caes.ucdavis.edu:27041] *** on communicator MPI_COMM_WORLD</div><div>[farm.caes.ucdavis.edu:27041] *** MPI_ERR_RANK: invalid rank</div><div>[farm.caes.ucdavis.edu:27041] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort)</div><div><br></div><div><div>Has this (or these) types of error occurred for other versions of Citcom using MPI_Isend (it seems that CitcomS uses</div><div>this command also). I'm not sure how to debug this error, especially since sometimes it just hangs with no error.</div><div><br></div><div>Any advice you have would be hepful,</div><div>Magali</div><div><br></div><div><br></div></div></div><div><div apple-content-edited="true"> <div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><div>-----------------------------</div><div>Associate Professor, U.C. Davis</div><div>Department of Geology/KeckCAVEs</div><div>Physical & Earth Sciences Bldg, rm 2129</div><div>Davis, CA 95616</div><div>-----------------</div><div><a href="mailto:mibillen@ucdavis.edu">mibillen@ucdavis.edu</a></div><div>(530) 754-5696</div><div><b>-----------------------------</b></div><div><b>** Note new e-mail, building, office</b></div><div><b> information as of Sept. 2009 **</b></div><div>-----------------------------</div></div></div></div> </div><br></div></div>_______________________________________________<br>CIG-MC mailing list<br><a href="mailto:CIG-MC@geodynamics.org">CIG-MC@geodynamics.org</a><br>http://geodynamics.org/cgi-bin/mailman/listinfo/cig-mc<br></blockquote></div><br><div apple-content-edited="true"> <span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: 'Lucida Grande'; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: 'Lucida Grande'; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><div>-----------------------------</div><div>Associate Professor, U.C. Davis</div><div>Department of Geology/KeckCAVEs</div><div>Physical & Earth Sciences Bldg, rm 2129</div><div>Davis, CA 95616</div><div>-----------------</div><div><a href="mailto:mibillen@ucdavis.edu">mibillen@ucdavis.edu</a></div><div>(530) 754-5696</div><div><b>-----------------------------</b></div><div><b>** Note new e-mail, building, office</b></div><div><b> information as of Sept. 2009 **</b></div><div>-----------------------------</div></div></div></span></div></span> </div><br></div></div></body></html>