[CIG-MC] [Fwd: CitcomCU MPI problem, perhaps]

[Thorsten Becker]

Ah, with apologies, I'll just answer half of my question myself.  In
the previous case, the ratio between processors was not the same as the
ratio between the mgunits, which now occurs to me I knew was not a good
idea with CitcomCU.

Using 

nprocx=8 nprocz=2 nprocy=2

or

nprocx=4 nprocz=1 nprocy=1

Everything works just honky dory. I still wonder why it would work once,
and then fail, but oh well. Sorry about that. 

Cheers


Thorsten




-------- Forwarded Message --------
From: Thorsten Becker <thorstinski at gmail.com>
To: Eh Tan <tan2 at geodynamics.org>, cig-mc at geodynamics.org
Cc: Lisa Alpert <lisaalpert4 at gmail.com>
Subject: CitcomCU MPI problem, perhaps
Date: Wed, 18 Aug 2010 15:11:23 -0700

Hi Eh,

Using the CIG SVN version of CitcomCU, we are experiencing some strange
problems that may be related to MPI calls. The attached model works
fine in lower resolution, as commented out in the input file. If we use
the higher  

levels=6 nx=1,257 nz=1,65 ny=1,65

resolution, the code works fine, too, until after the first time step.
After processing the initial velocities (by call to output), the code
hangs in the second assemble_forces step (upon reentry into the
solver). In particular, it seems to now like the exchange_id_d20 part
that I've given below. No error messages are produced, nor any core
dumps, but the code just hangs, presumably because because of some
wrong call to MPI. 

Do you see any simple solution to this, either by us fixing our input
file (sorry...), or by looking at the source code? 

Thanks much in advance!

Cheers

T

>>> part of exchange_id_d20 that causes the trouble



	idb = 0;
		for(k = 1; k <= 2; k++)
			if(E->parallel.NUM_PASS[levmax].bound[i][k])
			{
				for(j = 1; j <= E->parallel.NUM_NEQ[lev].pass[i][k]; j++)
					S[k][j - 1] = U[E->parallel.EXCHANGE_ID[lev][j].pass[i][k]];

				target_proc = E->parallel.PROCESSOR[lev].pass[i][k];
				if(target_proc != E->parallel.me)
				{
					idb++;
					MPI_Isend(S[k], E->parallel.NUM_NEQ[lev].pass[i][k], MPI_DOUBLE,
target_proc, 1, MPI_COMM_WORLD, &request[idb - 1]);
				}
			}					/* for k */
		

<<<


-- 
Thorsten W Becker           -       Department of Earth Sciences
University of Southern California,   3651 Trousdale Pkwy, MC0740
Los Angeles CA 90089-0740   -        geodynamics.usc.edu/~becker

-- 
Thorsten W Becker           -       Department of Earth Sciences
University of Southern California,   3651 Trousdale Pkwy, MC0740
Los Angeles CA 90089-0740   -        geodynamics.usc.edu/~becker