[CIG-MC] CitcomCU MPI problem, perhaps
Thorsten Becker
thorstinski at gmail.com
Wed Aug 18 15:11:23 PDT 2010
Hi Eh,
Using the CIG SVN version of CitcomCU, we are experiencing some strange
problems that may be related to MPI calls. The attached model works
fine in lower resolution, as commented out in the input file. If we use
the higher
levels=6 nx=1,257 nz=1,65 ny=1,65
resolution, the code works fine, too, until after the first time step.
After processing the initial velocities (by call to output), the code
hangs in the second assemble_forces step (upon reentry into the
solver). In particular, it seems to now like the exchange_id_d20 part
that I've given below. No error messages are produced, nor any core
dumps, but the code just hangs, presumably because because of some
wrong call to MPI.
Do you see any simple solution to this, either by us fixing our input
file (sorry...), or by looking at the source code?
Thanks much in advance!
Cheers
T
>>> part of exchange_id_d20 that causes the trouble
idb = 0;
for(k = 1; k <= 2; k++)
if(E->parallel.NUM_PASS[levmax].bound[i][k])
{
for(j = 1; j <= E->parallel.NUM_NEQ[lev].pass[i][k]; j++)
S[k][j - 1] = U[E->parallel.EXCHANGE_ID[lev][j].pass[i][k]];
target_proc = E->parallel.PROCESSOR[lev].pass[i][k];
if(target_proc != E->parallel.me)
{
idb++;
MPI_Isend(S[k], E->parallel.NUM_NEQ[lev].pass[i][k], MPI_DOUBLE,
target_proc, 1, MPI_COMM_WORLD, &request[idb - 1]);
}
} /* for k */
<<<
--
Thorsten W Becker - Department of Earth Sciences
University of Southern California, 3651 Trousdale Pkwy, MC0740
Los Angeles CA 90089-0740 - geodynamics.usc.edu/~becker
-------------- next part --------------
# Start up file for citcom using getpar.
#
# free side
#
##
# 1. Input and Output Files Information
datafile="../regtest/scratch/eh_test/" # directory and file head for output files.
use_scratch="local" # output files under current directory.
oldfile="" # directory and file head for restart files.
restart=0 # check Convection.c for different options
restart_timesteps=2000 # timestep to restart
stokes_flow_only=0 # 1: only solve velocity once; 0: time-dependent problem.
maxstep=200 # max time steps
storage_spacing=50 # write data every ...
# 2. Geometry, Ra numbers, Internal heating, Thermochemical/Purely thermal convection
Solver=multigrid node_assemble=1 # multigrid iterative method
rayleigh=5e6 # Rayleigh number using T=1890 and eta=12.*1e20
rayleigh_comp=0 # Compositional Rayleigh number. relevant for composition=1
composition=0 # 0: purely thermal convection; 1: thermochem
Q0=0 # Dimensionless internal heating rate
Q0_enriched=0 # Q0 for C=1 layer, when relevant
markers_per_ele=25 # number of particles per element
comp_depth=0.5 # initial depth of compositional interface
visc_heating=0 # 1: visc heating on; or 0: visc heating off.
adi_heating=0 # 1: adiabatic heating on; or 0: adiabatic heating off.
# 3. Grid And Multiprocessor Information
nprocx=4 # number of processors in x or theta dir
nprocz=2 # number of processors in z or r dir
nprocy=2 # number of processors in y or fi dir
nodex=257 nodez=65 nodey=65 # only relevant with conj-grad
mgunitx=8 # multigrid base level in x or t
mgunitz=2 # multigrid base level in z or r
mgunity=2 # multigrid base level in y or f
levels=6 nx=1,257 nz=1,65 ny=1,65
# levels=4 nx=1,65 nz=1,17 ny=1,17
# levels=5 nx=1,129 nz=1,33 ny=1,33
# 4. Coordinate Information
Geometry=cart3d # cart3d or Rsphere
# Geometry=Rsphere # cart3d or Rsphere
#CART: irrelevant if Geometry=Rsphere.
dimenx=4.0 # box size in x direction
dimenz=1.0 # box size in z direction
dimeny=1.0 # box size in y direction
z_grid_layers=2 # minus 1 is number of layers with uniform grid in z.
zz=0,1
#testing to see if I can get my lm back!
#lm back with mgunitsz=2, but not with 4
x_grid_layers=2
xx=0,4.0
y_grid_layers=2
yy=0,1.0
z_lmantle=0.5 # 1320 km-660 km
z_410=0.68493 # 1320 km-410 km
z_lith=0.9242 # 1320 100 km
# Regional SPHERICAL: irrelevant if Geometry=cart3d
radius_inner=0.55 radius_outer=1.0 # inner and outer radius
theta_north=73.5 theta_south=106.5 # colatitude in deg
fi_west=0 fi_east=36.0 # longitude in deg
r_grid_layers=4
rr=0.55,0.59,0.96,1.0
nr=1,6,44,49
t_grid_layers=2
tt=73.5,106.5
nt=1,49
f_grid_layers=2
ff=0,36
nf=1,49
r_lmantle=0.89482 # (Ro-670 km)/Ro
r_410=0.9356358
r_lith=0.984301 # (Ro-100 km)/Ro
# 5. Rheology
# rheol=4 # 0,1,2,... diff. option for diff. rheology. Check Visco....c
rheol=3 # test 9-11
TDEPV=on # on/off for temperature-dependent viscosity
VISC_UPDATE=on # on/off for updating viscosity or not.
update_every_steps=1 # update viscosity every n timesteps
num_mat=4 # number of material group with possibly different rheology
#num_mat=5 # number of material group with possibly different rheology
#visc0=1,1,1,1,100 # pre-exponential constant
visc0=1,1,1,100 # pre-exponential constant
#test 7,8 with lbyerlee=0.06 and no visc cutoff
#test 7,8 with lbyerlee=0.06 and no visc cutoff
#viscE=7,7,7,7
#viscE=8,8,8,8 #test 7 with lbyerlee=0.06 and no visc cutoff
viscE=10,10,10,10 #test 7,8 with lbyerlee=0.06 and no visc cutoff
#viscE=10,10,10,10,10 #test 11 added 50km "elastic" layer
#scaling temperature, for r=4, temp at top
viscT=1,1,1,1
#
#
viscZ=0,0,0,0
SDEPV=off # on/off for non-Newtonian
sdepv_misfit=0.010 # accuracy for non-Newtonian iteration
sdepv_expt=1,1,1,1 # for each mat group, stress exponent
sdepv_trns=1.e0,1.e0,1.e0,1.e0 # transition stress
sdepv_print_convergence=on
BDEPV=on # Byerlee, use defaults
lbyerlee=0.06,0.06,0.06,0.06,0.06 #test 2, 7,8,13
bbyerlee=3.4e7,3.4e7,3.4e7,3.4e7,3.4e7 #test 10,13 rheol 3
CDEPV=off # composition dependent viscosity
pre_comp=1,500
VMIN=off visc_min=1.0e-2 # viscosity lower cutoff
VMAX=off visc_max=1.0e4 # viscosity upper cutoff
visc_smooth_cycles=1 # how viscosity is smoothed in multigrid
Viscosity=system # always
# 6. DIMENSIONAL INFORMATION and Depth-dependence
layerd=1320000.0 #meter
radius=6370000.0
ReferenceT=2233.0
refvisc=1.0e20
density=3300.0
thermdiff=1.0e-6
gravacc=10.
thermexp=5e-5
cp=1250
wdensity=0.0
visc_factor=1.0
thermexp_factor=1.0
thermdiff_factor=1.00
dissipation_number=2.601
surf_temp=0.122257
# 7. phase changes: to turn off any of the phase changes, let Ra_XXX=0
Ra_410=0.0 #kg/m^3
Ra_670=0.0
clapeyron410=3.0e6 #Pa K-1
clapeyron670=-3.0e6
width410=3.5e4 #meter
width670=3.5e4
# 8. BOUNDARY CONDITIONS and Initial perturbations
topvbc=0 # velocity boundary conditions top and bottom
topvbxval=0.0
topvbyval=0.0
botvbc=0
botvbxval=0.0
botvbyval=0.0
# insulating bcs #
toptbc=0 bottbc=0 #
toptbcval=0
bottbcval=1.0 #
periodicx=off #
periodicy=off #
flowthroughx=off #
flowthroughy=off #
# temperature perturbations
num_perturbations=1 # N, Number of perturbations
perturbmag=0.001 # A list of N magnitudes
perturbk=1.0 # A list of N wavenumbers (/PI)
perturbl=6.0 # A list of N wavenumbers (/PI)
perturbm=0.0 # A list of N wavenumbers (/PI)
# 9. SOLVER RELATED MATTERS
Problem=convection # always, almost
aug_lagr=on
aug_number=1.0e3
precond=on
orthogonal=off
maxsub=1
viterations=2 # Uzawa iteration loops.
mg_cycle=1 # 1 = V cycle, 2 = W cycle, and so on.
down_heavy=3 # Increase downward smoothing by this factor.
up_heavy=3 # Increase upward smoothing by this factor.
vlowstep=25 # Enough to get very accurate soln at low level.
vhighstep=5 # Smoothing passes at highest level (finest grid).
piterations=300 # Uzawa iteration loops.
accuracy=1.0e-3 # Desired total accuracy
# Tuning of energy equation
adv_sub_iterations=2
finetunedt=0.75
ll_max=20
nlong=180
nlati=90
# Data input and program debugging
DESCRIBE=off #
BEGINNER=off #
VERBOSE=off #
verbose=off #
see_convergence=1
# output options
COMPRESS=1 #
vtkout=0
# vim:ts=8:sw=8
More information about the CIG-MC
mailing list