[CIG-SEISMO] computing error on cluster
胡元鑫
huanduh at gmail.com
Fri Jul 16 09:54:52 PDT 2010
Dear Dr.Pieyre,
After I made my model by using meshfem3D, I submitted it to cluster and got
error. But the same code worked well on my four-core computer by MPICH2.
After xmeshfem3D is finished (mpiexec -np 4 ./xmeshfem3D) on one node of
cluster, the error message are shown as follows:
"9825 150371
incorrect global numbering
Error detected, aborting MPI... proc 0
application called MPI_Abort(MPI_COMM_WORLD, 30) - process 0
rank 0 in job 1 logon01_50050 caused collective abort of all ranks
exit status of rank 0: return code 30 "
And the contents of output_mesher.txt are presented here:
"******************************************
*** Specfem3D MPI Mesher - f90 version ***
******************************************
Reading interface data from file DATA/interfaces.dat
to count the spectral elements
creating global slice addressing
Spatial distribution of slice numbers:
2 3
0 1
This is process 0
There are 4 MPI processes
Processes are numbered from 0 to 3
There are 64 elements along xi
There are 64 elements along eta
There are 15 elements along Z
There are 11 spectral elements along Z in layer 1
There are 4 spectral elements along Z in layer 2
There are 2 slices along xi
There are 2 slices along eta
There is a total of 4 slices
Shape functions defined by NGNOD = 8 control nodes
Surface shape functions defined by NGNOD2D = 4 control nodes
region selected:
latitude min = 31.0000000000000
latitude max = 31.2000000000000
longitude min = 103.100000000000
longitude max = 103.600000000000
this is mapped to UTM in region 48
UTM X min = 318594.992564268
UTM X max = 366618.290628533
UTM Y min = 3430953.24131916
UTM Y max = 3452412.76084654
UTM size of model along X is 48.0232980642641 km
UTM size of model along Y is 21.4595195273878 km
Bottom of the mesh is at a depth of 30.0000000000000 km
using UTM projection in region 48
Reading interface data from file DATA/interfaces.dat
**************************
creating mesh in the model
**************************
"
------------------------------------------------------------------------------------------------
constants.h:
! constants.h. Generated from constants.h.in by configure.
!
! solver in single or double precision depending on the machine (4 or 8
bytes)
!
! ALSO CHANGE FILE precision.h ACCORDINGLY
!
integer, parameter :: SIZE_REAL = 4
integer, parameter :: SIZE_DOUBLE = 8
! set to SIZE_REAL to run in single precision
! set to SIZE_DOUBLE to run in double precision (increases memory size by 2)
integer, parameter :: CUSTOM_REAL = SIZE_REAL
!----------- parameters that can be changed by the user -----------
! set to .false. if running on a Beowulf-type machine with local disks
! set to .true. if running on a shared-memory machine with common file
system
! if running on a Beowulf, also modify name of nodes in
filter_machine_file.f90
logical, parameter :: LOCAL_PATH_IS_ALSO_GLOBAL = .false.
! apply heuristic rule to modify doubling regions to balance angles
logical, parameter :: APPLY_HEURISTIC_RULE = .true.
! input, output and main MPI I/O files
integer, parameter :: ISTANDARD_OUTPUT = 6
integer, parameter :: IIN = 40,IOUT = 41
! uncomment this to write messages to a text file
integer, parameter :: IMAIN = 42
! uncomment this to write messages to the screen
! integer, parameter :: IMAIN = ISTANDARD_OUTPUT
! I/O unit for source and receiver vtk file
integer, parameter :: IOVTK = 98
! I/O unit for interface file
integer, parameter :: IIN_INTERFACES = 43
! ignore variable name field (junk) at the beginning of each input line
logical, parameter :: IGNORE_JUNK = .true.,DONT_IGNORE_JUNK = .false.
! minimum thickness in meters to include the effect of the oceans
! to avoid taking into account spurious oscillations in topography model
! double precision, parameter :: MINIMUM_THICKNESS_3D_OCEANS = 10.d0
! min and max density in the model
! double precision, parameter :: DENSITY_MAX = 3000.d0
! double precision, parameter :: DENSITY_MIN = 2000.d0
! density of sea water
! real(kind=CUSTOM_REAL), parameter :: RHO_OCEANS = 1020.0
! depth at which we start to honor the basement interface
! double precision, parameter :: Z_THRESHOLD_HONOR_BASEMENT = -4700.d0
!
---------------------------------------------------------------------------------------
! LQY -- Following 3 variables stays here temporarily,
! we need to move them to Par_file at a proper time
!
---------------------------------------------------------------------------------------
! save moho mesh and compute Moho boundary kernels
! logical, parameter :: SAVE_MOHO_MESH = .false.
!------------------------------------------------------
!----------- do not modify anything below -------------
!------------------------------------------------------
! on some processors (e.g. Pentiums) it is necessary to suppress underflows
! by using a small initial field instead of zero
logical, parameter :: FIX_UNDERFLOW_PROBLEM = .true.
! some useful constants
double precision, parameter :: PI = 3.141592653589793d0
double precision, parameter :: TWO_PI = 2.d0 * PI
! 3-D simulation
integer, parameter :: NDIM = 3
! dimension of the boundaries of the slices
integer, parameter :: NDIM2D = 2
! number of nodes for 2D and 3D shape functions for hexahedra
! we use 8-node mesh bricks, which are more stable than 27-node elements
integer, parameter :: NGNOD = 8, NGNOD2D = 4
! a few useful constants
double precision, parameter :: ZERO = 0.d0,ONE = 1.d0,TWO = 2.d0,HALF =
0.5d0
real(kind=CUSTOM_REAL), parameter :: &
ONE_THIRD = 1._CUSTOM_REAL/3._CUSTOM_REAL, &
FOUR_THIRDS = 4._CUSTOM_REAL/3._CUSTOM_REAL
! very large and very small values
double precision, parameter :: HUGEVAL = 1.d+30,TINYVAL = 1.d-9
! very large real value declared independently of the machine
real(kind=CUSTOM_REAL), parameter :: HUGEVAL_SNGL = 1.e+30_CUSTOM_REAL
! very large integer value
integer, parameter :: HUGEINT = 100000000
! define flag for elements
integer, parameter :: IFLAG_ONE_LAYER_TOPOGRAPHY = 1
integer, parameter :: IFLAG_BASEMENT_TOPO = 2
integer, parameter :: IFLAG_16km_BASEMENT = 3
integer, parameter :: IFLAG_MOHO_16km = 4
integer, parameter :: IFLAG_HALFSPACE_MOHO = 5
! define flag for regions for attenuation
integer, parameter :: NUM_REGIONS_ATTENUATION = 13
integer, parameter :: IATTENUATION_SEDIMENTS_40 = 1
integer, parameter :: IATTENUATION_SEDIMENTS_50 = 2
integer, parameter :: IATTENUATION_SEDIMENTS_60 = 3
integer, parameter :: IATTENUATION_SEDIMENTS_70 = 4
integer, parameter :: IATTENUATION_SEDIMENTS_80 = 5
integer, parameter :: IATTENUATION_SEDIMENTS_90 = 6
integer, parameter :: IATTENUATION_SEDIMENTS_100 = 7
integer, parameter :: IATTENUATION_SEDIMENTS_110 = 8
integer, parameter :: IATTENUATION_SEDIMENTS_120 = 9
integer, parameter :: IATTENUATION_SEDIMENTS_130 = 10
integer, parameter :: IATTENUATION_SEDIMENTS_140 = 11
integer, parameter :: IATTENUATION_SEDIMENTS_150 = 12
integer, parameter :: IATTENUATION_BEDROCK = 13
! Olsen's constant for Q_mu = constant * v_s attenuation rule
real, parameter :: OLSEN_ATTENUATION_RATIO = 0.05
! number of standard linear solids in parallel for attenuation
integer, parameter :: N_SLS = 3
! flag for the four edges of each slice and for the bottom edge
integer, parameter :: XI_MIN = 1
integer, parameter :: XI_MAX = 2
integer, parameter :: ETA_MIN = 3
integer, parameter :: ETA_MAX = 4
integer, parameter :: BOTTOM = 5
! number of GLL points not set in the mesher, do not modify this value
integer, parameter :: NGLLX = 2
integer, parameter :: NGLLY = NGLLX
integer, parameter :: NGLLZ = NGLLX
! number of points per surface element
integer, parameter :: NGLLSQUARE = NGLLX * NGLLY
! number of points per spectral element
integer, parameter :: NGLLCUBE = NGLLX * NGLLY * NGLLZ
! for vectorization of loops
integer, parameter :: NGLLSQUARE_NDIM = NGLLSQUARE * NDIM
integer, parameter :: NGLLCUBE_NDIM = NGLLCUBE * NDIM
! flag for projection from latitude/longitude to UTM, and back
integer, parameter :: ILONGLAT2UTM = 0, IUTM2LONGLAT = 1
! smallest real number on the Pentium and the SGI = 1.1754944E-38
! largest real number on the Pentium and the SGI = 3.4028235E+38
! small negligible initial value to avoid very slow underflow trapping
! but not too small to avoid trapping on velocity and acceleration in
Newmark
real(kind=CUSTOM_REAL), parameter :: VERYSMALLVAL = 1.E-24_CUSTOM_REAL
! displacement threshold above which we consider the code became unstable
real(kind=CUSTOM_REAL), parameter :: STABILITY_THRESHOLD =
1.E+25_CUSTOM_REAL
! geometrical tolerance for boundary detection
double precision, parameter :: SMALLVAL = 0.00001d0
! do not use tags for MPI messages, use dummy tag instead
integer, parameter :: itag = 0,itag2 = 0
! for the Gauss-Lobatto-Legendre points and weights
double precision, parameter :: GAUSSALPHA = 0.d0,GAUSSBETA = 0.d0
! number of lines per source in CMTSOLUTION file
integer, parameter :: NLINES_PER_CMTSOLUTION_SOURCE = 13
! number of iterations to solve the system for xi and eta
integer, parameter :: NUM_ITER = 4
! size of topography and bathymetry file for Southern California
integer, parameter :: NX_TOPO_SOCAL = 6001,NY_TOPO_SOCAL = 4001
double precision, parameter :: ORIG_LAT_TOPO_SOCAL = 30.d0
double precision, parameter :: ORIG_LONG_TOPO_SOCAL = 102.d0
double precision, parameter :: DEGREES_PER_CELL_TOPO_SOCAL = 2.d0 /
4000.d0
character(len=100), parameter :: TOPO_FILE_SOCAL =
'DATA/la_topography/topo_bathy_final.dat'
! size of Lupei Zhu's Moho map file for Southern California
integer, parameter :: NX_MOHO = 71,NY_MOHO = 51
double precision, parameter :: ORIG_LAT_MOHO = 32.d0
double precision, parameter :: ORIG_LONG_MOHO = -121.d0
double precision, parameter :: DEGREES_PER_CELL_MOHO = 0.1d0
! size of basement map file
integer, parameter :: NX_BASEMENT = 161,NY_BASEMENT = 144
double precision, parameter :: ORIG_X_BASEMENT = 316000.
double precision, parameter :: ORIG_Y_BASEMENT = 3655000.
double precision, parameter :: SPACING_X_BASEMENT = 1000.
double precision, parameter :: SPACING_Y_BASEMENT = 1000.
!
! new Gocad Voxets Peter July 29, 2002 - high-res and medium-res blocks
!
! size of the medium-resolution Gocad voxet
integer, parameter :: NX_GOCAD_MR = 194, NY_GOCAD_MR = 196, NZ_GOCAD_MR =
100
double precision, parameter :: ORIG_X_GOCAD_MR = 283000.
double precision, parameter :: ORIG_Y_GOCAD_MR = 3655000.
double precision, parameter :: ORIG_Z_GOCAD_MR = -15000.
double precision, parameter :: SPACING_X_GOCAD_MR = 1000.
double precision, parameter :: SPACING_Y_GOCAD_MR = 1000.
double precision, parameter :: SPACING_Z_GOCAD_MR = 200.
! maximum size of model for tapering of transition between Hauksson and MR
double precision, parameter :: END_X_GOCAD_MR = ORIG_X_GOCAD_MR +
SPACING_X_GOCAD_MR * (NX_GOCAD_MR - 1)
double precision, parameter :: END_Y_GOCAD_MR = ORIG_Y_GOCAD_MR +
SPACING_Y_GOCAD_MR * (NY_GOCAD_MR - 1)
! size of the high-resolution Gocad voxet
integer, parameter :: NX_GOCAD_HR = 185, NY_GOCAD_HR = 196, NZ_GOCAD_HR =
100
double precision, parameter :: ORIG_X_GOCAD_HR = 371052.25
double precision, parameter :: ORIG_Y_GOCAD_HR = 3725250.
double precision, parameter :: ORIG_Z_GOCAD_HR = -9500.
double precision, parameter :: SPACING_X_GOCAD_HR = 250.
double precision, parameter :: SPACING_Y_GOCAD_HR = 250.
double precision, parameter :: SPACING_Z_GOCAD_HR = 100.
! maximum size of model for tapering of transition between HR and MR
double precision, parameter :: END_X_GOCAD_HR = ORIG_X_GOCAD_HR +
SPACING_X_GOCAD_HR * (NX_GOCAD_HR - 1)
double precision, parameter :: END_Y_GOCAD_HR = ORIG_Y_GOCAD_HR +
SPACING_Y_GOCAD_HR * (NY_GOCAD_HR - 1)
! implement smooth transition between Hauksson, HR and MR Gocad blocks
logical, parameter :: TAPER_GOCAD_TRANSITIONS = .true.
! Salton Sea Gocad voxet
integer, parameter :: GOCAD_ST_NU = 638, GOCAD_ST_NV = 219, GOCAD_ST_NW =
76
double precision, parameter :: GOCAD_ST_O_X = 720844.0, GOCAD_ST_O_Y =
3401799.250, &
GOCAD_ST_O_Z = -6354.334
double precision, parameter :: GOCAD_ST_U_X = -209197.89, GOCAD_ST_U_Y =
320741.71
double precision, parameter :: GOCAD_ST_V_X = 109670.74, GOCAD_ST_V_Y =
71530.72
double precision, parameter :: GOCAD_ST_W_Z = 7666.334
double precision, parameter :: GOCAD_ST_NO_DATA_VALUE = -99999
!
!--- larger Hauksson model for entire So-Cal, 15 km resolution
!
! number of non-constant layers
integer, parameter :: NLAYERS_HAUKSSON = 9
! depth of layers
double precision, parameter :: Z_HAUKSSON_LAYER_1 = -1000.d0
double precision, parameter :: Z_HAUKSSON_LAYER_2 = -4000.d0
double precision, parameter :: Z_HAUKSSON_LAYER_3 = -6000.d0
double precision, parameter :: Z_HAUKSSON_LAYER_4 = -10000.d0
double precision, parameter :: Z_HAUKSSON_LAYER_5 = -15000.d0
double precision, parameter :: Z_HAUKSSON_LAYER_6 = -17000.d0
double precision, parameter :: Z_HAUKSSON_LAYER_7 = -22000.d0
double precision, parameter :: Z_HAUKSSON_LAYER_8 = -31000.d0
double precision, parameter :: Z_HAUKSSON_LAYER_9 = -33000.d0
integer, parameter :: NGRID_NEW_HAUKSSON = 201
! corners of new Hauksson's interpolated grid
double precision, parameter :: UTM_X_ORIG_HAUKSSON = 122035.012d0
double precision, parameter :: UTM_X_END_HAUKSSON = 766968.628d0
double precision, parameter :: UTM_Y_ORIG_HAUKSSON = 3547232.986d0
double precision, parameter :: UTM_Y_END_HAUKSSON = 4098868.501d0
double precision, parameter :: SPACING_UTM_X_HAUKSSON =
(UTM_X_END_HAUKSSON - UTM_X_ORIG_HAUKSSON) / (NGRID_NEW_HAUKSSON-1.d0)
double precision, parameter :: SPACING_UTM_Y_HAUKSSON =
(UTM_Y_END_HAUKSSON - UTM_Y_ORIG_HAUKSSON) / (NGRID_NEW_HAUKSSON-1.d0)
! layers in the So-Cal regional model
! DEPTH_MOHO_SOCAL = -35 km was based on Dreger and Helmberger (1990)
! and is (July 2007) the preferred Moho depth for Dreger.
! The depth of 32 km is used in the standard processing (Wald et al., 1995)
! of SoCal events and is the value in the original Kanamori-Hadley (1975)
model.
double precision, parameter :: DEPTH_5p5km_SOCAL = -5500.d0
double precision, parameter :: DEPTH_16km_SOCAL = -16000.d0
double precision, parameter :: DEPTH_MOHO_SOCAL = -32000.d0
! reference surface of the model before adding topography
double precision, parameter :: Z_SURFACE = 0.d0
! number of points in each AVS or OpenDX quadrangular cell for movies
integer, parameter :: NGNOD2D_AVS_DX = 4
! magic ratio for heuristic rule
! this gives 120 degree angles in doubling
! standard value 0.5 gives 135-135-90, which is not optimal
double precision, parameter :: MAGIC_RATIO = 0.6056d0
! type of elements for heuristic rule
integer, parameter :: ITYPE_UNUSUAL_1 = 1
integer, parameter :: ITYPE_UNUSUAL_1p = 2
integer, parameter :: ITYPE_UNUSUAL_4 = 3
integer, parameter :: ITYPE_UNUSUAL_4p = 4
! define number of spectral elements and points in basic symmetric mesh
doubling superbrick
integer, parameter :: NSPEC_DOUBLING_SUPERBRICK = 32
integer, parameter :: NGLOB_DOUBLING_SUPERBRICK = 67
integer, parameter :: NSPEC_SUPERBRICK_1L = 28
integer, parameter :: NGLOB_SUPERBRICK_1L = 58
integer, parameter :: NGNOD_EIGHT_CORNERS = 8
-----------------------------------------------------------------------------------------------------------------------------------------
Par_file:
# coordinates of mesh block in latitude/longitude and depth in km
LATITUDE_MIN = 31.d0
LATITUDE_MAX = 31.2d0
LONGITUDE_MIN = 103.1d0
LONGITUDE_MAX = 103.6d0
DEPTH_BLOCK_KM = 30.d0
UTM_PROJECTION_ZONE = 48
SUPPRESS_UTM_PROJECTION = .false.
# file that contains the interfaces of the model / mesh
INTERFACES_FILE = interfaces.dat
# number of elements at the surface along edges of the mesh at the surface
# (must be 8 * multiple of NPROC below if mesh is not regular and contains
mesh doublings)
# (must be multiple of NPROC below if mesh is regular)
NEX_XI = 64
NEX_ETA = 64
# number of MPI processors along xi and eta (can be different)
NPROC_XI = 2
NPROC_ETA = 2
# Regular/irregular mesh
USE_REGULAR_MESH = .false.
# Only for irregular meshes, number of doubling layers (1 or 2) and their
position
NDOUBLINGS = 1
# NZ_DOUGLING_1 is the parameter to set up if there is only one doubling
layer
NZ_DOUGLING_1 = 11
NZ_DOUGLING_2 = 0
# create mesh files for visualisation or further checking
CREATE_ABAQUS_FILES = .false.
CREATE_DX_FILES = .false.
# anticipated time step for simulation (in order to check the stability
condition)
DT = 0.01
# path to store the databases files
LOCAL_PATH = OUTPUT_FILES
# number of materials
NMATERIALS = 4
# define the different materials in the model as :
# #material_id #rho #vp #vs #Q_flag #anisotropy_flag #domain_id
# Q_flag : 0=no attenuation/1=IATTENUATION_SEDIMENTS_40,
2=..., 13=IATTENUATION_BEDROCK
# anisotropy_flag : 0=no anisotropy/ 1,2,.. check with implementation
in aniso_model.f90
# domain_id : 1=acoustic / 2=elastic / 3=poroelastic
1 1200 1500 750 1 0 2
2 1100 1600 800 3 0 1
3 1000 1500 0 0 0 2
4 1300 1400 700 2 0 3
# number of regions
NREGIONS = 4
# define the different regions of the model as :
#NEX_XI_BEGIN #NEX_XI_END #NEX_ETA_BEGIN #NEX_ETA_END #NZ_BEGIN #NZ_END
#material_id
1 64 1 64 1
4 1
1 64 1 64 5
5 2
1 64 1 64 6
15 3
14 25 7 19 7
10 4
-------------------------------------------------------------------------------------------------------------------------------------------------
interfaces.dat:
# number of interfaces
2
#
# We describe each interface below, structured as a 2D-grid, with several
parameters :
# number of points along XI and ETA, minimal XI ETA coordinates
# and spacing between points which must be constant.
# Then the records contain the Z coordinates of the NXI x NETA points.
#
# interface number 1
# SUPPRESS_UTM_PROJECTION NXI NETA LONG_MIN LAT_MIN SPACING_XI
SPACING_ETA
.true. 161 144 316000.d0 3655000.d0 20000.d0
20000.d0
interface1.dat
# interface number 2 (topography, top of the mesh)
.false. 6001 4001 102.d0 30.d0 0.0005d0 0.0005d0
interface2.dat
#
# for each layer, we give the number of spectral elements in the vertical
direction
# layer number 1 (bottom layer)
11
# layer number 2 (top layer)
4
-----------------------------------------------------------------------------------------------------------------------------------------
Please help me to solve the error. Many thanks.
Best regards,
Hu Yuanxin
2010-7-17
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://geodynamics.org/pipermail/cig-seismo/attachments/20100717/29ccd5d6/attachment-0001.htm
More information about the CIG-SEISMO
mailing list