Tutorial 2: Regional simulation
This is a step-by-step tutorial how to create a mesh for a regional simulation and run a forward simulation.
The following instructions assume that you have installed SPECFEM3D_GLOBE and familiarized yourself with how you will run the package based on your computer configuration, as detailed in the SPECFEM3D_GLOBE User manual (Chapter 2 provides installation help). Please make sure you have the package installed on your system.
The example is distributed with the package under the EXAMPLES/ directory. However, you might need to edit these example scripts slightly to launch them on your system.
Regional simulation
This is a step-by-step tutorial how to create a mesh for a regional simulation and run a single forward simulation.
Setting up example folder for simulations
We will set up the example folder for simulation runs:
* databases directory: create a directory DATABASES_MPI/ into which you will put the mesh partitions:
[]$ cd SPECFEM3D_GLOBE/EXAMPLES/regional_MiddleEast/ [regional_MiddleEast]$ mkdir -p DATABASES_MPI
Make sure this folder is accessible by the compute nodes on your system. The example will search for this folder in the current example directory. Depending on your system setup, you may also create this as a symbolic link to a local scratch directory /scratch/DATABASES_MPI.
* parameter files: make sure you have the parameter files in a local directory DATA/ for the example:
[regional_MiddleEast]$ ls -1 DATA/ Par_file CMTSOLUTION STATIONS
these files should already be provided in the example folder.
* executables: Copy the above example parameter files to the DATA/ directory in the root directory before compiling. Then compile the executables in the root directory:
[]$ cd SPECFEM3D_GLOBE/ [SPECFEM3D_GLOBE]$ make
this assumes that you run the configuration beforehand by ./configure (see the manual for further configuration options). In case the compilation was successful, it will create the main executables xmeshfem3D and xspecfem3D in the bin/ directory
Note: the solver executable xspecfem3D uses static allocations which will depend upon the parameters set in the Par_file. According to those parameters, a source header file OUTPUT_FILES/values_from_mesher.h will be created during the compilation process by the executable xcreate_header_file. The solver only runs for those parameter setups.
create a local directory bin/ to link the executables from the root directory:
[]$ cd SPECFEM3D_GLOBE/EXAMPLES/regional_MiddleEast [regional_MiddleEast]$ mkdir -p bin [regional_MiddleEast]$ cd bin/ [bin]$ ln -s ../../bin/xmeshfem3D [bin]$ ln -s ../../bin/xspecfem3D
All these steps and the following mesher and solver runs are put in a process.sh bash script file in the example folder. You can simply run the script:
[regional_MiddleEast]$ ./process.sh
to do the setup and following steps for you. The script uses a PBS scheduler to submit the corresponding cluster job. Please modify and adapt the script to your needs.
Meshing
You will need to create the mesh databases before you can run the solver.
1. the example will run the mesher in parallel on 64 CPUs. A script go_mesher_solver_pbs.bash is provided in this example folder to do this via a PBS job scheduler. You may adapt also one of the cluster scripts provided in the SPECFEM3D_GLOBE/UTILS/Cluster/ directory for this purpose.
Note: This example script will submit one single job script which contains both the meshing run and the forward solver run together. This is especially useful in case your DATABASES_MPI folder is only locally accessible on the cluster nodes (e.g. local scratch disk space) and allows you to run both meshing and forward simulation without the need to store all the created mesh database files in a globally accessible repository.
2. make sure, you have the parameter file Par_file in the local directory DATA/. This example will use the following parameter setup for the regional simulation:
# forward or adjoint simulation SIMULATION_TYPE = 1 # 1 = forward, 2 = adjoint, 3 = both simultaneously NOISE_TOMOGRAPHY = 0 # 0 = earthquake simulation, 1/2/3 = three steps in noise simulation SAVE_FORWARD = .false. # number of chunks (1,2,3 or 6) NCHUNKS = 1 # angular width of the first chunk (not used if full sphere with six chunks) ANGULAR_WIDTH_XI_IN_DEGREES = 45.d0 # angular size of a chunk ANGULAR_WIDTH_ETA_IN_DEGREES = 40.d0 CENTER_LATITUDE_IN_DEGREES = 29.d0 CENTER_LONGITUDE_IN_DEGREES = 55.d0 GAMMA_ROTATION_AZIMUTH = 0.d0 # number of elements at the surface along the two sides of the first chunk # (must be multiple of 16 and 8 * multiple of NPROC below) NEX_XI = 128 NEX_ETA = 128 # number of MPI processors along the two sides of the first chunk NPROC_XI = 8 NPROC_ETA = 8 ... # 3D model MODEL = s29ea ... # path to store the local database files on each node LOCAL_PATH = ./DATABASES_MPI ...
The number of spectral elements in this single chunk in both XI and ETA directions is 128. To obtain the shortest period resolved in the seismograms, you would determine:
shortest period ~ (256 / NEX_XI ) x (ANGULAR_WIDTH_XI_IN_DEGREES / 90 ) x 17
thus for this example, they are accurate down to approximately 17 s periods.
Once the mesher completed, check the output file OUTPUT_FILES/output_mesher.txt to see if the databases were generated successfully. The output file contains various mesh information, including the time step size used for your forward simulation:
time-stepping of the solver will be: 0.142911725716261
Forward simulation
To run a forward simulation, do the following:
1. set an earthquake source by modifying the file CMTSOLUTION in the directory DATA/. this example will use a CMT solution for a shallow earthquake (12.8 km depth) which took place in Iran in December, 2003.
You can obtain more CMT solutions from the Global Centroid-Moment Tensor project catalog.
2. place your stations by modifying the file STATIONS in the directory{{{ DATA/}}}. this example will use a very small subset of seismic stations in international networks IRIS/IDA (II) and IRIS/USGS (IU).
You can obtain more station information from the International Federation of Digital Seismograph Networks (FDSN).
Please see the SPECFEM3D_GLOBE User manual for more detailed informations of how to specify the source and stations.
3. make sure, you have the parameter files in the directory DATA/. Most parameters in the Par_file should be set before running the mesher xmeshfem3D. The following parameters are important for this example forward simulation:
# forward or adjoint simulation SIMULATION_TYPE = 1 # 1 = forward, 2 = adjoint, 3 = both simultaneously NOISE_TOMOGRAPHY = 0 # 0 = earthquake simulation, 1/2/3 = three steps in noise simulation SAVE_FORWARD = .false. # number of chunks (1,2,3 or 6) NCHUNKS = 1 ... # number of MPI processors along the two sides of the first chunk NPROC_XI = 8 NPROC_ETA = 8 ... # 3D model MODEL = s29ea # parameters describing the Earth model OCEANS = .true. ELLIPTICITY = .true. TOPOGRAPHY = .true. GRAVITY = .true. ROTATION = .true. ATTENUATION = .true. # absorbing boundary conditions for a regional simulation ABSORBING_CONDITIONS = .true. # record length in minutes RECORD_LENGTH_IN_MINUTES = 15.0d0 ... # path to store the local database files on each node LOCAL_PATH = ./DATABASES_MPI ... # output format for the seismograms (one can use either or all of the three formats) OUTPUT_SEISMOS_ASCII_TEXT = .true. ...
this example will use a single job script go_mesher_solver_pbs.bash to run mesher and solver together. The solver should take about 15 minutes to finish the simulation.
check the output file output_solver.txt in the output directory OUTPUT_FILES/ to see if the forward simulation was successfully finishing. the seismograms for station GAR, located in Garm, Tajikistan, should look like this, using gnuplot:
gnuplot> plot "GAR.II.MXZ.sem.ascii" w l,"GAR.II.MXE.sem.ascii" w l,"GAR.II.MXN.sem.ascii" w l
