[cig-commits] r16374 - in seismo/3D/SPECFEM3D_GLOBE/tags: . v5.0.0 v5.0.0/USER_MANUAL v5.0.0/USER_MANUAL/figures
dkomati1 at geodynamics.org
dkomati1 at geodynamics.org
Wed Mar 3 12:09:04 PST 2010
Author: dkomati1
Date: 2010-03-03 12:09:03 -0800 (Wed, 03 Mar 2010)
New Revision: 16374
Added:
seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/
seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/INSTALL
seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/README_SPECFEM3D_GLOBE
seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/USER_MANUAL/figures/PKPdf_all_15s500s.pdf
seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/USER_MANUAL/figures/bolivia_trans.pdf
seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/USER_MANUAL/figures/bolivia_vertical.pdf
seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/USER_MANUAL/figures/vanuatu_trans.pdf
seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/USER_MANUAL/figures/vanuatu_vertical.pdf
seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/USER_MANUAL/manual_SPECFEM3D_GLOBE.pdf
seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/USER_MANUAL/manual_SPECFEM3D_GLOBE.tex
Removed:
seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/INSTALL
seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/README_SPECFEM3D_GLOBE
seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/USER_MANUAL/figures/PKPdf_all_15s500s.pdf
seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/USER_MANUAL/figures/bolivia_trans.pdf
seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/USER_MANUAL/figures/bolivia_vertical.pdf
seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/USER_MANUAL/figures/vanuatu_trans.pdf
seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/USER_MANUAL/figures/vanuatu_vertical.pdf
seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/USER_MANUAL/manual_SPECFEM3D_GLOBE.pdf
seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/USER_MANUAL/manual_SPECFEM3D_GLOBE.tex
Log:
added tags/v5.0.0
Copied: seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0 (from rev 16372, seismo/3D/SPECFEM3D_GLOBE/trunk)
Deleted: seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/INSTALL
===================================================================
--- seismo/3D/SPECFEM3D_GLOBE/trunk/INSTALL 2010-03-03 18:09:09 UTC (rev 16372)
+++ seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/INSTALL 2010-03-03 20:09:03 UTC (rev 16374)
@@ -1,191 +0,0 @@
-
-!=======================================================================!
-! !
-! specfem3D is a 3-D spectral-element solver for the Earth. !
-! It uses a mesh generated by program meshfem3D !
-! !
-!=======================================================================!
-
-Instructions on how to install and use SPECFEM3D_GLOBE are
-available in the manual located in directory USER_MANUAL.
-
-Main developers: Dimitri Komatitsch, Jeroen Tromp, Qinya Liu and David Michea.
-
-
-
-Basic installation
-------------------
-
-1. untar the SPECFEM3D_GLOBE package:
-
- > tar -zxvf SPECFEM3D_GLOBE*.tar.gz
-
- in the following, we will assume you set the root directory of the
- package name to:
-
- SPECFEM3D_GLOBE/
-
-2. configure the software for your system:
-
- > cd SPECFEM3D_GLOBE/
- > ./configure
-
- by default, it uses gfortran as a Fortran compiler, mpif90 for MPI compilation
- and gcc as a C compiler.
-
- in order to use a different compiler, use for example:
-
- > ./configure FC=ifort
-
- in this case, it would make use of
-
- - ifort, Intel Fortran Compiler
- (see http://software.intel.com/en-us/intel-compilers/ )
- with which we see very good performance results.
-
- - a default MPI installation which provides mpif90 as fortran wrapper command.
- An excellent package is provided by Open MPI (see http://www.open-mpi.org/).
-
- for the example above, you would make sure that the mpif90 command
- would also use ifort as a Fortran compiler.
-
- - a C compiler provided from the GNU compiler collection (see http://gcc.gnu.org/).
-
- in case `configure` was successful, it will create the files:
-
- ./Makefile
- ./constants.h
- ./precision.h
- ./DATA/Par_file
- ./DATA/CMTSOLUTION
- ./DATA/STATIONS
-
- please make sure, your installation is working and that
- the created files 'constant.h' and 'Makefile' satisfy
- your needs.
-
- more information is given in the manual provided in USER_MANUAL.
-
-3. compile the package:
-
- > make
-
-
-well done!
-
-
-
-
-Configure options
------------------
-
-'configure' accepts the following options:
-
-FC Fortran90 compiler command name,
- default is 'gfortran'
-
-MPIFC MPI Fortran90 command name,
- default is 'mpif90'
-
-CC C compiler command name,
- default is 'gcc'
-
-FLAGS_CHECK Compiler flags for non-critical subroutines,
- default is '-O3'
-
-FLAGS_NO_CHECK Compiler flags for creating fast, production-run code for
- critical subroutines,
- default is '-O3'
-
-LOCAL_PATH_IS_ALSO_GLOBAL Directory LOCAL_PATH (specified in DATA/Par_file) on the
- cluster can be accessed by all compute nodes ('true') or not ('false'),
- default is 'true'
-
-
-and flags:
-
-
---enable-double-precision The package can run either in single or in double precision.
- default is single precision
-
---help Directs configure to print a usage screen
-
-
-run with
- ./configure <option>=<my_option> --<flag>
-
-
-Compiler specific flags are set in file 'flags.guess' in order to provide
-suitable default options for different compilers.
-
-you might however want to modify the created 'Makefile' and specify your best
-system compiler flags in order to ensure optimal performance of the code.
-
-
-
-
-
-Troobleshooting
----------------
-
-FAQ
-
-1. configuration fails:
-
- Examine the log file 'config.log'. It contains detailed informations.
- in many cases, the path's to these specific compiler commands F90,
- CC and MPIF90 won't be correct if `configure` fails.
-
- please make sure that you have a working installation of a Fortran compiler,
- a C compiler and an MPI implementation. you should be able to compile this
- little program code:
-
- program main
- include 'mpif.h'
- integer, parameter :: CUSTOM_MPI_TYPE = MPI_REAL
- integer ier
- call MPI_INIT(ier)
- call MPI_BARRIER(MPI_COMM_WORLD,ier)
- call MPI_FINALIZE(ier)
- end
-
-
-2. compilation fails stating :
- ...
- In file ./compute_element_properties.f90:44
-
- use meshfem3D_models_par
- 1
- Fatal Error: File 'meshfem3d_models_par.mod' opened at (1) is not a GFORTRAN module file
- ...
-
- make sure, you're pointing to the right 'mpif90' wrapper command.
-
- normally, this message will appear when you're mixing two different fortran
- compilers. that is, using e.g. gfortran to compile non-MPI files
- and mpif90, wrapper provided for e.g. ifort, to compile MPI-files.
- the module will be created by the wrapper, thus ifort, while the gfortran compiler
- is trying to read that module for the compilation of 'compute_element_properties.f90'.
-
- fix: e.g. specify > ./configure FC=gfortran MPIF90=/usr/local/openmpi-gfortran/bin/mpif90
-
-
-3. compilation fails with:
- ...
- obj/specfem3D.o: In function `MAIN__':
- specfem3D.f90:(.text+0xb66): relocation truncated to fit: R_X86_64_32S against `.bss'
- ...
-
- you're probably using some resolution settings in 'DATA/Par_file' which are
- too big for a single processor on your system. the solver tries to statically allocate arrays,
- which can not be handled by a 32-bit address anymore in this case.
-
- fix: check the static memory needed by the solver, which is outputted
- when you run: > make xcreate_header_file
- > ./xcreate_header_file
-
- the size of static arrays per slice has to fit on to a single processor.
-
- you can either try to use e.g. -mcmodel=medium as additional compiler flag,
- or better, change your resolution settings for NPROC_XI,NPROC_ETA and NEX_XI,NEX_ETA.
-
Copied: seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/INSTALL (from rev 16373, seismo/3D/SPECFEM3D_GLOBE/trunk/INSTALL)
===================================================================
--- seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/INSTALL (rev 0)
+++ seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/INSTALL 2010-03-03 20:09:03 UTC (rev 16374)
@@ -0,0 +1,191 @@
+
+!=======================================================================!
+! !
+! specfem3D is a 3-D spectral-element solver for the Earth. !
+! It uses a mesh generated by program meshfem3D !
+! !
+!=======================================================================!
+
+Instructions on how to install and use SPECFEM3D_GLOBE are
+available in the manual located in directory USER_MANUAL.
+
+Main developers: Dimitri Komatitsch, Jeroen Tromp, Qinya Liu and David Michea.
+
+
+
+Basic installation
+------------------
+
+1. untar the SPECFEM3D_GLOBE package:
+
+ > tar -zxvf SPECFEM3D_GLOBE*.tar.gz
+
+ in the following, we will assume you set the root directory of the
+ package name to:
+
+ SPECFEM3D_GLOBE/
+
+2. configure the software for your system:
+
+ > cd SPECFEM3D_GLOBE/
+ > ./configure
+
+ by default, it uses gfortran as a Fortran compiler, mpif90 for MPI compilation
+ and gcc as a C compiler.
+
+ in order to use a different compiler, use for example:
+
+ > ./configure FC=ifort
+
+ in this case, it would make use of
+
+ - ifort, Intel Fortran Compiler
+ (see http://software.intel.com/en-us/intel-compilers/ )
+ with which we see very good performance results.
+
+ - a default MPI installation which provides mpif90 as fortran wrapper command.
+ An excellent package is provided by Open MPI (see http://www.open-mpi.org/).
+
+ for the example above, you would make sure that the mpif90 command
+ would also use ifort as a Fortran compiler.
+
+ - a C compiler provided from the GNU compiler collection (see http://gcc.gnu.org/).
+
+ in case `configure` was successful, it will create the files:
+
+ ./Makefile
+ ./constants.h
+ ./precision.h
+ ./DATA/Par_file
+ ./DATA/CMTSOLUTION
+ ./DATA/STATIONS
+
+ please make sure, your installation is working and that
+ the created files 'constant.h' and 'Makefile' satisfy
+ your needs.
+
+ more information is given in the manual provided in USER_MANUAL.
+
+3. compile the package:
+
+ > make
+
+
+well done!
+
+
+
+
+Configure options
+-----------------
+
+'configure' accepts the following options:
+
+FC Fortran90 compiler command name,
+ default is 'gfortran'
+
+MPIFC MPI Fortran90 command name,
+ default is 'mpif90'
+
+CC C compiler command name,
+ default is 'gcc'
+
+FLAGS_CHECK Compiler flags for non-critical subroutines,
+ default is '-O3'
+
+FLAGS_NO_CHECK Compiler flags for creating fast, production-run code for
+ critical subroutines,
+ default is '-O3'
+
+LOCAL_PATH_IS_ALSO_GLOBAL Directory LOCAL_PATH (specified in DATA/Par_file) on the
+ cluster can be accessed by all compute nodes ('true') or not ('false'),
+ default is 'true'
+
+
+and flags:
+
+
+--enable-double-precision The package can run either in single or in double precision.
+ default is single precision
+
+--help Directs configure to print a usage screen
+
+
+run with
+ ./configure <option>=<my_option> --<flag>
+
+
+Compiler specific flags are set in file 'flags.guess' in order to provide
+suitable default options for different compilers.
+
+you might however want to modify the created 'Makefile' and specify your best
+system compiler flags in order to ensure optimal performance of the code.
+
+
+
+
+
+Troobleshooting
+---------------
+
+FAQ
+
+1. configuration fails:
+
+ Examine the log file 'config.log'. It contains detailed informations.
+ in many cases, the path's to these specific compiler commands F90,
+ CC and MPIF90 won't be correct if `configure` fails.
+
+ please make sure that you have a working installation of a Fortran compiler,
+ a C compiler and an MPI implementation. you should be able to compile this
+ little program code:
+
+ program main
+ include 'mpif.h'
+ integer, parameter :: CUSTOM_MPI_TYPE = MPI_REAL
+ integer ier
+ call MPI_INIT(ier)
+ call MPI_BARRIER(MPI_COMM_WORLD,ier)
+ call MPI_FINALIZE(ier)
+ end
+
+
+2. compilation fails stating :
+ ...
+ In file ./compute_element_properties.f90:44
+
+ use meshfem3D_models_par
+ 1
+ Fatal Error: File 'meshfem3d_models_par.mod' opened at (1) is not a GFORTRAN module file
+ ...
+
+ make sure, you're pointing to the right 'mpif90' wrapper command.
+
+ normally, this message will appear when you're mixing two different fortran
+ compilers. that is, using e.g. gfortran to compile non-MPI files
+ and mpif90, wrapper provided for e.g. ifort, to compile MPI-files.
+ the module will be created by the wrapper, thus ifort, while the gfortran compiler
+ is trying to read that module for the compilation of 'compute_element_properties.f90'.
+
+ fix: e.g. specify > ./configure FC=gfortran MPIF90=/usr/local/openmpi-gfortran/bin/mpif90
+
+
+3. compilation fails with:
+ ...
+ obj/specfem3D.o: In function `MAIN__':
+ specfem3D.f90:(.text+0xb66): relocation truncated to fit: R_X86_64_32S against `.bss'
+ ...
+
+ you're probably using some resolution settings in 'DATA/Par_file' which are
+ too big for a single processor on your system. the solver tries to statically allocate arrays,
+ which can not be handled by a 32-bit address anymore in this case.
+
+ fix: check the static memory needed by the solver, which is outputted
+ when you run: > make xcreate_header_file
+ > ./xcreate_header_file
+
+ the size of static arrays per slice has to fit on to a single processor.
+
+ you can either try to use e.g. -mcmodel=medium as additional compiler flag,
+ or better, change your resolution settings for NPROC_XI,NPROC_ETA and NEX_XI,NEX_ETA.
+
Deleted: seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/README_SPECFEM3D_GLOBE
===================================================================
--- seismo/3D/SPECFEM3D_GLOBE/trunk/README_SPECFEM3D_GLOBE 2010-03-03 18:09:09 UTC (rev 16372)
+++ seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/README_SPECFEM3D_GLOBE 2010-03-03 20:09:03 UTC (rev 16374)
@@ -1,118 +0,0 @@
-
-!=======================================================================!
-! !
-! specfem3D is a 3-D spectral-element solver for the Earth. !
-! It uses a mesh generated by program meshfem3D !
-! !
-!=======================================================================!
-
-Instructions on how to install and use SPECFEM3D_GLOBE are
-available in the manual located in directory USER_MANUAL.
-
-Main developers: Dimitri Komatitsch, Jeroen Tromp, Qinya Liu and David Michea.
-
-
-Installation
-------------
-
-In order to compile the package on your system, please consult the
-manual located in directory USER_MANUAL and the file './INSTALL' within this
-package directory.
-
-
-Default simulation
-------------------
-
-A short step-by-step procedure to run a default simulation is described
-here.
-
-1. To set up a default simulation, you need to have the files:
-
- DATA/Par_file
- DATA/CMTSOLUTION
- DATA/STATIONS
-
- we will use the default simulation files '*.default' provided in the package:
- > cd SPECFEM3D_GLOBE/
- > cp DATA/Par_file.default DATA/Par_file
- > cp DATA/CMTSOLUTION.default DATA/CMTSOLUTION
- > cp DATA/STATIONS.default DATA/STATIONS
-
- The default simulation is an earthquake that occurred in
- Bolivia 1994 with a moment magnitude of ~6.8.
-
- The simulation will run over the entire globe and require a total of 150 CPUs.
- The seismogram resolution will be accurate down to ~18 seconds.
-
-
-2. compile the mesher:
-
- > make xmeshfem3D
-
-
-3. run the mesher on your cluster:
-
- > qsub go_mesher_pbs.bash
-
- - Different cluster scripts are provided as examples in the directory
- UTILS/Cluster. Locate your script and copy it the package
- root directory:
- e.g. > cp UTILS/Cluster/pbs/go_mesher_pbs.bash ./
-
- The example above uses a version of the Portable Batch System (PBS).
- Example scripts are also provided for Load Sharing Facility (LSF)
- and Sun Grid Engine (SGE). make sure, you modify this script to
- fit to your specific cluster setup.
-
- - please, before you submit your job, make sure to have created the directory specified as
- 'LOCAL_PATH' in your DATA/Par_file. by default the DATA/Par_file specifies:
- ..
- # path to store the local database files on each node
- LOCAL_PATH = ./DATABASES_MPI
- ..
-
- make sure this points to the right location for your cluster and that
- you created the directory before submitting your job:
- e.g. > mkdir ./DATABASES_MPI
-
- (at this point, you should also know if your directory for LOCAL_PATH
- is accessible by all compute nodes or not. set the flag
- LOCAL_PATH_IS_ALSO_GLOBAL in constants.h accordingly )
-
- after a successful run, the mesher will have created a perfectly, load-balanced
- hexahedral spectral-element mesh stored in this directory.
-
- note: for the default example, the mesher will take about 30 minutes
- to create the mesh.
- the file disk space needed for this global mesh is about ~51 GB.
-
-
-4. compile the solver:
-
- > make xspecfem3D
-
- re-compilation is needed since the solver tries to use static memory allocation
- and compiler optimizations that will work out much better once the details of
- your simulation mesh are known.
-
-
-5. run the solver on your cluster:
-
- > qsub go_solver_pbs.bash
-
- - Again, different cluster scripts are provided in the directory
- UTILS/Cluster. modify the example scripts to fit to your specific
- cluster setup.
-
- after a successful simulation, all seismograms are stored into the directory:
- OUTPUT_FILES/
-
- note: the default simulation will take about 1 h 40 m.
- the file disk space for all seismograms in OUTPUT_FILES/ is about ~77 MB.
-
-
-
-well done!
-
-
-
Copied: seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/README_SPECFEM3D_GLOBE (from rev 16373, seismo/3D/SPECFEM3D_GLOBE/trunk/README_SPECFEM3D_GLOBE)
===================================================================
--- seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/README_SPECFEM3D_GLOBE (rev 0)
+++ seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/README_SPECFEM3D_GLOBE 2010-03-03 20:09:03 UTC (rev 16374)
@@ -0,0 +1,118 @@
+
+!=======================================================================!
+! !
+! specfem3D is a 3-D spectral-element solver for the Earth. !
+! It uses a mesh generated by program meshfem3D !
+! !
+!=======================================================================!
+
+Instructions on how to install and use SPECFEM3D_GLOBE are
+available in the manual located in directory USER_MANUAL.
+
+Main developers: Dimitri Komatitsch, Jeroen Tromp, Qinya Liu and David Michea.
+
+
+Installation
+------------
+
+In order to compile the package on your system, please consult the
+manual located in directory USER_MANUAL and the file './INSTALL' within this
+package directory.
+
+
+Default simulation
+------------------
+
+A short step-by-step procedure to run a default simulation is described
+here.
+
+1. To set up a default simulation, you need to have the files:
+
+ DATA/Par_file
+ DATA/CMTSOLUTION
+ DATA/STATIONS
+
+ we will use the default simulation files '*.default' provided in the package:
+ > cd SPECFEM3D_GLOBE/
+ > cp DATA/Par_file.default DATA/Par_file
+ > cp DATA/CMTSOLUTION.default DATA/CMTSOLUTION
+ > cp DATA/STATIONS.default DATA/STATIONS
+
+ The default simulation is an earthquake that occurred in
+ Bolivia 1994 with a moment magnitude of ~6.8.
+
+ The simulation will run over the entire globe and require a total of 150 CPUs.
+ The seismogram resolution will be accurate down to ~18 seconds.
+
+
+2. compile the mesher:
+
+ > make xmeshfem3D
+
+
+3. run the mesher on your cluster:
+
+ > qsub go_mesher_pbs.bash
+
+ - Different cluster scripts are provided as examples in the directory
+ UTILS/Cluster. Locate your script and copy it the package
+ root directory:
+ e.g. > cp UTILS/Cluster/pbs/go_mesher_pbs.bash ./
+
+ The example above uses a version of the Portable Batch System (PBS).
+ Example scripts are also provided for Load Sharing Facility (LSF)
+ and Sun Grid Engine (SGE). make sure, you modify this script to
+ fit to your specific cluster setup.
+
+ - please, before you submit your job, make sure to have created the directory specified as
+ 'LOCAL_PATH' in your DATA/Par_file. by default the DATA/Par_file specifies:
+ ..
+ # path to store the local database files on each node
+ LOCAL_PATH = ./DATABASES_MPI
+ ..
+
+ make sure this points to the right location for your cluster and that
+ you created the directory before submitting your job:
+ e.g. > mkdir ./DATABASES_MPI
+
+ (at this point, you should also know if your directory for LOCAL_PATH
+ is accessible by all compute nodes or not. set the flag
+ LOCAL_PATH_IS_ALSO_GLOBAL in constants.h accordingly )
+
+ after a successful run, the mesher will have created a perfectly, load-balanced
+ hexahedral spectral-element mesh stored in this directory.
+
+ note: for the default example, the mesher will take about 30 minutes
+ to create the mesh.
+ the file disk space needed for this global mesh is about ~51 GB.
+
+
+4. compile the solver:
+
+ > make xspecfem3D
+
+ re-compilation is needed since the solver tries to use static memory allocation
+ and compiler optimizations that will work out much better once the details of
+ your simulation mesh are known.
+
+
+5. run the solver on your cluster:
+
+ > qsub go_solver_pbs.bash
+
+ - Again, different cluster scripts are provided in the directory
+ UTILS/Cluster. modify the example scripts to fit to your specific
+ cluster setup.
+
+ after a successful simulation, all seismograms are stored into the directory:
+ OUTPUT_FILES/
+
+ note: the default simulation will take about 1 h 40 m.
+ the file disk space for all seismograms in OUTPUT_FILES/ is about ~77 MB.
+
+
+
+well done!
+
+
+
Deleted: seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/USER_MANUAL/figures/PKPdf_all_15s500s.pdf
===================================================================
(Binary files differ)
Copied: seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/USER_MANUAL/figures/PKPdf_all_15s500s.pdf (from rev 16373, seismo/3D/SPECFEM3D_GLOBE/trunk/USER_MANUAL/figures/PKPdf_all_15s500s.pdf)
===================================================================
(Binary files differ)
Deleted: seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/USER_MANUAL/figures/bolivia_trans.pdf
===================================================================
(Binary files differ)
Copied: seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/USER_MANUAL/figures/bolivia_trans.pdf (from rev 16373, seismo/3D/SPECFEM3D_GLOBE/trunk/USER_MANUAL/figures/bolivia_trans.pdf)
===================================================================
(Binary files differ)
Deleted: seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/USER_MANUAL/figures/bolivia_vertical.pdf
===================================================================
(Binary files differ)
Copied: seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/USER_MANUAL/figures/bolivia_vertical.pdf (from rev 16373, seismo/3D/SPECFEM3D_GLOBE/trunk/USER_MANUAL/figures/bolivia_vertical.pdf)
===================================================================
(Binary files differ)
Deleted: seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/USER_MANUAL/figures/vanuatu_trans.pdf
===================================================================
(Binary files differ)
Copied: seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/USER_MANUAL/figures/vanuatu_trans.pdf (from rev 16373, seismo/3D/SPECFEM3D_GLOBE/trunk/USER_MANUAL/figures/vanuatu_trans.pdf)
===================================================================
(Binary files differ)
Deleted: seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/USER_MANUAL/figures/vanuatu_vertical.pdf
===================================================================
(Binary files differ)
Copied: seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/USER_MANUAL/figures/vanuatu_vertical.pdf (from rev 16373, seismo/3D/SPECFEM3D_GLOBE/trunk/USER_MANUAL/figures/vanuatu_vertical.pdf)
===================================================================
(Binary files differ)
Deleted: seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/USER_MANUAL/manual_SPECFEM3D_GLOBE.pdf
===================================================================
(Binary files differ)
Copied: seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/USER_MANUAL/manual_SPECFEM3D_GLOBE.pdf (from rev 16373, seismo/3D/SPECFEM3D_GLOBE/trunk/USER_MANUAL/manual_SPECFEM3D_GLOBE.pdf)
===================================================================
(Binary files differ)
Deleted: seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/USER_MANUAL/manual_SPECFEM3D_GLOBE.tex
===================================================================
--- seismo/3D/SPECFEM3D_GLOBE/trunk/USER_MANUAL/manual_SPECFEM3D_GLOBE.tex 2010-03-03 18:09:09 UTC (rev 16372)
+++ seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/USER_MANUAL/manual_SPECFEM3D_GLOBE.tex 2010-03-03 20:09:03 UTC (rev 16374)
@@ -1,3300 +0,0 @@
-%% LyX 1.5.1 created this file. For more info, see www.lyx.org/.
-
-\documentclass[oneside,english]{book}
-\usepackage[T1]{fontenc}
-\usepackage[latin1]{inputenc}
-\usepackage{geometry}
-\geometry{verbose,letterpaper,tmargin=1in,bmargin=1in,lmargin=1in,rmargin=1in}
-\setcounter{secnumdepth}{3}
-\setcounter{tocdepth}{3}
-\usepackage{longtable}
-\usepackage{varioref}
-\usepackage{float}
-\usepackage{textcomp}
-\usepackage{amsmath}
-\usepackage{color}
-
-% figures
-\usepackage[pdftex]{graphicx}
-
-% we are running pdflatex, so convert .eps files to .pdf
-\usepackage{epstopdf}
-
-\usepackage{amssymb}
-\IfFileExists{url.sty}{\usepackage{url}}
- {\newcommand{\url}{\texttt}}
-\usepackage[authoryear]{natbib}
-
-\makeatletter
-
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% LyX specific LaTeX commands.
-%% Because html converters don't know tabularnewline
-\providecommand{\tabularnewline}{\\}
-
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Textclass specific LaTeX commands.
-\newenvironment{lyxcode}
-{\begin{list}{}{
-\setlength{\rightmargin}{\leftmargin}
-\setlength{\listparindent}{0pt}% needed for AMS classes
-\raggedright
-\setlength{\itemsep}{0pt}
-\setlength{\parsep}{0pt}
-\normalfont\ttfamily}%
- \item[]}
-{\end{list}}
-
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% User specified LaTeX commands.
-% fonts
-\usepackage{times}
-\usepackage{natbib}
-
-% date of last edit
-\date{\today}
-
-% create thumbnails
-\usepackage[pdftex]{thumbpdf}
-
-% hyperlinks to sections and references
-\usepackage[pdftex,bookmarks=true,bookmarksnumbered=true,pdfpagemode=None,pdfstartview=FitH,pdfpagelayout=SinglePage,pdfborder={0 0 0}]{hyperref}
-
-\let\myUrl\url
-\renewcommand{\url}[1]{(\myUrl{#1})}
-
-% biblio GJI
-\bibliographystyle{abbrvnat}
-
-\newcommand{\toall}[1]{\textbf{*** All: #1 ***}}
-\newcommand{\tojeroen}[1]{\textbf{*** Jeroen: #1 ***}}
-\newcommand{\tobrian}[1]{\textbf{*** Brian: #1 ***}}
-\newcommand{\tovala}[1]{\textbf{*** Vala: #1 ***}}
-\newcommand{\tovalabrian}[1]{\textbf{*** Vala \& Brian: #1 ***}}
-\newcommand{\tovalaqinya}[1]{\textbf{*** Vala \& Qinya: #1 ***}}
-\newcommand{\toqinya}[1]{\textbf{*** Qinya: #1 ***}}
-\newcommand{\tomin}[1]{\textbf{*** Min: #1 ***}}
-\newcommand{\toalessia}[1]{\textbf{*** Alessia: #1 ***}}
-\newcommand{\todimitri}[1]{\textbf{*** Dimitri: #1 ***}}
-
-\newcommand{\nexxi}{\mbox{\texttt{NEX\_XI}}}
-\newcommand{\nexeta}{\mbox{\texttt{NEX\_ETA}}}
-\newcommand{\nprocxi}{\mbox{\texttt{NPROC\_XI}}}
-\newcommand{\nproceta}{\mbox{\texttt{NPROC\_ETA}}}
-\newcommand{\nchunks}{\mbox{\texttt{NCHUNKS}}}
-
-\usepackage{babel}
-\makeatother
-
-\begin{document}
-\begin{center}
-\thispagestyle{empty}\textbf{}%
-\begin{figure}[H]
-\noindent \includegraphics[width=0.75\paperwidth]{figures/specfem_3d_globe-cover.pdf}
-\end{figure}
-
-\par\end{center}
-
-
-\title{\thispagestyle{empty}\textbf{SPECFEM3D\_GLOBE}\\
-\textbf{User Manual}}
-
-
-\author{$\copyright$ Princeton University / California Institute of Technology (USA) and\\
-University of Pau / CNRS / INRIA (France)\\
-Version 5.0.0 `Tiger'}
-
-
-\date{\noindent \today}
-
-\maketitle
-\tableofcontents{}
-
-
-\chapter{Introduction}
-
-The software package SPECFEM3D\_GLOBE simulates three-dimensional
-global and regional seismic wave propagation based upon the spectral-element
-method (SEM). Effects due to lateral variations in compressional-wave
-speed, shear-wave speed, density, a 3D crustal model, ellipticity,
-topography and bathymetry, the oceans, rotation, and self-gravitation
-are included. For a detailed introduction to the SEM as applied to
-global and regional seismic wave propagation, please consult \citet{KoVi98,KoTr99,Ch00,KoTr02a,KoTr02b,KoRiTr02,ChCaVi03,CaChViMo03,ChVa04,ChKoViCaVaFe07,TrKoLi08}.
-If you use 3D mantle model S20RTS, please cite \citet{RiVaWo99}.
-The package can accommodate full 21-parameter anisotropy \citep{ChTr07}
-as well as lateral variations in attenuation \citep{SavWiTr05}. Adjoint
-capabilities and finite-frequency kernel simulations are also included
-\citep{LiTr06,TrKoLi08,LiTr08}.
-
-All SPECFEM3D\_GLOBE software is written in Fortran90 with full portability
-in mind, and conforms strictly to the Fortran95 standard. It uses
-no obsolete or obsolescent features of Fortran77. The package uses
-parallel programming based upon the Message Passing Interface (MPI)
-\citep{GrLuSk94,Pac97}.
-
-SPECFEM3D won the Gordon Bell award for best performance at the SuperComputing~2003
-conference in Phoenix, Arizona (USA) by running at 5 teraflops (sustained)
-on 1944 processors of the Japanese Earth Simulator using 14.6 billion
-degrees of freedom stored in 2.5 terabytes of memory; see \cite{KoTsChTr03}
-and the Gordon Bell Awards News Release \url{www.sc-conference.org/sc2003/nr_finalaward.html }
-for details. It was a finalist again in 2008 for a run at 0.16 petaflops (sustained) on 149,784 processors of the `Jaguar' Cray XT5 system at Oak Ridge National Laboratories (USA) \citep{CaKoLaTiMiLeSnTr08}.
-
-Future improvements will include support for GPU graphics card acceleration \citep{KoMiEr09,KoErGoMi10,MiKo10}
-as well as Convolutional-Perfectly Matched absorbing Layers (C-PML) \citep{KoMa07,MaKoEz08,MaKo09,MaKoGeBr10}.
-
-\section{Citation}
-
-If you use SPECFEM3D\_GLOBE for your own research, please cite at
-least one of the following articles: \cite{TrKoLi08,ChKoViCaVaFe07,KoRiTr02,KoTr02a,KoTr02b,KoTr99}
-or \cite{KoVi98}. The corresponding Bib\TeX{} entries may be found
-in file \texttt{USER\_MANUAL/bibliography.bib} or in comments at the
-beginning of file \texttt{specfem3D.f90}.
-
-
-\section{Support}
-
-This material is based upon work supported by the U.S. National Science
-Foundation under Grants No. EAR-0406751 and EAR-0711177, by the French
-CNRS, French INRIA Sud-Ouest MAGIQUE-3D, French ANR NUMASIS under
-Grant No. ANR-05-CIGC-002, and European FP6 Marie Curie International
-Reintegration Grant No. MIRG-CT-2005-017461.
-Older versions of the code were initially developed by Dimitri Komatitsch at Institut de Physique du Globe (France)
-and then by Dimitri Komatitsch and Jeroen Tromp at Harvard University (USA).
-Any opinions, findings, and conclusions or recommendations expressed in this material are
-those of the authors and do not necessarily reflect the views of the
-U.S. National Science Foundation, CNRS, INRIA, ANR or the European
-Marie Curie program.
-
-
-\chapter{\label{cha:Getting-Started}Getting Started}
-
-The SPECFEM3D\_GLOBE software package comes in a gzipped tar ball.
-In the directory in which you want to install the package, type
-
-\begin{lyxcode}
-tar~-zxvf~SPECFEM3D\_GLOBE\_V4.0.3.tar.gz
-\end{lyxcode}
-The directory \texttt{SPECFEM3D\_GLOBE\_V4.0.3} will then contain
-the source code.
-
-To configure the software for your system, run the \texttt{configure}
-shell script. This script will attempt to guess the appropriate configuration
-values for your system. However, at a minimum, it is recommended that
-you explicitly specify the appropriate command names for your Fortran90
-compiler and MPI package:
-
-\begin{lyxcode}
-./configure~FC=ifort~MPIFC=mpif90
-\end{lyxcode}
-A summary of the most important configuration variables follows.
-
-\begin{description}
-\item [{\texttt{FC}}] Fortran90 compiler command name. By default, \texttt{configure}
-will execute the command names of various well-known Fortran compilers
-in succession, picking the first one it finds that works.
-\item [{\texttt{MPIFC}}] MPI Fortran90 command name. The default is \texttt{mpif90}.
-This must correspond to the same underlying compiler specified by
-\texttt{FC}; otherwise, you will encounter compilation or link errors
-when you attempt to build the code. If you are unsure about this,
-it is usually safe to set both \texttt{FC} and \texttt{MPIFC} to the
-MPI compiler command for your system:
-\end{description}
-\begin{lyxcode}
-./configure~FC=mpif90~MPIFC=mpif90
-\end{lyxcode}
-\begin{description}
-\item [{\texttt{FLAGS\_CHECK}}] Compiler flags for non-critical subroutines.
-\item [{\texttt{FLAGS\_NO\_CHECK}}] Compiler flags for creating fast, production-run
-code for critical subroutines.
-\item [{\texttt{LOCAL\_PATH\_IS\_ALSO\_GLOBAL}}] Set to \texttt{.false.}
-on most cluster applications. For reasons of speed, the (parallel)
-mesher typically writes a (parallel) database for the solver on the
-local disks of the compute nodes. Some systems have no local disks
-(e.g., BlueGene or the Earth Simulator) and other systems have a fast
-parallel file system, in which case this variable should be set to
-\texttt{.true.}. Note that this flag is not used by the mesher or
-the solver; it is only used for some of the post-processing.
-\end{description}
-In addition to reading configuration variables, \texttt{configure}
-accepts the following options:
-
-\begin{description}
-\item [{\texttt{-{}-enable-double-precision}}] The package can run either
-in single or in double precision. The default is single precision
-mode because this requires exactly half as much memory. To specify
-double precision mode, simply provide \texttt{-{}-enable-double-precision}
-as a command-line argument to \texttt{configure}. On a new system,
-it is definitely worth experimenting with single versus double precision
-simulations to determine which is faster. Note that on many current
-processors (e.g., Intel, AMD, IBM Power), single precision calculations
-are often significantly faster; the difference can typically be 10\%
-to 25\%. It is therefore worth trying single precision if you can.
-We recommend running the same calculation once in single precision
-and in double precision on your system and comparing the seismograms.
-If they are identical, you should probably select single precision
-for your future runs.
-\item [{\texttt{-{}-help}}] Directs \texttt{configure} to print a usage
-screen which provides a short description of all configuration variables
-and options. Note that the options relating to installation directories
-(e.g., \texttt{-{}-prefix}) do not apply to SPECFEM3D\_GLOBE.
-\end{description}
-The \texttt{configure} script runs a brief series of checks. Upon
-successful completion, it generates the files \texttt{Makefile}, \texttt{constants.h},
-and \texttt{precision.h} in the working directory.
-
-\begin{description}
-\item [{Note:}] If the \texttt{configure} script fails, and you don't know
-what went wrong, examine the log file \texttt{config.log}. This file
-contains a detailed transcript of all the checks \texttt{configure}
-performed. Most importantly, it includes the error output (if any)
-from your compiler.
-\end{description}
-The \texttt{configure} script automatically runs the script \texttt{flags.guess}.
-This helper script contains a number of suggested flags for various
-compilers; e.g., Portland, Intel, Absoft, NAG, Lahey, NEC, IBM and
-SGI. The software has run on a wide variety of compute platforms,
-e.g., various PC clusters and machines from Sun, SGI, IBM, Compaq,
-and NEC. The \texttt{flags.guess} script attempts to guess which compiler
-you are using (based upon the compiler command name) and choose the
-related optimization flags. The \texttt{configure} script then automatically
-inserts the suggested flags into \texttt{Makefile}. Note that \texttt{flags.guess}
-may fail to identify your compiler; and in any event, the default
-flags chosen by \texttt{flags.guess} are undoubtedly not optimal for
-your system. So, we encourage you to experiment with these flags (by
-editing the generated \texttt{Makefile} by hand) and to solicit advice
-from your system administrator. Selecting the right compiler and compiler
-flags can make a tremendous difference in terms of performance. We
-welcome feedback on your experience with various compilers and flags.
-
-On SGI systems, \texttt{flags.guess} automatically informs \texttt{configure}
-to insert `\texttt{`TRAP\_FPE=OFF}'' into the generated \texttt{Makefile}
-in order to turn underflow trapping off.
-
-Finally, before compiling, make sure that the subdirectories \texttt{obj},
-\texttt{bak} and \texttt{OUTPUT\_FILES} exist within the directory
-with the source code (\texttt{SPECFEM3D\_GLOBE\_V4.0.3}). The \texttt{go\_mesher}
-script discussed below automatically takes care of creating the \texttt{OUTPUT\_FILES}
-directory.
-
-Note that if you run very large meshes on a relatively small number
-of processors, the memory size needed on each processor might become
-greater than 2 gigabytes, which is the upper limit for 32-bit addressing;
-in this case, on some compilers you may need to add \texttt{``-mcmodel=medium}''
-to the compiler options otherwise the compiler will display an error
-message.
-
-
-\chapter{\label{cha:Running-the-Mesher}Running the Mesher \texttt{xmeshfem3D}}
-
-You are now ready to compile the mesher. In the directory with the
-source code, type `\texttt{make meshfem3D}'. If all paths and flags
-have been set correctly, the mesher should now compile and produce
-the executable \texttt{xmeshfem3D}.
-
-Input for the mesher (and the solver) is provided through the parameter
-file \texttt{Par\_file}, which resides in the subdirectory \texttt{DATA}.
-Before running the mesher, a number of parameters need to be set in
-the \texttt{Par\_file}. This requires a basic understanding of how
-the SEM is implemented, and we encourage you to read \citet{KoVi98,KoTr99,Ch00,KoTr02a,KoTr02b,KoRiTr02,ChCaVi03,CaChViMo03}
-and \citet{ChVa04}. A detailed theoretical analysis of the dispersion
-and stability properties of the SEM is available in \citet{DeSe07}
-and \citet{SeOl07}.
-
-In this chapter we will focus on simulations at the scale of the entire
-globe. Regional simulations will be addressed in Chapter~\ref{cha:Regional-Simulations}.
-The spectral-element mesh for a SPECFEM3D\_GLOBE simulation is based
-upon a mapping from the cube to the sphere called the \textit{cubed
-sphere} \citep{Sad72,RoIaPa96}. This cubed-sphere mapping breaks
-the globe into 6~chunks, each of which is further subdivided in terms
-of $n^{2}$ mesh slices, where $n\ge1$ is a positive integer, for
-a total of $6\times n^{2}$ slices (Figure~\ref{figure:mpi_slices}).
-Thus the minimum number of processors required for a global simulation
-is 6 (although it is theoretically possible to run more than one slice
-per processor). %
-\begin{figure}
-\centerline{ \begin{tabular}{cc}
-\includegraphics[width=0.45\textwidth]{figures/mpi_slices} & \includegraphics[width=0.45\textwidth]{figures/fullmesh_18} \tabularnewline
-\end{tabular}}
-
-\caption{Each of the 6~chunks that constitutes the cubed sphere is subdivided
-in terms of $n^{2}$~slices of elements, where $n\ge1$ is a positive
-integer, for a total of $6\times n^{2}$ slices (and therefore processors).
-The figure on the left shows a mesh that is divided in terms of $6\times5^{2}=150$
-slices as indicated by the various colors. In this cartoon, each slice
-contains $5\times5=25$ spectral elements at the Earth's surface.
-The figure on the right shows a mesh that is divided over $6\times18^{2}=1944$
-processors as indicated by the various colors. Regional simulations
-can be accommodated by using only 1, 2 or 3 chunks of the cubed sphere.
-One-chunk simulations may involve a mesh with lateral dimensions smaller
-than~$90^{\circ}$, thereby accommodating smaller-scale simulations. }
-
-
-\label{figure:mpi_slices}
-\end{figure}
-
-
-To run the mesher for a global simulation, the following parameters
-need to be set in the \texttt{Par\_file}:
-
-\begin{description}
-\item [{\texttt{SIMULATION\_TYPE}}] is set to 1 for forward simulations,
-2 for adjoint simulations (see Section \ref{sec:Adjoint-simulation-finite})
-and 3 for kernel simulations (see Section \ref{sec:Finite-Frequency-Kernels}).
-\item [{\texttt{SAVE\_FORWARD}}] is only set to \texttt{.true.} for a forward
-simulation with the last frame of the simulation saved, as part of
-the finite-frequency kernel calculations (see Section \ref{sec:Finite-Frequency-Kernels}).
-For a regular forward simulation, leave \texttt{SIMULATION\_TYPE}
-and \texttt{SAVE\_FORWARD} at their default values.
-\item [{\texttt{NCHUNKS}}] must be set to 6 for global simulations.
-\item [{\texttt{ANGULAR\_WIDTH\_XI\_IN\_DEGREES}}] Not needed for a global
-simulation. (See Chapter~\ref{cha:Regional-Simulations} for regional
-simulations.)
-\item [{\texttt{ANGULAR\_WIDTH\_ETA\_IN\_DEGREES}}] Not needed for a global
-simulation. (See Chapter~\ref{cha:Regional-Simulations} for regional
-simulations.)
-\item [{\texttt{CENTER\_LATITUDE\_IN\_DEGREES}}] Not needed for a global
-simulation. (See Chapter~\ref{cha:Regional-Simulations} for regional
-simulations.)
-\item [{\texttt{CENTER\_LONGITUDE\_IN\_DEGREES}}] Not needed for a global
-simulation. (See Chapter~\ref{cha:Regional-Simulations} for regional
-simulations.)
-\item [{\texttt{GAMMA\_ROTATION\_AZIMUTH}}] Not needed for a global simulation.
-(See Chapter~\ref{cha:Regional-Simulations} for regional simulations.)
-\item [{$\nexxi$}] The number of spectral elements along one side of a
-chunk in the cubed sphere (see Figure~\ref{figure:mpi_slices});
-this number \textit{must} be a multiple of 16 and 8~$\times$~a
-multiple of $\nprocxi$ defined below. We do not recommend using $\nexxi$
-less than 64 because the curvature of the Earth cannot be honored
-if one uses too few elements, and distorted elements can lead to inaccurate
-and unstable simulations, i.e., smaller values of $\nexxi$ are likely
-to result in spectral elements with a negative Jacobian, in which
-case the mesher will exit with an error message. Table~\ref{table:nex}
-summarizes various suitable choices for $\nexxi$ and the related
-values of $\nprocxi$. Based upon benchmarks against semi-analytical
-normal-mode synthetic seismograms, \citet{KoTr02a,KoTr02b} determined
-that a $\nexxi=256$ run is accurate to a shortest period of roughly
-17~s. Therefore, since accuracy is determined by the number of grid
-points per shortest wavelength, for any particular value of $\nexxi$
-the simulation will be accurate to a shortest period determined approximately
-by \begin{equation}
-\mbox{shortest period (s)}\simeq(256/\nexxi)\times17.\label{eq:shortest_period}\end{equation}
- The number of grid points in each orthogonal direction of the reference
-element, i.e., the number of Gauss-Lobatto-Legendre points, is determined
-by \texttt{NGLLX} in the \texttt{constants.h} file. In the globe we
-use $\mbox{\texttt{NGLLX}}=5$, for a total of $5^{3}=125$ points
-per elements. We suggest not to change this value.
-\item [{$\nexeta$}] For global simulations $\nexeta$ must be set to the
-same value as $\nexxi$.
-\item [{$\nprocxi$}] The number of processors or slices along one chunk
-of the cubed sphere (see Figure~\ref{figure:mpi_slices}); we must
-have $\nexxi=8\times c\times\nprocxi$, where $c\ge1$ is a positive
-integer. See Table~\ref{table:nex} for various suitable choices.
-\item [{$\nproceta$}] For global simulations $\nproceta$ must be set
-to the same value as $\nprocxi$.
-\item [{\texttt{MODEL}}] Must be set to one of the following:
-\item [{\textmd{1D~models~with~real~structure:}}]~
-
-\begin{description}
-\item [{\texttt{1D\_isotropic\_prem}}] Isotropic version of the spherically
-symmetric Preliminary Reference Earth Model (PREM) \citep{DzAn81}.
-\item [{\texttt{1D\_transversely\_isotropic\_prem}}] Transversely isotropic
-version of PREM.
-\item [{\texttt{1D\_iasp91}}] Spherically symmetric isotropic IASP91 model
-\citep{KeEn91}.
-\item [{\texttt{1D\_1066a}}] Spherically symmetric earth model 1066A \citep{gilbertdziewonski1975}.
-When \texttt{\small ATTENTUATION} is on, it uses an unpublished 1D
-attenuation model from Scripps.
-\item [{\texttt{1D\_ak135}}] Spherically symmetric isotropic AK135 model
-\citep{KeEnBu95}.
-\item [{\texttt{1}D\_ref}] A recent 1D Earth model developed by \citet{KuDzEk06}.
-This model is the 1D background model for the 3D models s362ani, s362wmani,
-s362ani\_prem, and s29ea.
-\end{description}
-\end{description}
-For historical reasons and to provide benchmarks against normal-mode
-synthetics, the mesher accommodates versions of various 1D models
-with a single crustal layer with the properties of the original upper
-crust. These `one-crust' models are:
-
-\texttt{1D\_isotropic\_prem\_onecrust}
-
-\texttt{1D\_transversely\_isotropic\_prem\_onecrust}
-
-\texttt{1D\_iasp91\_onecrust}, \texttt{1D\_1066a\_onecrust}
-
-\texttt{1D\_ak135\_onecrust}
-
-\begin{description}
-\item [{\textmd{Fully~3D~models:}}]~
-
-\begin{description}
-\item [{\texttt{transversely\_isotropic\_prem\_plus\_3D\_crust\_2.0}}] This
-model has CRUST2.0 \citep{BaLaMa00} on top of a transversely isotropic
-PREM. We first extrapolate PREM mantle velocity up to the surface,
-then overwrite the model with CRUST2.0
-\item [{\texttt{s20rts}}] By default, the code uses 3D mantle model S20RTS
-\citep{RiVaWo99} and 3D crustal model Crust2.0 \citep{BaLaMa00}.
-Note that S20RTS uses transversely isotropic PREM as a background
-model, and that we use the PREM radial attenuation model when \texttt{ATTENUATION}
-is incorporated. See Chapter~\ref{cha:-Changing-the} for a discussion
-on how to change 3D models.
-\item [{\texttt{\textcolor{black}{s362ani}}}] A global shear-wave speed
-model developed by \citet{KuDzEk06}. In this model, radial anisotropy
-is confined to the uppermost mantle. The model (and the corresponding
-mesh) incorporate tomography on the 650\textasciitilde{}km and 410\textasciitilde{}km
-discontinuities in the 1D reference model REF.
-\item [{\texttt{\textcolor{black}{s362wmani}}}] A version of S362ANI with
-anisotropy allowed throughout the mantle.
-\item [{\texttt{\textcolor{black}{s362ani\_prem}}}] A version of S362ANI
-calculated using PREM as the 1D reference model.
-\item [{\texttt{\textcolor{black}{s29ea}}}] A global model with higher
-resolution in the upper mantle beneath Eurasia calculated using REF
-as the 1D reference model.
-\item [{\texttt{3D\_anisotropic}}] See Chapter~\ref{cha:-Changing-the}
-for a discussion on how to specify your own 3D anisotropic model.
-\item [{\texttt{3D\_attenuation}}] See Chapter~\ref{cha:-Changing-the}
-for a discussion on how to specify your own 3D attenuation model.
-\end{description}
-\item [{\textmd{NOTE:}}]~\\
-When a 3D mantle model is chosen in \texttt{Par\_file}, the simulations are performed together with the 3D crustal model Crust2.0. Alternatively, Crust2.0 can be combined with a higher resolution European crustal model EUCrust07 \citep{EUCrust07}. This can be done by setting the crustal type to \texttt{ICRUST\_CRUSTMAPS} in the \texttt{constant.h} file.
-It is also possible to run simulations using a 3D mantle model with a 1D crustal model on top. This can be done by setting the model in \texttt{Par\_file} to \texttt{<3D mantle>\_1Dcrust}, e.g., \texttt{s20rts\_1Dcrust, s362ani\_1Dcrust}, etc. In this case, the 1D crustal model will be the one that is used in the 3D mantle model as a reference model (e.g., transversely isotropic PREM for s20rts, REF for s362ani, etc.).
-
-\item [{\texttt{OCEANS}}] Set to \texttt{.true.} if the effect of the oceans
-on seismic wave propagation should be incorporated based upon the
-approximate treatment discussed in \citet{KoTr02b}. This feature
-is inexpensive from a numerical perspective, both in terms of memory
-requirements and CPU time. This approximation is accurate at periods
-of roughly 20~s and longer. At shorter periods the effect of water
-phases/reverberations is not taken into account, even when the flag
-is on.
-\item [{\texttt{ELLIPTICITY}}] Set to \texttt{.true.} if the mesh should
-make the Earth model elliptical in shape according to Clairaut's equation
-\citep{DaTr98}. This feature adds no cost to the simulation.
-\item [{\texttt{TOPOGRAPHY}}] Set to \texttt{.true.} if topography and
-bathymetry should be incorporated based upon model ETOPO5 \citep{Etopo5}.
-This feature adds no cost to the simulation.
-\item [{\texttt{GRAVITY}}] Set to \texttt{.true.} if self-gravitation should
-be incorporated in the Cowling approximation \citep{KoTr02b,DaTr98}.
-Turning this feature on is relatively inexpensive, both from the perspective
-of memory requirements as well as in terms of computational speed.
-\item [{\texttt{ROTATION}}] Set to \texttt{.true.} if the Coriolis effect
-should be incorporated. Turning this feature on is relatively cheap
-numerically.
-\item [{\texttt{ATTENUATION}}] Set to \texttt{.true.} if attenuation should
-be incorporated. Turning this feature on increases the memory requirements
-significantly (roughly by a factor of~1.5), and is numerically fairly
-expensive. Of course for realistic simulations this flag should be
-turned on. See \citet{KoTr99,KoTr02a} for a discussion on the implementation
-of attenuation based upon standard linear solids.
-\item [{\texttt{ABSORBING\_CONDITIONS}}] Set to \texttt{.false.} for global
-simulations. See Chapter~\ref{cha:Regional-Simulations} for regional
-simulations.
-\item [{\texttt{RECORD\_LENGTH\_IN\_MINUTES}}] Choose the desired record
-length of the synthetic seismograms (in minutes). This controls the
-length of the numerical simulation, i.e., twice the record length
-requires twice as much CPU time. This feature is not used at the time
-of meshing but is required for the solver, i.e., you may change this
-parameter after running the mesher.
-\item [{\texttt{MOVIE\_SURFACE}}] Set to \texttt{.false.}, unless you want
-to create a movie of seismic wave propagation on the Earth's surface.
-Turning this option on generates large output files. See Section \ref{sec:Movies}
-for a discussion on the generation of movies. This feature is not
-used at the time of meshing but is relevant for the solver.
-\item [{\texttt{MOVIE\_VOLUME}}] Set to \texttt{.false.}, unless you want
-to create a movie of seismic wave propagation in the Earth's interior.
-Turning this option on generates huge output files. See Section \ref{sec:Movies}
-for a discussion on the generation of movies. This feature is not
-used at the time of meshing but is relevant for the solver.
-\item [{\texttt{NTSTEP\_BETWEEN\_FRAMES}}] Determines the number of timesteps
-between movie frames. Typically you want to save a snapshot every
-100 timesteps. The smaller you make this number the more output will
-be generated! See Section \ref{sec:Movies} for a discussion on the
-generation of movies. This feature is not used at the time of meshing
-but is relevant for the solver.
-\item [{\texttt{HDUR\_MOVIE}}] determines the half duration of the source
-time function for the movie simulations. When this parameter is set
-to be 0, a default half duration that corresponds to the accuracy
-of the simulation is provided.
-\item [{\texttt{SAVE\_MESH\_FILES}}] Set this flag to \texttt{.true}\texttt{\small .}
-to save AVS \url{www.avs.com}, OpenDX \url{www.opendx.org}, or ParaView \url{www.paraview.org}
-mesh files for subsequent viewing. Turning the flag on generates large
-(distributed) files in the \texttt{LOCAL\_PATH} directory. See Section~\ref{sec:Meshes}
-for a discussion of mesh viewing features.
-\item [{\texttt{NUMBER\_OF\_RUNS}}] On machines with a run-time limit,
-for instance for a batch/queue system, a simulation may need to be
-completed in stages. This option allows you to select the number of
-stages in which the simulation will be completed (1, 2 or 3). Choose
-1 for a run without restart files. This feature is not used at the
-time of meshing but is required for the solver. At the end of the
-first or second stage of a multi-stage simulation, large files are
-written to the file system to save the current state of the simulation.
-This state is read back from the file system at the beginning of the
-next stage of the multi-stage run. Reading and writing the states
-can be very time consuming depending on the nature of the network
-and the file system (in this case writing to the local file system,
-i.e., the disk on a node, is preferable).
-\item [{\texttt{NUMBER\_OF\_THIS\_RUN}}] If you choose to perform the run
-in stages, you need to tell the solver what stage run to perform.
-This feature is not used at the time of meshing but is required for
-the solver.
-\item [{\texttt{LOCAL\_PATH}}] Directory in which the databases generated
-by the mesher will be written. Generally one uses a directory on the
-local disk of the compute nodes, although on some machines these databases
-are written on a parallel (global) file system (see also the earlier
-discussion of the \texttt{LOCAL\_PATH\_IS\_ALSO\_GLOBAL} flag in Chapter~\ref{cha:Getting-Started}).
-The mesher generates the necessary databases in parallel, one set
-for each of the $6\times\nprocxi^{2}$ slices that constitutes the
-mesh (see Figure~\ref{figure:mpi_slices}). After the mesher finishes,
-you can log in to one of the compute nodes and view the contents of
-the \texttt{LOCAL\_PATH} directory to see the (many) files generated
-by the mesher.
-\item [{\texttt{NTSTEP\_BETWEEN\_OUTPUT\_INFO}}] This parameter specifies
-the interval at which basic information about a run is written to
-the file system (\texttt{timestamp{*}} files in the \texttt{OUTPUT\_FILES}
-directory). If you have access to a fast machine, set \texttt{NTSTEP\_BETWEEN\_OUTPUT\_INFO}
-to a relatively high value (e.g., at least 100, or even 1000 or more)
-to avoid writing output text files too often. This feature is not
-used at the time of meshing. One can set this parameter to a larger
-value than the number of time steps to avoid writing output during
-the run.
-\item [{\texttt{NTSTEP\_BETWEEN\_OUTPUT\_SEISMOS}}] This parameter specifies
-the interval at which synthetic seismograms are written in the \texttt{LOCAL\_PATH}
-directory. The seismograms can be created in three different formats
-by setting the parameters \texttt{OUTPUT\_SEISMOS\_ASCII\_TEXT}, \texttt{OUTPUT\_SEISMOS\_SAC\_ALPHANUM}
-and \texttt{OUTPUT\_SEI-}~\\
-\texttt{SMOS\_SAC\_BINARY}. One can choose any combination of these
-parameters (details on the formats follow in the description of each
-parameter). SAC \url{www.iris.edu/software/sac/} is a signal-processing software
-package. If a run crashes, you may still find usable (but shorter
-than requested) seismograms in this directory. On a fast machine set
-\texttt{NTSTEP\_BETWEEN\_OUTPUT\_SEISMOS} to a relatively high value
-to avoid writing to the seismograms too often. This feature is not
-used at the time of meshing.
-\item [{\texttt{NTSTEP\_BETWEEN\_READ\_ADJSRC}}] The number of adjoint
-sources read in each time for an adjoint simulation.
-\item [{\texttt{OUTPUT\_SEISMOS\_ASCII\_TEXT}}] Set this flag to \texttt{.true.}
-if you want to have the synthetic seismograms written in two-column
-ASCII format (the first column contains time in seconds and the second
-column the displacement in meters of the recorded signal, no header
-information). Files will be named with extension \texttt{.ascii}.
-\item [{\texttt{OUTPUT\_SEISMOS\_SAC\_ALPHANUM}}] Set this flag to \texttt{.true.}
-if you want to have the synthetic seismograms written in alphanumeric
-(human readable) SAC format, which includes header information on
-the source and receiver parameters (e.g., source/receiver coordinates,
-station name, etc., see Appendix~\ref{cha:SAC-headers}). For details on the format, please check the SAC webpage \url{www.iris.edu/software/sac/}. Files will be named with extension \texttt{.sacan}.
-\item [{\texttt{OUTPUT\_SEISMOS\_SAC\_BINARY}}] Set this flag to \texttt{.true.}
-if you want to have the synthetic seismograms written in binary SAC
-format. The header information included is the same as for the alphanumeric
-SAC format (see Appendix~\ref{cha:SAC-headers}). Using this format requires the least disk space, which may be particulary important if you have a large number of stations.
-For details on the binary format please also check the SAC webpage \url{www.iris.edu/software/sac/} and Appendix~\ref{cha:SAC-headers}. Files will be named with extension \texttt{.sac}.
-\item [{\texttt{ROTATE\_SEISMOGRAMS\_RT}}] Set this flag to \texttt{.true.}
-if you want to have radial (R) and transverse (T) horizontal components
-of the synthetic seismograms (default is \texttt{.false.} $\rightarrow$
-East (E) and North (N) components).
-\item [{\texttt{WRITE\_SEISMOGRAMS\_BY\_MASTER}}] Set this flag to \texttt{.true.}
-if you want to have all the seismograms written by the master (no
-need to collect them on the nodes after the run).
-\item [{\texttt{SAVE\_ALL\_SEISMOS\_IN\_ONE\_FILE}}] Set this flag to \texttt{.true.}
-if you want to have all the seismograms saved in one large combined
-file instead of one file per seismogram to avoid overloading shared
-non-local file systems such as GPFS for instance.
-\item [{\texttt{USE\_BINARY\_FOR\_LARGE\_FILE}}] Set this flag to \texttt{.true.}
-if you want to use binary instead of ASCII for that large file (not
-used if SAVE\_ALL\_SEISMOS\_IN \_ONE\_FILE = .false.)
-\item [{\texttt{RECEIVERS\_CAN\_BE\_BURIED}}] This flag accommodates stations
-with instruments that are buried, i.e., the solver will calculate
-seismograms at the burial depth specified in the \texttt{STATIONS}
-file. This feature is not used at the time of meshing.
-\item [{\texttt{PRINT\_SOURCE\_TIME\_FUNCTION}}] Turn this flag on to print
-information about the source time function in the file \texttt{OUTPUT\_FILES/plot\_source\_time\_function.txt}.
-This feature is not used at the time of meshing.
-\end{description}
-\noindent \begin{center}
-\label{table:nex} \begin{longtable}{|c|c|c|c|c|c|c|c|c|c|c|c|}
-\hline
-\nprocxi & processors & \multicolumn{10}{c|}{\nexxi}\tabularnewline
-\hline
-\endhead
-\hline
-1 & 6 & 64 & 80 & 96 & 112 & 128 & 144 & 160 & 176 & 192 & 208\tabularnewline
-\hline
-2 & 24 & 64 & 80 & 96 & 112 & 128 & 144 & 160 & 176 & 192 & 208\tabularnewline
-\hline
-3 & 54 & 96 & 144 & 192 & 240 & 288 & 336 & 384 & 432 & 480 & 528\tabularnewline
-\hline
-4 & 96 & 64 & 96 & 128 & 160 & 192 & 224 & 256 & 288 & 320 & 352\tabularnewline
-\hline
-5 & 150 & 80 & 160 & 240 & 320 & 400 & 480 & 560 & 640 & 720 & 800\tabularnewline
-\hline
-6 & 216 & 96 & 144 & 192 & 240 & 288 & 336 & 384 & 432 & 480 & 528\tabularnewline
-\hline
-7 & 294 & 112 & 224 & 336 & 448 & 560 & 672 & 784 & 896 & 1008 & 1120\tabularnewline
-\hline
-8 & 384 & 64 & 128 & 192 & 256 & 320 & 384 & 448 & 512 & 576 & 640\tabularnewline
-\hline
-9 & 486 & 144 & 288 & 432 & 576 & 720 & 864 & 1008 & 1152 & 1296 & 1440\tabularnewline
-\hline
-10 & 600 & 80 & 160 & 240 & 320 & 400 & 480 & 560 & 640 & 720 & 800\tabularnewline
-\hline
-11 & 726 & 176 & 352 & 528 & 704 & 880 & 1056 & 1232 & 1408 & 1584 & 1760\tabularnewline
-\hline
-12 & 864 & 96 & 192 & 288 & 384 & 480 & 576 & 672 & 768 & 864 & 960\tabularnewline
-\hline
-13 & 1014 & 208 & 416 & 624 & 832 & 1040 & 1248 & 1456 & 1664 & 1872 & 2080\tabularnewline
-\hline
-14 & 1176 & 112 & 224 & 336 & 448 & 560 & 672 & 784 & 896 & 1008 & 1120\tabularnewline
-\hline
-15 & 1350 & 240 & 480 & 720 & 960 & 1200 & 1440 & 1680 & 1920 & 2160 & 2400\tabularnewline
-\hline
-16 & 1536 & 128 & 256 & 384 & 512 & 640 & 768 & 896 & 1024 & 1152 & 1280\tabularnewline
-\hline
-17 & 1734 & 272 & 544 & 816 & 1088 & 1360 & 1632 & 1904 & 2176 & 2448 & 2720\tabularnewline
-\hline
-18 & 1944 & 144 & 288 & 432 & 576 & 720 & 864 & 1008 & 1152 & 1296 & 1440\tabularnewline
-\hline
-19 & 2166 & 304 & 608 & 912 & 1216 & 1520 & 1824 & 2128 & 2432 & 2736 & 3040\tabularnewline
-\hline
-20 & 2400 & 160 & 320 & 480 & 640 & 800 & 960 & 1120 & 1280 & 1440 & 1600\tabularnewline
-\hline
-21 & 2646 & 336 & 672 & 1008 & 1344 & 1680 & 2016 & 2352 & 2688 & 3024 & 3360\tabularnewline
-\hline
-22 & 2904 & 176 & 352 & 528 & 704 & 880 & 1056 & 1232 & 1408 & 1584 & 1760\tabularnewline
-\hline
-23 & 3174 & 368 & 736 & 1104 & 1472 & 1840 & 2208 & 2576 & 2944 & 3312 & 3680\tabularnewline
-\hline
-24 & 3456 & 192 & 384 & 576 & 768 & 960 & 1152 & 1344 & 1536 & 1728 & 1920\tabularnewline
-\hline
-25 & 3750 & 400 & 800 & 1200 & 1600 & 2000 & 2400 & 2800 & 3200 & 3600 & 4000\tabularnewline
-\hline
-26 & 4056 & 208 & 416 & 624 & 832 & 1040 & 1248 & 1456 & 1664 & 1872 & 2080\tabularnewline
-\hline
-27 & 4374 & 432 & 864 & 1296 & 1728 & 2160 & 2592 & 3024 & 3456 & 3888 & 4320\tabularnewline
-\hline
-28 & 4704 & 224 & 448 & 672 & 896 & 1120 & 1344 & 1568 & 1792 & 2016 & 2240\tabularnewline
-\hline
-29 & 5046 & 464 & 928 & 1392 & 1856 & 2320 & 2784 & 3248 & 3712 & 4176 & 4640\tabularnewline
-\hline
-30 & 5400 & 240 & 480 & 720 & 960 & 1200 & 1440 & 1680 & 1920 & 2160 & 2400\tabularnewline
-\hline
-31 & 5766 & 496 & 992 & 1488 & 1984 & 2480 & 2976 & 3472 & 3968 & 4464 & 4960\tabularnewline
-\hline
-32 & 6144 & 256 & 512 & 768 & 1024 & 1280 & 1536 & 1792 & 2048 & 2304 & 2560\tabularnewline
-\hline
-33 & 6534 & 528 & 1056 & 1584 & 2112 & 2640 & 3168 & 3696 & 4224 & 4752 & 5280\tabularnewline
-\hline
-34 & 6936 & 272 & 544 & 816 & 1088 & 1360 & 1632 & 1904 & 2176 & 2448 & 2720\tabularnewline
-\hline
-35 & 7350 & 560 & 1120 & 1680 & 2240 & 2800 & 3360 & 3920 & 4480 & 5040 & 5600\tabularnewline
-\hline
-36 & 7776 & 288 & 576 & 864 & 1152 & 1440 & 1728 & 2016 & 2304 & 2592 & 2880\tabularnewline
-\hline
-37 & 8214 & 592 & 1184 & 1776 & 2368 & 2960 & 3552 & 4144 & 4736 & 5328 & 5920\tabularnewline
-\hline
-38 & 8664 & 304 & 608 & 912 & 1216 & 1520 & 1824 & 2128 & 2432 & 2736 & 3040\tabularnewline
-\hline
-39 & 9126 & 624 & 1248 & 1872 & 2496 & 3120 & 3744 & 4368 & 4992 & 5616 & 6240\tabularnewline
-\hline
-40 & 9600 & 320 & 640 & 960 & 1280 & 1600 & 1920 & 2240 & 2560 & 2880 & 3200\tabularnewline
-\hline
-41 & 10086 & 656 & 1312 & 1968 & 2624 & 3280 & 3936 & 4592 & 5248 & 5904 & 6560\tabularnewline
-\hline
-42 & 10584 & 336 & 672 & 1008 & 1344 & 1680 & 2016 & 2352 & 2688 & 3024 & 3360\tabularnewline
-\hline
-43 & 11094 & 688 & 1376 & 2064 & 2752 & 3440 & 4128 & 4816 & 5504 & 6192 & 6880\tabularnewline
-\hline
-44 & 11616 & 352 & 704 & 1056 & 1408 & 1760 & 2112 & 2464 & 2816 & 3168 & 3520\tabularnewline
-\hline
-45 & 12150 & 720 & 1440 & 2160 & 2880 & 3600 & 4320 & 5040 & 5760 & 6480 & 7200\tabularnewline
-\hline
-46 & 12696 & 368 & 736 & 1104 & 1472 & 1840 & 2208 & 2576 & 2944 & 3312 & 3680\tabularnewline
-\hline
-47 & 13254 & 752 & 1504 & 2256 & 3008 & 3760 & 4512 & 5264 & 6016 & 6768 & 7520\tabularnewline
-\hline
-48 & 13824 & 384 & 768 & 1152 & 1536 & 1920 & 2304 & 2688 & 3072 & 3456 & 3840\tabularnewline
-\hline
-49 & 14406 & 784 & 1568 & 2352 & 3136 & 3920 & 4704 & 5488 & 6272 & 7056 & 7840\tabularnewline
-\hline
-50 & 15000 & 400 & 800 & 1200 & 1600 & 2000 & 2400 & 2800 & 3200 & 3600 & 4000\tabularnewline
-\hline
-51 & 15606 & 816 & 1632 & 2448 & 3264 & 4080 & 4896 & 5712 & 6528 & 7344 & 8160\tabularnewline
-\hline
-52 & 16224 & 416 & 832 & 1248 & 1664 & 2080 & 2496 & 2912 & 3328 & 3744 & 4160\tabularnewline
-\hline
-53 & 16854 & 848 & 1696 & 2544 & 3392 & 4240 & 5088 & 5936 & 6784 & 7632 & 8480\tabularnewline
-\hline
-54 & 17496 & 432 & 864 & 1296 & 1728 & 2160 & 2592 & 3024 & 3456 & 3888 & 4320\tabularnewline
-\hline
-55 & 18150 & 880 & 1760 & 2640 & 3520 & 4400 & 5280 & 6160 & 7040 & 7920 & 8800\tabularnewline
-\hline
-56 & 18816 & 448 & 896 & 1344 & 1792 & 2240 & 2688 & 3136 & 3584 & 4032 & 4480\tabularnewline
-\hline
-57 & 19494 & 912 & 1824 & 2736 & 3648 & 4560 & 5472 & 6384 & 7296 & 8208 & 9120\tabularnewline
-\hline
-58 & 20184 & 464 & 928 & 1392 & 1856 & 2320 & 2784 & 3248 & 3712 & 4176 & 4640\tabularnewline
-\hline
-59 & 20886 & 944 & 1888 & 2832 & 3776 & 4720 & 5664 & 6608 & 7552 & 8496 & 9440\tabularnewline
-\hline
-60 & 21600 & 480 & 960 & 1440 & 1920 & 2400 & 2880 & 3360 & 3840 & 4320 & 4800\tabularnewline
-\hline
-61 & 22326 & 976 & 1952 & 2928 & 3904 & 4880 & 5856 & 6832 & 7808 & 8784 & 9760\tabularnewline
-\hline
-62 & 23064 & 496 & 992 & 1488 & 1984 & 2480 & 2976 & 3472 & 3968 & 4464 & 4960\tabularnewline
-\hline
-63 & 23814 & 1008 & 2016 & 3024 & 4032 & 5040 & 6048 & 7056 & 8064 & 9072 & 10080\tabularnewline
-\hline
-64 & 24576 & 512 & 1024 & 1536 & 2048 & 2560 & 3072 & 3584 & 4096 & 4608 & 5120\tabularnewline
-\hline
-65 & 25350 & 1040 & 2080 & 3120 & 4160 & 5200 & 6240 & 7280 & 8320 & 9360 & 10400\tabularnewline
-\hline
-66 & 26136 & 528 & 1056 & 1584 & 2112 & 2640 & 3168 & 3696 & 4224 & 4752 & 5280\tabularnewline
-\hline
-67 & 26934 & 1072 & 2144 & 3216 & 4288 & 5360 & 6432 & 7504 & 8576 & 9648 & 10720\tabularnewline
-\hline
-68 & 27744 & 544 & 1088 & 1632 & 2176 & 2720 & 3264 & 3808 & 4352 & 4896 & 5440\tabularnewline
-\hline
-69 & 28566 & 1104 & 2208 & 3312 & 4416 & 5520 & 6624 & 7728 & 8832 & 9936 & 11040\tabularnewline
-\hline
-70 & 29400 & 560 & 1120 & 1680 & 2240 & 2800 & 3360 & 3920 & 4480 & 5040 & 5600\tabularnewline
-\hline
-71 & 30246 & 1136 & 2272 & 3408 & 4544 & 5680 & 6816 & 7952 & 9088 & 10224 & 11360\tabularnewline
-\hline
-72 & 31104 & 576 & 1152 & 1728 & 2304 & 2880 & 3456 & 4032 & 4608 & 5184 & 5760\tabularnewline
-\hline
-73 & 31974 & 1168 & 2336 & 3504 & 4672 & 5840 & 7008 & 8176 & 9344 & 10512 & 11680\tabularnewline
-\hline
-74 & 32856 & 592 & 1184 & 1776 & 2368 & 2960 & 3552 & 4144 & 4736 & 5328 & 5920\tabularnewline
-\hline
-75 & 33750 & 1200 & 2400 & 3600 & 4800 & 6000 & 7200 & 8400 & 9600 & 10800 & 12000\tabularnewline
-\hline
-76 & 34656 & 608 & 1216 & 1824 & 2432 & 3040 & 3648 & 4256 & 4864 & 5472 & 6080\tabularnewline
-\hline
-77 & 35574 & 1232 & 2464 & 3696 & 4928 & 6160 & 7392 & 8624 & 9856 & 11088 & 12320\tabularnewline
-\hline
-78 & 36504 & 624 & 1248 & 1872 & 2496 & 3120 & 3744 & 4368 & 4992 & 5616 & 6240\tabularnewline
-\hline
-79 & 37446 & 1264 & 2528 & 3792 & 5056 & 6320 & 7584 & 8848 & 10112 & 11376 & 12640\tabularnewline
-\hline
-80 & 38400 & 640 & 1280 & 1920 & 2560 & 3200 & 3840 & 4480 & 5120 & 5760 & 6400\tabularnewline
-\hline
-81 & 39366 & 1296 & 2592 & 3888 & 5184 & 6480 & 7776 & 9072 & 10368 & 11664 & 12960\tabularnewline
-\hline
-82 & 40344 & 656 & 1312 & 1968 & 2624 & 3280 & 3936 & 4592 & 5248 & 5904 & 6560\tabularnewline
-\hline
-83 & 41334 & 1328 & 2656 & 3984 & 5312 & 6640 & 7968 & 9296 & 10624 & 11952 & 13280\tabularnewline
-\hline
-84 & 42336 & 672 & 1344 & 2016 & 2688 & 3360 & 4032 & 4704 & 5376 & 6048 & 6720\tabularnewline
-\hline
-85 & 43350 & 1360 & 2720 & 4080 & 5440 & 6800 & 8160 & 9520 & 10880 & 12240 & 13600\tabularnewline
-\hline
-86 & 44376 & 688 & 1376 & 2064 & 2752 & 3440 & 4128 & 4816 & 5504 & 6192 & 6880\tabularnewline
-\hline
-87 & 45414 & 1392 & 2784 & 4176 & 5568 & 6960 & 8352 & 9744 & 11136 & 12528 & 13920\tabularnewline
-\hline
-88 & 46464 & 704 & 1408 & 2112 & 2816 & 3520 & 4224 & 4928 & 5632 & 6336 & 7040\tabularnewline
-\hline
-89 & 47526 & 1424 & 2848 & 4272 & 5696 & 7120 & 8544 & 9968 & 11392 & 12816 & 14240\tabularnewline
-\hline
-90 & 48600 & 720 & 1440 & 2160 & 2880 & 3600 & 4320 & 5040 & 5760 & 6480 & 7200\tabularnewline
-\hline
-91 & 49686 & 1456 & 2912 & 4368 & 5824 & 7280 & 8736 & 10192 & 11648 & 13104 & 14560\tabularnewline
-\hline
-92 & 50784 & 736 & 1472 & 2208 & 2944 & 3680 & 4416 & 5152 & 5888 & 6624 & 7360\tabularnewline
-\hline
-93 & 51894 & 1488 & 2976 & 4464 & 5952 & 7440 & 8928 & 10416 & 11904 & 13392 & 14880\tabularnewline
-\hline
-94 & 53016 & 752 & 1504 & 2256 & 3008 & 3760 & 4512 & 5264 & 6016 & 6768 & 7520\tabularnewline
-\hline
-95 & 54150 & 1520 & 3040 & 4560 & 6080 & 7600 & 9120 & 10640 & 12160 & 13680 & 15200\tabularnewline
-\hline
-96 & 55296 & 768 & 1536 & 2304 & 3072 & 3840 & 4608 & 5376 & 6144 & 6912 & 7680\tabularnewline
-\hline
-97 & 56454 & 1552 & 3104 & 4656 & 6208 & 7760 & 9312 & 10864 & 12416 & 13968 & 15520\tabularnewline
-\hline
-98 & 57624 & 784 & 1568 & 2352 & 3136 & 3920 & 4704 & 5488 & 6272 & 7056 & 7840\tabularnewline
-\hline
-99 & 58806 & 1584 & 3168 & 4752 & 6336 & 7920 & 9504 & 11088 & 12672 & 14256 & 15840\tabularnewline
-\hline
-100 & 60000 & 800 & 1600 & 2400 & 3200 & 4000 & 4800 & 5600 & 6400 & 7200 & 8000\tabularnewline
-\hline
-101 & 61206 & 1616 & 3232 & 4848 & 6464 & 8080 & 9696 & 11312 & 12928 & 14544 & 16160\tabularnewline
-\hline
-102 & 62424 & 816 & 1632 & 2448 & 3264 & 4080 & 4896 & 5712 & 6528 & 7344 & 8160\tabularnewline
-\hline
-103 & 63654 & 1648 & 3296 & 4944 & 6592 & 8240 & 9888 & 11536 & 13184 & 14832 & 16480\tabularnewline
-\hline
-104 & 64896 & 832 & 1664 & 2496 & 3328 & 4160 & 4992 & 5824 & 6656 & 7488 & 8320\tabularnewline
-\hline
-105 & 66150 & 1680 & 3360 & 5040 & 6720 & 8400 & 10080 & 11760 & 13440 & 15120 & 16800\tabularnewline
-\hline
-106 & 67416 & 848 & 1696 & 2544 & 3392 & 4240 & 5088 & 5936 & 6784 & 7632 & 8480\tabularnewline
-\hline
-107 & 68694 & 1712 & 3424 & 5136 & 6848 & 8560 & 10272 & 11984 & 13696 & 15408 & 17120\tabularnewline
-\hline
-108 & 69984 & 864 & 1728 & 2592 & 3456 & 4320 & 5184 & 6048 & 6912 & 7776 & 8640\tabularnewline
-\hline
-109 & 71286 & 1744 & 3488 & 5232 & 6976 & 8720 & 10464 & 12208 & 13952 & 15696 & 17440\tabularnewline
-\hline
-110 & 72600 & 880 & 1760 & 2640 & 3520 & 4400 & 5280 & 6160 & 7040 & 7920 & 8800\tabularnewline
-\hline
-111 & 73926 & 1776 & 3552 & 5328 & 7104 & 8880 & 10656 & 12432 & 14208 & 15984 & 17760\tabularnewline
-\hline
-112 & 75264 & 896 & 1792 & 2688 & 3584 & 4480 & 5376 & 6272 & 7168 & 8064 & 8960\tabularnewline
-\hline
-113 & 76614 & 1808 & 3616 & 5424 & 7232 & 9040 & 10848 & 12656 & 14464 & 16272 & 18080\tabularnewline
-\hline
-114 & 77976 & 912 & 1824 & 2736 & 3648 & 4560 & 5472 & 6384 & 7296 & 8208 & 9120\tabularnewline
-\hline
-115 & 79350 & 1840 & 3680 & 5520 & 7360 & 9200 & 11040 & 12880 & 14720 & 16560 & 18400\tabularnewline
-\hline
-116 & 80736 & 928 & 1856 & 2784 & 3712 & 4640 & 5568 & 6496 & 7424 & 8352 & 9280\tabularnewline
-\hline
-117 & 82134 & 1872 & 3744 & 5616 & 7488 & 9360 & 11232 & 13104 & 14976 & 16848 & 18720\tabularnewline
-\hline
-118 & 83544 & 944 & 1888 & 2832 & 3776 & 4720 & 5664 & 6608 & 7552 & 8496 & 9440\tabularnewline
-\hline
-119 & 84966 & 1904 & 3808 & 5712 & 7616 & 9520 & 11424 & 13328 & 15232 & 17136 & 19040\tabularnewline
-\hline
-120 & 86400 & 960 & 1920 & 2880 & 3840 & 4800 & 5760 & 6720 & 7680 & 8640 & 9600\tabularnewline
-\hline
-121 & 87846 & 1936 & 3872 & 5808 & 7744 & 9680 & 11616 & 13552 & 15488 & 17424 & 19360\tabularnewline
-\hline
-122 & 89304 & 976 & 1952 & 2928 & 3904 & 4880 & 5856 & 6832 & 7808 & 8784 & 9760\tabularnewline
-\hline
-123 & 90774 & 1968 & 3936 & 5904 & 7872 & 9840 & 11808 & 13776 & 15744 & 17712 & 19680\tabularnewline
-\hline
-124 & 92256 & 992 & 1984 & 2976 & 3968 & 4960 & 5952 & 6944 & 7936 & 8928 & 9920\tabularnewline
-\hline
-125 & 93750 & 2000 & 4000 & 6000 & 8000 & 10000 & 12000 & 14000 & 16000 & 18000 & 20000\tabularnewline
-\hline
-126 & 95256 & 1008 & 2016 & 3024 & 4032 & 5040 & 6048 & 7056 & 8064 & 9072 & 10080\tabularnewline
-\hline
-127 & 96774 & 2032 & 4064 & 6096 & 8128 & 10160 & 12192 & 14224 & 16256 & 18288 & 20320\tabularnewline
-\hline
-128 & 98304 & 1024 & 2048 & 3072 & 4096 & 5120 & 6144 & 7168 & 8192 & 9216 & 10240\tabularnewline
-\hline
-129 & 99846 & 2064 & 4128 & 6192 & 8256 & 10320 & 12384 & 14448 & 16512 & 18576 & 20640\tabularnewline
-\hline
-\end{longtable}
-\par\end{center}
-
-\begin{center}
-%
-\begin{table}[H]
-\caption{Sample choices for $\nexxi$ given $\nprocxi$ based upon the relationship
-$\nexxi=8\times c\times\nprocxi$, where the integer $c\ge1$. The
-number of MPI slices, i.e., the total number of required processors,
-is $6\times\nprocxi^{2}$, as illustrated in Figure~\ref{figure:mpi_slices}.
-The approximate shortest period at which the global simulation is
-accurate for a given value of $\nexxi$ can be estimated by running
-the small serial program \texttt{xcreate\_header\_file}.}
-
-\end{table}
-
-\par\end{center}
-
-Finally, you need to provide a file that tells MPI what compute nodes
-to use for the simulations. The file must have a number of entries
-(one entry per line) at least equal to the number of processors needed
-for the run. A sample file is provided in the file \texttt{mymachines}.
-This file is not used by the mesher or solver, but is required by
-the \texttt{go\_mesher} and \texttt{go\_solver} default job submission
-scripts. See Chapter \ref{cha:Running-Scheduler} for information
-about running the code on a system with a scheduler, e.g., LSF.
-
-Now that you have set the appropriate parameters in the \texttt{Par\_file}
-and have compiled the mesher, you are ready to launch it! This is
-most easily accomplished based upon the \texttt{go\_mesher} script.
-When you run on a PC cluster, the script assumes that the nodes are
-named n001, n002, etc. If this is not the case, change the \texttt{tr
--d `n'} line in the script. You may also need to edit the last command
-at the end of the script that invokes the \texttt{mpirun} command.
-
-Mesher output is provided in the \texttt{OUTPUT\_FILES} directory
-in \texttt{output\_mesher.txt}; this file provides lots of details
-about the mesh that was generated. Alternatively, output can be directed
-to the screen instead by uncommenting a line in \texttt{constants.h}:
-
-\begin{lyxcode}
-!~uncomment~this~to~write~messages~to~the~screen~
-
-!~integer,~parameter~::~IMAIN~=~ISTANDARD\_OUTPUT~~
-\end{lyxcode}
-Note that on very fast machines, writing to the screen may slow down
-the code.
-
-Another file generated by the mesher is the header file \texttt{OUTPUT\_FILES/values\_from\_mesher.h}.
-This file specifies a number of constants and flags needed by the
-solver. These values are passed statically to the solver for reasons
-of speed. Some useful statistics about the mesh are also provided
-in this file.
-
-For a given model, set of nodes, and set of parameters in \texttt{Par\_file},
-one only needs to run the mesher once and for all, even if one wants
-to run several simulations with different sources and/or receivers
-(the source and receiver information is used in the solver only).
-
-Please note that it is difficult to correctly sample S waves in the
-inner core of the Earth because S-wave velocity is very small there.
-Therefore, correctly sampling S waves in the inner core would require
-a very dense mesh, which in turn would drastically reduce the time
-step of the explicit time scheme because the P wave velocity is very
-high in the inner core (Poisson's ratio is roughly equal to 0.44).
-Because shear wave attenuation is very high in the inner core ($Q_{\mu}$
-is approximately equal to 85), we have therefore decided to design
-the inner core mesh such that P waves are very well sampled but S
-waves are right at the sampling limit or even slightly below. This
-works fine because spurious numerical oscillations due to S-wave subsampling
-are almost completely suppressed by attenuation. However, this implies
-that one should not use SPECFEM3D\_GLOBE with the regular mesh and
-period estimates of Table \ref{table:nex} to study the PKJKP phase
-very precisely. If one is interested in that phase, one should use
-typically 1.5 times to twice the number of elements NEX indicated
-in the table.
-
-Regarding fluid/solid coupling at the CMB and ICB, in SPECFEM3D\_GLOBE
-we do not use the fluid-solid formulation of \citet{KoTr02a} and
-\citet{KoTr02b} anymore, we now use a displacement potential in the
-fluid (rather than a velocity potential as in \citet{KoTr02a} and
-\citet{KoTr02b}). This leads to the simpler fluid-solid matching
-condition introduced by \citet{ChVa04} with no numerical iterations
-at the CMB and ICB.
-
-For accuracy reasons, in the mesher the coordinates of the mesh points
-(arrays xstore, ystore and zstore) are always created in double precision.
-If the solver is compiled in single precision mode, the mesh coordinates
-are converted to single precision before being saved in the local
-mesh files.
-
-
-\section{Memory requirements}
-
-The SPECFEM3D\_GLOBE memory requirements can be estimated before or
-after running the mesher using the small serial program \texttt{\small xcreate\_header\_file},
-which reads the input file \texttt{\small DATA/Par\_file} and displays
-the total amount of memory that will be needed by the mesher and the
-solver to run it. This way, users can easily modify the parameters
-and check that their simulation will fit in memory on their machine.
-The file created by \texttt{\small xcreate\_header\_file} is called
-\texttt{OUTPUT\_FILES/values\_from\_mesher.h} and contains even more
-details about the future simulation.
-
-
-\section{Checking the MPI Buffers (Optional)}
-
-The mesher writes MPI communication tables in the \texttt{OUTPUT\_FILES}
-subdirectory in the files \texttt{addressing.txt}, \texttt{list\_messages\_corners.txt}
-and \texttt{list\_messages\_faces.txt}, and MPI communication buffers
-to the local disks. Use the four serial codes
-
-\begin{lyxcode}
-check\_buffers\_2D.f90
-
-check\_buffers\_1D.f90
-
-check\_buffers\_faces\_chunks.f90
-
-check\_buffers\_corners\_chunks.f90~
-\end{lyxcode}
-to check that all the MPI buffers created by the mesher have been
-generated correctly. For example, typing `\texttt{make check\_buffers\_2D}'
-and then `\texttt{xcheck\_buffers\_2D}' checks the communication buffers
-between faces common to the mesh slices. `\texttt{xcheck\_buffers\_1D}'
-checks the communication buffers between edges common to the mesh
-slices. `\texttt{xcheck\_buffers\_faces\_chunks}' checks the communication
-buffers between faces common to the mesh chunks, i.e., the faces of
-the six blocks of the cubed-sphere mesh. `\texttt{xcheck\_buffers\_corners\_chunks}'
-checks the communication buffers between edges common to the mesh
-chunks, which must be treated separately in MPI because they are of
-valence 3 (i.e., they are shared between three chunks).
-
-Please note that running these codes is optional because no information
-needed by the solver is generated.
-
-\chapter{\label{cha:Running-the-Solver}Running the Solver \texttt{xspecfem3D}}
-
-Now that you have successfully run the mesher, you are ready to compile
-the solver. For reasons of speed, the solver uses static memory allocation.
-Therefore it needs to be recompiled (type `\texttt{make clean}' and
-`\texttt{make specfem3D}') every time one reruns the mesher with different
-parameters. To compile the solver one needs a file generated by the
-mesher in the directory \texttt{OUTPUT\_FILES} called \texttt{values\_from\_mesher.h},
-which contains parameters describing the static size of the arrays
-as well as the setting of certain flags.
-
-The solver needs three input files in the \texttt{DATA} directory
-to run: the \texttt{Par\_file} that was discussed in detail in Chapter~\ref{cha:Running-the-Mesher},
-the earthquake source parameter file \texttt{CMTSOLUTION}, and the
-stations file \texttt{STATIONS}. Most parameters in the \texttt{Par\_file}
-should be set prior to running the mesher. Only the following parameters
-may be changed after running the mesher:
-
-\begin{itemize}
-\item the simulation type control parameters: \texttt{SIMULATION\_TYPE}
-and \texttt{SAVE\_FORWARD}
-\item the record length \texttt{RECORD\_LENGTH\_IN\_MINUTES}
-\item the movie control parameters \texttt{MOVIE\_SURFACE}, \texttt{MOVIE\_VOLUME},
-and \texttt{NTSTEPS\_BETWEEN\_FRAMES}
-\item the multi-stage simulation parameters \texttt{NUMBER\_OF\_RUNS} and
-\texttt{NUMBER\_OF\_THIS\_RUN}
-\item the output information parameters \texttt{NTSTEP\_BETWEEN\_OUTPUT\_INFO,
-NTSTEP\_BETWEEN\_OUTPUT\_}~\\
-\texttt{SEISMOS, OUTPUT\_SEISMOS\_ASCII\_TEXT, OUTPUT\_SEISMOS\_SAC\_ALPHANUM,
-OUTPUT\_}~\\
-\texttt{SEISMOS\_SAC\_BINARY and ROTATE\_SEISMOGRAMS\_RT}
-\item the \texttt{RECEIVERS\_CAN\_BE\_BURIED} and \texttt{PRINT\_SOURCE\_TIME\_FUNCTION}
-flags
-\end{itemize}
-Any other change to the \texttt{Par\_file} implies rerunning both
-the mesher and the solver.
-
-For any particular earthquake, the \texttt{CMTSOLUTION} file that
-represents the point source may be obtained directly from the Harvard Centroid-Moment Tensor (CMT) web page \url{www.globalcmt.org}.
-It looks like this:
-
-\begin{lyxcode}
-{\small }%
-\begin{figure}[H]
-\noindent \begin{centering}
-{\small \includegraphics[width=1\textwidth]{figures/Denali_CMT} }
-\par\end{centering}{\small \par}
-
-\caption{\texttt{CMTSOLUTION} file obtained from the Harvard CMT catalog. The
-top line is the initial estimate of the source, which is used as a
-starting point for the CMT solution. \textbf{M} is the moment tensor,
-$M_{0}${\small{} }is the seismic moment, and $M_{w}$ is the moment
-magnitude.}
-
-
-\label{fig:CMTSOLUTION-file}
-\end{figure}
-{\small \par}
-\end{lyxcode}
-The \texttt{CMTSOLUTION} should be edited in the following way:
-
-\begin{itemize}
-\item For point-source simulations (see finite sources, page \pageref{To-simulate-a})
-we recommend setting the source half-duration parameter \texttt{half
-duration} equal to zero, which corresponds to simulating a step source-time
-function, i.e., a moment-rate function that is a delta function. If
-\texttt{half duration} is not set to zero, the code will use a Gaussian
-(i.e., a signal with a shape similar to a `smoothed triangle', as
-explained in \citet{KoTr02a} and shown in Fig~\ref{fig:gauss.vs.triangle})
-source-time function with half-width \texttt{half duration}. We prefer
-to run the solver with \texttt{half duration} set to zero and convolve
-the resulting synthetic seismograms in post-processing after the run,
-because this way it is easy to use a variety of source-time functions
-(see Section \ref{sec:Process-data-and-syn}). \citet{KoTr02a} determined
-that the noise generated in the simulation by using a step source
-time function may be safely filtered out afterward based upon a convolution
-with the desired source time function and/or low-pass filtering. Use
-the postprocessing script \texttt{process\_syn.pl} (see Section \ref{sub:process_syn.pl})
-with the \texttt{-h} flag, or the serial code \texttt{convolve\_source\_timefunction.f90}
-and the script \texttt{UTILS/convolve\_source\_timefunction.csh} for
-this purpose, or alternatively use signal-processing software packages
-such as SAC \url{www.iris.edu/software/sac/}. Type
-
-\begin{lyxcode}
-make~convolve\_source\_timefunction
-\end{lyxcode}
-to compile the code and then set the parameter \texttt{hdur} in \texttt{UTILS/convolve\_source\_timefunction.csh}
-to the desired half-duration.
-
-\item The zero time of the simulation corresponds to the center of the triangle/Gaussian,
-or the centroid time of the earthquake. The start time of the simulation
-is $t=-1.5*\texttt{half duration}$ (the 1.5 is to make sure the moment
-rate function is very close to zero when starting the simulation).
-To convert to absolute time $t_{\mathrm{abs}}$, set
-
-\begin{lyxcode}
-$t_{\mathrm{abs}}=t_{\mathrm{pde}}+\texttt{time shift}+t_{\mathrm{synthetic}}$
-\end{lyxcode}
-where $t_{\mathrm{pde}}$ is the time given in the first line of the
-\texttt{CMTSOLUTION}, \texttt{time shift} is the corresponding value
-from the original \texttt{CMTSOLUTION} file and $t_{\mathrm{synthetic}}$
-is the time in the first column of the output seismogram.
-
-\end{itemize}
-%
-\begin{figure}
-\noindent \begin{centering}
-\includegraphics[width=3in]{figures/gauss_vs_triangle_mod}
-\par\end{centering}
-
-\caption{Comparison of the shape of a triangle and the Gaussian function actually
-used.}
-
-
-\label{fig:gauss.vs.triangle}
-\end{figure}
-
-
-Centroid latitude and longitude should be provided in geographical
-coordinates. The code converts these coordinates to geocentric coordinates~\citep{DaTr98}.
-Of course you may provide your own source representations by designing
-your own \texttt{CMTSOLUTION} file. Just make sure that the resulting
-file adheres to the Harvard CMT conventions (see Appendix~\ref{cha:Reference-Frame-Convention}).
-Note that the first line in the \texttt{CMTSOLUTION} file is the Preliminary Determination of Earthquakes (PDE) solution performed by the USGS NEIC, which is used as a seed for the Harvard CMT inversion. The PDE solution is based upon P waves and often gives the hypocenter of the earthquake, i.e., the rupture initiation point, whereas the CMT solution gives the `centroid location', which is the location with dominant moment release. The PDE solution is not used by our software package but must be present anyway in the first line of the file.
-
-In the current version of the code, the solver can run with a non-zero \texttt{time shift} in the \texttt{CMTSOLUTION} file. Thus one does not need to set \texttt{time shift} to zero as it was the case for previous versions of the code. \texttt{time shift} is only used for writing centroid time in the SAC headers (see Appendix~\ref{cha:SAC-headers}). CMT time is obtained by adding \texttt{time shift} to the PDE time given in the first line in the \texttt{CMTSOLUTION} file. Therefore it is recommended not to modify \texttt{time shift} to have the correct timing information in the SAC headers without any post-processing of seismograms.
-
-\label{To-simulate-a}To simulate a kinematic rupture, i.e., a finite-source
-event, represented in terms of $N_{\mathrm{sources}}$ point sources,
-provide a \texttt{CMTSOLUTION} file that has $N_{\mathrm{sources}}$
-entries, one for each subevent (i.e., concatenate $N_{\mathrm{sources}}$
-\texttt{CMTSOLUTION} files to a single \texttt{CMTSOLUTION} file).
-At least one entry (not necessarily the first) must have a zero \texttt{time
-shift}, and all the other entries must have non-negative \texttt{time
-shift}. If none of the entries has a zero \texttt{time shift} in the \texttt{CMTSOLUTION} file, the smallest \texttt{time shift} is subtracted from all sources to initiate the simulation. Each subevent can have its own half duration, latitude, longitude,
-depth, and moment tensor (effectively, the local moment-density tensor).
-
-Note that the zero in the synthetics does NOT represent the hypocentral
-time or centroid time in general, but the timing of the \textit{center}
-of the source triangle with zero \texttt{time shift} (Fig~\ref{fig:source_timing}).
-
-Although it is convenient to think of each source as a triangle, in
-the simulation they are actually Gaussians (as they have better frequency
-characteristics). The relationship between the triangle and the Gaussian
-used is shown in Fig~\ref{fig:gauss.vs.triangle}. For finite fault
-simulations it is usually not advisable to use a zero half duration
-and convolve afterwards, since the half duration is generally fixed
-by the finite fault model.
-
-{\small }%
-\begin{figure}[H]
-\noindent \begin{centering}
-{\small \includegraphics[width=5in]{figures/source_timing} }
-\par\end{centering}{\small \par}
-
-\caption{Example of timing for three sources. The center of the first source
-triangle is defined to be time zero. Note that this is NOT in general
-the hypocentral time, or the start time of the source (marked as tstart).
-The parameter \texttt{time shift} in the \texttt{CMTSOLUTION} file
-would be t1(=0), t2, t3 in this case, and the parameter \texttt{half
-duration} would be hdur1, hdur2, hdur3 for the sources 1, 2, 3 respectively.}
-
-
-{\small \label{fig:source_timing} }
-\end{figure}
-{\small \par}
-
-The solver can calculate seismograms at any number of stations for
-basically the same numerical cost, so the user is encouraged to include
-as many stations as conceivably useful in the \texttt{STATIONS} file,
-which looks like this:
-
-{\small }%
-\begin{figure}[H]
-\noindent \begin{centering}
-{\small \includegraphics{figures/STATIONS_global_explained} }
-\par\end{centering}{\small \par}
-
-\caption{Sample \texttt{STATIONS} file. Station latitude and longitude should
-be provided in geographical coordinates. The width of the station
-label should be no more than 32 characters (see \texttt{MAX\_LENGTH\_STATION\_NAME}
-in the \texttt{constants.h} file), and the network label should be
-no more than 8 characters (see \texttt{MAX\_LENGTH\_NETWORK\_NAME}
-in the \texttt{constants.h} file).}
-
-\end{figure}
-{\small \par}
-
-Each line represents one station in the following format:
-
-\begin{lyxcode}
-\noindent {\small Station~Network~Latitude~(degrees)~Longitude~(degrees)~Elevation~(m)~burial~(m)~}{\small \par}
-\end{lyxcode}
-If you want to put a station on the ocean floor, just set
-elevation and burial depth in the STATIONS file to 0.
-Equivalently you can also set elevation to a negative value equal
-to the ocean depth, and burial depth to 0.
-
-Solver output is provided in the \texttt{OUTPUT\_FILES} directory
-in the \texttt{output\_solver.txt} file. Output can be directed to
-the screen instead by uncommenting a line in \texttt{constants.h}:
-
-\begin{lyxcode}
-!~uncomment~this~to~write~messages~to~the~screen~
-
-!~integer,~parameter~::~IMAIN~=~ISTANDARD\_OUTPUT~
-\end{lyxcode}
-Note that on very fast machines, writing to the screen may slow down
-the code.
-
-While the solver is running, its progress may be tracked by monitoring
-the `\texttt{timestamp{*}}' files in the \texttt{OUTPUT\_FILES} directory.
-These tiny files look something like this:
-
-\begin{lyxcode}
-Time~step~\#~~~~~~~~~~~200~
-
-Time:~~~~0.6956667~~~~~~minutes~
-
-Elapsed~time~in~seconds~=~~~~~252.6748970000000~
-
-Elapsed~time~in~hh:mm:ss~=~~~~0~h~04~m~12~s~
-
-Mean~elapsed~time~per~time~step~in~seconds~=~~~~~1.263374485000000~
-
-Max~norm~displacement~vector~U~in~solid~in~all~slices~(m)~=~~~~1.9325~
-
-Max~non-dimensional~potential~Ufluid~in~fluid~in~all~slices~=~1.1058885E-22~
-\end{lyxcode}
-The \texttt{timestamp{*}} files provide the \texttt{Mean elapsed time
-per time step in seconds}, which may be used to assess performance
-on various machines (assuming you are the only user on a node), as
-well as the \texttt{\small Max}{\small{} }\texttt{\small norm}{\small{}
-}\texttt{\small displacement}{\small{} }\texttt{\small vector}{\small{}
-}\texttt{\small U}{\small{} }\texttt{\small in}{\small{} }\texttt{\small solid}{\small{}
-}\texttt{\small in}{\small{} }\texttt{\small all}{\small{} }\texttt{\small slices~(m)}{\small{}
-}\texttt{\small and}{\small{} }\texttt{\small Max}{\small{} }\texttt{\small non-dimensional}{\small{}
-}\texttt{\small potential}{\small{} }\texttt{\small Ufluid}{\small{}
-}\texttt{\small in}{\small{} }\texttt{\small fluid}{\small{} }\texttt{\small in}{\small{}
-}\texttt{\small all}{\small{} }\texttt{\small slices}. If something
-is wrong with the model, the mesh, or the source, you will see the
-code become unstable through exponentionally growing values of the
-displacement and/or fluid potential with time, and ultimately the
-run will be terminated by the program when either of these values
-becomes greater than \texttt{STABILITY\_THRESHOLD} defined in \texttt{constants.h}.
-You can control the rate at which the timestamp files are written
-based upon the parameter \texttt{NTSTEP\_BETWEEN\_OUTPUT\_INFO} in
-the \texttt{Par\_file}.
-
-Having set the \texttt{Par\_file} parameters, and having provided
-the \texttt{CMTSOLUTION} and \texttt{STATIONS} files, you are now
-ready to launch the solver! This is most easily accomplished based
-upon the \texttt{go\_solver} script (see Chapter~\ref{cha:Running-Scheduler}
-for information about running the code through a scheduler, e.g.,
-LSF). You may need to edit the last command at the end of the script
-that invokes the \texttt{mpirun} command. Another option is to use
-the \texttt{runall} script, which compiles and runs both mesher and
-solver in sequence. This is a safe approach that ensures using the
-correct combination of mesher output and solver input.
-
-It is important to realize that the CPU and memory requirements of
-the solver are closely tied to choices about attenuation (\texttt{ATTENUATION})
-and the nature of the model (i.e., isotropic models are cheaper than
-anisotropic models). We encourage you to run a variety of simulations
-with various flags turned on or off to develop a sense for what is
-involved.
-
-For the same model, one can rerun the solver for different events
-by simply changing the \texttt{CMTSOLUTION} file, and/or for different
-stations by changing the \texttt{STATIONS} file. There is no need
-to rerun the mesher. Of course it is best to include as many stations
-as possible, since this does not add significantly to the cost of
-the simulation.
-
-
-\chapter{\label{cha:Regional-Simulations}Regional Simulations}
-
-The code has the option of running in 1-, 2-, 3- or 6-chunk mode.
-The 1- or 2-chunk options may be used for higher resolution regional
-simulations. A one-chunk mesh may have lateral dimensions other than
-the customary $90^{\circ}$ per chunk, which can further increase
-the resolution of the mesh, and thus reduce the shortest period in
-the synthetic seismograms (but of course then also reducing the time
-step in order for the simulation to remain stable). A disadvantage
-of regional simulations is that one needs to use approximate absorbing
-boundary conditions on the side and bottom edges of the model (e.g.,
-see \citet{KoTr99} for a description of the paraxial boundary conditions
-used). Figure~\vref{fig:3D-spectral-element-mesh} and Figure~\vref{fig:Close-up-view-of}
-show an example of a one-chunk mesh centered on the Japan subduction
-zone, applied in the Japan regional waveform simulation \citep{ChTrHeKa07}.
-
-
-\section{One-Chunk Simulations\label{sec:One-Chunk-Simulations}}
-
-For a one-chunk regional simulation the following parameters need
-to be set in the \texttt{Par\_file}:
-
-\begin{description}
-\item [{$\nchunks$}] Must be set to 1.
-\item [{\texttt{ANGULAR\_WIDTH\_XI\_IN\_DEGREES}}] Denotes the width of
-one side of the chunk ($90^{\circ}$or less).
-\item [{\texttt{ANGULAR\_WIDTH\_ETA\_IN\_DEGREES}}] Denotes the width of
-the second side of the chunk ($90^{\circ}$or less). Note that this
-value may be different from \texttt{ANGULAR\_WIDTH\_XI\_IN\_DEGREES}.
-\item [{\texttt{CENTER\_LATITUDE\_IN\_DEGREES}}] Defines the latitude of
-the center of the chunk (degrees).
-\item [{\texttt{CENTER\_LONGITUDE\_IN\_DEGREES}}] Defines the longitude
-of the center of the chunk (degrees).
-\item [{\texttt{GAMMA\_ROTATION\_AZIMUTH}}] Defines the rotation angle
-of the chunk about its center measured counter clockwise from due
-North (degrees). The corners of the mesh are output in \texttt{\small OUTPUT\_FILES/values\_from\_mesher.h}.
-The output corner progression in \texttt{\small OUTPUT\_FILES/values\_from\_mesher.h}
-is bottom left, bottom right, top left, top right. The rotation azimuth
-can be changed in the \texttt{Par\_file} and the corners output \texttt{(}\texttt{\small OUTPUT\_FILES/}~\\
-\texttt{\small values\_from\_mesher.h}\texttt{)} by using{\small{}
-}\texttt{\small xcreate\_header\_file}. It is important to note that
-the mesher or the solver does not need to be run to determine the
-limits of a 1-chunk simulation.
-\item [{$\nexxi$}] The number of spectral elements along the $\xi$ side
-of the chunk. This number \textit{must} be 8~$\times$~a multiple
-of $\nprocxi$ defined below. For a $90^{\circ}$ chunk, we do not
-recommend using $\nexxi$ less than~64 because the curvature of the
-Earth cannot be honored if one uses too few elements, which results
-in inaccurate and unstable simulations.
-\item [{$\nexeta$}] The number of spectral elements along the $\eta$
-side of the chunk. This number \textit{must} be 8~$\times$~a multiple
-of $\nproceta$ defined below. Note that in order to get elements
-that are close to square on the Earth's surface, the following ratios
-should be similar:
-
-\begin{lyxcode}
-$\texttt{ANGULAR\_WIDTH\_XI\_IN\_DEGREES}/\nexxi$
-
-$\texttt{ANGULAR\_WIDTH\_ETA\_IN\_DEGREES}/\nexeta$~
-\end{lyxcode}
-Because of the geometry of the cubed sphere, the option of having
-different values for $\nexxi$ and $\nexeta$ is available only for
-regional simulations when $\nchunks=1$ (1/6th of the sphere).
-
-\item [{$\nprocxi$}] The number of processors or mesh slices along the
-$\xi$ side of the chunk. To accommodate the mesh doubling layers,
-we must have $\nexxi=8\times c\times\nprocxi$, where $c\ge1$ is
-a positive integer. See Table~\ref{table:nex} for various suitable
-choices.
-\item [{$\nproceta$}] The number of processors or slices along the $\eta$
-side of the chunk; we must have $\nexeta=8\times c\times\nproceta$,
-where $c\ge1$ is a positive integer. $\nprocxi$ and $\nproceta$
-must be equal when $\nchunks=6$.
-\end{description}
-%
-\begin{figure}[H]
-\begin{centering}
-\includegraphics[scale=0.65]{figures/fig5a}
-\par\end{centering}
-
-\caption{\textbf{\label{fig:3D-spectral-element-mesh}}S-wave velocity anomalies
-from the global tomographic model s20rts \citep{RiVa00} are superimposed
-on the mesh. For parallel computing purposes, the one-chunk SEM simulation
-is subdivided in terms of 64 slices. The center of the chunk is at
-(38.5$^{\circ}$ N, 137.5$^{\circ}$ E), and the lateral dimensions
-are 30$^{\circ}$ $\times$ 30$^{\circ}$. Two doubling layers are
-indicated at a depth of 25~km (PREM Moho depth) and a depth of about
-1650~km. Shows full view of 25 neighboring slices; see Figure~\ref{fig:Close-up-view-of}
-for close-up of upper mantle mesh.}
-
-\end{figure}
-%
-\begin{figure}[H]
-\begin{centering}
-\includegraphics[scale=0.65]{figures/fig5b}
-\par\end{centering}
-
-\caption{\textbf{\label{fig:Close-up-view-of}}Close-up view of the upper mantle
-mesh shown in Figure~\ref{fig:3D-spectral-element-mesh}. Note that
-the element size in the crust (top layer) is 13~km $\times$ 13~km,
-and that the size of the spectral elements is doubled in the upper
-mantle. The velocity variation is captured by NGLL = 5 grid points
-in each direction of the elements \citep{KoTr02a,KoTr02b}.}
-
-\end{figure}
-
-
-\begin{description}
-\item [{\texttt{ABSORBING\_CONDITIONS}}] Set to \texttt{.true.}{\small{}
-}for regional simulations. For instance, see \citet{KoTr99} for a
-description of the paraxial boundary conditions used. Note that these
-conditions are never perfect, and in particular surface waves may
-partially reflect off the artificial boundaries. Note also that certain
-arrivals, e.g., PKIKPPKIKP, will be missing from the synthetics.
-\end{description}
-When the width of the chunk is different from $90^{\circ}$ (or the
-number of elements is greater than 1248), the radial distribution
-of elements needs to be adjusted as well to maintain spectral elements
-that are as cube-like as possible. The code attempts to do this, but
-be sure to view the mesh with your favorite graphics package to make
-sure that the element are well behaved. Remember: a high-quality mesh is paramount for accurate simulations. In addition to a reorganization of the radial distribution of elements,
-the time stepping and period range in which the attenuation is applied
-is automatically determined. The minimum and maximum periods for attenuation
-are:
-
-\[
-\omega_{max}=\omega_{min}\times10^{W_{3}}\]
-
-
-\noindent where $W_{3}$ is the optimal width in frequency for 3 Standard
-Linear Solids, about 1.75. See \texttt{\small read\_compute\_parameters.f90}
-for more details.
-
-The time stepping is determined in a similar fashion as Equation (48)
-in \citet{KoTr02a}:
-
-\begin{lyxcode}
-dt~=~$S_{c}$~Element~Width~in~km~($r=$ICB)~/~Velocity~($r=$ICB)
-\end{lyxcode}
-where $S_{c}$ is the stability condition (about 0.4). We use the
-radius at the inner core boundary because this is where the maximum
-velocity/element width occurs. Again, see \texttt{\small read\_compute\_parameters}\texttt{.f90}
-for all the details.
-
-The approximate shortest period at which a regional simulation is
-accurate may be determined based upon the relation \begin{equation}
-\mbox{shortest period (s)}\simeq(256/\nexxi)\times(\texttt{ANGULAR\_WIDTH\_XI\_IN\_DEGREES}/90)\times17.\label{eq:shortest_period_regional}\end{equation}
-
-
-
-\section{Two-Chunk Simulations}
-
-For a two-chunk regional simulation the following parameters need
-to be set in the \texttt{Par\_file}:
-
-\begin{description}
-\item [{$\nchunks$}] Must be set to 2
-\item [{\texttt{ANGULAR\_WIDTH\_XI\_IN\_DEGREES}}] Denotes the width of
-one side of the chunk, and it has to be 90 degrees.
-\item [{\texttt{ANGULAR\_WIDTH\_ETA\_IN\_DEGREES}}] Denotes the width of
-the second side of the chunk, and it also has to be 90 degrees.
-\end{description}
-\texttt{NEX\_XI} and \texttt{NEX\_ETA} follow the same description
-in Section \ref{sec:One-Chunk-Simulations}, however, they need to
-be the same in this case. All other parameters are similar to the
-one-chunk simulations, refer to Section \ref{sec:One-Chunk-Simulations}
-for details.
-
-%
-\begin{figure}[H]
-\noindent \begin{centering}
-\includegraphics[width=0.6\textwidth]{figures/2-chunk-surface}
-\par\end{centering}
-
-\caption{Geometry of a 2-chunk simulation, where the first chunk (\texttt{CHUNK\_AB})
-centers at $40^{\circ}$ latitude, $10^{\circ}$ longitude, and has
-been rotated by $20^{\circ}$ counter clockwise, and the second chunk
-(\texttt{CHUNK\_AC}) connects to the first chunk through one face.}
-
-\end{figure}
-
-
-
-\chapter{\label{cha:Adjoint-Simulations}Adjoint Simulations}
-
-Adjoint simulations are generally performed for two distinct applications.
-First, they can be used for earthquake source inversions, especially
-earthquakes with large ruptures such as the Sumatra-Andaman event
-\citep{LayKanamoriAmmon2005,AmmonJiThio2005,ParkSongTromp2005}. Second,
-they can be used to generate finite-frequency sensitivity kernels
-that are a critical part of tomographic inversions based upon 3D reference
-models \citep{trompetal2005,LiTr06,TrKoLi08,LiTr08}. In either case, source
-parameter or velocity structure updates are sought to minimize a specific
-misfit function (e.g., waveform or traveltime differences), and the
-adjoint simulation provides a means of computing the gradient of the
-misfit function and further reducing it in successive iterations.
-Applications and procedures pertaining to source studies and finite-frequency
-kernels are discussed in Sections~\ref{sec:Adjoint-simulation-sources}
-and \ref{sec:Adjoint-simulation-finite}, respectively. The two related
-parameters in the \texttt{Par\_file} are \texttt{SIMULATION\_TYPE}
-(1, 2 or 3) and \texttt{SAVE\_FORWARD} (boolean).
-
-
-\section{\label{sec:Adjoint-simulation-sources}Adjoint Simulations for Sources}
-
-In the case where a specific misfit function is minimized to invert
-for the earthquake source parameters, the gradient of the misfit function
-with respect to these source parameters can be computed by placing
-time-reversed seismograms at the receivers and using them as sources
-in an adjoint simulation, and then the value of the gradient is obtained
-from the adjoint seismograms recorded at the original earthquake location.
-
-\begin{enumerate}
-\item \textbf{Prepare the adjoint sources} \label{enu:Prepare-the-adjoint}
-
-\begin{enumerate}
-\item First, run a regular forward simlation (\texttt{SIMULATION\_TYPE =
-1} and \texttt{SAVE\_FORWARD = .false.}). You can automatically set
-these two variables using the \texttt{\small UTILS/change\_simulation\_type.pl}
-script:
-
-\begin{lyxcode}
-UTILS/change\_simulation\_type.pl~-f~
-\end{lyxcode}
-and then collect the recorded seismograms at all the stations given
-in \texttt{DATA/STATIONS}.
-
-\item Then select the stations for which you want to compute the time-reversed
-adjoint sources and run the adjoint simulation, and compile them into
-the \texttt{DATA/STATIONS\_ADJOINT} file, which has the same format
-as the regular \texttt{DATA/STATIONS} file.
-
-\begin{itemize}
-\item Depending on what type of misfit function is used for the source inversion,
-adjoint sources need to be computed from the original recorded seismograms
-for the selected stations and saved in the \texttt{SEM/} directory
-with the format \texttt{STA.NT.LH?.adj}, where \texttt{STA}, \texttt{NT}
-are the station name and network code given in the \texttt{DATA/STATIONS\_ADJOINT}
-file, and \texttt{LH?} represents the component name of a particular
-adjoint seismogram.
-\item The adjoint seismograms are in the same format as the original seismogram
-(\texttt{STA.NT.LH?.sem?}), with the same start time, time interval
-and record length.
-\end{itemize}
-\item Notice that even if you choose to time reverse only one component
-from one specific station, you still need to supply all three components
-because the code is expecting them (you can set the other two components
-to be zero).
-\item Also note that since time-reversal is done in the code itself, no
-explicit time-reversing is needed for the preparation of the adjoint
-sources, i.e., the adjoint sources are in the same forward time sense
-as the original recorded seismograms.
-\end{enumerate}
-\item \textbf{Set the related parameters and run the adjoint simulation}\\
-In the \texttt{DATA/Par\_file}, set the two related parameters to
-be \texttt{SIMULATION\_TYPE = 2} and \texttt{SAVE\_FORWARD = .false.}.
-More conveniently, use the scripts \texttt{UTILS/change\_simulation\_type.pl}
-to modify the \texttt{Par\_file} automatically (\texttt{change\_simulation\_type.pl
--a}). Then run the solver to launch the adjoint simulation.
-\item \textbf{Collect the seismograms at the original source location}
-
-
-After the adjoint simulation has completed successfully, get the seismograms
-from directory \texttt{OUTPUT\_FILES}.
-
-\begin{itemize}
-\item These adjoint seismograms are recorded at the locations of the original
-earthquake sources given by the \texttt{DATA/CMTSOLUTION} file, and
-have names of the form \texttt{S?????.NT.S??.sem} for the six-component
-strain tensor (\texttt{SNN,SEE,SZZ,SNE,SNZ,SEZ}) at these locations,
-and \texttt{S?????.NT.LH?.sem} for the three-component displacements
-(\texttt{LHN,LHE,LHZ}) recorded at these locations.
-\item \texttt{S?????} denotes the source number; for example, if the original
-\texttt{CMTSOLUTION} provides only a point source, then the seismograms
-collected will start with \texttt{S00001}.
-\item These adjoint seismograms provide critical information for the computation
-of the gradient of the misfit function.
-\end{itemize}
-\end{enumerate}
-
-\section{\label{sec:Adjoint-simulation-finite}Adjoint Simulations for Finite-Frequency
-Kernels (Kernel Simulation)}
-
-Finite-frequency sensitivity kernels are computed in two successive
-simulations (please refer to \citet{LiTr06} and \citet{TrKoLi08} for details).
-
-\begin{enumerate}
-\item \textbf{Run a forward simulation with the state variables saved at
-the end of the simulation}
-
-
-Prepare the \texttt{\small CMTSOLUTION} and \texttt{\small STATIONS}
-files, set the parameters \texttt{\small SIMULATION\_TYPE}{\small{}
-}\texttt{\small =}{\small{} }\texttt{\small 1} and \texttt{\small SAVE\_FORWARD
-=}{\small{} }\texttt{\small .true.} in the \texttt{Par\_file} (\texttt{change\_simulation\_type
--F}), and run the solver.
-
-\begin{itemize}
-\item Notice that attenuation is not implemented yet for the computation
-of finite-frequency kernels; therefore set \texttt{ATTENUATION = .false.}
-in the \texttt{Par\_file}.
-\item We also suggest you modify the half duration of the \texttt{CMTSOLUTION}
-to be similar to the accuracy of the simulation (see Equation \ref{eq:shortest_period}
-or \ref{eq:shortest_period_regional}) to avoid too much high-frequency
-noise in the forward wavefield, although theoretically the high-frequency
-noise should be eliminated when convolved with an adjoint wavefield
-with the proper frequency content.
-\item This forward simulation differs from the regular simulations (\texttt{\small SIMULATION\_TYPE}{\small{}
-}\texttt{\small =}{\small{} }\texttt{\small 1} and \texttt{\small SAVE\_FORWARD}{\small{}
-}\texttt{\small =}{\small{} }\texttt{\small .false.}) described in
-the previous chapters in that the state variables for the last time
-step of the simulation, including wavefields of the displacement,
-velocity, acceleration, etc., are saved to the \texttt{LOCAL\_PATH}
-to be used for the subsequent simulation.
-\item For regional simulations, the files recording the absorbing boundary
-contribution are also written to the \texttt{LOCAL\_PATH} when \texttt{SAVE\_FORWARD
-= .true.}.
-\end{itemize}
-\item \textbf{Prepare the adjoint sources}
-
-
-The adjoint sources need to be prepared the same way as described
-in Section~\ref{sec:Adjoint-simulation-sources}, item~\ref{enu:Prepare-the-adjoint}.
-
-\begin{itemize}
-\item In the case of travel-time finite-frequency kernel for one source-receiver
-pair, i.e., point source from the \texttt{CMTSOLUTION}, and one station
-in the \texttt{STATIONS\_ADJOINT} list, we supply a sample program
-in \texttt{UTILS/cut\_velocity} to cut a certain portion of the original
-displacement seismograms and convert them into the proper adjoint
-source to compute the finite-frequency kernel.
-
-\begin{lyxcode}
-cut\_velocity~t1~t2~ifile{[}0-5]~E/N/Z-ascii-files~{[}baz]
-\end{lyxcode}
-where \texttt{t1} and \texttt{t2} are the start and end time of the
-portion you are interested in, \texttt{ifile} denotes the component
-of the seismograms to be used (0 for all three components, 1 for east,
-2 for north, and 3 for vertical, 4 for transverse, and 5 for radial
-component), \texttt{E/N/Z-ascii-files} indicate the three-component
-displacement seismograms in the right order, and \texttt{baz} is the
-back-azimuth of the station from the event location. Note that \texttt{baz}
-is only supplied when \texttt{ifile} = 4 or 5.
-
-\end{itemize}
-\item \textbf{Run the kernel simulation}
-
-
-With the successful forward simulation and the adjoint source ready
-in \texttt{SEM/}, set \texttt{SIMULATION\_TYPE = 3} and \texttt{SAVE\_FORWARD
-= .false.} in the \texttt{Par\_file(change\_simulation\_type.pl -b)},
-and rerun the solver.
-
-\begin{itemize}
-\item The adjoint simulation is launched together with the back reconstruction
-of the original forward wavefield from the state variables saved from
-the previous forward simulation, and the finite-frequency kernels
-are computed by the interaction of the reconstructed forward wavefield
-and the adjoint wavefield.
-\item The back-reconstructed seismograms at the original station locations
-are saved to the \texttt{OUTPUT\_FILES} directory at the end of the
-kernel simulations.
-\item These back-constructed seismograms can be compared with the time-reversed
-original seismograms to assess the accuracy of the backward reconstruction,
-and they should match very well (in the time-reversed sense).
-\item The files containing the density, P-wave speed and S-wave speed kernels
-are saved in the \texttt{LOCAL\_PATH} with the names of \texttt{proc??????\_reg\_?\_rho(alpha,beta)\_kernel.bin},
-where \texttt{proc??????} represents the processor number, and \texttt{reg\_?}
-denotes the region these kernels are for, including mantle (\texttt{reg\_1}),
-outer core (\texttt{reg\_2}), and inner core (\texttt{reg\_3}). The
-output kernels are in the unit of $s/km^{3}$.
-\end{itemize}
-\item \textbf{Run the boundary kernel simulation}
-
-
-\noindent If you set the \texttt{SAVE\_BOUNDARY\_MESH = .true.} in
-the \texttt{constants.h} file before the simulations, i.e., at the
-beginning of step 1, you will get not only the volumetric kernels
-as described in step 3, but also boundary kernels for the Earth's
-internal discontinuities, such as Moho, 410-km discontinuity, 670-km
-discontinuity, CMB and ICB. These kernel files are also saved in the
-local scratch directory defined by \texttt{LOCAL\_PATH }and have names
-such as \texttt{proc??????\_reg\_1(2)\_Moho(d400,d670,CMB,ICB)\_kernel.bin}.
-For a theoretical derivation of the boundary kernels, refer to \citet{trompetal2005},
-and for the visualization of the boundary kernels, refer to Section
-\ref{sec:Finite-Frequency-Kernels}.
-
-\item \textbf{Run the anisotropic kernel simulation}
-
-
-Instead of the kernels for the isotropic wave speeds, you can also
-compute the kernels for the 21 independent components $C_{IJ},\, I,J=1,...,6$
-(using Voigt's notation) of the elastic tensor in the (spherical)
-geographical coordinate system. This is done by setting \texttt{ANISOTROPIC\_KL}
-\texttt{=} \texttt{.true.} in \texttt{constants.h} before step 3.
-The definition of the parameters $C_{IJ}$ in terms of the corresponding
-components $c_{ijkl},ijkl,i,j,k,l=1,2,3$ of the elastic tensor in
-spherical coordinates follows \citet{ChTr07}. The computation of
-the anisotropic kernels is only implemented in the crust and mantle
-regions. The 21 anisotropic kernels are saved in the \texttt{LOCAL\_PATH}
-in one file with the name of \texttt{proc??????\_reg1\_cijkl\_kernel.bin}
-(with \texttt{proc??????} the processor number). The output kernels
-correspond to perturbation $\delta C_{IJ}$ of the elastic parameters
-and their unit is in $s/GPa/km^{3}$. For consistency, the output
-density kernels with this option turned on are for a perturbation
-$\delta\rho$ (and not $\frac{\delta\rho}{\rho}$) and their unit
-is in s / (kg/m$^{3}$) / km$^{3}$. These `primary' anisotropic kernels
-can then be combined to obtain the kernels related to other descriptions
-of anisotropy. This can be done, for example, when combining the kernel
-files from slices into one mesh file (see Section~\ref{sec:Finite-Frequency-Kernels}).
-
-\end{enumerate}
-In general, the first three steps need to be run sequentially to ensure
-proper access to the necessary files at different stages. If the simulations
-are run through some cluster scheduling system (e.g., LSF), and the
-forward simulation and the subsequent kernel simulations cannot be
-assigned to the same set of computer nodes, the kernel simulation
-will not be able to access the database files saved by the forward
-simulation. Solutions for this problem are provided in Chapter~\ref{cha:Running-Scheduler}.
-Visualization of the finite-frequency kernels is discussed in Section~\ref{sec:Finite-Frequency-Kernels}.
-
-
-\chapter{Graphics}
-
-
-\section{\label{sec:Meshes}Meshes}
-
-Use the serial code \texttt{combine\_AVS\_DX.f90} (type `\texttt{make
-combine\_AVS\_DX}' and then `\texttt{xcombine\_AVS\_DX}') to generate
-AVS \url{www.avs.com} output files (in AVS UCD format) or OpenDX \url{www.opendx.org}
-output files showing the mesh, the MPI partition (slices), the $\nchunks$
-chunks, the source and receiver location, etc. Use the AVS UCD files
-\texttt{AVS\_continent\_boundaries.inp} and \texttt{AVS\_plate\_boundaries.inp}
-or the OpenDX files \texttt{DX\_continent\_boundaries.dx} and \texttt{DX\_plate\_boundaries.dx}
-(that can be created using Perl scripts located in \texttt{UTILS/Visualization/opendx\_AVS})
-for reference.
-
-
-\section{\label{sec:Movies}Movies}
-
-To make a surface or volume movie of the simulation, set parameters
-\texttt{MOVIE\_SURFACE}, \texttt{MOVIE\_VOLUME}, and \texttt{NTSTEP\_BETWEEN\_FRAMES}
-in the \texttt{Par\_file}. Turning on the movie flags, in particular
-\texttt{MOVIE\_VOLUME}, produces large output files. \texttt{MOVIE\_VOLUME}
-files are saved in the \texttt{LOCAL\_PATH} directory, whereas \texttt{MOVIE\_SURFACE}
-output files are saved in the \texttt{OUTPUT\_FILES} directory. We
-save the velocity field. The look of a movie is determined by the
-half-duration of the source. The half-duration should be large enough
-so that the movie does not contain frequencies that are not resolved
-by the mesh, i.e., it should not contain numerical noise. This can
-be accomplished by selecting a CMT \texttt{HALF\_DURATION} > 1.1 $\times$
-smallest period (see figure \ref{fig:CMTSOLUTION-file}). When \texttt{\small MOVIE\_SURFACE}
-= \texttt{\small .true.} or \texttt{\small MOVIE\_VOLUME}{\small{}
-}\texttt{\small =}{\small{} }\texttt{\small .true.}, the half duration
-of each source in the \texttt{CMTSOLUTION} file is replaced by
-
-\begin{quote}
-\[
-\sqrt{(}\mathrm{\mathtt{HALF\_DURATIO}\mathtt{N}^{2}}+\mathrm{\mathtt{HDUR\_MOVI}\mathtt{E}^{2}})\]
-\textbf{NOTE:} If \texttt{HDUR\_MOVIE} is set to 0.0, the code will
-select the appropriate value of 1.1 $\times$ smallest period. As
-usual, for a point source one can set \texttt{HALF\_DURATION} in the
-\texttt{Par\_file} to be 0.0 and \texttt{HDUR\_MOVIE} = 0.0 to get
-the highest frequencies resolved by the simulation, but for a finite
-source one would keep all the \texttt{HALF\_DURATION}s as prescribed
-by the finite source model and set \texttt{HDUR\_MOVIE} = 0.0.
-\end{quote}
-
-\subsection{Movie Surface}
-
-When running \texttt{xspecfem3D} with the \texttt{MOVIE\_SURFACE}
-flag turned on the code outputs \texttt{moviedata??????} files in
-the \texttt{OUTPUT\_FILES} directory. The files are in a fairly complicated
-binary format, but there are two programs provided to convert the
-output into more user friendly formats. The first one, \texttt{create\_movie\_AVS\_DX.f90}
-outputs data in ASCII, OpenDX, AVS, or ParaView format. Run the code
-from the source directory (type `\texttt{make} \texttt{create\_movie\_AVS\_DX}'
-first) to create an input file in your format of choice. The code
-will prompt the user for input parameters. The second program \texttt{create\_movie\_GMT\_global.f90}
-outputs ASCII xyz files, convenient for use with GMT. This codes uses
-significantly less memory than \texttt{create\_movie\_AVS\_DX.f90}
-and is therefore useful for high resolution runs.
-
-%
-\begin{figure}[H]
-\noindent \begin{centering}
-\includegraphics[scale=0.75]{figures/geo-poster4_small}
-\par\end{centering}
-
-\caption{Snapshots from a global movie for the December 26, 2004, M=9.2 Sumatra-Andaman
-earthquake. Time runs down successive columns.}
-
-\end{figure}
-
-
-
-\subsection{\label{sub:Movie-Volume}Movie Volume}
-
-When running xspecfem3D with the \texttt{\small MOVIE\_VOLUME} flag
-turned on, the code outputs several files in \texttt{\small LOCAL\_DIR}.
-As the files can be very large, there are several flags in the \texttt{\small Par\_file}
-that control the region in space and time that is saved. These are:
-\texttt{\small MOVIE\_TOP\_KM}, \texttt{\small MOVIE\_BOTTOM\_KM},
-\texttt{\small MOVIE\_WEST\_DEG}, \texttt{\small MOVIE\_EAST\_DEG},
-\texttt{\small MOVIE\_NORTH\_DEG}, \texttt{\small MOVIE\_SOUTH\_DEG},
-\texttt{\small MOVIE\_START} and \texttt{\small MOVIE\_STOP}. The
-code will save a given element if the center of the element is in
-the prescribed volume.
-
-\begin{description}
-\item [{The~Top/Bottom:}] Depth below the surface in kilometers, use \texttt{\small MOVIE\_TOP}
-\texttt{\small =} \texttt{\small -100.0} to make sure the surface
-is stored.
-\item [{West/East:}] Longitude, degrees East {[}-180.0/180.0]
-\item [{North/South:}] Latitute, degrees North {[}-90.0/90.0]
-\item [{Start/Stop:}] Frames will be stored at \texttt{\small MOVIE\_START}
-\texttt{\small +} \texttt{\small i{*}NSTEP\_BETWEEN\_FRAMES}, where
-\texttt{\small i=(0,1,2..)} while \texttt{\small i{*}NSTEP\_BETWEEN\_FRAMES}
-\texttt{\small <=} \texttt{\small MOVIE\_STOP}{\small \par}
-\end{description}
-The code saves several files, and the output is saved by each processor.
-The first is \texttt{\small proc??????\_movie3D\_info.txt} which contains
-two numbers, first the number of points within the prescribed volume
-within this particular slice, and second the number of elements. The
-next files are \texttt{\small proc??????\_movie3D\_x.bin}, \texttt{\small proc??????\_movie3D\_y.bin},
-\texttt{\small proc??????\_movie3D\_z.bin} which store the locations
-of the points in the 3D mesh.
-
-Finally the code stores the ``value'' at each of the points. Which
-value is determined by \texttt{\small MOVIE\_VOLUME\_TYPE} in the
-\texttt{\small Par\_file}. Choose 1 to save the strain, 2 to save
-the time integral of strain, and 3 to save $\mu${*}time integral
-of strain in the subvolume. Choosing 4 causes the code to save the
-trace of the stress and the deviatoric stress in the whole volume
-(not the subvolume in space), at the time steps specified. The name
-of the output file will depend on the \texttt{\small MOVIE\_VOLUME\_TYPE}
-chosen.
-
-Setting \texttt{\small MOVIE\_VOLUME\_COARSE} \texttt{\small =} \texttt{\small .true.}
-will make the code save only the corners of the elements, not all
-the points within each element for \texttt{\small MOVIE\_VOLUME\_TYPE}
-\texttt{\small =} \texttt{\small 1,2,3}.
-
-To make the code output your favorite ``value'' simply add a new \texttt{\small MOVIE\_VOLUME\_TYPE},
-a new subroutine to \texttt{\small write\_movie\_volume.f90} and a
-subroutine call to \texttt{\small specfem3D.f90}.
-
-A utility program to combine the files produced by \texttt{\small MOVIE\_VOLUME\_TYPE}
-\texttt{\small =} \texttt{\small 1,2,3} is provided in \texttt{\small combine\_paraview}~\\
-\texttt{\small \_strain\_data.f90}. Type \texttt{\small xcombine\_paraview\_strain\_data}
-to get the usage statement. The program \texttt{\small combine\_vol}~\\
-\texttt{\small \_data.f90} can be used for \texttt{\small MOVIE\_VOLUME\_TYPE}
-\texttt{\small =} \texttt{\small 4}.
-
-
-\section{\label{sec:Finite-Frequency-Kernels}Finite-Frequency Kernels}
-
-The finite-frequency kernels computed as explained in Section \ref{sec:Adjoint-simulation-finite}
-are saved in the \texttt{LOCAL\_PATH} at the end of the simulation.
-Therefore, we first need to collect these files on the front end,
-combine them into one mesh file, and visualize them with some auxilliary
-programs.
-
-\begin{enumerate}
-\item \textbf{Create slice files}
-
-
-We will only discuss the case of one source-receiver pair, i.e., the
-so-called banana-doughnut kernels. Although it is possible to collect
-the kernel files from all slices onto the front end, it usually takes
-up too much storage space (at least tens of gigabytes). Since the
-sensitivity kernels are the strongest along the source-receiver great
-circle path, it is sufficient to collect only the slices that are
-along or close to the great circle path.
-
-A Perl script \texttt{UTILS/Visualization/Paraview/global\_slice\_number.pl} can
-help to figure out the slice numbers that lie along the great circle
-path (both the minor and major arcs), as well as the slice numbers
-required to produce a full picture of the inner core if your kernel
-also illuminates the inner core.
-
-\begin{enumerate}
-\item You need to first compile the utility programs provided in the \texttt{UTILS/Visualization/Paraview/}~\\
-\texttt{global\_slice\_util
-}directory. Then copy the \texttt{CMTSOLUTION} file, \texttt{STATIONS\_ADJOINT},
-and \texttt{Par\_file}, and run:
-
-\begin{lyxcode}
-global\_slice\_number.pl~CMTSOLUTION~STATIONS\_ADJOINT~Par\_file
-\end{lyxcode}
-In the case of visualization boundary kernels or spherical cross-sections
-of the volumetric kernels, it is necessary to obtain the slice numbers
-that cover a belt along the source and receiver great circle path,
-and you can use the hybrid version:
-
-\begin{lyxcode}
-globe\_slice\_number2.pl~CMTSOLUTION~STATIONS\_ADJOINT~
-
-~~~~~Par\_file~belt\_width\_in\_degrees
-\end{lyxcode}
-A typical value for \texttt{belt\_width\_in\_degrees} can be 20.
-
-\item For a full 6-chunk simulation, this script will generate the \texttt{slice\_minor},
-\texttt{slice\_major}, \texttt{slice\_ic} files, but for a one- or
-two-chunk simulation, this script only generates the \texttt{slice\_minor}
-file.
-\item For cases with multiple sources and multiple receivers, you need to
-provide a slice file before proceeding to the next step.
-\end{enumerate}
-\item \textbf{Collect the kernel files}
-
-
-After obtaining the slice files, you can collect the corresponding
-kernel files from the given slices.
-
-\begin{enumerate}
-\item To accomplish this, you can use or modify the scripts in \texttt{UTILS/collect\_database
-}directory:
-
-\begin{lyxcode}
-{\small copy\_m(oc,ic)\_globe\_database.pl~slice\_file~lsf\_machine\_file~filename~{[}jobid]}{\small \par}
-\end{lyxcode}
-for volumetric kernels, where \texttt{\small lsf\_machine\_file} is
-the machine file generated by the LSF scheduler, \texttt{\small filename}
-is the kernel name (e.g., \texttt{\small rho\_kernel}, \texttt{\small alpha\_kernel}
-and \texttt{\small beta\_kernel}), and the optional \texttt{\small jobid}
-is the name of the subdirectory under \texttt{\small LOCAL\_PATH}
-where all the kernel files are stored. For boundary kernels, you need
-to use
-
-\begin{lyxcode}
-{\small copy\_surf\_globe\_database.pl~slice\_file~lsf\_machine\_file~filename~{[}jobid]}{\small \par}
-\end{lyxcode}
-where the filename can be \texttt{\small Moho\_kernel}, \texttt{\small d400\_kernel},
-\texttt{\small d670\_kernel}, \texttt{\small CMB\_kernel} and \texttt{\small ICB\_kernel}.
-
-\item After executing this script, all the necessary mesh topology files
-as well as the kernel array files are collected to the local directory
-on the front end.
-\end{enumerate}
-\item \textbf{Combine kernel files into one mesh file}
-
-
-We use an auxiliary program \texttt{combine\_vol\_data.f90} to combine
-the volumetric kernel files from all slices into one mesh file, and
-\texttt{combine\_surf\_data.f90 }to combine the surface kernel files.
-
-\begin{enumerate}
-\item Compile it in the global code directory:
-
-\begin{lyxcode}
-{\footnotesize make~combine\_vol\_data~}{\footnotesize \par}
-
-{\footnotesize xcombine\_vol\_data}~{\footnotesize slice\_list}~{\footnotesize filename}~{\footnotesize input\_dir}~{\footnotesize output\_dir}~{\footnotesize high/low-resolution}~{\footnotesize {[}region]}{\footnotesize \par}
-\end{lyxcode}
-where \texttt{input\_dir} is the directory where all the individual
-kernel files are stored, and \texttt{output\_dir} is where the mesh
-file will be written.
-
-\begin{lyxcode}
-{\footnotesize xcombine\_surf\_data~slice\_list~filename~surfname~input\_dir~output\_dir~hi~gh/low-resolution~2D/3D}{\footnotesize \par}
-\end{lyxcode}
-where \texttt{surfname} should correspond to the specific kernel file
-name, and can be chosen from \texttt{Moho}, \texttt{400}, \texttt{670},
-\texttt{CMB} and \texttt{ICB}.
-
-\item Use 1 for a high-resolution mesh, outputting all the GLL points to
-the mesh file, or use 0 for low resolution, outputting only the corner
-points of the elements to the mesh file. Use 0 for 2D surface kernel
-files and 1 for 3D volumetric kernel files.
-\item Use region = 1 for the mantle, region =2 for the outer core, region
-= 3 for the inner core, and region = 0 for all regions.
-\item The output mesh file will have the name \texttt{reg\_?\_rho(alpha,beta)\_kernel.mesh,}
-or\texttt{ }~\\
-\texttt{reg\_?\_Moho(d400,d670,CMB,ICB)\_kernel.surf.}
-\end{enumerate}
-\item \textbf{Convert mesh files into .vtu files}
-
-\begin{enumerate}
-\item We next convert the \texttt{.mesh} file into the VTU (Unstructured
-grid file) format which can be viewed in ParaView, for example:
-
-\begin{lyxcode}
-mesh2vtu.pl~-i~file.mesh~-o~file.vtu
-\end{lyxcode}
-\item Notice that this Perl script uses a program \texttt{mesh2vtu} in the
-\texttt{UTILS/Visualization/Paraview/}~\\
-\texttt{mesh2vtu} directory, which further uses the
-VTK \url{www.vtk.org} run-time library for its execution. Therefore,
-make sure you have them properly set in the script.
-\end{enumerate}
-\item \textbf{Copy over the source and receiver .vtk file}
-
-
-In the case of a single source and a single receiver, the simulation
-also generates the \texttt{OUTPUT\_FILES/sr.vtk} file to describe
-the source and receiver locations, which can be viewed in Paraview
-in the next step.
-
-\item \textbf{View the mesh in ParaView}
-
-
-Finally, we can view the mesh in ParaView \url{www.paraview.org}.
-
-\begin{enumerate}
-\item Open ParaView.
-\item From the top menu, \textsf{File} $\rightarrow$\textsf{ Open data},
-select \texttt{file.vtu}, and click the \textsf{Accept} button.
-
-\begin{itemize}
-\item If the mesh file is of moderate size, it shows up on the screen; otherwise,
-only the outline is shown.
-\end{itemize}
-\item Click \textsf{Display Tab} $\rightarrow$ \textsf{Display Style} $\rightarrow$
-\textsf{Representation} and select \textsf{wireframe of surface} to
-display it.
-\item To create a cross-section of the volumetric mesh, choose \textsf{Filter}
-$\rightarrow$ \textsf{cut}, and under \textsf{Parameters Tab}, choose
-\textsf{Cut Function} $\rightarrow$ \textsf{plane}.
-\item Fill in center and normal information given by the standard output
-from \texttt{global\_slice\_number.pl} script.
-\item To change the color scale, go to \textsf{Display Tab} $\rightarrow$
-\textsf{Color} $\rightarrow$ \textsf{Edit Color Map} and reselect
-lower and upper limits, or change the color scheme.
-\item Now load in the source and receiver location file by \textsf{File}
-$\rightarrow$\textsf{ Open data}, select \texttt{sr.vt}k, and click
-the \textsf{Accept} button. Choose \textsf{Filter} $\rightarrow$\textsf{
-Glyph}, and represent the points by `\textsf{spheres}'.
-\item For more information about ParaView, see the ParaView Users Guide \url{www.paraview.org/files/v1.6/ParaViewUsersGuide.PDF}.
-\end{enumerate}
-\end{enumerate}
-For illustration purposes, Figure \ref{fig:P-wave-speed-finite-frequency}
-shows the P-wave speed finite-frequency kernel for a P arrival recorded
-at an epicental distance of $60^{\circ}$ for a deep event.
-
-%
-\begin{figure}[H]
-\includegraphics[clip,scale=0.9]{figures/P_alpha_60d_18s}
-
-\caption{\label{fig:P-wave-speed-finite-frequency}P-wave speed finite-frequency
-kernel for a P arrival recorded at an epicentral distance of $60^{\circ}$.}
-
-\end{figure}
-
-
-
-\chapter{\label{cha:Running-Scheduler}Running through a Scheduler}
-
-The code is usually run on large parallel machines, often PC clusters,
-most of which use schedulers, i.e., queuing or batch management systems
-to manage the running of jobs from a large number of users. The following
-considerations need to be taken into account when running on a system
-that uses a scheduler:
-
-\begin{itemize}
-\item The processors/nodes to be used for each run are assigned dynamically
-by the scheduler, based on availability. Therefore, in order for the
-mesher and the solver (or between successive runs of the solver) to
-have access to the same database files (if they are stored on hard
-drives local to the nodes on which the code is run), they must be
-launched in sequence as a single job.
-\item On some systems, the nodes to which running jobs are assigned are
-not configured for compilation. It may therefore be necessary to pre-compile
-both the mesher and the solver. A small program provided in the distribution
-called \texttt{\small create\_header\_file.f90} can be used to directly
-create\texttt{\small{} OUTPUT\_FILES/values\_from\_mesher.h} using
-the information in the \texttt{\small DATA/Par\_file} without having
-to run the mesher (type \texttt{\small `make}{\small{} }\texttt{\small create\_header\_}~\\
-\texttt{\small file}' to compile it and `\texttt{\small xcreate\_header\_file}'
-to run it; refer to the sample scripts below). The solver can now
-be compiled as explained above.
-\item One feature of schedulers/queuing systems is that they allow submission
-of multiple jobs in a {}``launch and forget'' mode. In order to
-take advantage of this property, care needs to be taken that output
-and intermediate files from separate jobs do not overwrite each other,
-or otherwise interfere with other running jobs.
-\end{itemize}
-We describe here in some detail a job submission procedure for the
-Caltech 1024-node cluster, CITerra, under the LSF scheduling system.
-We consider the submission of a regular forward simulation. The two
-main scripts are \texttt{\small run\_lsf.bash}, which compiles the
-Fortran code and submits the job to the scheduler, and \texttt{\small go\_mesher\_solver\_lsf}~\\
-\texttt{\small .bash}, which contains the instructions that make up
-the job itself. These scripts can be found in \texttt{\small UTILS/Cluster/lsf}
-directory and can straightforwardly be modified and adapted to meet
-more specific running needs.
-
-
-\section{\texttt{run\_lsf.bash}}
-
-This script first sets the job queue to be `normal'. It then compiles
-the mesher and solver together, figures out the number of processors
-required for this simulation from \texttt{DATA/Par\_file}, and submits
-the LSF job.
-
-\begin{lyxcode}
-\#!/bin/bash
-
-\#~use~the~normal~queue~unless~otherwise~directed~queue=\char`\"{}-q~normal\char`\"{}~
-
-if~{[}~\$\#~-eq~1~];~then
-
-~~~~~~~~echo~\char`\"{}Setting~the~queue~to~\$1\char`\"{}
-
-~~~~~~~~queue=\char`\"{}-q~\$1\char`\"{}~
-
-fi~\\
-~\\
-\#~compile~the~mesher~and~the~solver~
-
-d=`date'~echo~\char`\"{}Starting~compilation~\$d\char`\"{}~
-
-make~clean~
-
-make~meshfem3D~
-
-make~create\_header\_file~
-
-xcreate\_header\_file~
-
-make~specfem3D~
-
-d=`date'~
-
-echo~\char`\"{}Finished~compilation~\$d\char`\"{}~\\
-~\\
-\#~compute~total~number~of~nodes~needed~
-
-NPROC\_XI=`grep~NPROC\_XI~DATA/Par\_file~|~cut~-c~34-~'~
-
-NPROC\_ETA=`grep~NPROC\_ETA~DATA/Par\_file~|~cut~-c~34-~'~
-
-NCHUNKS=`grep~NCHUNKS~DATA/Par\_file~|~cut~-c~34-~'~\\
-~\\
-\#~total~number~of~nodes~is~the~product~of~the~values~read~
-
-numnodes=\$((~\$NCHUNKS~{*}~\$NPROC\_XI~{*}~\$NPROC\_ETA~))~\\
-~\\
-echo~\char`\"{}Submitting~job\char`\"{}~
-
-bsub~\$queue~-n~\$numnodes~-W~60~-K~<go\_mesher\_solver\_lsf\_globe.bash~
-\end{lyxcode}
-
-\section{\texttt{go\_mesher\_solver\_lsf\_globe.bash}}
-
-This script describes the job itself, including setup steps that can
-only be done once the scheduler has assigned a job-ID and a set of
-compute nodes to the job, the \texttt{run\_lsf.bash} commands used
-to run the mesher and the solver, and calls to scripts that collect
-the output seismograms from the compute nodes and perform clean-up
-operations.
-
-\begin{enumerate}
-\item First the script directs the scheduler to save its own output and
-output from \texttt{stdout} into \texttt{\small OUTPUT\_FILES/\%J.o},
-where \texttt{\%J} is short-hand for the job-ID; it also tells the
-scheduler what version of \texttt{mpich} to use (\texttt{mpich\_gm})
-and how to name this job (\texttt{go\_mesher\_solver\_lsf}).
-\item The script then creates a list of the nodes allocated to this job
-by echoing the value of a dynamically set environment variable \texttt{LSB\_MCPU\_HOSTS}
-and parsing the output into a one-column list using the Perl script
-\texttt{UTILS/Cluster/lsf/remap\_lsf\_machines.pl}. It then creates a set of scratch
-directories on these nodes (\texttt{\small /scratch/}~\\
-\texttt{\small \$USER/DATABASES\_MPI}) to be used as the \texttt{LOCAL\_PATH}
-for temporary storage of the database files. The scratch directories
-are created using \texttt{shmux}, a shell multiplexor that can execute
-the same commands on many hosts in parallel. \texttt{shmux} is available
-from Shmux \url{web.taranis.org/shmux/}. Make sure that the \texttt{LOCAL\_PATH}
-parameter in \texttt{DATA/Par\_file} is also set properly.
-\item The next portion of the script launches the mesher and then the solver
-using \texttt{run\_lsf.bash}.
-\item The final portion of the script performs clean up on the nodes using
-the Perl script \texttt{cleanmulti.pl}
-\end{enumerate}
-\begin{lyxcode}
-\#!/bin/bash~-v
-
-\#BSUB~-o~OUTPUT\_FILES/\%J.o
-
-\#BSUB~-a~mpich\_gm
-
-\#BSUB~-J~go\_mesher\_solver\_lsf
-
-BASEMPIDIR=/scratch/\$USER/DATABASES\_MPI
-
-echo~\char`\"{}\$LSB\_MCPU\_HOSTS\char`\"{}~>~OUTPUT\_FILES/lsf\_machines
-
-echo~\char`\"{}\$LSB\_JOBID\char`\"{}~>~OUTPUT\_FILES/jobid
-
-./remap\_lsf\_machines.pl~OUTPUT\_FILES/lsf\_machines~>OUTPUT\_FILES/machines
-
-\#~Modif~:~create~a~directory~for~this~job
-
-shmux~-M50~-Sall~-c~\char`\"{}mkdir~-p~/scratch/\$USER;
-
-mkdir~-p~\$BASEMPIDIR.\$LSB\_JOBID\char`\"{}~-~<~OUTPUT\_FILES/machines~>/dev/null
-
-\#~Set~the~local~path~in~Par\_file
-
-sed~-e~\char`\"{}s:\textasciicircum{}LOCAL\_PATH~.{*}:LOCAL\_PATH~=~\$BASEMPIDIR.\$LSB\_JOBID:\char`\"{}
-
-<~DATA/Par\_file~>~DATA/Par\_file.tmp
-
-mv~DATA/Par\_file.tmp~DATA/Par\_file
-
-current\_pwd=\$PWD
-
-mpirun.lsf~~-{}-gm-no-shmem~-{}-gm-copy-env~\$current\_pwd/xmeshfem3D
-
-mpirun.lsf~-{}-gm-no-shmem~-{}-gm-copy-env~\$current\_pwd/xspecfem3D
-
-\#~clean~up
-
-cleanbase\_jobid.pl~OUTPUT\_FILES/machines~DATA/Par\_file
-\end{lyxcode}
-
-\section{\texttt{run\_lsf.kernel} and \texttt{go\_mesher\_solver\_globe.kernel}}
-
-For kernel simulations, you can use the sample run scripts \texttt{run\_lsf.kernel}
-and \texttt{go\_mesher\_solver\_globe}~\\
-\texttt{.kernel} provided in \texttt{UTILS/Cluster} directory, and modify
-the command-line arguments of \texttt{xcut\_velocity} in \texttt{go\_mesher\_}~\\
-\texttt{solver\_globe.kernel }according to the start and end time
-of the specific portion of the forward seismograms you are interested
-in.
-
-
-\chapter{{\normalsize \label{cha:-Changing-the}} Changing the Model}
-
-In this section we explain how to change the crustal, mantle, or inner
-core models. These changes involve contributing specific subroutines
-that replace existing subroutines in the \texttt{SPECFEM3D\_GLOBE}
-package.
-
-
-\section{{\normalsize \label{sec:Changing-the-Crustal}}Changing the Crustal
-Model}
-
-The 3D crustal model Crust2.0 \citep{BaLaMa00} is superimposed onto
-the mesh by the subroutine \texttt{model\_crust}~\\
-\texttt{.f90}. To accomplish this, the flag \texttt{CRUSTAL}, set
-in the subroutine \texttt{get\_model\_parameters.f90}, is used
-to indicate a 3D crustal model. When this flag is set to \texttt{.true.},
-the crust on top of the 1D reference model (PREM, IASP91, or AK135)
-is removed and replaced by extending the mantle. The 3D crustal model
-is subsequently overprinted onto the crust-less 1D reference model.
-The call to the 3D crustal routine is of the form
-
-\begin{lyxcode}
-call~model\_crust(lat,lon,r,vp,vs,rho,moho,foundcrust,CM\_V,elem\_in\_crust)
-\end{lyxcode}
-Input to this routine consists of:
-
-\begin{description}
-\item [{\texttt{lat}}] Latitude in degrees.
-\item [{\texttt{lon}}] Longitude in degrees.
-\item [{\texttt{r}}] Non-dimensionalized radius ($0<\texttt{r}<1$).
-\end{description}
-Output from the routine consists of:
-
-\begin{description}
-\item [{\texttt{vp}}] Non-dimensionalized compressional wave speed at location
-(\texttt{lat},\texttt{lon},\texttt{r}).
-\item [{\texttt{vs}}] Non-dimensionalized shear wave speed.
-\item [{\texttt{rho}}] Non-dimensionalized density.
-\item [{\texttt{moho}}] Non-dimensionalized Moho depth.
-\item [{\texttt{found\_crust}}] Logical that is set to \texttt{.true.}
-only if crust exists at location (\texttt{lat},\texttt{lon},\texttt{r}),
-i.e., \texttt{.false.} for radii \texttt{r} in the mantle. This flags
-determines whether or not a particular location is in the crust and,
-if so, what parameters to assign to the mesh at this location.
-\item [{\texttt{CM\_V}}] Fortran structure that contains the parameters,
-variables and arrays that describe the model.
-\item [{\texttt{elem\_in\_crust}}] Logical that is used to force the
-routine to return crustal values, even if the location would
-be below the crust.
-\end{description}
-All output needs to be non-dimensionalized according to the convention
-summarized in Appendix~\ref{cha:Non-Dimensionalization-Conventions}.
-You can replace this subroutine by your own routine \textit{provided
-you do not change the call structure of the routine}, i.e., the new
-routine should take exactly the same input and produce the required,
-properly non-dimensionalized output.
-
-Part of the file \texttt{model\_crust.f90} is the subroutine \texttt{model\_crust\_broadcast}.
-The call to this routine takes argument \texttt{CM\_V} and is used
-to once-and-for-all read in the databases related to Crust2.0 and
-broadcast the model to all parallel processes. If
-you replace the file \texttt{model\_crust.f90} with your own implementation,
-you \textit{must} provide a \texttt{model\_crust\_broadcast} routine,
-even if it does nothing. Model constants and variables read by the
-routine \texttt{model\_crust\_broadcast} are passed to the subroutine
-\texttt{read\_crust\_model} through the structure \texttt{CM\_V}.
-An alternative crustal model could use the same construct. Please
-feel free to contribute subroutines for new models and send them to
-us so that they can be included in future releases of the software.
-
-\begin{quote}
-\textbf{NOTE:} If you decide to create your own version of file \texttt{model\_crust.f90},
-you must add calls to \texttt{MPI\_BCAST} in the subroutine \texttt{model\_crust\_broadcast}
-after the call to the \texttt{read\_crust\_model} subroutine that
-reads the isotropic mantle model once and for all in the mesher. This
-is done in order to read the (potentially large) model data files
-on the master node (which is the processor of rank 0 in our code)
-only and then send a copy to all the other nodes using an MPI broadcast,
-rather than using an implementation in which all the nodes would read
-the same model data files from a remotely-mounted home file system,
-which could create a bottleneck on the network in the case of a large
-number of nodes. For example, in the current call to that routine
-from \texttt{model\_crust.f90,} we write:
-\end{quote}
-\begin{lyxcode}
-{\footnotesize !~the~variables~read~are~declared~and~stored~in~structure~CM\_V~~~~~}{\footnotesize \par}
-
-{\footnotesize{}~~if(myrank~==~0)~call~read\_crust\_model(CM\_V)~}{\footnotesize \par}
-
-{\footnotesize !~broadcast~the~information~read~on~the~master~to~the~nodes~~~~~}{\footnotesize \par}
-
-{\footnotesize{}~~call~MPI\_BCAST(CM\_V\%thlr,NKEYS\_CRUST{*}NLAYERS\_CRUST,MPI\_DOUBLE\_PRECISION,}{\footnotesize \par}
-
-{\footnotesize{}~~~~~~~~~~~~~~~~~~~0,MPI\_COMM\_WORLD,ier)~~~~~}{\footnotesize \par}
-
-{\footnotesize{}~~call~MPI\_BCAST(CM\_V\%velocp,NKEYS\_CRUST{*}NLAYERS\_CRUST,MPI\_DOUBLE\_PRECISION,}{\footnotesize \par}
-
-{\footnotesize{}~~~~~~~~~~~~~~~~~~~0,MPI\_COMM\_WORLD,ier)~~~~~}{\footnotesize \par}
-
-{\footnotesize{}~~call~MPI\_BCAST(CM\_V\%velocs,NKEYS\_CRUST{*}NLAYERS\_CRUST,MPI\_DOUBLE\_PRECISION,}{\footnotesize \par}
-
-{\footnotesize{}~~~~~~~~~~~~~~~~~~~0,MPI\_COMM\_WORLD,ier)~~~~~}{\footnotesize \par}
-
-{\footnotesize{}~~call~MPI\_BCAST(CM\_V\%dens,NKEYS\_CRUST{*}NLAYERS\_CRUST,MPI\_DOUBLE\_PRECISION,}{\footnotesize \par}
-
-{\footnotesize{}~~~~~~~~~~~~~~~~~~~0,MPI\_COMM\_WORLD,ier)~~~~~}{\footnotesize \par}
-
-{\footnotesize{}~~call~MPI\_BCAST(CM\_V\%abbreviation,NCAP\_CRUST{*}NCAP\_CRUST,MPI\_CHARACTER,}{\footnotesize \par}
-
-{\footnotesize{}~~~~~~~~~~~~~~~~~~~0,MPI\_COMM\_WORLD,ier)~~~~~}{\footnotesize \par}
-
-{\footnotesize{}~~call~MPI\_BCAST(CM\_V\%code,2{*}NKEYS\_CRUST,MPI\_CHARACTER,0,MPI\_COMM\_WORLD,ier)~}{\footnotesize \par}
-\end{lyxcode}
-
-\section{{\normalsize \label{sec:Changing-the-Mantle}}Changing the Mantle
-Model}
-
-This section discusses how to change isotropic and anisotropic 3D
-mantle models. Usually such changes go hand-in-hand with changing
-the 3D crustal model.
-
-
-\subsection{{\normalsize \label{sub:Isotropic-Models}Isotropic Models}}
-
-The 3D mantle model S20RTS \citep{RiVaWo99} is superimposed onto
-the mantle mesh by the subroutine \texttt{model\_s20rts.f90}. The
-call to this subroutine is of the form
-
-\begin{lyxcode}
-call~model\_s20rts(radius,theta,phi,dvs,dvp,drho,D3MM\_V)~
-\end{lyxcode}
-Input to this routine consists of:
-
-\begin{description}
-\item [{\texttt{radius}}] Non-dimensionalized radius ($\texttt{RCMB/R\_ EARTH}<\texttt{r}<\texttt{RMOHO/R\_ EARTH}$;
-for a given 1D reference model, the constants \texttt{RCMB} and \texttt{RMOHO}
-are set in the \texttt{\small get\_model\_parameters}\texttt{.f90}
-file). The code expects the isotropic mantle model to be defined between
-the Moho (with radius \texttt{RMOHO} in m) and the core-mantle boundary
-(CMB; radius \texttt{RCMB} in m) of a 1D reference model. When a 3D
-crustal model is superimposed, as will usually be the case, the 3D
-mantle model is stretched to fill any potential gap between the radius
-of the Moho in the 1D reference model and the Moho in the 3D crustal
-model. Thus, when the Moho in the 3D crustal model is shallower than
-the Moho in the reference model, e.g., typically below the oceans,
-the mantle model is extended to fill this gap.
-\item [{\texttt{theta}}] Colatitude in radians.
-\item [{\texttt{phi}}] Longitude in radians.
-\end{description}
-Output from the routine are the following non-dimensional perturbations:
-
-\begin{description}
-\item [{\texttt{dvs}}] Relative shear-wave speed perturbations $\delta\beta/\beta$
-at location (\texttt{radius},\texttt{theta},\texttt{phi}).
-\item [{\texttt{dvp}}] Relative compressional-wave speed perturbations
-$\delta\alpha/\alpha$.
-\item [{\texttt{drho}}] Relative density perturbations $\delta\rho/\rho$.
-\item [{\texttt{D3MM\_V}}] Fortran structure that contains the parameters,
-variables and arrays that describe the model.
-\end{description}
-You can replace the \texttt{model\_s20rts.f90} file with your own
-version \textit{provided you do not change the call structure of the
-routine}, i.e., the new routine should take exactly the same input
-and produce the required relative output.
-
-Part of the file \texttt{model\_s20rts.f90} is the subroutine
-\texttt{model\_s20rts\_broadcast}.
-The call to this routine takes argument\texttt{ D3MM\_V} and is used
-to once-and-for-all read in the databases related to S20RTS. If you
-replace the file \texttt{model\_s20rts.f90} with your own implementation,
-you \textit{must} provide a \texttt{model\_s20rts\_broadcast} routine,
-even if it does nothing. Model constants and variables read by the
-routine \texttt{model\_s20rts\_broadcast} are passed to the subroutine
-\texttt{read\_model\_s20rts} through the structure \texttt{D3MM\_V.}
-An alternative mantle model should use the same construct.
-
-\begin{quote}
-\textbf{NOTE:} If you decide to create your own version of file \texttt{model\_s20rts.f90},
-you must add calls to \texttt{MPI\_BCAST} in the subroutine \texttt{model\_s20rts\_broadcast}
-after the call to the \texttt{read\_model\_s20rts} subroutine that
-reads the isotropic mantle model once and for all in the mesher. This
-is done in order to read the (potentially large) model data files
-on the master node (which is the processor of rank 0 in our code)
-only and then send a copy to all the other nodes using an MPI broadcast,
-rather than using an implementation in which all the nodes would read
-the same model data files from a remotely-mounted home file system,
-which could create a bottleneck on the network in the case of a large
-number of nodes. For example, in the current call to that routine
-from \texttt{model\_s20rts.f90,} we write:
-\end{quote}
-\begin{lyxcode}
-{\footnotesize !~the~variables~read~are~declared~and~stored~in~structure~D3MM\_V}{\footnotesize \par}
-
-{\footnotesize{}~~if(myrank~==~0)~call~read\_model\_s20rts(D3MM\_V)~}{\footnotesize \par}
-
-{\footnotesize !~broadcast~the~information~read~on~the~master~to~the~nodes~~~~~~~}{\footnotesize \par}
-
-{\footnotesize{}~~call~MPI\_BCAST(D3MM\_V\%dvs\_a,(NK+1){*}(NS+1){*}(NS+1),MPI\_DOUBLE\_PRECISION,}{\footnotesize \par}
-
-{\footnotesize{}~~~~~~~~~~0,MPI\_COMM\_WORLD,ier)}{\footnotesize \par}
-
-{\footnotesize{}~~call~MPI\_BCAST(D3MM\_V\%dvs\_b,(NK+1){*}(NS+1){*}(NS+1),MPI\_DOUBLE\_PRECISION,}{\footnotesize \par}
-
-{\footnotesize{}~~~~~~~~~~0,MPI\_COMM\_WORLD,ier)}{\footnotesize \par}
-
-{\footnotesize{}~~call~MPI\_BCAST(D3MM\_V\%dvp\_a,(NK+1){*}(NS+1){*}(NS+1),MPI\_DOUBLE\_PRECISION,}{\footnotesize \par}
-
-{\footnotesize{}~~~~~~~~~~0,MPI\_COMM\_WORLD,ier)}{\footnotesize \par}
-
-{\footnotesize{}~~call~MPI\_BCAST(D3MM\_V\%dvp\_b,(NK+1){*}(NS+1){*}(NS+1),MPI\_DOUBLE\_PRECISION,}{\footnotesize \par}
-
-{\footnotesize{}~~~~~~~~~~0,MPI\_COMM\_WORLD,ier)~~~~~~}{\footnotesize \par}
-
-{\footnotesize{}~~call~MPI\_BCAST(D3MM\_V\%spknt,NK+1,MPI\_DOUBLE\_PRECISION,0,MPI\_COMM\_WORLD,ier)~~~~~~~}{\footnotesize \par}
-
-{\footnotesize{}~~call~MPI\_BCAST(D3MM\_V\%qq0,(NK+1){*}(NK+1),MPI\_DOUBLE\_PRECISION,0,MPI\_COMM\_WORLD,ier)~}{\footnotesize \par}
-
-{\footnotesize{}~~call~MPI\_BCAST(D3MM\_V\%qq,3{*}(NK+1){*}(NK+1),MPI\_DOUBLE\_PRECISION,0,MPI\_COMM\_WORLD,ier)~}{\footnotesize \par}
-\end{lyxcode}
-
-\subsection{{\normalsize \label{sub:Anisotropic-Models}Anisotropic Models}}
-
-Three-dimensional anisotropic mantle models may be superimposed on
-the mesh based upon the subroutine
-
-\begin{lyxcode}
-model\_aniso\_mantle.f90
-\end{lyxcode}
-The call to this subroutine is of the form
-
-\begin{lyxcode}
-call~model\_aniso\_mantle(r,theta,phi,rho,~\&~
-
-~~~~c11,c12,c13,c14,c15,c16,c22,c23,c24,c25,c26,~\&~
-
-~~~~c33,c34,c35,c36,c44,c45,c46,c55,c56,c66,AMM\_V)~
-\end{lyxcode}
-Input to this routine consists of:
-
-\begin{description}
-\item [{\texttt{r}}] Non-dimensionalized radius ($\texttt{RCMB/R\_ EARTH}<\texttt{r}<\texttt{RMOHO/R\_ EARTH}$;
-for a given 1D reference model, the constants \texttt{RCMB} and \texttt{RMOHO}
-are set in the \texttt{\small get\_model\_parameters}\texttt{.f90}
-file). The code expects the anisotropic mantle model to be defined
-between the Moho and the core-mantle boundary (CMB). When a 3D crustal
-model is superimposed, as will usually be the case, the 3D mantle
-model is stretched to fill any potential gap between the radius of
-the Moho in the 1D reference model and the Moho in the 3D crustal
-model. Thus, when the Moho in the 3D crustal model is shallower than
-the Moho in the reference model, e.g., typically below the oceans,
-the mantle model is extended to fill this gap.
-\item [{\texttt{theta}}] Colatitude in radians.
-\item [{\texttt{phi}}] Longitude in radians.
-\end{description}
-Output from the routine consists of the following non-dimensional
-model parameters:
-
-\begin{description}
-\item [{\texttt{rho}}] Non-dimensionalized density $\rho$.
-\item [{\texttt{c11},}] \textbf{$\cdots$,} \texttt{\textbf{c66}} 21 non-dimensionalized
-anisotropic elastic parameters.
-\item [{\texttt{AMM\_V}}] Fortran structure that contains the parameters,
-variables and arrays that describe the model.
-\end{description}
-You can replace the \texttt{model\_aniso\_mantle.f90} file by
-your own version \textit{provided you do not change the call structure
-of the routine}, i.e., the new routine should take exactly the same
-input and produce the required relative output. Part of the file \texttt{model\_aniso\_mantle.f90}
-is the subroutine \texttt{model\_aniso\_mantle\_broadcast}. The call to
-this routine takes argument \texttt{AMM\_V} and is used to once-and-for-all
-read in the static databases related to the anisotropic model. When
-you choose to replace the file \texttt{model\_aniso\_mantle.f90}
-with your own implementation you \textit{must} provide a \texttt{model\_aniso\_mantle\_broadcast}
-routine, even if it does nothing. Model constants and variables read
-by the routine \texttt{model\_aniso\_mantle\_broadcast} are passed through the
-structure \texttt{AMM\_V}. An alternative anisotropic mantle model
-should use the same construct.
-
-\begin{quote}
-\textbf{NOTE:} If you decide to create your own version of file \texttt{model\_aniso\_mantle.f90},
-you must add calls to \texttt{MPI\_BCAST} in file \texttt{model\_aniso\_mantle.f90}
-after the call to the \texttt{read\_aniso\_mantle\_model} subroutine
-that reads the anisotropic mantle model once and for all in the mesher.
-This is done in order to read the (potentially large) model data files
-on the master node (which is the processor of rank 0 in our code)
-only and then send a copy to all the other nodes using an MPI broadcast,
-rather than using an implementation in which all the nodes would read
-the same model data files from a remotely-mounted home file system,
-which could create a bottleneck on the network in the case of a large
-number of nodes. For example, in the current call to that routine
-from \texttt{model\_aniso\_mantle.f90,} we write:
-\end{quote}
-\begin{lyxcode}
-{\footnotesize !~the~variables~read~are~declared~and~stored~in~structure~AMM\_V}{\footnotesize \par}
-
-{\footnotesize{}~~if(myrank~==~0)~call~read\_aniso\_mantle\_model(AMM\_V)}{\footnotesize \par}
-
-{\footnotesize !~broadcast~the~information~read~on~the~master~to~the~nodes}{\footnotesize \par}
-
-{\footnotesize{}~~call~MPI\_BCAST(AMM\_V\%npar1,1,MPI\_INTEGER,0,MPI\_COMM\_WORLD,ier)~~~~}{\footnotesize \par}
-
-{\footnotesize{}~~call~MPI\_BCAST(AMM\_V\%beta,14{*}34{*}37{*}73,MPI\_DOUBLE\_PRECISION,0,MPI\_COMM\_WORLD,ier)}{\footnotesize \par}
-
-{\footnotesize{}~~call~MPI\_BCAST(AMM\_V\%pro,47,MPI\_DOUBLE\_PRECISION,0,MPI\_COMM\_WORLD,ier)}{\footnotesize \par}
-\end{lyxcode}
-Rotation of the anisotropic tensor elements from one coordinate system
-to another coordinate system may be accomplished based upon the subroutine
-\texttt{rotate\_aniso\_tensor}. Use of this routine requires understanding
-the coordinate system used in \texttt{SPECFEM3D\_GLOBE}, as discussed
-in Appendix~\ref{cha:Reference-Frame-Convention}.
-
-
-\section{{\normalsize \label{sec:Anelastic-Models}}Anelastic Models}
-
-Three-dimensional anelastic (attenuation) models may be superimposed
-onto the mesh based upon your subroutine .
-\texttt{model\_atten3D.f90}.
-The call to this routine would be as follows
-
-\begin{lyxcode}
-call~model\_atten3D(radius,~colatitude,~longitude,~Qmu,~QRFSI12\_Q,~idoubling)
-\end{lyxcode}
-Input to this routine consists of:
-
-\begin{description}
-\item [{\texttt{radius}}] scaled radius of the earth: $0\,(\mathrm{center})<=r\,<=1$(surface)
-\item [{\texttt{latitude}}] Colatitude in degrees: $0^{\circ}<=\theta<=180^{\circ}$
-\item [{\texttt{longitude}}] Longitude in degrees: $-180^{\circ}<=\phi<=180^{\circ}$
-\item [{\texttt{QRFSI12\_Q}}] Fortran structure that contains the parameters,
-variables and arrays that describe the model
-\item [{\texttt{idoubling}}] value of the doubling index flag in each radial region of the mesh
-\end{description}
-Output to this routine consists of:
-
-\begin{description}
-\item [{\texttt{Qmu}}] Shear wave quality factor: $0<Q_{\mu}<5000$
-\end{description}
-A 3-D attenuation model QRFSI12 \citep{DaEkDz08}
-is provided, as well as 1-D models with a PREM and a 1DREF
-attenuation structure. By default the PREM attenuation model is taken,
-using the routine \texttt{model\_attenuation\_1D\_PREM},
-found in \texttt{model\_attenuation.f90}.
-
-To create your own three-dimensional attenuation model, you add your model
-using a routine like the ~\\
-\texttt{model\_atten3D\_QRFSI12} subroutine
-and the example routine above as a guide and replace the call in file
-\texttt{meshfem3D\_models.f90} to your own subroutine.
-
-Note that the resolution and maximum value of anelastic models are
-truncated. This speeds the construction of the standard linear solids
-during the meshing stage. To change the resolution, currently at one
-significant figure following the decimal, or the maximum value (5000),
-consult \texttt{constants.h}. In order to prevent unexpected results,
-quality factors $Q_{\mu}$ should never be equal to 0 outside of the
-inner core.
-
-
-\chapter{Post-Processing Scripts}
-
-Several post-processing scripts/programs are provided in the \texttt{UTILS/seis\_process}
-directory, most of which need to be adjusted for different
-systems, for example, the path of the executable programs. Here we
-only list the available scripts and provide a brief description, and
-you can either refer to the related sections for detailed usage or,
-in many cases, type the script/program name without arguments
-to see its usage.
-
-
-\section{Clean Local Database}
-
-After all the simulations are done, you may need to clean the local
-scratch disks for the next simulation. This is especially important
-in the case of 1- or 2-chunk kernel simulations, where very large files
-are generated for the absorbing boundaries to help with the reconstruction
-of the regular forward wavefield. A sample script is provided in \texttt{UTILS/Cluster/lsf}:
-
-\begin{lyxcode}
-cleanbase.pl~machines
-\end{lyxcode}
-
-\section{Process Data and Synthetics\label{sec:Process-data-and-syn}}
-
-In many cases, the SEM synthetics are calculated and compared to observed
-seismograms recorded at seismic stations. Since the SEM synthetics
-are accurate for a certain frequency range, both the original data
-and the synthetics need to be processed before a comparison can be
-made. For such comparisons, the following steps are recommended:
-
-\begin{enumerate}
-\item Make sure that both synthetic and observed seismograms have the correct station/event and timing information.
-\item Convolve synthetic seismograms with a source time function with the half duration specified in the \texttt{CMTSOLUTION} file, provided, as recommended, you used a zero half duration in the SEM simulations.
-\item Resample both observed and synthetic seismograms to a common sampling rate.
-\item Cut the records using the same window.
-\item Remove the trend and mean from the records and taper them.
-\item Remove the instrument response from the observed seismograms (recommended) or convolve the synthetic seismograms with the instrument response.
-\item Make sure that you apply the same filters to both observed and synthetic seismograms. Preferably, avoid filtering your records more than once.
-\item Now, you are ready to compare your synthetic and observed seismograms.
-\end{enumerate}
-
-We generally use the following scripts for processing:
-
-\subsection{\texttt{process\_data.pl}}
-
-This script cuts a given portion of the original data, filters it,
-transfers the data into a displacement record, and picks the first
-P and S arrivals. For more functionality, type `\texttt{process\_data.pl}'
-without any argument. An example of the usage of the script:
-
-\begin{lyxcode}
-{\footnotesize process\_data.pl~-m~CMTSOLUTION~-s~1.0~-l~0/4000~-i~-f~-t~40/500~-p~-x~bp~DATA/1999.330{*}.LH?.SAC}{\footnotesize \par}
-\end{lyxcode}
-
-\noindent which has resampled the SAC files to a sampling rate of 1 seconds, cut them between 0 and 4000 seconds, transfered them into displacement
-records and filtered them between 40 and 500 seconds, picked the first P and S arrivals, and added suffix `\texttt{bp}' to the file names.
-
-Note that all of the scripts in this section actually use SAC,
-saclst and/or IASP91 to do the core operations; therefore make sure
-that the SAC, saclst and IASP91 packages are installed on your
-system, and that all the environment variables are set properly before
-running these scripts.
-
-
-\subsection{\texttt{\label{sub:process_syn.pl}process\_syn.pl}}
-
-This script converts the synthetic output from the SEM code from ASCII
-to SAC format, and performs similar operations as `\texttt{process\_data.pl}'.
-An example of the usage of the script:
-
-\begin{lyxcode}
-{\footnotesize process\_syn.pl~-m~CMTSOLUTION~-h~-a~STATIONS~-s~1.0~-l~0/4000~-f~-t~40/500~-p~-x~bp~SEM/{*}.LH?.sem}{\footnotesize \par}
-\end{lyxcode}
-which will convolve the synthetics with a triangular source-time function
-from the \texttt{CMTSOLUTION} file, convert the synthetics into SAC
-format, add event and station information into the SAC headers, resample the SAC files with a sampling rate of 1 seconds, cut them between 0 and 4000 seconds, filter them between 40 and
-500 seconds with the same filter used for the observed data, pick the first P and S arrivals, and add the suffix `\texttt{bp}'
-to the file names.
-
-More options are available for this script, such as adding a time shift
-to the origin time of the synthetics, convolving the synthetics with
-a triangular source time function with a given half duration, etc.
-Type \texttt{process\_syn.pl} without any argument for detailed
-usage.
-
-
-\subsection{\texttt{rotate.pl}}
-
-To rotate the horizontal components of both the data and the synthetics
-(LHN and LHE) to the transverse and radial directions (LHT and LHR),\texttt{\small{}
-}use{\small{} }\texttt{\small rotate.pl}:
-
-\begin{lyxcode}
-rotate.pl~-l~0~-L~4000~-d~DATA/{*}.LHE.SAC.bp~
-
-rotate.pl~-l~0~-L~4000~SEM/{*}.LHE.sem.sac.bp~
-\end{lyxcode}
-where the first command performs rotation on the SAC data obtained
-through IRIS (which may have timing information written in the filename),
-while the second command rotates the processed synthetics.
-
-For synthetics, another (simpler) option is to set flag \texttt{ROTATE\_SEISMOGRAMS\_RT}
-to \texttt{.true.} in the parameter file \texttt{DATA/Par\_file}.
-
-
-\subsection{\texttt{clean\_sac\_headers\_after\_crash.sh}}
-
-Note: You need to have the \texttt{sismoutil-0.9b} package installed
-on your computer if you want to run this script on binary SAC files.
-The software is available via the ORFEUS web site \url{www.orfeus-eu.org}.
-
-In case the simulation crashes during run-time without computing and
-writing all time steps, the SAC files (if flags \texttt{OUTPUT\_SEISMOS\_SAC\_ALPHANUM}
-or \texttt{OUTPUT\_SEISMOS\_SAC\_BINARY} have been set to \texttt{.true.})
-are corrupted and cannot be used in SAC. If the simulation
-ran long enough so that the synthetic data may still be of use, you
-can run the script called \texttt{clean\_sac\_headers\_after\_crash.sh}
-(located in the \texttt{UTILS} directory) on the SAC files to correct
-the header variable NPTS to the actually written number of time steps.
-The script must be called from the \texttt{SPECFEM3D} main directory,
-and the input argument to this script is simply a list of SAC seismogram
-files.
-
-
-\section{Map Local Database}
-
-A sample program \texttt{remap\_database} is provided to map the local
-database from a set of machines to another set of machines. This is
-especially useful when you want to run mesher and solver, or different
-types of solvers separately through a scheduler (refer to Chapter~\ref{cha:Running-Scheduler}).
-
-\begin{lyxcode}
-run\_lsf.bash~-{}-gm-no-shmem~-{}-gm-copy-env~remap\_database
-
-old\_machines~150~{[}old\_jobid~new\_jobid]
-\end{lyxcode}
-where \texttt{old\_machines} is the LSF machine file used in the previous
-simulation, and \texttt{150} is the number of processors in total.
-Note that you need to supply \texttt{old\_jobid} and \texttt{new\_jobid(\%J)}
-which are the LSF job-IDs for the old and new run if your databases
-are stored in a sub-directory named after the jobid on the scratch
-disk.
-
-
-\chapter*{\label{cha:Bug-Reports-and}Bug Reports and Suggestions for Improvements}
-
-To report bugs or suggest improvements to the code, please send an
-e-mail to the CIG Computational Seismology Mailing List \url{cig-seismo at geodynamics.org}
-or Jeroen Tromp \url{jtromp-AT-princeton.edu}, and/or use our online
-bug tracking system Roundup \url{www.geodynamics.org/roundup}.
-
-
-\chapter*{\label{cha:Notes-and-Acknowledgements}Notes and Acknowledgements}
-
-In order to keep the software package thread-safe in case a multithreaded
-implementation of MPI is used, developers should not add modules or
-common blocks to the source code but rather use regular subroutine
-arguments (which can be grouped in ``derived types'' if needed for
-clarity).
-
-The Gauss-Lobatto-Legendre subroutines in \texttt{gll\_library.f90}
-are based in part on software libraries from the Massachusetts Institute
-of Technology, Department of Mechanical Engineering (Cambridge, Massachusetts, USA).
-The non-structured global numbering software was provided by Paul
-F. Fischer (Brown University, Providence, Rhode Island, USA, now at Argonne National Laboratory, USA).
-
-OpenDX \url{www.opendx.org} is open-source based on IBM Data Explorer,
-AVS \url{www.avs.com} is a trademark of Advanced Visualization Systems,
-and ParaView \url{www.paraview.com} is an open-source visualization
-platform.{\small{} }{\small \par}
-
-The main developers of the \texttt{SPECFEM3D\_GLOBE} source code are
-Dimitri Komatitsch, Jeroen Tromp, Qinya Liu and David Mich\'ea. The
-following individuals (listed in alphabetical order) have also contributed
-to the development or improvement of the source code: Ebru Bozda\u g, Min Chen, Vala Hj\"orleifsd\'ottir,
-Nicolas Le Goff, Jes\'us Labarta, Matthias Meschede, Daniel Peter, Brian Savage, Bernhard Schuberth, Anne Sieminski,
-Leif Strand, Peter van Keken and Hejun Zhu. The following individuals (listed
-in alphabetical order) contributed to this manual: Ebru Bozda\u g, Min Chen, Vala
-Hj\"orleifsd\'ottir, Sue Kientz, Dimitri Komatitsch, Qinya Liu, Alessia
-Maggi, David Mich\'ea, Daniel Peter, Brian Savage, Anne Sieminski, Carl Tape, and
-Jeroen Tromp. The manual's cover graphic was created by Santiago Lombeyda
-from Caltech's Center for Advanced Computing Research (CACR) \url{www.cacr.caltech.edu}.
-Older versions of the code were initially developed by Dimitri Komatitsch at Institut de Physique du Globe (France)
-and then by Dimitri Komatitsch and Jeroen Tromp at Harvard University (USA).
-
-Please e-mail your feedback, questions, comments, and suggestions
-to Jeroen Tromp \url{jtromp-AT-princeton.edu} or to the CIG Computational Seismology Mailing List \url{cig-seismo at geodynamics.org}.
-
-
-\chapter*{\label{cha:Copyright}Copyright}
-
-Main authors: Dimitri Komatitsch and Jeroen Tromp
-
-Princeton University, USA,
-and University of Pau / CNRS / INRIA, France
-
-$\copyright$ Princeton University / California Institute of Technology and University of Pau / CNRS
-/ INRIA, March 2010
-
-This program is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published
-by the Free Software Foundation (see Appendix \ref{cha:License}).
-
-{\small \bibliography{bibliography}
-}{\small \par}
-
-\appendix
-
-\chapter{\label{cha:Reference-Frame-Convention}Reference Frame Convention}
-
-The code uses the following convention for the Cartesian reference
-frame:
-
-\begin{itemize}
-\item the $x$ axis points East
-\item the $y$ axis points North
-\item the $z$ axis points up
-\end{itemize}
-Note that this convention is different from both the \citet{AkRi80}
-convention and the Harvard Centroid-Moment Tensor (CMT) convention.
-The Aki \& Richards convention is
-
-\begin{itemize}
-\item the $x$ axis points North
-\item the $y$ axis points East
-\item the $z$ axis points down
-\end{itemize}
-and the Harvard CMT convention is
-
-\begin{itemize}
-\item the $x$ axis points South
-\item the $y$ axis points East
-\item the $z$ axis points up
-\end{itemize}
-
-\chapter{\label{cha:Non-Dimensionalization-Conventions}Non-Dimensionalization
-Conventions{\small{} }}
-
-All physical parameters used by the code are non-dimensionalized according
-to the conventions summarized in Table~{\small \ref{table:conventions}.
-}%
-\begin{table}[ht]
-\noindent \begin{centering}
-{\small }\begin{tabular}{|c|c|}
-\hline
-quantity (units) & non-dimensionalized with \tabularnewline
-\hline
-distance (m) & \texttt{R\_EARTH} \tabularnewline
-time (s) & $\sqrt{\texttt{PI}\times\texttt{GRAV}\times\texttt{RHOAV}}$ \tabularnewline
-density (kg/m$^{3}$) & \texttt{RHOAV} \tabularnewline
-\hline
-\end{tabular}
-\par\end{centering}{\small \par}
-
-\caption{Non-dimensionalization employed by the code. The constants \texttt{R\_EARTH}
-(the radius of the Earth), \texttt{PI} (the number $\pi$), \texttt{GRAV}
-(the universal gravitational constant), and \texttt{RHOAV} (the Earth's
-average density) are defined in the \texttt{constants.h} file. }
-
-
-{\small \label{table:conventions} }
-\end{table}
-{\small \par}
-
-
-\chapter{Benchmarks}
-
-\citet{KoTr02a,KoTr02b} carefully benchmarked the spectral-element
-simulations of global seismic waves against normal-mode seismograms.
-Version 4.0 of \texttt{SPECFEM3D\_GLOBE} has been benchmarked again
-following the same procedure.
-
-In this appendix we present two tests: a `long-period' (periods longer
-than 17~s) simulation of a shallow event in isotropic PREM \citep{DzAn81}
-without the ocean layer, without attenuation but including the effects
-of self-gravitation (in the Cowling approximation) (Figures \ref{fig:Vanuatu-with-Vertical}
-and \ref{fig:Vanuatu-with-Transverse}), and a `short-period' (periods
-longer than 9~s) simulation of a deep event in transversely isotropic
-PREM without the ocean layer and including the effects of self-gravitation
-and attenuation (Figures \ref{fig:Bolivia-with-Vertical}, \ref{fig:Bolivia-with-Transverse}
-and \ref{fig:Bolivia-PKP}).
-
-%
-\begin{figure}[ht]
-\noindent \begin{centering}
-\includegraphics[scale=0.75]{figures/vanuatu_vertical.eps}\caption{\label{fig:Vanuatu-with-Vertical} Normal-mode (blue) and SEM (red)
-vertical displacements in isotropic PREM considering the effects of
-self-gravitation but not attenuation for 13 stations at increasing
-distance from the 1999 November 26th Vanuatu event located at 15~km
-depth. The SEM computation is accurate for periods longer than 17~s.
-The seismograms have been filtered between 50~s and 500~s. The station
-names are indicated on the left. }
-
-\par\end{centering}
-\end{figure}
-%
-\begin{figure}[ht]
-\noindent \begin{centering}
-\includegraphics[scale=0.75]{figures/vanuatu_trans.eps}\caption{\label{fig:Vanuatu-with-Transverse}Same as in Figure \ref{fig:Vanuatu-with-Vertical}
-for the transverse displacements.}
-
-\par\end{centering}
-\end{figure}
-%
-\begin{figure}[ht]
-\noindent \begin{centering}
-\includegraphics[scale=0.75]{figures/bolivia_vertical.eps}\caption{\label{fig:Bolivia-with-Vertical}Normal-mode (blue) and SEM (red)
-vertical displacements in transversely isotropic PREM considering
-the effects of self-gravitation and attenuation for 12 stations at
-increasing distance from the 1994 June 9th Bolivia event located at
-647~km depth. The SEM computation is accurate for periods longer
-than 9~s. The seismograms have been filtered between 10~s and 500~s.
-The station names are indicated on the left.}
-
-\par\end{centering}
-\end{figure}
-%
-\begin{figure}[ht]
-\noindent \begin{centering}
-\includegraphics[scale=0.75]{figures/bolivia_trans.eps}\caption{\label{fig:Bolivia-with-Transverse}Same as in Figure \ref{fig:Bolivia-with-Vertical}
-for the transverse displacements.}
-
-\par\end{centering}
-\end{figure}
-
-
-%
-\begin{figure}[ht]
-\noindent \begin{centering}
-\includegraphics[scale=0.75]{figures/PKPdf_all_15s500s.eps}\caption{\label{fig:Bolivia-PKP}Seismograms recorded between 130 degrees and
-230 degrees, showing in particular the good agreement for core phases
-such as PKP. This figure is similar to Figure 24 of \citet{KoTr02a}.
-The results have been filtered between 15~s and 500~s.}
-
-\par\end{centering}
-\end{figure}
-
-
-The normal-mode synthetics are calculated with the code \texttt{QmXD}
-using mode catalogs with a shortest period of 8~s generated by the
-code \texttt{OBANI}. No free-air, tilt, or gravitational potential
-corrections were applied \citep{DaTr98}. We also turned off the effect
-of the oceans in \texttt{QmXD}.
-
-The normal-mode and SEM displacement seismograms are first calculated
-for a step source-time function, i.e., setting the parameter \texttt{half}
-\texttt{duration} in the \texttt{CMTSOLUTION} file to zero for the
-SEM simulations. Both sets of seismograms are subsequently convolved
-with a triangular source-time function using the processing script
-\texttt{UTILS/seis\_}~\\
-\texttt{process/process\_syn.pl}. They are also band-pass filtered
-and the horizontal components are rotated to the radial and transverse
-directions (with the script \texttt{UTILS/seis\_process/rotate.pl}).
-
-The match between the normal-mode and SEM seismograms is quite remarkable
-for the experiment with attenuation, considering the very different
-implementations of attenuation in the two computations (e.g., frequency
-domain versus time domain, constant Q versus absorption bands).
-
-Further tests can be found in the \texttt{EXAMPLES} directory. It
-contains the normal-mode and SEM seismograms, and the parameters (\texttt{STATIONS},
-\texttt{CMTSOLUTION} and \texttt{Par\_file}) for the SEM simulations.
-
-\chapter{\label{cha:SAC-headers}SAC Headers}
-
-Most information about the simulations (i.e., event/station information, sampling rate, etc.) are written in the headers of the seismograms in SAC format. The list of headers and their explanations may be found in Figure \ref{fig:SAC-headers}. Please check the SAC webpages \url{www.iris.edu/software/sac/} for further information. Please note that the reference time KZTIME is the centroid time ($t_\text{CMT}=t_\text{PDE}+\texttt{time shift}$) which corresponds to zero time in the synthetics. For kinematic rupture simulations, KZTIME equals to the CMT time of the source having the minimum time-shift in the \texttt{CMTSOLUTION} file, and coordinates, depth and half-duration of the event are not provided in the headers.
-
-\begin{figure}[ht]
-\noindent \begin{centering}
-\includegraphics[scale=0.8]{figures/headers_sem_explained.pdf}\caption{\label{fig:SAC-headers}List of SAC headers and their explanations for a sample seismogram.}
-
-\par\end{centering}
-\end{figure}
-
-\chapter{\label{cha:License}License}
-
-\textbf{GNU GENERAL PUBLIC LICENSE Version 2, June 1991. Copyright
-(C) 1989, 1991 Free Software Foundation, Inc. 59 Temple Place, Suite
-330, Boston, MA 02111-1307 USA} \\
-Everyone is permitted to copy and distribute verbatim copies of this
-license document, but changing it is not allowed.
-
-
-\section*{Preamble}
-
-The licenses for most software are designed to take away your freedom
-to share and change it. By contrast, the GNU General Public License
-is intended to guarantee your freedom to share and change free software
--- to make sure the software is free for all its users. This General
-Public License applies to most of the Free Software Foundation's software
-and to any other program whose authors commit to using it. (Some other
-Free Software Foundation software is covered by the GNU Library General
-Public License instead.) You can apply it to your programs, too.
-
-When we speak of free software, we are referring to freedom, not price.
-Our General Public Licenses are designed to make sure that you have
-the freedom to distribute copies of free software (and charge for
-this service if you wish), that you receive source code or can get
-it if you want it, that you can change the software or use pieces
-of it in new free programs; and that you know you can do these things.
-
-To protect your rights, we need to make restrictions that forbid anyone
-to deny you these rights or to ask you to surrender the rights. These
-restrictions translate to certain responsibilities for you if you
-distribute copies of the software, or if you modify it.
-
-For example, if you distribute copies of such a program, whether gratis
-or for a fee, you must give the recipients all the rights that you
-have. You must make sure that they, too, receive or can get the source
-code. And you must show them these terms so they know their rights.
-
-We protect your rights with two steps:
-
-\begin{enumerate}
-\item Copyright the software, and
-\item Offer you this license which gives you legal permission to copy, distribute
-and/or modify the software.
-\end{enumerate}
-Also, for each author's protection and ours, we want to make certain
-that everyone understands that there is no warranty for this free
-software. If the software is modified by someone else and passed on,
-we want its recipients to know that what they have is not the original,
-so that any problems introduced by others will not reflect on the
-original authors' reputations.
-
-Finally, any free program is threatened constantly by software patents.
-We wish to avoid the danger that redistributors of a free program
-will individually obtain patent licenses, in effect making the program
-proprietary. To prevent this, we have made it clear that any patent
-must be licensed for everyone's free use or not licensed at all.
-
-The precise terms and conditions for copying, distribution and modification
-follow.
-
-
-\section*{GNU GENERAL PUBLIC LICENSE TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION
-AND MODIFICATION }
-
-\begin{itemize}
-
-\item[0.]This License applies to any program or other work which
-contains a notice placed by the copyright holder saying it may be
-distributed under the terms of this General Public License. The ``Program''
-below refers to any such program or work, and a ``work based on the
-Program'' means either the Program or any derivative work under copyright
-law: that is to say, a work containing the Program or a portion of
-it, either verbatim or with modifications and/or translated into another
-language. (Hereinafter, translation is included without limitation
-in the term ``modification.'') Each licensee is addressed as ``you.''\\
-\\
-Activities other than copying, distribution and modification are not
-covered by this License; they are outside its scope. The act of running
-the Program is not restricted, and the output from the Program is
-covered only if its contents constitute a work based on the Program
-(independent of having been made by running the Program). Whether
-that is true depends on what the Program does.
-
-\end{itemize}
-
-\begin{enumerate}
-\item You may copy and distribute verbatim copies of the Program's source
-code as you receive it, in any medium, provided that you conspicuously
-and appropriately publish on each copy an appropriate copyright notice
-and disclaimer of warranty; keep intact all the notices that refer
-to this License and to the absence of any warranty; and give any other
-recipients of the Program a copy of this License along with the Program.
-
-
-You may charge a fee for the physical act of transferring a copy,
-and you may at your option offer warranty protection in exchange for
-a fee.
-
-\item You may modify your copy or copies of the Program or any portion of
-it, thus forming a work based on the Program, and copy and distribute
-such modifications or work under the terms of Section 1 above, provided
-that you also meet all of these conditions:
-
-\begin{enumerate}
-\item You must cause the modified files to carry prominent notices stating
-that you changed the files and the date of any change.
-\item You must cause any work that you distribute or publish, that in whole
-or in part contains or is derived from the Program or any part thereof,
-to be licensed as a whole at no charge to all third parties under
-the terms of this License.
-\item If the modified program normally reads commands interactively when
-run, you must cause it, when started running for such interactive
-use in the most ordinary way, to print or display an announcement
-including an appropriate copyright notice and a notice that there
-is no warranty (or else, saying that you provide a warranty) and that
-users may redistribute the program under these conditions, and telling
-the user how to view a copy of this License. (Exception: if the Program
-itself is interactive but does not normally print such an announcement,
-your work based on the Program is not required to print an announcement.)
-\end{enumerate}
-These requirements apply to the modified work as a whole. If identifiable
-sections of that work are not derived from the Program, and can be
-reasonably considered independent and separate works in themselves,
-then this License, and its terms, do not apply to those sections when
-you distribute them as separate works. But when you distribute the
-same sections as part of a whole which is a work based on the Program,
-the distribution of the whole must be on the terms of this License,
-whose permissions for other licensees extend to the entire whole,
-and thus to each and every part regardless of who wrote it.
-
-Thus, it is not the intent of this section to claim rights or contest
-your rights to work written entirely by you; rather, the intent is
-to exercise the right to control the distribution of derivative or
-collective works based on the Program.
-
-In addition, mere aggregation of another work not based on the Program
-with the Program (or with a work based on the Program) on a volume
-of a storage or distribution medium does not bring the other work
-under the scope of this License.
-
-\item You may copy and distribute the Program (or a work based on it, under
-Section 2) in object code or executable form under the terms of Sections
-1 and 2 above provided that you also do one of the following:
-
-\begin{enumerate}
-\item Accompany it with the complete corresponding machine-readable source
-code, which must be distributed under the terms of Sections 1 and
-2 above on a medium customarily used for software interchange; or,
-\item Accompany it with a written offer, valid for at least three years,
-to give any third party, for a charge no more than your cost of physically
-performing source distribution, a complete machine-readable copy of
-the corresponding source code, to be distributed under the terms of
-Sections 1 and 2 above on a medium customarily used for software interchange;
-or,
-\item Accompany it with the information you received as to the offer to
-distribute corresponding source code. (This alternative is allowed
-only for noncommercial distribution and only if you received the program
-in object code or executable form with such an offer, in accord with
-Subsection b above.)
-\end{enumerate}
-The source code for a work means the preferred form of the work for
-making modifications to it. For an executable work, complete source
-code means all the source code for all modules it contains, plus any
-associated interface definition files, plus the scripts used to control
-compilation and installation of the executable. However, as a special
-exception, the source code distributed need not include anything that
-is normally distributed (in either source or binary form) with the
-major components (compiler, kernel, and so on) of the operating system
-on which the executable runs, unless that component itself accompanies
-the executable.
-
-If distribution of executable or object code is made by offering access
-to copy from a designated place, then offering equivalent access to
-copy the source code from the same place counts as distribution of
-the source code, even though third parties are not compelled to copy
-the source along with the object code.
-
-\item You may not copy, modify, sublicense, or distribute the Program except
-as expressly provided under this License. Any attempt otherwise to
-copy, modify, sublicense or distribute the Program is void, and will
-automatically terminate your rights under this License. However, parties
-who have received copies, or rights, from you under this License will
-not have their licenses terminated so long as such parties remain
-in full compliance.
-\item You are not required to accept this License, since you have not signed
-it. However, nothing else grants you permission to modify or distribute
-the Program or its derivative works. These actions are prohibited
-by law if you do not accept this License. Therefore, by modifying
-or distributing the Program (or any work based on the Program), you
-indicate your acceptance of this License to do so, and all its terms
-and conditions for copying, distributing or modifying the Program
-or works based on it.
-\item Each time you redistribute the Program (or any work based on the Program),
-the recipient automatically receives a license from the original licensor
-to copy, distribute or modify the Program subject to these terms and
-conditions. You may not impose any further restrictions on the recipients'
-exercise of the rights granted herein. You are not responsible for
-enforcing compliance by third parties to this License.
-\item If, as a consequence of a court judgment or allegation of patent infringement
-or for any other reason (not limited to patent issues), conditions
-are imposed on you (whether by court order, agreement or otherwise)
-that contradict the conditions of this License, they do not excuse
-you from the conditions of this License. If you cannot distribute
-so as to satisfy simultaneously your obligations under this License
-and any other pertinent obligations, then as a consequence you may
-not distribute the Program at all. For example, if a patent license
-would not permit royalty-free redistribution of the Program by all
-those who receive copies directly or indirectly through you, then
-the only way you could satisfy both it and this License would be to
-refrain entirely from distribution of the Program.
-
-
-If any portion of this section is held invalid or unenforceable under
-any particular circumstance, the balance of the section is intended
-to apply and the section as a whole is intended to apply in other
-circumstances.
-
-It is not the purpose of this section to induce you to infringe any
-patents or other property right claims or to contest validity of any
-such claims; this section has the sole purpose of protecting the integrity
-of the free software distribution system, which is implemented by
-public license practices. Many people have made generous contributions
-to the wide range of software distributed through that system in reliance
-on consistent application of that system; it is up to the author/donor
-to decide if he or she is willing to distribute software through any
-other system and a licensee cannot impose that choice.
-
-This section is intended to make thoroughly clear what is believed
-to be a consequence of the rest of this License.
-
-\item If the distribution and/or use of the Program is restricted in certain
-countries either by patents or by copyrighted interfaces, the original
-copyright holder who places the Program under this License may add
-an explicit geographical distribution limitation excluding those countries,
-so that distribution is permitted only in or among countries not thus
-excluded. In such case, this License incorporates the limitation as
-if written in the body of this License.
-\item The Free Software Foundation may publish revised and/or new versions
-of the General Public License from time to time. Such new versions
-will be similar in spirit to the present version, but may differ in
-detail to address new problems or concerns.
-
-
-Each version is given a distinguishing version number. If the Program
-specifies a version number of this License which applies to it and
-``any later version,'' you have the option of following the terms
-and conditions either of that version or of any later version published
-by the Free Software Foundation. If the Program does not specify a
-version number of this License, you may choose any version ever published
-by the Free Software Foundation.
-
-\item If you wish to incorporate parts of the Program into other free programs
-whose distribution conditions are different, write to the author to
-ask for permission. For software which is copyrighted by the Free
-Software Foundation, write to the Free Software Foundation; we sometimes
-make exceptions for this. Our decision will be guided by the two goals
-of preserving the free status of all derivatives of our free software
-and of promoting the sharing and reuse of software generally.
-\end{enumerate}
-
-\subsection*{NO WARRANTY }
-
-\begin{itemize}
-
-\item[11.]BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS
-NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE
-LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS
-AND/OR OTHER PARTIES PROVIDE THE PROGRAM ``AS IS'' WITHOUT WARRANTY
-OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED
-TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
-PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE
-PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME
-THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
-
-\item[12.]IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED
-TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY
-MODIFY AND/OR REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE
-TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR
-CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE
-PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED
-INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE
-OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER
-OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
-
-\end{itemize}
-
-
-\section*{END OF TERMS AND CONDITIONS }
-
-
-\subsection*{How to Apply These Terms to Your New Programs}
-
-If you develop a new program, and you want it to be of the greatest
-possible use to the public, the best way to achieve this is to make
-it free software which everyone can redistribute and change under
-these terms.
-
-To do so, attach the following notices to the program. It is safest
-to attach them to the start of each source file to most effectively
-convey the exclusion of warranty; and each file should have at least
-the ``copyright'' line and a pointer to where the full notice is found.
-For example:
-
-\begin{quote}
-One line to give the program's name and a brief idea of what it does.
-Copyright {\footnotesize $\copyright$ (}year) (name of author)
-
-This program is free software; you can redistribute it and/or modify
-it under the terms of the GNU General Public License as published
-by the Free Software Foundation; either version 2 of the License,
-or (at your option) any later version.
-
-This program is distributed in the hope that it will be useful, but
-WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
-or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
-for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program; if not, write to the Free Software Foundation,
-Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
-\end{quote}
-Also add information on how to contact you by electronic and paper
-mail.
-
-If the program is interactive, make it output a short notice like
-this when it starts in an interactive mode:
-
-\begin{quote}
-Gnomovision version 69, Copyright $\copyright$ year name of author Gnomovision
-comes with ABSOLUTELY NO WARRANTY; for details type `show w'. This
-is free software, and you are welcome to redistribute it under certain
-conditions; type `show c' for details.
-\end{quote}
-The hypothetical commands `show w' and `show c' should show the appropriate
-parts of the General Public License. Of course, the commands you use
-may be called something other than `show w' and `show c'; they could
-even be mouse-clicks or menu items -- whatever suits your program.
-
-You should also get your employer (if you work as a programmer) or
-your school, if any, to sign a ``copyright disclaimer'' for the program,
-if necessary. Here is a sample; alter the names:
-
-\begin{quote}
-Yoyodyne, Inc., hereby disclaims all copyright interest in the program
-`Gnomovision' (which makes passes at compilers) written by James Hacker.
-
-(signature of Ty Coon)\\
-1 April 1989 \\
-Ty Coon, President of Vice
-\end{quote}
-This General Public License does not permit incorporating your program
-into proprietary programs. If your program is a subroutine library,
-you may consider it more useful to permit linking proprietary applications
-with the library. If this is what you want to do, use the GNU Library
-General Public License instead of this License.
-\end{document}
Copied: seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/USER_MANUAL/manual_SPECFEM3D_GLOBE.tex (from rev 16373, seismo/3D/SPECFEM3D_GLOBE/trunk/USER_MANUAL/manual_SPECFEM3D_GLOBE.tex)
===================================================================
--- seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/USER_MANUAL/manual_SPECFEM3D_GLOBE.tex (rev 0)
+++ seismo/3D/SPECFEM3D_GLOBE/tags/v5.0.0/USER_MANUAL/manual_SPECFEM3D_GLOBE.tex 2010-03-03 20:09:03 UTC (rev 16374)
@@ -0,0 +1,3367 @@
+%% LyX 1.5.1 created this file. For more info, see www.lyx.org/.
+
+\documentclass[oneside,english]{book}
+\usepackage[T1]{fontenc}
+\usepackage[latin1]{inputenc}
+\usepackage{geometry}
+\geometry{verbose,letterpaper,tmargin=1in,bmargin=1in,lmargin=1in,rmargin=1in}
+\setcounter{secnumdepth}{3}
+\setcounter{tocdepth}{3}
+\usepackage{longtable}
+\usepackage{varioref}
+\usepackage{float}
+\usepackage{textcomp}
+\usepackage{amsmath}
+\usepackage{color}
+
+% figures
+\usepackage[pdftex]{graphicx}
+
+% we are running pdflatex, so convert .eps files to .pdf
+\usepackage{epstopdf}
+
+\usepackage{amssymb}
+\IfFileExists{url.sty}{\usepackage{url}}
+ {\newcommand{\url}{\texttt}}
+\usepackage[authoryear]{natbib}
+
+\makeatletter
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% LyX specific LaTeX commands.
+%% Because html converters don't know tabularnewline
+\providecommand{\tabularnewline}{\\}
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Textclass specific LaTeX commands.
+\newenvironment{lyxcode}
+{\begin{list}{}{
+\setlength{\rightmargin}{\leftmargin}
+\setlength{\listparindent}{0pt}% needed for AMS classes
+\raggedright
+\setlength{\itemsep}{0pt}
+\setlength{\parsep}{0pt}
+\normalfont\ttfamily}%
+ \item[]}
+{\end{list}}
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% User specified LaTeX commands.
+% fonts
+\usepackage{times}
+\usepackage{natbib}
+
+% date of last edit
+\date{\today}
+
+% create thumbnails
+\usepackage[pdftex]{thumbpdf}
+
+% hyperlinks to sections and references
+\usepackage[pdftex,bookmarks=true,bookmarksnumbered=true,pdfpagemode=None,pdfstartview=FitH,pdfpagelayout=SinglePage,pdfborder={0 0 0}]{hyperref}
+
+\let\myUrl\url
+\renewcommand{\url}[1]{(\myUrl{#1})}
+
+% biblio GJI
+\bibliographystyle{abbrvnat}
+
+\newcommand{\toall}[1]{\textbf{*** All: #1 ***}}
+\newcommand{\tojeroen}[1]{\textbf{*** Jeroen: #1 ***}}
+\newcommand{\tobrian}[1]{\textbf{*** Brian: #1 ***}}
+\newcommand{\tovala}[1]{\textbf{*** Vala: #1 ***}}
+\newcommand{\tovalabrian}[1]{\textbf{*** Vala \& Brian: #1 ***}}
+\newcommand{\tovalaqinya}[1]{\textbf{*** Vala \& Qinya: #1 ***}}
+\newcommand{\toqinya}[1]{\textbf{*** Qinya: #1 ***}}
+\newcommand{\tomin}[1]{\textbf{*** Min: #1 ***}}
+\newcommand{\toalessia}[1]{\textbf{*** Alessia: #1 ***}}
+\newcommand{\todimitri}[1]{\textbf{*** Dimitri: #1 ***}}
+
+\newcommand{\nexxi}{\mbox{\texttt{NEX\_XI}}}
+\newcommand{\nexeta}{\mbox{\texttt{NEX\_ETA}}}
+\newcommand{\nprocxi}{\mbox{\texttt{NPROC\_XI}}}
+\newcommand{\nproceta}{\mbox{\texttt{NPROC\_ETA}}}
+\newcommand{\nchunks}{\mbox{\texttt{NCHUNKS}}}
+
+\usepackage{babel}
+\makeatother
+
+\begin{document}
+\begin{center}
+\thispagestyle{empty}\textbf{}%
+\begin{figure}[H]
+\noindent \includegraphics[width=0.75\paperwidth]{figures/specfem_3d_globe-cover.pdf}
+\end{figure}
+
+\par\end{center}
+
+
+\title{\thispagestyle{empty}\textbf{SPECFEM3D\_GLOBE}\\
+\textbf{User Manual}}
+
+
+\author{$\copyright$ Princeton University / California Institute of Technology (USA) and\\
+University of Pau / CNRS / INRIA (France)\\
+Version 5.0.0 `Tiger'}
+
+
+\date{\noindent \today}
+
+\maketitle
+\tableofcontents{}
+
+
+\chapter{Introduction}
+
+The software package SPECFEM3D\_GLOBE simulates three-dimensional
+global and regional seismic wave propagation based upon the spectral-element
+method (SEM). Effects due to lateral variations in compressional-wave
+speed, shear-wave speed, density, a 3D crustal model, ellipticity,
+topography and bathymetry, the oceans, rotation, and self-gravitation
+are included. For a detailed introduction to the SEM as applied to
+global and regional seismic wave propagation, please consult \citet{KoVi98,KoTr99,Ch00,KoTr02a,KoTr02b,KoRiTr02,ChCaVi03,CaChViMo03,ChVa04,ChKoViCaVaFe07,TrKoLi08}.
+If you use 3D mantle model S20RTS, please cite \citet{RiVaWo99}.
+The package can accommodate full 21-parameter anisotropy \citep{ChTr07}
+as well as lateral variations in attenuation \citep{SavWiTr05}. Adjoint
+capabilities and finite-frequency kernel simulations are also included
+\citep{LiTr06,TrKoLi08,LiTr08}.
+
+All SPECFEM3D\_GLOBE software is written in Fortran90 with full portability
+in mind, and conforms strictly to the Fortran95 standard. It uses
+no obsolete or obsolescent features of Fortran77. The package uses
+parallel programming based upon the Message Passing Interface (MPI)
+\citep{GrLuSk94,Pac97}.
+
+SPECFEM3D won the Gordon Bell award for best performance at the SuperComputing~2003
+conference in Phoenix, Arizona (USA) by running at 5 teraflops (sustained)
+on 1944 processors of the Japanese Earth Simulator using 14.6 billion
+degrees of freedom stored in 2.5 terabytes of memory; see \cite{KoTsChTr03}
+and the Gordon Bell Awards News Release \url{www.sc-conference.org/sc2003/nr_finalaward.html }
+for details. It was a finalist again in 2008 for a run at 0.16 petaflops (sustained) on 149,784 processors of the `Jaguar' Cray XT5 system at Oak Ridge National Laboratories (USA) \citep{CaKoLaTiMiLeSnTr08}.
+
+Future improvements will include support for GPU graphics card acceleration \citep{KoMiEr09,KoErGoMi10,MiKo10}
+as well as Convolutional-Perfectly Matched absorbing Layers (C-PML) \citep{KoMa07,MaKoEz08,MaKo09,MaKoGeBr10}.
+
+\section{Citation}
+
+If you use SPECFEM3D\_GLOBE for your own research, please cite at
+least one of the following articles: \cite{TrKoLi08,ChKoViCaVaFe07,KoRiTr02,KoTr02a,KoTr02b,KoTr99}
+or \cite{KoVi98}. The corresponding Bib\TeX{} entries may be found
+in file \texttt{USER\_MANUAL/bibliography.bib} or in comments at the
+beginning of file \texttt{specfem3D.f90}.
+
+
+\section{Support}
+
+This material is based upon work supported by the U.S. National Science
+Foundation under Grants No. EAR-0406751 and EAR-0711177, by the French
+CNRS, French INRIA Sud-Ouest MAGIQUE-3D, French ANR NUMASIS under
+Grant No. ANR-05-CIGC-002, and European FP6 Marie Curie International
+Reintegration Grant No. MIRG-CT-2005-017461.
+Older versions of the code were initially developed by Dimitri Komatitsch at Institut de Physique du Globe (France)
+and then by Dimitri Komatitsch and Jeroen Tromp at Harvard University (USA).
+Any opinions, findings, and conclusions or recommendations expressed in this material are
+those of the authors and do not necessarily reflect the views of the
+U.S. National Science Foundation, CNRS, INRIA, ANR or the European
+Marie Curie program.
+
+
+\chapter{\label{cha:Getting-Started}Getting Started}
+
+\section{Configuring and compiling the source code}
+
+The SPECFEM3D\_GLOBE software package comes in a gzipped tar ball.
+In the directory in which you want to install the package, type
+
+\begin{lyxcode}
+tar~-zxvf~SPECFEM3D\_GLOBE\_V4.0.3.tar.gz
+\end{lyxcode}
+The directory \texttt{SPECFEM3D\_GLOBE\_V4.0.3} will then contain
+the source code.
+
+To configure the software for your system, run the \texttt{configure}
+shell script. This script will attempt to guess the appropriate configuration
+values for your system. However, at a minimum, it is recommended that
+you explicitly specify the appropriate command names for your Fortran90
+compiler and MPI package:
+
+\begin{lyxcode}
+./configure~FC=ifort~MPIFC=mpif90
+\end{lyxcode}
+A summary of the most important configuration variables follows.
+
+\begin{description}
+\item [{\texttt{FC}}] Fortran90 compiler command name. By default, \texttt{configure}
+will execute the command names of various well-known Fortran compilers
+in succession, picking the first one it finds that works.
+\item [{\texttt{MPIFC}}] MPI Fortran90 command name. The default is \texttt{mpif90}.
+This must correspond to the same underlying compiler specified by
+\texttt{FC}; otherwise, you will encounter compilation or link errors
+when you attempt to build the code. If you are unsure about this,
+it is usually safe to set both \texttt{FC} and \texttt{MPIFC} to the
+MPI compiler command for your system:
+\end{description}
+\begin{lyxcode}
+./configure~FC=mpif90~MPIFC=mpif90
+\end{lyxcode}
+\begin{description}
+\item [{\texttt{FLAGS\_CHECK}}] Compiler flags for non-critical subroutines.
+\item [{\texttt{FLAGS\_NO\_CHECK}}] Compiler flags for creating fast, production-run
+code for critical subroutines.
+\item [{\texttt{LOCAL\_PATH\_IS\_ALSO\_GLOBAL}}] Set to \texttt{.false.}
+on most cluster applications. For reasons of speed, the (parallel)
+mesher typically writes a (parallel) database for the solver on the
+local disks of the compute nodes. Some systems have no local disks
+(e.g., BlueGene or the Earth Simulator) and other systems have a fast
+parallel file system, in which case this variable should be set to
+\texttt{.true.}. Note that this flag is not used by the mesher or
+the solver; it is only used for some of the post-processing.
+\end{description}
+In addition to reading configuration variables, \texttt{configure}
+accepts the following options:
+
+\begin{description}
+\item [{\texttt{-{}-enable-double-precision}}] The package can run either
+in single or in double precision. The default is single precision
+mode because this requires exactly half as much memory. To specify
+double precision mode, simply provide \texttt{-{}-enable-double-precision}
+as a command-line argument to \texttt{configure}. On a new system,
+it is definitely worth experimenting with single versus double precision
+simulations to determine which is faster. Note that on many current
+processors (e.g., Intel, AMD, IBM Power), single precision calculations
+are often significantly faster; the difference can typically be 10\%
+to 25\%. It is therefore worth trying single precision if you can.
+We recommend running the same calculation once in single precision
+and in double precision on your system and comparing the seismograms.
+If they are identical, you should probably select single precision
+for your future runs.
+\item [{\texttt{-{}-help}}] Directs \texttt{configure} to print a usage
+screen which provides a short description of all configuration variables
+and options. Note that the options relating to installation directories
+(e.g., \texttt{-{}-prefix}) do not apply to SPECFEM3D\_GLOBE.
+\end{description}
+The \texttt{configure} script runs a brief series of checks. Upon
+successful completion, it generates the files \texttt{Makefile}, \texttt{constants.h},
+and \texttt{precision.h} in the working directory.
+
+\begin{description}
+\item [{Note:}] If the \texttt{configure} script fails, and you don't know
+what went wrong, examine the log file \texttt{config.log}. This file
+contains a detailed transcript of all the checks \texttt{configure}
+performed. Most importantly, it includes the error output (if any)
+from your compiler.
+\end{description}
+The \texttt{configure} script automatically runs the script \texttt{flags.guess}.
+This helper script contains a number of suggested flags for various
+compilers; e.g., Portland, Intel, Absoft, NAG, Lahey, NEC, IBM and
+SGI. The software has run on a wide variety of compute platforms,
+e.g., various PC clusters and machines from Sun, SGI, IBM, Compaq,
+and NEC. The \texttt{flags.guess} script attempts to guess which compiler
+you are using (based upon the compiler command name) and choose the
+related optimization flags. The \texttt{configure} script then automatically
+inserts the suggested flags into \texttt{Makefile}. Note that \texttt{flags.guess}
+may fail to identify your compiler; and in any event, the default
+flags chosen by \texttt{flags.guess} are undoubtedly not optimal for
+your system. So, we encourage you to experiment with these flags (by
+editing the generated \texttt{Makefile} by hand) and to solicit advice
+from your system administrator. Selecting the right compiler and compiler
+flags can make a tremendous difference in terms of performance. We
+welcome feedback on your experience with various compilers and flags.
+
+On SGI systems, \texttt{flags.guess} automatically informs \texttt{configure}
+to insert `\texttt{`TRAP\_FPE=OFF}'' into the generated \texttt{Makefile}
+in order to turn underflow trapping off.
+
+Finally, before compiling, make sure that the subdirectories \texttt{obj},
+\texttt{bak} and \texttt{OUTPUT\_FILES} exist within the directory
+with the source code (\texttt{SPECFEM3D\_GLOBE\_V4.0.3}). The \texttt{go\_mesher}
+script discussed below automatically takes care of creating the \texttt{OUTPUT\_FILES}
+directory.
+
+\section{Running meshes that require more than 2 gigabytes of memory per processor core}
+
+If you run very large meshes on a relatively small number
+of processors, the memory size needed on each processor might become
+greater than 2 gigabytes, which is the upper limit for 32-bit addressing.
+In this case, on some compilers you may need to add \texttt{``-mcmodel=medium}''
+to the compiler options otherwise the compiler will display an error
+message (for instance \texttt{``relocation truncated to fit: R\_X86\_64\_PC32 against .bss''} or
+something similar).
+
+\section{Using a cross compiler}
+
+The \texttt{``configure''} script assumes that you will compile the code on the same kind of hardware
+as the machine on which you will run it. On some systems (for instance IBM Blue Gene) this might not be the case
+and you may compile the code using a cross compiler on a frontend computer that does not have the same
+architecture. In such a case, typing \texttt{``make all''} on the frontend will fail, but you can use one of these two solutions: \\
+
+\noindent
+1/ create a script that runs \texttt{``make all''} on a node instead of on the frontend, if the compiler is also installed on the nodes \\
+
+\noindent
+2/ after running the \texttt{``configure''} script, create two copies of the Makefile:
+
+\begin{verbatim}
+mv Makefile Makefile1
+cp Makefile1 Makefile2
+\end{verbatim}
+
+\noindent
+In \texttt{Makefile1} put this instead of the current values:
+
+\begin{verbatim}
+FLAGS_CHECK = -O0
+FLAGS_NO_CHECK = -O0
+\end{verbatim}
+
+\noindent
+and replace
+
+\begin{verbatim}
+create_header_file: $O/program_create_header_file.o $(LIBSPECFEM)
+ ${FCCOMPILE_CHECK} -o xcreate_header_file $O/program_create_header_file.o $(LIBSPECFEM)
+\end{verbatim}
+
+\noindent
+with
+
+\begin{verbatim}
+create_header_file: $O/program_create_header_file.o $O/exit_mpi.o $(LIBSPECFEM)
+ ${MPIFCCOMPILE_CHECK} -o xcreate_header_file $O/program_create_header_file.o \
+ $O/exit_mpi.o $(LIBSPECFEM)
+\end{verbatim}
+
+\noindent
+In \texttt{Makefile2} comment out these two lines:
+
+\begin{verbatim}
+#OUTPUT_FILES/values_from_mesher.h: xcreate_header_file
+# ./xcreate_header_file
+\end{verbatim}
+
+\noindent
+Then:
+
+\begin{verbatim}
+ make -f Makefile1 clean
+ make -f Makefile1 create_header_file
+ ./xcreate_header_file
+ make -f Makefile2 clean
+ make -f Makefile2 meshfem3D
+ make -f Makefile2 specfem3D
+\end{verbatim}
+
+\noindent
+should work.
+
+\chapter{\label{cha:Running-the-Mesher}Running the Mesher \texttt{xmeshfem3D}}
+
+You are now ready to compile the mesher. In the directory with the
+source code, type `\texttt{make meshfem3D}'. If all paths and flags
+have been set correctly, the mesher should now compile and produce
+the executable \texttt{xmeshfem3D}.
+
+Input for the mesher (and the solver) is provided through the parameter
+file \texttt{Par\_file}, which resides in the subdirectory \texttt{DATA}.
+Before running the mesher, a number of parameters need to be set in
+the \texttt{Par\_file}. This requires a basic understanding of how
+the SEM is implemented, and we encourage you to read \citet{KoVi98,KoTr99,Ch00,KoTr02a,KoTr02b,KoRiTr02,ChCaVi03,CaChViMo03}
+and \citet{ChVa04}. A detailed theoretical analysis of the dispersion
+and stability properties of the SEM is available in \citet{DeSe07}
+and \citet{SeOl07}.
+
+In this chapter we will focus on simulations at the scale of the entire
+globe. Regional simulations will be addressed in Chapter~\ref{cha:Regional-Simulations}.
+The spectral-element mesh for a SPECFEM3D\_GLOBE simulation is based
+upon a mapping from the cube to the sphere called the \textit{cubed
+sphere} \citep{Sad72,RoIaPa96}. This cubed-sphere mapping breaks
+the globe into 6~chunks, each of which is further subdivided in terms
+of $n^{2}$ mesh slices, where $n\ge1$ is a positive integer, for
+a total of $6\times n^{2}$ slices (Figure~\ref{figure:mpi_slices}).
+Thus the minimum number of processors required for a global simulation
+is 6 (although it is theoretically possible to run more than one slice
+per processor). %
+\begin{figure}
+\centerline{ \begin{tabular}{cc}
+\includegraphics[width=0.45\textwidth]{figures/mpi_slices} & \includegraphics[width=0.45\textwidth]{figures/fullmesh_18} \tabularnewline
+\end{tabular}}
+
+\caption{Each of the 6~chunks that constitutes the cubed sphere is subdivided
+in terms of $n^{2}$~slices of elements, where $n\ge1$ is a positive
+integer, for a total of $6\times n^{2}$ slices (and therefore processors).
+The figure on the left shows a mesh that is divided in terms of $6\times5^{2}=150$
+slices as indicated by the various colors. In this cartoon, each slice
+contains $5\times5=25$ spectral elements at the Earth's surface.
+The figure on the right shows a mesh that is divided over $6\times18^{2}=1944$
+processors as indicated by the various colors. Regional simulations
+can be accommodated by using only 1, 2 or 3 chunks of the cubed sphere.
+One-chunk simulations may involve a mesh with lateral dimensions smaller
+than~$90^{\circ}$, thereby accommodating smaller-scale simulations. }
+
+
+\label{figure:mpi_slices}
+\end{figure}
+
+
+To run the mesher for a global simulation, the following parameters
+need to be set in the \texttt{Par\_file}:
+
+\begin{description}
+\item [{\texttt{SIMULATION\_TYPE}}] is set to 1 for forward simulations,
+2 for adjoint simulations (see Section \ref{sec:Adjoint-simulation-finite})
+and 3 for kernel simulations (see Section \ref{sec:Finite-Frequency-Kernels}).
+\item [{\texttt{SAVE\_FORWARD}}] is only set to \texttt{.true.} for a forward
+simulation with the last frame of the simulation saved, as part of
+the finite-frequency kernel calculations (see Section \ref{sec:Finite-Frequency-Kernels}).
+For a regular forward simulation, leave \texttt{SIMULATION\_TYPE}
+and \texttt{SAVE\_FORWARD} at their default values.
+\item [{\texttt{NCHUNKS}}] must be set to 6 for global simulations.
+\item [{\texttt{ANGULAR\_WIDTH\_XI\_IN\_DEGREES}}] Not needed for a global
+simulation. (See Chapter~\ref{cha:Regional-Simulations} for regional
+simulations.)
+\item [{\texttt{ANGULAR\_WIDTH\_ETA\_IN\_DEGREES}}] Not needed for a global
+simulation. (See Chapter~\ref{cha:Regional-Simulations} for regional
+simulations.)
+\item [{\texttt{CENTER\_LATITUDE\_IN\_DEGREES}}] Not needed for a global
+simulation. (See Chapter~\ref{cha:Regional-Simulations} for regional
+simulations.)
+\item [{\texttt{CENTER\_LONGITUDE\_IN\_DEGREES}}] Not needed for a global
+simulation. (See Chapter~\ref{cha:Regional-Simulations} for regional
+simulations.)
+\item [{\texttt{GAMMA\_ROTATION\_AZIMUTH}}] Not needed for a global simulation.
+(See Chapter~\ref{cha:Regional-Simulations} for regional simulations.)
+\item [{$\nexxi$}] The number of spectral elements along one side of a
+chunk in the cubed sphere (see Figure~\ref{figure:mpi_slices});
+this number \textit{must} be a multiple of 16 and 8~$\times$~a
+multiple of $\nprocxi$ defined below. We do not recommend using $\nexxi$
+less than 64 because the curvature of the Earth cannot be honored
+if one uses too few elements, and distorted elements can lead to inaccurate
+and unstable simulations, i.e., smaller values of $\nexxi$ are likely
+to result in spectral elements with a negative Jacobian, in which
+case the mesher will exit with an error message. Table~\ref{table:nex}
+summarizes various suitable choices for $\nexxi$ and the related
+values of $\nprocxi$. Based upon benchmarks against semi-analytical
+normal-mode synthetic seismograms, \citet{KoTr02a,KoTr02b} determined
+that a $\nexxi=256$ run is accurate to a shortest period of roughly
+17~s. Therefore, since accuracy is determined by the number of grid
+points per shortest wavelength, for any particular value of $\nexxi$
+the simulation will be accurate to a shortest period determined approximately
+by \begin{equation}
+\mbox{shortest period (s)}\simeq(256/\nexxi)\times17.\label{eq:shortest_period}\end{equation}
+ The number of grid points in each orthogonal direction of the reference
+element, i.e., the number of Gauss-Lobatto-Legendre points, is determined
+by \texttt{NGLLX} in the \texttt{constants.h} file. In the globe we
+use $\mbox{\texttt{NGLLX}}=5$, for a total of $5^{3}=125$ points
+per elements. We suggest not to change this value.
+\item [{$\nexeta$}] For global simulations $\nexeta$ must be set to the
+same value as $\nexxi$.
+\item [{$\nprocxi$}] The number of processors or slices along one chunk
+of the cubed sphere (see Figure~\ref{figure:mpi_slices}); we must
+have $\nexxi=8\times c\times\nprocxi$, where $c\ge1$ is a positive
+integer. See Table~\ref{table:nex} for various suitable choices.
+\item [{$\nproceta$}] For global simulations $\nproceta$ must be set
+to the same value as $\nprocxi$.
+\item [{\texttt{MODEL}}] Must be set to one of the following:
+\item [{\textmd{1D~models~with~real~structure:}}]~
+
+\begin{description}
+\item [{\texttt{1D\_isotropic\_prem}}] Isotropic version of the spherically
+symmetric Preliminary Reference Earth Model (PREM) \citep{DzAn81}.
+\item [{\texttt{1D\_transversely\_isotropic\_prem}}] Transversely isotropic
+version of PREM.
+\item [{\texttt{1D\_iasp91}}] Spherically symmetric isotropic IASP91 model
+\citep{KeEn91}.
+\item [{\texttt{1D\_1066a}}] Spherically symmetric earth model 1066A \citep{gilbertdziewonski1975}.
+When \texttt{\small ATTENTUATION} is on, it uses an unpublished 1D
+attenuation model from Scripps.
+\item [{\texttt{1D\_ak135}}] Spherically symmetric isotropic AK135 model
+\citep{KeEnBu95}.
+\item [{\texttt{1}D\_ref}] A recent 1D Earth model developed by \citet{KuDzEk06}.
+This model is the 1D background model for the 3D models s362ani, s362wmani,
+s362ani\_prem, and s29ea.
+\end{description}
+\end{description}
+For historical reasons and to provide benchmarks against normal-mode
+synthetics, the mesher accommodates versions of various 1D models
+with a single crustal layer with the properties of the original upper
+crust. These `one-crust' models are:
+
+\texttt{1D\_isotropic\_prem\_onecrust}
+
+\texttt{1D\_transversely\_isotropic\_prem\_onecrust}
+
+\texttt{1D\_iasp91\_onecrust}, \texttt{1D\_1066a\_onecrust}
+
+\texttt{1D\_ak135\_onecrust}
+
+\begin{description}
+\item [{\textmd{Fully~3D~models:}}]~
+
+\begin{description}
+\item [{\texttt{transversely\_isotropic\_prem\_plus\_3D\_crust\_2.0}}] This
+model has CRUST2.0 \citep{BaLaMa00} on top of a transversely isotropic
+PREM. We first extrapolate PREM mantle velocity up to the surface,
+then overwrite the model with CRUST2.0
+\item [{\texttt{s20rts}}] By default, the code uses 3D mantle model S20RTS
+\citep{RiVaWo99} and 3D crustal model Crust2.0 \citep{BaLaMa00}.
+Note that S20RTS uses transversely isotropic PREM as a background
+model, and that we use the PREM radial attenuation model when \texttt{ATTENUATION}
+is incorporated. See Chapter~\ref{cha:-Changing-the} for a discussion
+on how to change 3D models.
+\item [{\texttt{\textcolor{black}{s362ani}}}] A global shear-wave speed
+model developed by \citet{KuDzEk06}. In this model, radial anisotropy
+is confined to the uppermost mantle. The model (and the corresponding
+mesh) incorporate tomography on the 650\textasciitilde{}km and 410\textasciitilde{}km
+discontinuities in the 1D reference model REF.
+\item [{\texttt{\textcolor{black}{s362wmani}}}] A version of S362ANI with
+anisotropy allowed throughout the mantle.
+\item [{\texttt{\textcolor{black}{s362ani\_prem}}}] A version of S362ANI
+calculated using PREM as the 1D reference model.
+\item [{\texttt{\textcolor{black}{s29ea}}}] A global model with higher
+resolution in the upper mantle beneath Eurasia calculated using REF
+as the 1D reference model.
+\item [{\texttt{3D\_anisotropic}}] See Chapter~\ref{cha:-Changing-the}
+for a discussion on how to specify your own 3D anisotropic model.
+\item [{\texttt{3D\_attenuation}}] See Chapter~\ref{cha:-Changing-the}
+for a discussion on how to specify your own 3D attenuation model.
+\end{description}
+\item [{\textmd{NOTE:}}]~\\
+When a 3D mantle model is chosen in \texttt{Par\_file}, the simulations are performed together with the 3D crustal model Crust2.0. Alternatively, Crust2.0 can be combined with a higher resolution European crustal model EUCrust07 \citep{EUCrust07}. This can be done by setting the crustal type to \texttt{ICRUST\_CRUSTMAPS} in the \texttt{constant.h} file.
+It is also possible to run simulations using a 3D mantle model with a 1D crustal model on top. This can be done by setting the model in \texttt{Par\_file} to \texttt{<3D mantle>\_1Dcrust}, e.g., \texttt{s20rts\_1Dcrust, s362ani\_1Dcrust}, etc. In this case, the 1D crustal model will be the one that is used in the 3D mantle model as a reference model (e.g., transversely isotropic PREM for s20rts, REF for s362ani, etc.).
+
+\item [{\texttt{OCEANS}}] Set to \texttt{.true.} if the effect of the oceans
+on seismic wave propagation should be incorporated based upon the
+approximate treatment discussed in \citet{KoTr02b}. This feature
+is inexpensive from a numerical perspective, both in terms of memory
+requirements and CPU time. This approximation is accurate at periods
+of roughly 20~s and longer. At shorter periods the effect of water
+phases/reverberations is not taken into account, even when the flag
+is on.
+\item [{\texttt{ELLIPTICITY}}] Set to \texttt{.true.} if the mesh should
+make the Earth model elliptical in shape according to Clairaut's equation
+\citep{DaTr98}. This feature adds no cost to the simulation.
+\item [{\texttt{TOPOGRAPHY}}] Set to \texttt{.true.} if topography and
+bathymetry should be incorporated based upon model ETOPO5 \citep{Etopo5}.
+This feature adds no cost to the simulation.
+\item [{\texttt{GRAVITY}}] Set to \texttt{.true.} if self-gravitation should
+be incorporated in the Cowling approximation \citep{KoTr02b,DaTr98}.
+Turning this feature on is relatively inexpensive, both from the perspective
+of memory requirements as well as in terms of computational speed.
+\item [{\texttt{ROTATION}}] Set to \texttt{.true.} if the Coriolis effect
+should be incorporated. Turning this feature on is relatively cheap
+numerically.
+\item [{\texttt{ATTENUATION}}] Set to \texttt{.true.} if attenuation should
+be incorporated. Turning this feature on increases the memory requirements
+significantly (roughly by a factor of~1.5), and is numerically fairly
+expensive. Of course for realistic simulations this flag should be
+turned on. See \citet{KoTr99,KoTr02a} for a discussion on the implementation
+of attenuation based upon standard linear solids.
+\item [{\texttt{ABSORBING\_CONDITIONS}}] Set to \texttt{.false.} for global
+simulations. See Chapter~\ref{cha:Regional-Simulations} for regional
+simulations.
+\item [{\texttt{RECORD\_LENGTH\_IN\_MINUTES}}] Choose the desired record
+length of the synthetic seismograms (in minutes). This controls the
+length of the numerical simulation, i.e., twice the record length
+requires twice as much CPU time. This feature is not used at the time
+of meshing but is required for the solver, i.e., you may change this
+parameter after running the mesher.
+\item [{\texttt{MOVIE\_SURFACE}}] Set to \texttt{.false.}, unless you want
+to create a movie of seismic wave propagation on the Earth's surface.
+Turning this option on generates large output files. See Section \ref{sec:Movies}
+for a discussion on the generation of movies. This feature is not
+used at the time of meshing but is relevant for the solver.
+\item [{\texttt{MOVIE\_VOLUME}}] Set to \texttt{.false.}, unless you want
+to create a movie of seismic wave propagation in the Earth's interior.
+Turning this option on generates huge output files. See Section \ref{sec:Movies}
+for a discussion on the generation of movies. This feature is not
+used at the time of meshing but is relevant for the solver.
+\item [{\texttt{NTSTEP\_BETWEEN\_FRAMES}}] Determines the number of timesteps
+between movie frames. Typically you want to save a snapshot every
+100 timesteps. The smaller you make this number the more output will
+be generated! See Section \ref{sec:Movies} for a discussion on the
+generation of movies. This feature is not used at the time of meshing
+but is relevant for the solver.
+\item [{\texttt{HDUR\_MOVIE}}] determines the half duration of the source
+time function for the movie simulations. When this parameter is set
+to be 0, a default half duration that corresponds to the accuracy
+of the simulation is provided.
+\item [{\texttt{SAVE\_MESH\_FILES}}] Set this flag to \texttt{.true}\texttt{\small .}
+to save AVS \url{www.avs.com}, OpenDX \url{www.opendx.org}, or ParaView \url{www.paraview.org}
+mesh files for subsequent viewing. Turning the flag on generates large
+(distributed) files in the \texttt{LOCAL\_PATH} directory. See Section~\ref{sec:Meshes}
+for a discussion of mesh viewing features.
+\item [{\texttt{NUMBER\_OF\_RUNS}}] On machines with a run-time limit,
+for instance for a batch/queue system, a simulation may need to be
+completed in stages. This option allows you to select the number of
+stages in which the simulation will be completed (1, 2 or 3). Choose
+1 for a run without restart files. This feature is not used at the
+time of meshing but is required for the solver. At the end of the
+first or second stage of a multi-stage simulation, large files are
+written to the file system to save the current state of the simulation.
+This state is read back from the file system at the beginning of the
+next stage of the multi-stage run. Reading and writing the states
+can be very time consuming depending on the nature of the network
+and the file system (in this case writing to the local file system,
+i.e., the disk on a node, is preferable).
+\item [{\texttt{NUMBER\_OF\_THIS\_RUN}}] If you choose to perform the run
+in stages, you need to tell the solver what stage run to perform.
+This feature is not used at the time of meshing but is required for
+the solver.
+\item [{\texttt{LOCAL\_PATH}}] Directory in which the databases generated
+by the mesher will be written. Generally one uses a directory on the
+local disk of the compute nodes, although on some machines these databases
+are written on a parallel (global) file system (see also the earlier
+discussion of the \texttt{LOCAL\_PATH\_IS\_ALSO\_GLOBAL} flag in Chapter~\ref{cha:Getting-Started}).
+The mesher generates the necessary databases in parallel, one set
+for each of the $6\times\nprocxi^{2}$ slices that constitutes the
+mesh (see Figure~\ref{figure:mpi_slices}). After the mesher finishes,
+you can log in to one of the compute nodes and view the contents of
+the \texttt{LOCAL\_PATH} directory to see the (many) files generated
+by the mesher.
+\item [{\texttt{NTSTEP\_BETWEEN\_OUTPUT\_INFO}}] This parameter specifies
+the interval at which basic information about a run is written to
+the file system (\texttt{timestamp{*}} files in the \texttt{OUTPUT\_FILES}
+directory). If you have access to a fast machine, set \texttt{NTSTEP\_BETWEEN\_OUTPUT\_INFO}
+to a relatively high value (e.g., at least 100, or even 1000 or more)
+to avoid writing output text files too often. This feature is not
+used at the time of meshing. One can set this parameter to a larger
+value than the number of time steps to avoid writing output during
+the run.
+\item [{\texttt{NTSTEP\_BETWEEN\_OUTPUT\_SEISMOS}}] This parameter specifies
+the interval at which synthetic seismograms are written in the \texttt{LOCAL\_PATH}
+directory. The seismograms can be created in three different formats
+by setting the parameters \texttt{OUTPUT\_SEISMOS\_ASCII\_TEXT}, \texttt{OUTPUT\_SEISMOS\_SAC\_ALPHANUM}
+and \texttt{OUTPUT\_SEI-}~\\
+\texttt{SMOS\_SAC\_BINARY}. One can choose any combination of these
+parameters (details on the formats follow in the description of each
+parameter). SAC \url{www.iris.edu/software/sac/} is a signal-processing software
+package. If a run crashes, you may still find usable (but shorter
+than requested) seismograms in this directory. On a fast machine set
+\texttt{NTSTEP\_BETWEEN\_OUTPUT\_SEISMOS} to a relatively high value
+to avoid writing to the seismograms too often. This feature is not
+used at the time of meshing.
+\item [{\texttt{NTSTEP\_BETWEEN\_READ\_ADJSRC}}] The number of adjoint
+sources read in each time for an adjoint simulation.
+\item [{\texttt{OUTPUT\_SEISMOS\_ASCII\_TEXT}}] Set this flag to \texttt{.true.}
+if you want to have the synthetic seismograms written in two-column
+ASCII format (the first column contains time in seconds and the second
+column the displacement in meters of the recorded signal, no header
+information). Files will be named with extension \texttt{.ascii}.
+\item [{\texttt{OUTPUT\_SEISMOS\_SAC\_ALPHANUM}}] Set this flag to \texttt{.true.}
+if you want to have the synthetic seismograms written in alphanumeric
+(human readable) SAC format, which includes header information on
+the source and receiver parameters (e.g., source/receiver coordinates,
+station name, etc., see Appendix~\ref{cha:SAC-headers}). For details on the format, please check the SAC webpage \url{www.iris.edu/software/sac/}. Files will be named with extension \texttt{.sacan}.
+\item [{\texttt{OUTPUT\_SEISMOS\_SAC\_BINARY}}] Set this flag to \texttt{.true.}
+if you want to have the synthetic seismograms written in binary SAC
+format. The header information included is the same as for the alphanumeric
+SAC format (see Appendix~\ref{cha:SAC-headers}). Using this format requires the least disk space, which may be particulary important if you have a large number of stations.
+For details on the binary format please also check the SAC webpage \url{www.iris.edu/software/sac/} and Appendix~\ref{cha:SAC-headers}. Files will be named with extension \texttt{.sac}.
+\item [{\texttt{ROTATE\_SEISMOGRAMS\_RT}}] Set this flag to \texttt{.true.}
+if you want to have radial (R) and transverse (T) horizontal components
+of the synthetic seismograms (default is \texttt{.false.} $\rightarrow$
+East (E) and North (N) components).
+\item [{\texttt{WRITE\_SEISMOGRAMS\_BY\_MASTER}}] Set this flag to \texttt{.true.}
+if you want to have all the seismograms written by the master (no
+need to collect them on the nodes after the run).
+\item [{\texttt{SAVE\_ALL\_SEISMOS\_IN\_ONE\_FILE}}] Set this flag to \texttt{.true.}
+if you want to have all the seismograms saved in one large combined
+file instead of one file per seismogram to avoid overloading shared
+non-local file systems such as GPFS for instance.
+\item [{\texttt{USE\_BINARY\_FOR\_LARGE\_FILE}}] Set this flag to \texttt{.true.}
+if you want to use binary instead of ASCII for that large file (not
+used if SAVE\_ALL\_SEISMOS\_IN \_ONE\_FILE = .false.)
+\item [{\texttt{RECEIVERS\_CAN\_BE\_BURIED}}] This flag accommodates stations
+with instruments that are buried, i.e., the solver will calculate
+seismograms at the burial depth specified in the \texttt{STATIONS}
+file. This feature is not used at the time of meshing.
+\item [{\texttt{PRINT\_SOURCE\_TIME\_FUNCTION}}] Turn this flag on to print
+information about the source time function in the file \texttt{OUTPUT\_FILES/plot\_source\_time\_function.txt}.
+This feature is not used at the time of meshing.
+\end{description}
+%
+\clearpage
+\noindent \begin{center}
+\label{table:nex} \begin{longtable}{|c|c|c|c|c|c|c|c|c|c|c|c|}
+\hline
+\nprocxi & processors & \multicolumn{10}{c|}{\nexxi}\tabularnewline
+\hline
+\endhead
+\hline
+1 & 6 & 64 & 80 & 96 & 112 & 128 & 144 & 160 & 176 & 192 & 208\tabularnewline
+\hline
+2 & 24 & 64 & 80 & 96 & 112 & 128 & 144 & 160 & 176 & 192 & 208\tabularnewline
+\hline
+3 & 54 & 96 & 144 & 192 & 240 & 288 & 336 & 384 & 432 & 480 & 528\tabularnewline
+\hline
+4 & 96 & 64 & 96 & 128 & 160 & 192 & 224 & 256 & 288 & 320 & 352\tabularnewline
+\hline
+5 & 150 & 80 & 160 & 240 & 320 & 400 & 480 & 560 & 640 & 720 & 800\tabularnewline
+\hline
+6 & 216 & 96 & 144 & 192 & 240 & 288 & 336 & 384 & 432 & 480 & 528\tabularnewline
+\hline
+7 & 294 & 112 & 224 & 336 & 448 & 560 & 672 & 784 & 896 & 1008 & 1120\tabularnewline
+\hline
+8 & 384 & 64 & 128 & 192 & 256 & 320 & 384 & 448 & 512 & 576 & 640\tabularnewline
+\hline
+9 & 486 & 144 & 288 & 432 & 576 & 720 & 864 & 1008 & 1152 & 1296 & 1440\tabularnewline
+\hline
+10 & 600 & 80 & 160 & 240 & 320 & 400 & 480 & 560 & 640 & 720 & 800\tabularnewline
+\hline
+11 & 726 & 176 & 352 & 528 & 704 & 880 & 1056 & 1232 & 1408 & 1584 & 1760\tabularnewline
+\hline
+12 & 864 & 96 & 192 & 288 & 384 & 480 & 576 & 672 & 768 & 864 & 960\tabularnewline
+\hline
+13 & 1014 & 208 & 416 & 624 & 832 & 1040 & 1248 & 1456 & 1664 & 1872 & 2080\tabularnewline
+\hline
+14 & 1176 & 112 & 224 & 336 & 448 & 560 & 672 & 784 & 896 & 1008 & 1120\tabularnewline
+\hline
+15 & 1350 & 240 & 480 & 720 & 960 & 1200 & 1440 & 1680 & 1920 & 2160 & 2400\tabularnewline
+\hline
+16 & 1536 & 128 & 256 & 384 & 512 & 640 & 768 & 896 & 1024 & 1152 & 1280\tabularnewline
+\hline
+17 & 1734 & 272 & 544 & 816 & 1088 & 1360 & 1632 & 1904 & 2176 & 2448 & 2720\tabularnewline
+\hline
+18 & 1944 & 144 & 288 & 432 & 576 & 720 & 864 & 1008 & 1152 & 1296 & 1440\tabularnewline
+\hline
+19 & 2166 & 304 & 608 & 912 & 1216 & 1520 & 1824 & 2128 & 2432 & 2736 & 3040\tabularnewline
+\hline
+20 & 2400 & 160 & 320 & 480 & 640 & 800 & 960 & 1120 & 1280 & 1440 & 1600\tabularnewline
+\hline
+21 & 2646 & 336 & 672 & 1008 & 1344 & 1680 & 2016 & 2352 & 2688 & 3024 & 3360\tabularnewline
+\hline
+22 & 2904 & 176 & 352 & 528 & 704 & 880 & 1056 & 1232 & 1408 & 1584 & 1760\tabularnewline
+\hline
+23 & 3174 & 368 & 736 & 1104 & 1472 & 1840 & 2208 & 2576 & 2944 & 3312 & 3680\tabularnewline
+\hline
+24 & 3456 & 192 & 384 & 576 & 768 & 960 & 1152 & 1344 & 1536 & 1728 & 1920\tabularnewline
+\hline
+25 & 3750 & 400 & 800 & 1200 & 1600 & 2000 & 2400 & 2800 & 3200 & 3600 & 4000\tabularnewline
+\hline
+26 & 4056 & 208 & 416 & 624 & 832 & 1040 & 1248 & 1456 & 1664 & 1872 & 2080\tabularnewline
+\hline
+27 & 4374 & 432 & 864 & 1296 & 1728 & 2160 & 2592 & 3024 & 3456 & 3888 & 4320\tabularnewline
+\hline
+28 & 4704 & 224 & 448 & 672 & 896 & 1120 & 1344 & 1568 & 1792 & 2016 & 2240\tabularnewline
+\hline
+29 & 5046 & 464 & 928 & 1392 & 1856 & 2320 & 2784 & 3248 & 3712 & 4176 & 4640\tabularnewline
+\hline
+30 & 5400 & 240 & 480 & 720 & 960 & 1200 & 1440 & 1680 & 1920 & 2160 & 2400\tabularnewline
+\hline
+31 & 5766 & 496 & 992 & 1488 & 1984 & 2480 & 2976 & 3472 & 3968 & 4464 & 4960\tabularnewline
+\hline
+32 & 6144 & 256 & 512 & 768 & 1024 & 1280 & 1536 & 1792 & 2048 & 2304 & 2560\tabularnewline
+\hline
+33 & 6534 & 528 & 1056 & 1584 & 2112 & 2640 & 3168 & 3696 & 4224 & 4752 & 5280\tabularnewline
+\hline
+34 & 6936 & 272 & 544 & 816 & 1088 & 1360 & 1632 & 1904 & 2176 & 2448 & 2720\tabularnewline
+\hline
+35 & 7350 & 560 & 1120 & 1680 & 2240 & 2800 & 3360 & 3920 & 4480 & 5040 & 5600\tabularnewline
+\hline
+36 & 7776 & 288 & 576 & 864 & 1152 & 1440 & 1728 & 2016 & 2304 & 2592 & 2880\tabularnewline
+\hline
+37 & 8214 & 592 & 1184 & 1776 & 2368 & 2960 & 3552 & 4144 & 4736 & 5328 & 5920\tabularnewline
+\hline
+38 & 8664 & 304 & 608 & 912 & 1216 & 1520 & 1824 & 2128 & 2432 & 2736 & 3040\tabularnewline
+\hline
+39 & 9126 & 624 & 1248 & 1872 & 2496 & 3120 & 3744 & 4368 & 4992 & 5616 & 6240\tabularnewline
+\hline
+40 & 9600 & 320 & 640 & 960 & 1280 & 1600 & 1920 & 2240 & 2560 & 2880 & 3200\tabularnewline
+\hline
+41 & 10086 & 656 & 1312 & 1968 & 2624 & 3280 & 3936 & 4592 & 5248 & 5904 & 6560\tabularnewline
+\hline
+42 & 10584 & 336 & 672 & 1008 & 1344 & 1680 & 2016 & 2352 & 2688 & 3024 & 3360\tabularnewline
+\hline
+43 & 11094 & 688 & 1376 & 2064 & 2752 & 3440 & 4128 & 4816 & 5504 & 6192 & 6880\tabularnewline
+\hline
+44 & 11616 & 352 & 704 & 1056 & 1408 & 1760 & 2112 & 2464 & 2816 & 3168 & 3520\tabularnewline
+\hline
+45 & 12150 & 720 & 1440 & 2160 & 2880 & 3600 & 4320 & 5040 & 5760 & 6480 & 7200\tabularnewline
+\hline
+46 & 12696 & 368 & 736 & 1104 & 1472 & 1840 & 2208 & 2576 & 2944 & 3312 & 3680\tabularnewline
+\hline
+47 & 13254 & 752 & 1504 & 2256 & 3008 & 3760 & 4512 & 5264 & 6016 & 6768 & 7520\tabularnewline
+\hline
+48 & 13824 & 384 & 768 & 1152 & 1536 & 1920 & 2304 & 2688 & 3072 & 3456 & 3840\tabularnewline
+\hline
+49 & 14406 & 784 & 1568 & 2352 & 3136 & 3920 & 4704 & 5488 & 6272 & 7056 & 7840\tabularnewline
+\hline
+50 & 15000 & 400 & 800 & 1200 & 1600 & 2000 & 2400 & 2800 & 3200 & 3600 & 4000\tabularnewline
+\hline
+51 & 15606 & 816 & 1632 & 2448 & 3264 & 4080 & 4896 & 5712 & 6528 & 7344 & 8160\tabularnewline
+\hline
+52 & 16224 & 416 & 832 & 1248 & 1664 & 2080 & 2496 & 2912 & 3328 & 3744 & 4160\tabularnewline
+\hline
+53 & 16854 & 848 & 1696 & 2544 & 3392 & 4240 & 5088 & 5936 & 6784 & 7632 & 8480\tabularnewline
+\hline
+54 & 17496 & 432 & 864 & 1296 & 1728 & 2160 & 2592 & 3024 & 3456 & 3888 & 4320\tabularnewline
+\hline
+55 & 18150 & 880 & 1760 & 2640 & 3520 & 4400 & 5280 & 6160 & 7040 & 7920 & 8800\tabularnewline
+\hline
+56 & 18816 & 448 & 896 & 1344 & 1792 & 2240 & 2688 & 3136 & 3584 & 4032 & 4480\tabularnewline
+\hline
+57 & 19494 & 912 & 1824 & 2736 & 3648 & 4560 & 5472 & 6384 & 7296 & 8208 & 9120\tabularnewline
+\hline
+58 & 20184 & 464 & 928 & 1392 & 1856 & 2320 & 2784 & 3248 & 3712 & 4176 & 4640\tabularnewline
+\hline
+59 & 20886 & 944 & 1888 & 2832 & 3776 & 4720 & 5664 & 6608 & 7552 & 8496 & 9440\tabularnewline
+\hline
+60 & 21600 & 480 & 960 & 1440 & 1920 & 2400 & 2880 & 3360 & 3840 & 4320 & 4800\tabularnewline
+\hline
+61 & 22326 & 976 & 1952 & 2928 & 3904 & 4880 & 5856 & 6832 & 7808 & 8784 & 9760\tabularnewline
+\hline
+62 & 23064 & 496 & 992 & 1488 & 1984 & 2480 & 2976 & 3472 & 3968 & 4464 & 4960\tabularnewline
+\hline
+63 & 23814 & 1008 & 2016 & 3024 & 4032 & 5040 & 6048 & 7056 & 8064 & 9072 & 10080\tabularnewline
+\hline
+64 & 24576 & 512 & 1024 & 1536 & 2048 & 2560 & 3072 & 3584 & 4096 & 4608 & 5120\tabularnewline
+\hline
+65 & 25350 & 1040 & 2080 & 3120 & 4160 & 5200 & 6240 & 7280 & 8320 & 9360 & 10400\tabularnewline
+\hline
+66 & 26136 & 528 & 1056 & 1584 & 2112 & 2640 & 3168 & 3696 & 4224 & 4752 & 5280\tabularnewline
+\hline
+67 & 26934 & 1072 & 2144 & 3216 & 4288 & 5360 & 6432 & 7504 & 8576 & 9648 & 10720\tabularnewline
+\hline
+68 & 27744 & 544 & 1088 & 1632 & 2176 & 2720 & 3264 & 3808 & 4352 & 4896 & 5440\tabularnewline
+\hline
+69 & 28566 & 1104 & 2208 & 3312 & 4416 & 5520 & 6624 & 7728 & 8832 & 9936 & 11040\tabularnewline
+\hline
+70 & 29400 & 560 & 1120 & 1680 & 2240 & 2800 & 3360 & 3920 & 4480 & 5040 & 5600\tabularnewline
+\hline
+71 & 30246 & 1136 & 2272 & 3408 & 4544 & 5680 & 6816 & 7952 & 9088 & 10224 & 11360\tabularnewline
+\hline
+72 & 31104 & 576 & 1152 & 1728 & 2304 & 2880 & 3456 & 4032 & 4608 & 5184 & 5760\tabularnewline
+\hline
+73 & 31974 & 1168 & 2336 & 3504 & 4672 & 5840 & 7008 & 8176 & 9344 & 10512 & 11680\tabularnewline
+\hline
+74 & 32856 & 592 & 1184 & 1776 & 2368 & 2960 & 3552 & 4144 & 4736 & 5328 & 5920\tabularnewline
+\hline
+75 & 33750 & 1200 & 2400 & 3600 & 4800 & 6000 & 7200 & 8400 & 9600 & 10800 & 12000\tabularnewline
+\hline
+76 & 34656 & 608 & 1216 & 1824 & 2432 & 3040 & 3648 & 4256 & 4864 & 5472 & 6080\tabularnewline
+\hline
+77 & 35574 & 1232 & 2464 & 3696 & 4928 & 6160 & 7392 & 8624 & 9856 & 11088 & 12320\tabularnewline
+\hline
+78 & 36504 & 624 & 1248 & 1872 & 2496 & 3120 & 3744 & 4368 & 4992 & 5616 & 6240\tabularnewline
+\hline
+79 & 37446 & 1264 & 2528 & 3792 & 5056 & 6320 & 7584 & 8848 & 10112 & 11376 & 12640\tabularnewline
+\hline
+80 & 38400 & 640 & 1280 & 1920 & 2560 & 3200 & 3840 & 4480 & 5120 & 5760 & 6400\tabularnewline
+\hline
+81 & 39366 & 1296 & 2592 & 3888 & 5184 & 6480 & 7776 & 9072 & 10368 & 11664 & 12960\tabularnewline
+\hline
+82 & 40344 & 656 & 1312 & 1968 & 2624 & 3280 & 3936 & 4592 & 5248 & 5904 & 6560\tabularnewline
+\hline
+83 & 41334 & 1328 & 2656 & 3984 & 5312 & 6640 & 7968 & 9296 & 10624 & 11952 & 13280\tabularnewline
+\hline
+84 & 42336 & 672 & 1344 & 2016 & 2688 & 3360 & 4032 & 4704 & 5376 & 6048 & 6720\tabularnewline
+\hline
+85 & 43350 & 1360 & 2720 & 4080 & 5440 & 6800 & 8160 & 9520 & 10880 & 12240 & 13600\tabularnewline
+\hline
+86 & 44376 & 688 & 1376 & 2064 & 2752 & 3440 & 4128 & 4816 & 5504 & 6192 & 6880\tabularnewline
+\hline
+87 & 45414 & 1392 & 2784 & 4176 & 5568 & 6960 & 8352 & 9744 & 11136 & 12528 & 13920\tabularnewline
+\hline
+88 & 46464 & 704 & 1408 & 2112 & 2816 & 3520 & 4224 & 4928 & 5632 & 6336 & 7040\tabularnewline
+\hline
+89 & 47526 & 1424 & 2848 & 4272 & 5696 & 7120 & 8544 & 9968 & 11392 & 12816 & 14240\tabularnewline
+\hline
+90 & 48600 & 720 & 1440 & 2160 & 2880 & 3600 & 4320 & 5040 & 5760 & 6480 & 7200\tabularnewline
+\hline
+91 & 49686 & 1456 & 2912 & 4368 & 5824 & 7280 & 8736 & 10192 & 11648 & 13104 & 14560\tabularnewline
+\hline
+92 & 50784 & 736 & 1472 & 2208 & 2944 & 3680 & 4416 & 5152 & 5888 & 6624 & 7360\tabularnewline
+\hline
+93 & 51894 & 1488 & 2976 & 4464 & 5952 & 7440 & 8928 & 10416 & 11904 & 13392 & 14880\tabularnewline
+\hline
+94 & 53016 & 752 & 1504 & 2256 & 3008 & 3760 & 4512 & 5264 & 6016 & 6768 & 7520\tabularnewline
+\hline
+95 & 54150 & 1520 & 3040 & 4560 & 6080 & 7600 & 9120 & 10640 & 12160 & 13680 & 15200\tabularnewline
+\hline
+96 & 55296 & 768 & 1536 & 2304 & 3072 & 3840 & 4608 & 5376 & 6144 & 6912 & 7680\tabularnewline
+\hline
+97 & 56454 & 1552 & 3104 & 4656 & 6208 & 7760 & 9312 & 10864 & 12416 & 13968 & 15520\tabularnewline
+\hline
+98 & 57624 & 784 & 1568 & 2352 & 3136 & 3920 & 4704 & 5488 & 6272 & 7056 & 7840\tabularnewline
+\hline
+99 & 58806 & 1584 & 3168 & 4752 & 6336 & 7920 & 9504 & 11088 & 12672 & 14256 & 15840\tabularnewline
+\hline
+100 & 60000 & 800 & 1600 & 2400 & 3200 & 4000 & 4800 & 5600 & 6400 & 7200 & 8000\tabularnewline
+\hline
+101 & 61206 & 1616 & 3232 & 4848 & 6464 & 8080 & 9696 & 11312 & 12928 & 14544 & 16160\tabularnewline
+\hline
+102 & 62424 & 816 & 1632 & 2448 & 3264 & 4080 & 4896 & 5712 & 6528 & 7344 & 8160\tabularnewline
+\hline
+103 & 63654 & 1648 & 3296 & 4944 & 6592 & 8240 & 9888 & 11536 & 13184 & 14832 & 16480\tabularnewline
+\hline
+104 & 64896 & 832 & 1664 & 2496 & 3328 & 4160 & 4992 & 5824 & 6656 & 7488 & 8320\tabularnewline
+\hline
+105 & 66150 & 1680 & 3360 & 5040 & 6720 & 8400 & 10080 & 11760 & 13440 & 15120 & 16800\tabularnewline
+\hline
+106 & 67416 & 848 & 1696 & 2544 & 3392 & 4240 & 5088 & 5936 & 6784 & 7632 & 8480\tabularnewline
+\hline
+107 & 68694 & 1712 & 3424 & 5136 & 6848 & 8560 & 10272 & 11984 & 13696 & 15408 & 17120\tabularnewline
+\hline
+108 & 69984 & 864 & 1728 & 2592 & 3456 & 4320 & 5184 & 6048 & 6912 & 7776 & 8640\tabularnewline
+\hline
+109 & 71286 & 1744 & 3488 & 5232 & 6976 & 8720 & 10464 & 12208 & 13952 & 15696 & 17440\tabularnewline
+\hline
+110 & 72600 & 880 & 1760 & 2640 & 3520 & 4400 & 5280 & 6160 & 7040 & 7920 & 8800\tabularnewline
+\hline
+111 & 73926 & 1776 & 3552 & 5328 & 7104 & 8880 & 10656 & 12432 & 14208 & 15984 & 17760\tabularnewline
+\hline
+112 & 75264 & 896 & 1792 & 2688 & 3584 & 4480 & 5376 & 6272 & 7168 & 8064 & 8960\tabularnewline
+\hline
+113 & 76614 & 1808 & 3616 & 5424 & 7232 & 9040 & 10848 & 12656 & 14464 & 16272 & 18080\tabularnewline
+\hline
+114 & 77976 & 912 & 1824 & 2736 & 3648 & 4560 & 5472 & 6384 & 7296 & 8208 & 9120\tabularnewline
+\hline
+115 & 79350 & 1840 & 3680 & 5520 & 7360 & 9200 & 11040 & 12880 & 14720 & 16560 & 18400\tabularnewline
+\hline
+116 & 80736 & 928 & 1856 & 2784 & 3712 & 4640 & 5568 & 6496 & 7424 & 8352 & 9280\tabularnewline
+\hline
+117 & 82134 & 1872 & 3744 & 5616 & 7488 & 9360 & 11232 & 13104 & 14976 & 16848 & 18720\tabularnewline
+\hline
+118 & 83544 & 944 & 1888 & 2832 & 3776 & 4720 & 5664 & 6608 & 7552 & 8496 & 9440\tabularnewline
+\hline
+119 & 84966 & 1904 & 3808 & 5712 & 7616 & 9520 & 11424 & 13328 & 15232 & 17136 & 19040\tabularnewline
+\hline
+120 & 86400 & 960 & 1920 & 2880 & 3840 & 4800 & 5760 & 6720 & 7680 & 8640 & 9600\tabularnewline
+\hline
+121 & 87846 & 1936 & 3872 & 5808 & 7744 & 9680 & 11616 & 13552 & 15488 & 17424 & 19360\tabularnewline
+\hline
+122 & 89304 & 976 & 1952 & 2928 & 3904 & 4880 & 5856 & 6832 & 7808 & 8784 & 9760\tabularnewline
+\hline
+123 & 90774 & 1968 & 3936 & 5904 & 7872 & 9840 & 11808 & 13776 & 15744 & 17712 & 19680\tabularnewline
+\hline
+124 & 92256 & 992 & 1984 & 2976 & 3968 & 4960 & 5952 & 6944 & 7936 & 8928 & 9920\tabularnewline
+\hline
+125 & 93750 & 2000 & 4000 & 6000 & 8000 & 10000 & 12000 & 14000 & 16000 & 18000 & 20000\tabularnewline
+\hline
+126 & 95256 & 1008 & 2016 & 3024 & 4032 & 5040 & 6048 & 7056 & 8064 & 9072 & 10080\tabularnewline
+\hline
+127 & 96774 & 2032 & 4064 & 6096 & 8128 & 10160 & 12192 & 14224 & 16256 & 18288 & 20320\tabularnewline
+\hline
+128 & 98304 & 1024 & 2048 & 3072 & 4096 & 5120 & 6144 & 7168 & 8192 & 9216 & 10240\tabularnewline
+\hline
+129 & 99846 & 2064 & 4128 & 6192 & 8256 & 10320 & 12384 & 14448 & 16512 & 18576 & 20640\tabularnewline
+\hline
+\end{longtable}
+%
+\begin{table}[h]
+\caption{Sample choices for $\nexxi$ given $\nprocxi$ based upon the relationship
+$\nexxi=8\times c\times\nprocxi$, where the integer $c\ge1$. The
+number of MPI slices, i.e., the total number of required processors,
+is $6\times\nprocxi^{2}$, as illustrated in Figure~\ref{figure:mpi_slices}.
+The approximate shortest period at which the global simulation is
+accurate for a given value of $\nexxi$ can be estimated by running
+the small serial program \texttt{xcreate\_header\_file}.}
+\end{table}
+\end{center}
+
+Finally, you need to provide a file that tells MPI what compute nodes
+to use for the simulations. The file must have a number of entries
+(one entry per line) at least equal to the number of processors needed
+for the run. A sample file is provided in the file \texttt{mymachines}.
+This file is not used by the mesher or solver, but is required by
+the \texttt{go\_mesher} and \texttt{go\_solver} default job submission
+scripts. See Chapter \ref{cha:Running-Scheduler} for information
+about running the code on a system with a scheduler, e.g., LSF.
+
+Now that you have set the appropriate parameters in the \texttt{Par\_file}
+and have compiled the mesher, you are ready to launch it! This is
+most easily accomplished based upon the \texttt{go\_mesher} script.
+When you run on a PC cluster, the script assumes that the nodes are
+named n001, n002, etc. If this is not the case, change the \texttt{tr
+-d `n'} line in the script. You may also need to edit the last command
+at the end of the script that invokes the \texttt{mpirun} command.
+
+Mesher output is provided in the \texttt{OUTPUT\_FILES} directory
+in \texttt{output\_mesher.txt}; this file provides lots of details
+about the mesh that was generated. Alternatively, output can be directed
+to the screen instead by uncommenting a line in \texttt{constants.h}:
+
+\begin{lyxcode}
+!~uncomment~this~to~write~messages~to~the~screen~
+
+!~integer,~parameter~::~IMAIN~=~ISTANDARD\_OUTPUT~~
+\end{lyxcode}
+Note that on very fast machines, writing to the screen may slow down
+the code.
+
+Another file generated by the mesher is the header file \texttt{OUTPUT\_FILES/values\_from\_mesher.h}.
+This file specifies a number of constants and flags needed by the
+solver. These values are passed statically to the solver for reasons
+of speed. Some useful statistics about the mesh are also provided
+in this file.
+
+For a given model, set of nodes, and set of parameters in \texttt{Par\_file},
+one only needs to run the mesher once and for all, even if one wants
+to run several simulations with different sources and/or receivers
+(the source and receiver information is used in the solver only).
+
+Please note that it is difficult to correctly sample S waves in the
+inner core of the Earth because S-wave velocity is very small there.
+Therefore, correctly sampling S waves in the inner core would require
+a very dense mesh, which in turn would drastically reduce the time
+step of the explicit time scheme because the P wave velocity is very
+high in the inner core (Poisson's ratio is roughly equal to 0.44).
+Because shear wave attenuation is very high in the inner core ($Q_{\mu}$
+is approximately equal to 85), we have therefore decided to design
+the inner core mesh such that P waves are very well sampled but S
+waves are right at the sampling limit or even slightly below. This
+works fine because spurious numerical oscillations due to S-wave subsampling
+are almost completely suppressed by attenuation. However, this implies
+that one should not use SPECFEM3D\_GLOBE with the regular mesh and
+period estimates of Table \ref{table:nex} to study the PKJKP phase
+very precisely. If one is interested in that phase, one should use
+typically 1.5 times to twice the number of elements NEX indicated
+in the table.
+
+Regarding fluid/solid coupling at the CMB and ICB, in SPECFEM3D\_GLOBE
+we do not use the fluid-solid formulation of \citet{KoTr02a} and
+\citet{KoTr02b} anymore, we now use a displacement potential in the
+fluid (rather than a velocity potential as in \citet{KoTr02a} and
+\citet{KoTr02b}). This leads to the simpler fluid-solid matching
+condition introduced by \citet{ChVa04} with no numerical iterations
+at the CMB and ICB.
+
+For accuracy reasons, in the mesher the coordinates of the mesh points
+(arrays xstore, ystore and zstore) are always created in double precision.
+If the solver is compiled in single precision mode, the mesh coordinates
+are converted to single precision before being saved in the local
+mesh files.
+
+
+\section{Memory requirements}
+
+The SPECFEM3D\_GLOBE memory requirements can be estimated before or
+after running the mesher using the small serial program \texttt{\small xcreate\_header\_file},
+which reads the input file \texttt{\small DATA/Par\_file} and displays
+the total amount of memory that will be needed by the mesher and the
+solver to run it. This way, users can easily modify the parameters
+and check that their simulation will fit in memory on their machine.
+The file created by \texttt{\small xcreate\_header\_file} is called
+\texttt{OUTPUT\_FILES/values\_from\_mesher.h} and contains even more
+details about the future simulation.
+
+
+\section{Checking the MPI Buffers (Optional)}
+
+The mesher writes MPI communication tables in the \texttt{OUTPUT\_FILES}
+subdirectory in the files \texttt{addressing.txt}, \texttt{list\_messages\_corners.txt}
+and \texttt{list\_messages\_faces.txt}, and MPI communication buffers
+to the local disks. Use the four serial codes
+
+\begin{lyxcode}
+check\_buffers\_2D.f90
+
+check\_buffers\_1D.f90
+
+check\_buffers\_faces\_chunks.f90
+
+check\_buffers\_corners\_chunks.f90~
+\end{lyxcode}
+to check that all the MPI buffers created by the mesher have been
+generated correctly. For example, typing `\texttt{make check\_buffers\_2D}'
+and then `\texttt{xcheck\_buffers\_2D}' checks the communication buffers
+between faces common to the mesh slices. `\texttt{xcheck\_buffers\_1D}'
+checks the communication buffers between edges common to the mesh
+slices. `\texttt{xcheck\_buffers\_faces\_chunks}' checks the communication
+buffers between faces common to the mesh chunks, i.e., the faces of
+the six blocks of the cubed-sphere mesh. `\texttt{xcheck\_buffers\_corners\_chunks}'
+checks the communication buffers between edges common to the mesh
+chunks, which must be treated separately in MPI because they are of
+valence 3 (i.e., they are shared between three chunks).
+
+Please note that running these codes is optional because no information
+needed by the solver is generated.
+
+\chapter{\label{cha:Running-the-Solver}Running the Solver \texttt{xspecfem3D}}
+
+Now that you have successfully run the mesher, you are ready to compile
+the solver. For reasons of speed, the solver uses static memory allocation.
+Therefore it needs to be recompiled (type `\texttt{make clean}' and
+`\texttt{make specfem3D}') every time one reruns the mesher with different
+parameters. To compile the solver one needs a file generated by the
+mesher in the directory \texttt{OUTPUT\_FILES} called \texttt{values\_from\_mesher.h},
+which contains parameters describing the static size of the arrays
+as well as the setting of certain flags.
+
+The solver needs three input files in the \texttt{DATA} directory
+to run: the \texttt{Par\_file} that was discussed in detail in Chapter~\ref{cha:Running-the-Mesher},
+the earthquake source parameter file \texttt{CMTSOLUTION}, and the
+stations file \texttt{STATIONS}. Most parameters in the \texttt{Par\_file}
+should be set prior to running the mesher. Only the following parameters
+may be changed after running the mesher:
+
+\begin{itemize}
+\item the simulation type control parameters: \texttt{SIMULATION\_TYPE}
+and \texttt{SAVE\_FORWARD}
+\item the record length \texttt{RECORD\_LENGTH\_IN\_MINUTES}
+\item the movie control parameters \texttt{MOVIE\_SURFACE}, \texttt{MOVIE\_VOLUME},
+and \texttt{NTSTEPS\_BETWEEN\_FRAMES}
+\item the multi-stage simulation parameters \texttt{NUMBER\_OF\_RUNS} and
+\texttt{NUMBER\_OF\_THIS\_RUN}
+\item the output information parameters \texttt{NTSTEP\_BETWEEN\_OUTPUT\_INFO,
+NTSTEP\_BETWEEN\_OUTPUT\_}~\\
+\texttt{SEISMOS, OUTPUT\_SEISMOS\_ASCII\_TEXT, OUTPUT\_SEISMOS\_SAC\_ALPHANUM,
+OUTPUT\_}~\\
+\texttt{SEISMOS\_SAC\_BINARY and ROTATE\_SEISMOGRAMS\_RT}
+\item the \texttt{RECEIVERS\_CAN\_BE\_BURIED} and \texttt{PRINT\_SOURCE\_TIME\_FUNCTION}
+flags
+\end{itemize}
+Any other change to the \texttt{Par\_file} implies rerunning both
+the mesher and the solver.
+
+For any particular earthquake, the \texttt{CMTSOLUTION} file that
+represents the point source may be obtained directly from the Harvard Centroid-Moment Tensor (CMT) web page \url{www.globalcmt.org}.
+It looks like this:
+
+\begin{lyxcode}
+{\small }%
+\begin{figure}[H]
+\noindent \begin{centering}
+{\small \includegraphics[width=1\textwidth]{figures/Denali_CMT} }
+\par\end{centering}{\small \par}
+
+\caption{\texttt{CMTSOLUTION} file obtained from the Harvard CMT catalog. The
+top line is the initial estimate of the source, which is used as a
+starting point for the CMT solution. \textbf{M} is the moment tensor,
+$M_{0}${\small{} }is the seismic moment, and $M_{w}$ is the moment
+magnitude.}
+
+
+\label{fig:CMTSOLUTION-file}
+\end{figure}
+{\small \par}
+\end{lyxcode}
+The \texttt{CMTSOLUTION} should be edited in the following way:
+
+\begin{itemize}
+\item For point-source simulations (see finite sources, page \pageref{To-simulate-a})
+we recommend setting the source half-duration parameter \texttt{half
+duration} equal to zero, which corresponds to simulating a step source-time
+function, i.e., a moment-rate function that is a delta function. If
+\texttt{half duration} is not set to zero, the code will use a Gaussian
+(i.e., a signal with a shape similar to a `smoothed triangle', as
+explained in \citet{KoTr02a} and shown in Fig~\ref{fig:gauss.vs.triangle})
+source-time function with half-width \texttt{half duration}. We prefer
+to run the solver with \texttt{half duration} set to zero and convolve
+the resulting synthetic seismograms in post-processing after the run,
+because this way it is easy to use a variety of source-time functions
+(see Section \ref{sec:Process-data-and-syn}). \citet{KoTr02a} determined
+that the noise generated in the simulation by using a step source
+time function may be safely filtered out afterward based upon a convolution
+with the desired source time function and/or low-pass filtering. Use
+the postprocessing script \texttt{process\_syn.pl} (see Section \ref{sub:process_syn.pl})
+with the \texttt{-h} flag, or the serial code \texttt{convolve\_source\_timefunction.f90}
+and the script \texttt{UTILS/convolve\_source\_timefunction.csh} for
+this purpose, or alternatively use signal-processing software packages
+such as SAC \url{www.iris.edu/software/sac/}. Type
+
+\begin{lyxcode}
+make~convolve\_source\_timefunction
+\end{lyxcode}
+to compile the code and then set the parameter \texttt{hdur} in \texttt{UTILS/convolve\_source\_timefunction.csh}
+to the desired half-duration.
+
+\item The zero time of the simulation corresponds to the center of the triangle/Gaussian,
+or the centroid time of the earthquake. The start time of the simulation
+is $t=-1.5*\texttt{half duration}$ (the 1.5 is to make sure the moment
+rate function is very close to zero when starting the simulation).
+To convert to absolute time $t_{\mathrm{abs}}$, set
+
+\begin{lyxcode}
+$t_{\mathrm{abs}}=t_{\mathrm{pde}}+\texttt{time shift}+t_{\mathrm{synthetic}}$
+\end{lyxcode}
+where $t_{\mathrm{pde}}$ is the time given in the first line of the
+\texttt{CMTSOLUTION}, \texttt{time shift} is the corresponding value
+from the original \texttt{CMTSOLUTION} file and $t_{\mathrm{synthetic}}$
+is the time in the first column of the output seismogram.
+
+\end{itemize}
+%
+\begin{figure}
+\noindent \begin{centering}
+\includegraphics[width=3in]{figures/gauss_vs_triangle_mod}
+\par\end{centering}
+
+\caption{Comparison of the shape of a triangle and the Gaussian function actually
+used.}
+
+
+\label{fig:gauss.vs.triangle}
+\end{figure}
+
+
+Centroid latitude and longitude should be provided in geographical
+coordinates. The code converts these coordinates to geocentric coordinates~\citep{DaTr98}.
+Of course you may provide your own source representations by designing
+your own \texttt{CMTSOLUTION} file. Just make sure that the resulting
+file adheres to the Harvard CMT conventions (see Appendix~\ref{cha:Reference-Frame-Convention}).
+Note that the first line in the \texttt{CMTSOLUTION} file is the Preliminary Determination of Earthquakes (PDE) solution performed by the USGS NEIC, which is used as a seed for the Harvard CMT inversion. The PDE solution is based upon P waves and often gives the hypocenter of the earthquake, i.e., the rupture initiation point, whereas the CMT solution gives the `centroid location', which is the location with dominant moment release. The PDE solution is not used by our software package but must be present anyway in the first line of the file.
+
+In the current version of the code, the solver can run with a non-zero \texttt{time shift} in the \texttt{CMTSOLUTION} file. Thus one does not need to set \texttt{time shift} to zero as it was the case for previous versions of the code. \texttt{time shift} is only used for writing centroid time in the SAC headers (see Appendix~\ref{cha:SAC-headers}). CMT time is obtained by adding \texttt{time shift} to the PDE time given in the first line in the \texttt{CMTSOLUTION} file. Therefore it is recommended not to modify \texttt{time shift} to have the correct timing information in the SAC headers without any post-processing of seismograms.
+
+\label{To-simulate-a}To simulate a kinematic rupture, i.e., a finite-source
+event, represented in terms of $N_{\mathrm{sources}}$ point sources,
+provide a \texttt{CMTSOLUTION} file that has $N_{\mathrm{sources}}$
+entries, one for each subevent (i.e., concatenate $N_{\mathrm{sources}}$
+\texttt{CMTSOLUTION} files to a single \texttt{CMTSOLUTION} file).
+At least one entry (not necessarily the first) must have a zero \texttt{time
+shift}, and all the other entries must have non-negative \texttt{time
+shift}. If none of the entries has a zero \texttt{time shift} in the \texttt{CMTSOLUTION} file, the smallest \texttt{time shift} is subtracted from all sources to initiate the simulation. Each subevent can have its own half duration, latitude, longitude,
+depth, and moment tensor (effectively, the local moment-density tensor).
+
+Note that the zero in the synthetics does NOT represent the hypocentral
+time or centroid time in general, but the timing of the \textit{center}
+of the source triangle with zero \texttt{time shift} (Fig~\ref{fig:source_timing}).
+
+Although it is convenient to think of each source as a triangle, in
+the simulation they are actually Gaussians (as they have better frequency
+characteristics). The relationship between the triangle and the Gaussian
+used is shown in Fig~\ref{fig:gauss.vs.triangle}. For finite fault
+simulations it is usually not advisable to use a zero half duration
+and convolve afterwards, since the half duration is generally fixed
+by the finite fault model.
+
+{\small }%
+\begin{figure}[H]
+\noindent \begin{centering}
+{\small \includegraphics[width=5in]{figures/source_timing} }
+\par\end{centering}{\small \par}
+
+\caption{Example of timing for three sources. The center of the first source
+triangle is defined to be time zero. Note that this is NOT in general
+the hypocentral time, or the start time of the source (marked as tstart).
+The parameter \texttt{time shift} in the \texttt{CMTSOLUTION} file
+would be t1(=0), t2, t3 in this case, and the parameter \texttt{half
+duration} would be hdur1, hdur2, hdur3 for the sources 1, 2, 3 respectively.}
+
+
+{\small \label{fig:source_timing} }
+\end{figure}
+{\small \par}
+
+The solver can calculate seismograms at any number of stations for
+basically the same numerical cost, so the user is encouraged to include
+as many stations as conceivably useful in the \texttt{STATIONS} file,
+which looks like this:
+
+{\small }%
+\begin{figure}[H]
+\noindent \begin{centering}
+{\small \includegraphics{figures/STATIONS_global_explained} }
+\par\end{centering}{\small \par}
+
+\caption{Sample \texttt{STATIONS} file. Station latitude and longitude should
+be provided in geographical coordinates. The width of the station
+label should be no more than 32 characters (see \texttt{MAX\_LENGTH\_STATION\_NAME}
+in the \texttt{constants.h} file), and the network label should be
+no more than 8 characters (see \texttt{MAX\_LENGTH\_NETWORK\_NAME}
+in the \texttt{constants.h} file).}
+
+\end{figure}
+{\small \par}
+
+Each line represents one station in the following format:
+
+\begin{lyxcode}
+\noindent {\small Station~Network~Latitude~(degrees)~Longitude~(degrees)~Elevation~(m)~burial~(m)~}{\small \par}
+\end{lyxcode}
+If you want to put a station on the ocean floor, just set
+elevation and burial depth in the STATIONS file to 0.
+Equivalently you can also set elevation to a negative value equal
+to the ocean depth, and burial depth to 0.
+
+Solver output is provided in the \texttt{OUTPUT\_FILES} directory
+in the \texttt{output\_solver.txt} file. Output can be directed to
+the screen instead by uncommenting a line in \texttt{constants.h}:
+
+\begin{lyxcode}
+!~uncomment~this~to~write~messages~to~the~screen~
+
+!~integer,~parameter~::~IMAIN~=~ISTANDARD\_OUTPUT~
+\end{lyxcode}
+Note that on very fast machines, writing to the screen may slow down
+the code.
+
+While the solver is running, its progress may be tracked by monitoring
+the `\texttt{timestamp{*}}' files in the \texttt{OUTPUT\_FILES} directory.
+These tiny files look something like this:
+
+\begin{lyxcode}
+Time~step~\#~~~~~~~~~~~200~
+
+Time:~~~~0.6956667~~~~~~minutes~
+
+Elapsed~time~in~seconds~=~~~~~252.6748970000000~
+
+Elapsed~time~in~hh:mm:ss~=~~~~0~h~04~m~12~s~
+
+Mean~elapsed~time~per~time~step~in~seconds~=~~~~~1.263374485000000~
+
+Max~norm~displacement~vector~U~in~solid~in~all~slices~(m)~=~~~~1.9325~
+
+Max~non-dimensional~potential~Ufluid~in~fluid~in~all~slices~=~1.1058885E-22~
+\end{lyxcode}
+The \texttt{timestamp{*}} files provide the \texttt{Mean elapsed time
+per time step in seconds}, which may be used to assess performance
+on various machines (assuming you are the only user on a node), as
+well as the \texttt{\small Max}{\small{} }\texttt{\small norm}{\small{}
+}\texttt{\small displacement}{\small{} }\texttt{\small vector}{\small{}
+}\texttt{\small U}{\small{} }\texttt{\small in}{\small{} }\texttt{\small solid}{\small{}
+}\texttt{\small in}{\small{} }\texttt{\small all}{\small{} }\texttt{\small slices~(m)}{\small{}
+}\texttt{\small and}{\small{} }\texttt{\small Max}{\small{} }\texttt{\small non-dimensional}{\small{}
+}\texttt{\small potential}{\small{} }\texttt{\small Ufluid}{\small{}
+}\texttt{\small in}{\small{} }\texttt{\small fluid}{\small{} }\texttt{\small in}{\small{}
+}\texttt{\small all}{\small{} }\texttt{\small slices}. If something
+is wrong with the model, the mesh, or the source, you will see the
+code become unstable through exponentionally growing values of the
+displacement and/or fluid potential with time, and ultimately the
+run will be terminated by the program when either of these values
+becomes greater than \texttt{STABILITY\_THRESHOLD} defined in \texttt{constants.h}.
+You can control the rate at which the timestamp files are written
+based upon the parameter \texttt{NTSTEP\_BETWEEN\_OUTPUT\_INFO} in
+the \texttt{Par\_file}.
+
+Having set the \texttt{Par\_file} parameters, and having provided
+the \texttt{CMTSOLUTION} and \texttt{STATIONS} files, you are now
+ready to launch the solver! This is most easily accomplished based
+upon the \texttt{go\_solver} script (see Chapter~\ref{cha:Running-Scheduler}
+for information about running the code through a scheduler, e.g.,
+LSF). You may need to edit the last command at the end of the script
+that invokes the \texttt{mpirun} command. Another option is to use
+the \texttt{runall} script, which compiles and runs both mesher and
+solver in sequence. This is a safe approach that ensures using the
+correct combination of mesher output and solver input.
+
+It is important to realize that the CPU and memory requirements of
+the solver are closely tied to choices about attenuation (\texttt{ATTENUATION})
+and the nature of the model (i.e., isotropic models are cheaper than
+anisotropic models). We encourage you to run a variety of simulations
+with various flags turned on or off to develop a sense for what is
+involved.
+
+For the same model, one can rerun the solver for different events
+by simply changing the \texttt{CMTSOLUTION} file, and/or for different
+stations by changing the \texttt{STATIONS} file. There is no need
+to rerun the mesher. Of course it is best to include as many stations
+as possible, since this does not add significantly to the cost of
+the simulation.
+
+
+\chapter{\label{cha:Regional-Simulations}Regional Simulations}
+
+The code has the option of running in 1-, 2-, 3- or 6-chunk mode.
+The 1- or 2-chunk options may be used for higher resolution regional
+simulations. A one-chunk mesh may have lateral dimensions other than
+the customary $90^{\circ}$ per chunk, which can further increase
+the resolution of the mesh, and thus reduce the shortest period in
+the synthetic seismograms (but of course then also reducing the time
+step in order for the simulation to remain stable). A disadvantage
+of regional simulations is that one needs to use approximate absorbing
+boundary conditions on the side and bottom edges of the model (e.g.,
+see \citet{KoTr99} for a description of the paraxial boundary conditions
+used). Figure~\vref{fig:3D-spectral-element-mesh} and Figure~\vref{fig:Close-up-view-of}
+show an example of a one-chunk mesh centered on the Japan subduction
+zone, applied in the Japan regional waveform simulation \citep{ChTrHeKa07}.
+
+
+\section{One-Chunk Simulations\label{sec:One-Chunk-Simulations}}
+
+For a one-chunk regional simulation the following parameters need
+to be set in the \texttt{Par\_file}:
+
+\begin{description}
+\item [{$\nchunks$}] Must be set to 1.
+\item [{\texttt{ANGULAR\_WIDTH\_XI\_IN\_DEGREES}}] Denotes the width of
+one side of the chunk ($90^{\circ}$or less).
+\item [{\texttt{ANGULAR\_WIDTH\_ETA\_IN\_DEGREES}}] Denotes the width of
+the second side of the chunk ($90^{\circ}$or less). Note that this
+value may be different from \texttt{ANGULAR\_WIDTH\_XI\_IN\_DEGREES}.
+\item [{\texttt{CENTER\_LATITUDE\_IN\_DEGREES}}] Defines the latitude of
+the center of the chunk (degrees).
+\item [{\texttt{CENTER\_LONGITUDE\_IN\_DEGREES}}] Defines the longitude
+of the center of the chunk (degrees).
+\item [{\texttt{GAMMA\_ROTATION\_AZIMUTH}}] Defines the rotation angle
+of the chunk about its center measured counter clockwise from due
+North (degrees). The corners of the mesh are output in \texttt{\small OUTPUT\_FILES/values\_from\_mesher.h}.
+The output corner progression in \texttt{\small OUTPUT\_FILES/values\_from\_mesher.h}
+is bottom left, bottom right, top left, top right. The rotation azimuth
+can be changed in the \texttt{Par\_file} and the corners output \texttt{(}\texttt{\small OUTPUT\_FILES/}~\\
+\texttt{\small values\_from\_mesher.h}\texttt{)} by using{\small{}
+}\texttt{\small xcreate\_header\_file}. It is important to note that
+the mesher or the solver does not need to be run to determine the
+limits of a 1-chunk simulation.
+\item [{$\nexxi$}] The number of spectral elements along the $\xi$ side
+of the chunk. This number \textit{must} be 8~$\times$~a multiple
+of $\nprocxi$ defined below. For a $90^{\circ}$ chunk, we do not
+recommend using $\nexxi$ less than~64 because the curvature of the
+Earth cannot be honored if one uses too few elements, which results
+in inaccurate and unstable simulations.
+\item [{$\nexeta$}] The number of spectral elements along the $\eta$
+side of the chunk. This number \textit{must} be 8~$\times$~a multiple
+of $\nproceta$ defined below. Note that in order to get elements
+that are close to square on the Earth's surface, the following ratios
+should be similar:
+
+\begin{lyxcode}
+$\texttt{ANGULAR\_WIDTH\_XI\_IN\_DEGREES}/\nexxi$
+
+$\texttt{ANGULAR\_WIDTH\_ETA\_IN\_DEGREES}/\nexeta$~
+\end{lyxcode}
+Because of the geometry of the cubed sphere, the option of having
+different values for $\nexxi$ and $\nexeta$ is available only for
+regional simulations when $\nchunks=1$ (1/6th of the sphere).
+
+\item [{$\nprocxi$}] The number of processors or mesh slices along the
+$\xi$ side of the chunk. To accommodate the mesh doubling layers,
+we must have $\nexxi=8\times c\times\nprocxi$, where $c\ge1$ is
+a positive integer. See Table~\ref{table:nex} for various suitable
+choices.
+\item [{$\nproceta$}] The number of processors or slices along the $\eta$
+side of the chunk; we must have $\nexeta=8\times c\times\nproceta$,
+where $c\ge1$ is a positive integer. $\nprocxi$ and $\nproceta$
+must be equal when $\nchunks=6$.
+\end{description}
+%
+\begin{figure}[H]
+\begin{centering}
+\includegraphics[scale=0.65]{figures/fig5a}
+\par\end{centering}
+
+\caption{\textbf{\label{fig:3D-spectral-element-mesh}}S-wave velocity anomalies
+from the global tomographic model s20rts \citep{RiVa00} are superimposed
+on the mesh. For parallel computing purposes, the one-chunk SEM simulation
+is subdivided in terms of 64 slices. The center of the chunk is at
+(38.5$^{\circ}$ N, 137.5$^{\circ}$ E), and the lateral dimensions
+are 30$^{\circ}$ $\times$ 30$^{\circ}$. Two doubling layers are
+indicated at a depth of 25~km (PREM Moho depth) and a depth of about
+1650~km. Shows full view of 25 neighboring slices; see Figure~\ref{fig:Close-up-view-of}
+for close-up of upper mantle mesh.}
+
+\end{figure}
+%
+\begin{figure}[H]
+\begin{centering}
+\includegraphics[scale=0.65]{figures/fig5b}
+\par\end{centering}
+
+\caption{\textbf{\label{fig:Close-up-view-of}}Close-up view of the upper mantle
+mesh shown in Figure~\ref{fig:3D-spectral-element-mesh}. Note that
+the element size in the crust (top layer) is 13~km $\times$ 13~km,
+and that the size of the spectral elements is doubled in the upper
+mantle. The velocity variation is captured by NGLL = 5 grid points
+in each direction of the elements \citep{KoTr02a,KoTr02b}.}
+
+\end{figure}
+
+
+\begin{description}
+\item [{\texttt{ABSORBING\_CONDITIONS}}] Set to \texttt{.true.}{\small{}
+}for regional simulations. For instance, see \citet{KoTr99} for a
+description of the paraxial boundary conditions used. Note that these
+conditions are never perfect, and in particular surface waves may
+partially reflect off the artificial boundaries. Note also that certain
+arrivals, e.g., PKIKPPKIKP, will be missing from the synthetics.
+\end{description}
+When the width of the chunk is different from $90^{\circ}$ (or the
+number of elements is greater than 1248), the radial distribution
+of elements needs to be adjusted as well to maintain spectral elements
+that are as cube-like as possible. The code attempts to do this, but
+be sure to view the mesh with your favorite graphics package to make
+sure that the element are well behaved. Remember: a high-quality mesh is paramount for accurate simulations. In addition to a reorganization of the radial distribution of elements,
+the time stepping and period range in which the attenuation is applied
+is automatically determined. The minimum and maximum periods for attenuation
+are:
+
+\[
+\omega_{max}=\omega_{min}\times10^{W_{3}}\]
+
+
+\noindent where $W_{3}$ is the optimal width in frequency for 3 Standard
+Linear Solids, about 1.75. See \texttt{\small read\_compute\_parameters.f90}
+for more details.
+
+The time stepping is determined in a similar fashion as Equation (48)
+in \citet{KoTr02a}:
+
+\begin{lyxcode}
+dt~=~$S_{c}$~Element~Width~in~km~($r=$ICB)~/~Velocity~($r=$ICB)
+\end{lyxcode}
+where $S_{c}$ is the stability condition (about 0.4). We use the
+radius at the inner core boundary because this is where the maximum
+velocity/element width occurs. Again, see \texttt{\small read\_compute\_parameters}\texttt{.f90}
+for all the details.
+
+The approximate shortest period at which a regional simulation is
+accurate may be determined based upon the relation \begin{equation}
+\mbox{shortest period (s)}\simeq(256/\nexxi)\times(\texttt{ANGULAR\_WIDTH\_XI\_IN\_DEGREES}/90)\times17.\label{eq:shortest_period_regional}\end{equation}
+
+
+
+\section{Two-Chunk Simulations}
+
+For a two-chunk regional simulation the following parameters need
+to be set in the \texttt{Par\_file}:
+
+\begin{description}
+\item [{$\nchunks$}] Must be set to 2
+\item [{\texttt{ANGULAR\_WIDTH\_XI\_IN\_DEGREES}}] Denotes the width of
+one side of the chunk, and it has to be 90 degrees.
+\item [{\texttt{ANGULAR\_WIDTH\_ETA\_IN\_DEGREES}}] Denotes the width of
+the second side of the chunk, and it also has to be 90 degrees.
+\end{description}
+\texttt{NEX\_XI} and \texttt{NEX\_ETA} follow the same description
+in Section \ref{sec:One-Chunk-Simulations}, however, they need to
+be the same in this case. All other parameters are similar to the
+one-chunk simulations, refer to Section \ref{sec:One-Chunk-Simulations}
+for details.
+
+%
+\begin{figure}[H]
+\noindent \begin{centering}
+\includegraphics[width=0.6\textwidth]{figures/2-chunk-surface}
+\par\end{centering}
+
+\caption{Geometry of a 2-chunk simulation, where the first chunk (\texttt{CHUNK\_AB})
+centers at $40^{\circ}$ latitude, $10^{\circ}$ longitude, and has
+been rotated by $20^{\circ}$ counter clockwise, and the second chunk
+(\texttt{CHUNK\_AC}) connects to the first chunk through one face.}
+
+\end{figure}
+
+
+
+\chapter{\label{cha:Adjoint-Simulations}Adjoint Simulations}
+
+Adjoint simulations are generally performed for two distinct applications.
+First, they can be used for earthquake source inversions, especially
+earthquakes with large ruptures such as the Sumatra-Andaman event
+\citep{LayKanamoriAmmon2005,AmmonJiThio2005,ParkSongTromp2005}. Second,
+they can be used to generate finite-frequency sensitivity kernels
+that are a critical part of tomographic inversions based upon 3D reference
+models \citep{trompetal2005,LiTr06,TrKoLi08,LiTr08}. In either case, source
+parameter or velocity structure updates are sought to minimize a specific
+misfit function (e.g., waveform or traveltime differences), and the
+adjoint simulation provides a means of computing the gradient of the
+misfit function and further reducing it in successive iterations.
+Applications and procedures pertaining to source studies and finite-frequency
+kernels are discussed in Sections~\ref{sec:Adjoint-simulation-sources}
+and \ref{sec:Adjoint-simulation-finite}, respectively. The two related
+parameters in the \texttt{Par\_file} are \texttt{SIMULATION\_TYPE}
+(1, 2 or 3) and \texttt{SAVE\_FORWARD} (boolean).
+
+
+\section{\label{sec:Adjoint-simulation-sources}Adjoint Simulations for Sources}
+
+In the case where a specific misfit function is minimized to invert
+for the earthquake source parameters, the gradient of the misfit function
+with respect to these source parameters can be computed by placing
+time-reversed seismograms at the receivers and using them as sources
+in an adjoint simulation, and then the value of the gradient is obtained
+from the adjoint seismograms recorded at the original earthquake location.
+
+\begin{enumerate}
+\item \textbf{Prepare the adjoint sources} \label{enu:Prepare-the-adjoint}
+
+\begin{enumerate}
+\item First, run a regular forward simlation (\texttt{SIMULATION\_TYPE =
+1} and \texttt{SAVE\_FORWARD = .false.}). You can automatically set
+these two variables using the \texttt{\small UTILS/change\_simulation\_type.pl}
+script:
+
+\begin{lyxcode}
+UTILS/change\_simulation\_type.pl~-f~
+\end{lyxcode}
+and then collect the recorded seismograms at all the stations given
+in \texttt{DATA/STATIONS}.
+
+\item Then select the stations for which you want to compute the time-reversed
+adjoint sources and run the adjoint simulation, and compile them into
+the \texttt{DATA/STATIONS\_ADJOINT} file, which has the same format
+as the regular \texttt{DATA/STATIONS} file.
+
+\begin{itemize}
+\item Depending on what type of misfit function is used for the source inversion,
+adjoint sources need to be computed from the original recorded seismograms
+for the selected stations and saved in the \texttt{SEM/} directory
+with the format \texttt{STA.NT.LH?.adj}, where \texttt{STA}, \texttt{NT}
+are the station name and network code given in the \texttt{DATA/STATIONS\_ADJOINT}
+file, and \texttt{LH?} represents the component name of a particular
+adjoint seismogram.
+\item The adjoint seismograms are in the same format as the original seismogram
+(\texttt{STA.NT.LH?.sem?}), with the same start time, time interval
+and record length.
+\end{itemize}
+\item Notice that even if you choose to time reverse only one component
+from one specific station, you still need to supply all three components
+because the code is expecting them (you can set the other two components
+to be zero).
+\item Also note that since time-reversal is done in the code itself, no
+explicit time-reversing is needed for the preparation of the adjoint
+sources, i.e., the adjoint sources are in the same forward time sense
+as the original recorded seismograms.
+\end{enumerate}
+\item \textbf{Set the related parameters and run the adjoint simulation}\\
+In the \texttt{DATA/Par\_file}, set the two related parameters to
+be \texttt{SIMULATION\_TYPE = 2} and \texttt{SAVE\_FORWARD = .false.}.
+More conveniently, use the scripts \texttt{UTILS/change\_simulation\_type.pl}
+to modify the \texttt{Par\_file} automatically (\texttt{change\_simulation\_type.pl
+-a}). Then run the solver to launch the adjoint simulation.
+\item \textbf{Collect the seismograms at the original source location}
+
+
+After the adjoint simulation has completed successfully, get the seismograms
+from directory \texttt{OUTPUT\_FILES}.
+
+\begin{itemize}
+\item These adjoint seismograms are recorded at the locations of the original
+earthquake sources given by the \texttt{DATA/CMTSOLUTION} file, and
+have names of the form \texttt{S?????.NT.S??.sem} for the six-component
+strain tensor (\texttt{SNN,SEE,SZZ,SNE,SNZ,SEZ}) at these locations,
+and \texttt{S?????.NT.LH?.sem} for the three-component displacements
+(\texttt{LHN,LHE,LHZ}) recorded at these locations.
+\item \texttt{S?????} denotes the source number; for example, if the original
+\texttt{CMTSOLUTION} provides only a point source, then the seismograms
+collected will start with \texttt{S00001}.
+\item These adjoint seismograms provide critical information for the computation
+of the gradient of the misfit function.
+\end{itemize}
+\end{enumerate}
+
+\section{\label{sec:Adjoint-simulation-finite}Adjoint Simulations for Finite-Frequency
+Kernels (Kernel Simulation)}
+
+Finite-frequency sensitivity kernels are computed in two successive
+simulations (please refer to \citet{LiTr06} and \citet{TrKoLi08} for details).
+
+\begin{enumerate}
+\item \textbf{Run a forward simulation with the state variables saved at
+the end of the simulation}
+
+
+Prepare the \texttt{\small CMTSOLUTION} and \texttt{\small STATIONS}
+files, set the parameters \texttt{\small SIMULATION\_TYPE}{\small{}
+}\texttt{\small =}{\small{} }\texttt{\small 1} and \texttt{\small SAVE\_FORWARD
+=}{\small{} }\texttt{\small .true.} in the \texttt{Par\_file} (\texttt{change\_simulation\_type
+-F}), and run the solver.
+
+\begin{itemize}
+\item Notice that attenuation is not implemented yet for the computation
+of finite-frequency kernels; therefore set \texttt{ATTENUATION = .false.}
+in the \texttt{Par\_file}.
+\item We also suggest you modify the half duration of the \texttt{CMTSOLUTION}
+to be similar to the accuracy of the simulation (see Equation \ref{eq:shortest_period}
+or \ref{eq:shortest_period_regional}) to avoid too much high-frequency
+noise in the forward wavefield, although theoretically the high-frequency
+noise should be eliminated when convolved with an adjoint wavefield
+with the proper frequency content.
+\item This forward simulation differs from the regular simulations (\texttt{\small SIMULATION\_TYPE}{\small{}
+}\texttt{\small =}{\small{} }\texttt{\small 1} and \texttt{\small SAVE\_FORWARD}{\small{}
+}\texttt{\small =}{\small{} }\texttt{\small .false.}) described in
+the previous chapters in that the state variables for the last time
+step of the simulation, including wavefields of the displacement,
+velocity, acceleration, etc., are saved to the \texttt{LOCAL\_PATH}
+to be used for the subsequent simulation.
+\item For regional simulations, the files recording the absorbing boundary
+contribution are also written to the \texttt{LOCAL\_PATH} when \texttt{SAVE\_FORWARD
+= .true.}.
+\end{itemize}
+\item \textbf{Prepare the adjoint sources}
+
+
+The adjoint sources need to be prepared the same way as described
+in Section~\ref{sec:Adjoint-simulation-sources}, item~\ref{enu:Prepare-the-adjoint}.
+
+\begin{itemize}
+\item In the case of travel-time finite-frequency kernel for one source-receiver
+pair, i.e., point source from the \texttt{CMTSOLUTION}, and one station
+in the \texttt{STATIONS\_ADJOINT} list, we supply a sample program
+in \texttt{UTILS/cut\_velocity} to cut a certain portion of the original
+displacement seismograms and convert them into the proper adjoint
+source to compute the finite-frequency kernel.
+
+\begin{lyxcode}
+cut\_velocity~t1~t2~ifile{[}0-5]~E/N/Z-ascii-files~{[}baz]
+\end{lyxcode}
+where \texttt{t1} and \texttt{t2} are the start and end time of the
+portion you are interested in, \texttt{ifile} denotes the component
+of the seismograms to be used (0 for all three components, 1 for east,
+2 for north, and 3 for vertical, 4 for transverse, and 5 for radial
+component), \texttt{E/N/Z-ascii-files} indicate the three-component
+displacement seismograms in the right order, and \texttt{baz} is the
+back-azimuth of the station from the event location. Note that \texttt{baz}
+is only supplied when \texttt{ifile} = 4 or 5.
+
+\end{itemize}
+\item \textbf{Run the kernel simulation}
+
+
+With the successful forward simulation and the adjoint source ready
+in \texttt{SEM/}, set \texttt{SIMULATION\_TYPE = 3} and \texttt{SAVE\_FORWARD
+= .false.} in the \texttt{Par\_file(change\_simulation\_type.pl -b)},
+and rerun the solver.
+
+\begin{itemize}
+\item The adjoint simulation is launched together with the back reconstruction
+of the original forward wavefield from the state variables saved from
+the previous forward simulation, and the finite-frequency kernels
+are computed by the interaction of the reconstructed forward wavefield
+and the adjoint wavefield.
+\item The back-reconstructed seismograms at the original station locations
+are saved to the \texttt{OUTPUT\_FILES} directory at the end of the
+kernel simulations.
+\item These back-constructed seismograms can be compared with the time-reversed
+original seismograms to assess the accuracy of the backward reconstruction,
+and they should match very well (in the time-reversed sense).
+\item The files containing the density, P-wave speed and S-wave speed kernels
+are saved in the \texttt{LOCAL\_PATH} with the names of \texttt{proc??????\_reg\_?\_rho(alpha,beta)\_kernel.bin},
+where \texttt{proc??????} represents the processor number, and \texttt{reg\_?}
+denotes the region these kernels are for, including mantle (\texttt{reg\_1}),
+outer core (\texttt{reg\_2}), and inner core (\texttt{reg\_3}). The
+output kernels are in the unit of $s/km^{3}$.
+\end{itemize}
+\item \textbf{Run the boundary kernel simulation}
+
+
+\noindent If you set the \texttt{SAVE\_BOUNDARY\_MESH = .true.} in
+the \texttt{constants.h} file before the simulations, i.e., at the
+beginning of step 1, you will get not only the volumetric kernels
+as described in step 3, but also boundary kernels for the Earth's
+internal discontinuities, such as Moho, 410-km discontinuity, 670-km
+discontinuity, CMB and ICB. These kernel files are also saved in the
+local scratch directory defined by \texttt{LOCAL\_PATH }and have names
+such as \texttt{proc??????\_reg\_1(2)\_Moho(d400,d670,CMB,ICB)\_kernel.bin}.
+For a theoretical derivation of the boundary kernels, refer to \citet{trompetal2005},
+and for the visualization of the boundary kernels, refer to Section
+\ref{sec:Finite-Frequency-Kernels}.
+
+\item \textbf{Run the anisotropic kernel simulation}
+
+
+Instead of the kernels for the isotropic wave speeds, you can also
+compute the kernels for the 21 independent components $C_{IJ},\, I,J=1,...,6$
+(using Voigt's notation) of the elastic tensor in the (spherical)
+geographical coordinate system. This is done by setting \texttt{ANISOTROPIC\_KL}
+\texttt{=} \texttt{.true.} in \texttt{constants.h} before step 3.
+The definition of the parameters $C_{IJ}$ in terms of the corresponding
+components $c_{ijkl},ijkl,i,j,k,l=1,2,3$ of the elastic tensor in
+spherical coordinates follows \citet{ChTr07}. The computation of
+the anisotropic kernels is only implemented in the crust and mantle
+regions. The 21 anisotropic kernels are saved in the \texttt{LOCAL\_PATH}
+in one file with the name of \texttt{proc??????\_reg1\_cijkl\_kernel.bin}
+(with \texttt{proc??????} the processor number). The output kernels
+correspond to perturbation $\delta C_{IJ}$ of the elastic parameters
+and their unit is in $s/GPa/km^{3}$. For consistency, the output
+density kernels with this option turned on are for a perturbation
+$\delta\rho$ (and not $\frac{\delta\rho}{\rho}$) and their unit
+is in s / (kg/m$^{3}$) / km$^{3}$. These `primary' anisotropic kernels
+can then be combined to obtain the kernels related to other descriptions
+of anisotropy. This can be done, for example, when combining the kernel
+files from slices into one mesh file (see Section~\ref{sec:Finite-Frequency-Kernels}).
+
+\end{enumerate}
+In general, the first three steps need to be run sequentially to ensure
+proper access to the necessary files at different stages. If the simulations
+are run through some cluster scheduling system (e.g., LSF), and the
+forward simulation and the subsequent kernel simulations cannot be
+assigned to the same set of computer nodes, the kernel simulation
+will not be able to access the database files saved by the forward
+simulation. Solutions for this problem are provided in Chapter~\ref{cha:Running-Scheduler}.
+Visualization of the finite-frequency kernels is discussed in Section~\ref{sec:Finite-Frequency-Kernels}.
+
+
+\chapter{Graphics}
+
+
+\section{\label{sec:Meshes}Meshes}
+
+Use the serial code \texttt{combine\_AVS\_DX.f90} (type `\texttt{make
+combine\_AVS\_DX}' and then `\texttt{xcombine\_AVS\_DX}') to generate
+AVS \url{www.avs.com} output files (in AVS UCD format) or OpenDX \url{www.opendx.org}
+output files showing the mesh, the MPI partition (slices), the $\nchunks$
+chunks, the source and receiver location, etc. Use the AVS UCD files
+\texttt{AVS\_continent\_boundaries.inp} and \texttt{AVS\_plate\_boundaries.inp}
+or the OpenDX files \texttt{DX\_continent\_boundaries.dx} and \texttt{DX\_plate\_boundaries.dx}
+(that can be created using Perl scripts located in \texttt{UTILS/Visualization/opendx\_AVS})
+for reference.
+
+
+\section{\label{sec:Movies}Movies}
+
+To make a surface or volume movie of the simulation, set parameters
+\texttt{MOVIE\_SURFACE}, \texttt{MOVIE\_VOLUME}, and \texttt{NTSTEP\_BETWEEN\_FRAMES}
+in the \texttt{Par\_file}. Turning on the movie flags, in particular
+\texttt{MOVIE\_VOLUME}, produces large output files. \texttt{MOVIE\_VOLUME}
+files are saved in the \texttt{LOCAL\_PATH} directory, whereas \texttt{MOVIE\_SURFACE}
+output files are saved in the \texttt{OUTPUT\_FILES} directory. We
+save the velocity field. The look of a movie is determined by the
+half-duration of the source. The half-duration should be large enough
+so that the movie does not contain frequencies that are not resolved
+by the mesh, i.e., it should not contain numerical noise. This can
+be accomplished by selecting a CMT \texttt{HALF\_DURATION} > 1.1 $\times$
+smallest period (see figure \ref{fig:CMTSOLUTION-file}). When \texttt{\small MOVIE\_SURFACE}
+= \texttt{\small .true.} or \texttt{\small MOVIE\_VOLUME}{\small{}
+}\texttt{\small =}{\small{} }\texttt{\small .true.}, the half duration
+of each source in the \texttt{CMTSOLUTION} file is replaced by
+
+\begin{quote}
+\[
+\sqrt{(}\mathrm{\mathtt{HALF\_DURATIO}\mathtt{N}^{2}}+\mathrm{\mathtt{HDUR\_MOVI}\mathtt{E}^{2}})\]
+\textbf{NOTE:} If \texttt{HDUR\_MOVIE} is set to 0.0, the code will
+select the appropriate value of 1.1 $\times$ smallest period. As
+usual, for a point source one can set \texttt{HALF\_DURATION} in the
+\texttt{Par\_file} to be 0.0 and \texttt{HDUR\_MOVIE} = 0.0 to get
+the highest frequencies resolved by the simulation, but for a finite
+source one would keep all the \texttt{HALF\_DURATION}s as prescribed
+by the finite source model and set \texttt{HDUR\_MOVIE} = 0.0.
+\end{quote}
+
+\subsection{Movie Surface}
+
+When running \texttt{xspecfem3D} with the \texttt{MOVIE\_SURFACE}
+flag turned on the code outputs \texttt{moviedata??????} files in
+the \texttt{OUTPUT\_FILES} directory. The files are in a fairly complicated
+binary format, but there are two programs provided to convert the
+output into more user friendly formats. The first one, \texttt{create\_movie\_AVS\_DX.f90}
+outputs data in ASCII, OpenDX, AVS, or ParaView format. Run the code
+from the source directory (type `\texttt{make} \texttt{create\_movie\_AVS\_DX}'
+first) to create an input file in your format of choice. The code
+will prompt the user for input parameters. The second program \texttt{create\_movie\_GMT\_global.f90}
+outputs ASCII xyz files, convenient for use with GMT. This codes uses
+significantly less memory than \texttt{create\_movie\_AVS\_DX.f90}
+and is therefore useful for high resolution runs.
+
+%
+\begin{figure}[H]
+\noindent \begin{centering}
+\includegraphics[scale=0.75]{figures/geo-poster4_small}
+\par\end{centering}
+
+\caption{Snapshots from a global movie for the December 26, 2004, M=9.2 Sumatra-Andaman
+earthquake. Time runs down successive columns.}
+
+\end{figure}
+
+
+
+\subsection{\label{sub:Movie-Volume}Movie Volume}
+
+When running xspecfem3D with the \texttt{\small MOVIE\_VOLUME} flag
+turned on, the code outputs several files in \texttt{\small LOCAL\_DIR}.
+As the files can be very large, there are several flags in the \texttt{\small Par\_file}
+that control the region in space and time that is saved. These are:
+\texttt{\small MOVIE\_TOP\_KM}, \texttt{\small MOVIE\_BOTTOM\_KM},
+\texttt{\small MOVIE\_WEST\_DEG}, \texttt{\small MOVIE\_EAST\_DEG},
+\texttt{\small MOVIE\_NORTH\_DEG}, \texttt{\small MOVIE\_SOUTH\_DEG},
+\texttt{\small MOVIE\_START} and \texttt{\small MOVIE\_STOP}. The
+code will save a given element if the center of the element is in
+the prescribed volume.
+
+\begin{description}
+\item [{The~Top/Bottom:}] Depth below the surface in kilometers, use \texttt{\small MOVIE\_TOP}
+\texttt{\small =} \texttt{\small -100.0} to make sure the surface
+is stored.
+\item [{West/East:}] Longitude, degrees East {[}-180.0/180.0]
+\item [{North/South:}] Latitute, degrees North {[}-90.0/90.0]
+\item [{Start/Stop:}] Frames will be stored at \texttt{\small MOVIE\_START}
+\texttt{\small +} \texttt{\small i{*}NSTEP\_BETWEEN\_FRAMES}, where
+\texttt{\small i=(0,1,2..)} while \texttt{\small i{*}NSTEP\_BETWEEN\_FRAMES}
+\texttt{\small <=} \texttt{\small MOVIE\_STOP}{\small \par}
+\end{description}
+The code saves several files, and the output is saved by each processor.
+The first is \texttt{\small proc??????\_movie3D\_info.txt} which contains
+two numbers, first the number of points within the prescribed volume
+within this particular slice, and second the number of elements. The
+next files are \texttt{\small proc??????\_movie3D\_x.bin}, \texttt{\small proc??????\_movie3D\_y.bin},
+\texttt{\small proc??????\_movie3D\_z.bin} which store the locations
+of the points in the 3D mesh.
+
+Finally the code stores the ``value'' at each of the points. Which
+value is determined by \texttt{\small MOVIE\_VOLUME\_TYPE} in the
+\texttt{\small Par\_file}. Choose 1 to save the strain, 2 to save
+the time integral of strain, and 3 to save $\mu${*}time integral
+of strain in the subvolume. Choosing 4 causes the code to save the
+trace of the stress and the deviatoric stress in the whole volume
+(not the subvolume in space), at the time steps specified. The name
+of the output file will depend on the \texttt{\small MOVIE\_VOLUME\_TYPE}
+chosen.
+
+Setting \texttt{\small MOVIE\_VOLUME\_COARSE} \texttt{\small =} \texttt{\small .true.}
+will make the code save only the corners of the elements, not all
+the points within each element for \texttt{\small MOVIE\_VOLUME\_TYPE}
+\texttt{\small =} \texttt{\small 1,2,3}.
+
+To make the code output your favorite ``value'' simply add a new \texttt{\small MOVIE\_VOLUME\_TYPE},
+a new subroutine to \texttt{\small write\_movie\_volume.f90} and a
+subroutine call to \texttt{\small specfem3D.f90}.
+
+A utility program to combine the files produced by \texttt{\small MOVIE\_VOLUME\_TYPE}
+\texttt{\small =} \texttt{\small 1,2,3} is provided in \texttt{\small combine\_paraview}~\\
+\texttt{\small \_strain\_data.f90}. Type \texttt{\small xcombine\_paraview\_strain\_data}
+to get the usage statement. The program \texttt{\small combine\_vol}~\\
+\texttt{\small \_data.f90} can be used for \texttt{\small MOVIE\_VOLUME\_TYPE}
+\texttt{\small =} \texttt{\small 4}.
+
+
+\section{\label{sec:Finite-Frequency-Kernels}Finite-Frequency Kernels}
+
+The finite-frequency kernels computed as explained in Section \ref{sec:Adjoint-simulation-finite}
+are saved in the \texttt{LOCAL\_PATH} at the end of the simulation.
+Therefore, we first need to collect these files on the front end,
+combine them into one mesh file, and visualize them with some auxilliary
+programs.
+
+\begin{enumerate}
+\item \textbf{Create slice files}
+
+
+We will only discuss the case of one source-receiver pair, i.e., the
+so-called banana-doughnut kernels. Although it is possible to collect
+the kernel files from all slices onto the front end, it usually takes
+up too much storage space (at least tens of gigabytes). Since the
+sensitivity kernels are the strongest along the source-receiver great
+circle path, it is sufficient to collect only the slices that are
+along or close to the great circle path.
+
+A Perl script \texttt{UTILS/Visualization/Paraview/global\_slice\_number.pl} can
+help to figure out the slice numbers that lie along the great circle
+path (both the minor and major arcs), as well as the slice numbers
+required to produce a full picture of the inner core if your kernel
+also illuminates the inner core.
+
+\begin{enumerate}
+\item You need to first compile the utility programs provided in the \texttt{UTILS/Visualization/Paraview/}~\\
+\texttt{global\_slice\_util
+}directory. Then copy the \texttt{CMTSOLUTION} file, \texttt{STATIONS\_ADJOINT},
+and \texttt{Par\_file}, and run:
+
+\begin{lyxcode}
+global\_slice\_number.pl~CMTSOLUTION~STATIONS\_ADJOINT~Par\_file
+\end{lyxcode}
+In the case of visualization boundary kernels or spherical cross-sections
+of the volumetric kernels, it is necessary to obtain the slice numbers
+that cover a belt along the source and receiver great circle path,
+and you can use the hybrid version:
+
+\begin{lyxcode}
+globe\_slice\_number2.pl~CMTSOLUTION~STATIONS\_ADJOINT~
+
+~~~~~Par\_file~belt\_width\_in\_degrees
+\end{lyxcode}
+A typical value for \texttt{belt\_width\_in\_degrees} can be 20.
+
+\item For a full 6-chunk simulation, this script will generate the \texttt{slice\_minor},
+\texttt{slice\_major}, \texttt{slice\_ic} files, but for a one- or
+two-chunk simulation, this script only generates the \texttt{slice\_minor}
+file.
+\item For cases with multiple sources and multiple receivers, you need to
+provide a slice file before proceeding to the next step.
+\end{enumerate}
+\item \textbf{Collect the kernel files}
+
+
+After obtaining the slice files, you can collect the corresponding
+kernel files from the given slices.
+
+\begin{enumerate}
+\item To accomplish this, you can use or modify the scripts in \texttt{UTILS/collect\_database
+}directory:
+
+\begin{lyxcode}
+{\small copy\_m(oc,ic)\_globe\_database.pl~slice\_file~lsf\_machine\_file~filename~{[}jobid]}{\small \par}
+\end{lyxcode}
+for volumetric kernels, where \texttt{\small lsf\_machine\_file} is
+the machine file generated by the LSF scheduler, \texttt{\small filename}
+is the kernel name (e.g., \texttt{\small rho\_kernel}, \texttt{\small alpha\_kernel}
+and \texttt{\small beta\_kernel}), and the optional \texttt{\small jobid}
+is the name of the subdirectory under \texttt{\small LOCAL\_PATH}
+where all the kernel files are stored. For boundary kernels, you need
+to use
+
+\begin{lyxcode}
+{\small copy\_surf\_globe\_database.pl~slice\_file~lsf\_machine\_file~filename~{[}jobid]}{\small \par}
+\end{lyxcode}
+where the filename can be \texttt{\small Moho\_kernel}, \texttt{\small d400\_kernel},
+\texttt{\small d670\_kernel}, \texttt{\small CMB\_kernel} and \texttt{\small ICB\_kernel}.
+
+\item After executing this script, all the necessary mesh topology files
+as well as the kernel array files are collected to the local directory
+on the front end.
+\end{enumerate}
+\item \textbf{Combine kernel files into one mesh file}
+
+
+We use an auxiliary program \texttt{combine\_vol\_data.f90} to combine
+the volumetric kernel files from all slices into one mesh file, and
+\texttt{combine\_surf\_data.f90 }to combine the surface kernel files.
+
+\begin{enumerate}
+\item Compile it in the global code directory:
+
+\begin{lyxcode}
+{\footnotesize make~combine\_vol\_data~}{\footnotesize \par}
+
+{\footnotesize xcombine\_vol\_data}~{\footnotesize slice\_list}~{\footnotesize filename}~{\footnotesize input\_dir}~{\footnotesize output\_dir}~{\footnotesize high/low-resolution}~{\footnotesize {[}region]}{\footnotesize \par}
+\end{lyxcode}
+where \texttt{input\_dir} is the directory where all the individual
+kernel files are stored, and \texttt{output\_dir} is where the mesh
+file will be written.
+
+\begin{lyxcode}
+{\footnotesize xcombine\_surf\_data~slice\_list~filename~surfname~input\_dir~output\_dir~hi~gh/low-resolution~2D/3D}{\footnotesize \par}
+\end{lyxcode}
+where \texttt{surfname} should correspond to the specific kernel file
+name, and can be chosen from \texttt{Moho}, \texttt{400}, \texttt{670},
+\texttt{CMB} and \texttt{ICB}.
+
+\item Use 1 for a high-resolution mesh, outputting all the GLL points to
+the mesh file, or use 0 for low resolution, outputting only the corner
+points of the elements to the mesh file. Use 0 for 2D surface kernel
+files and 1 for 3D volumetric kernel files.
+\item Use region = 1 for the mantle, region =2 for the outer core, region
+= 3 for the inner core, and region = 0 for all regions.
+\item The output mesh file will have the name \texttt{reg\_?\_rho(alpha,beta)\_kernel.mesh,}
+or\texttt{ }~\\
+\texttt{reg\_?\_Moho(d400,d670,CMB,ICB)\_kernel.surf.}
+\end{enumerate}
+\item \textbf{Convert mesh files into .vtu files}
+
+\begin{enumerate}
+\item We next convert the \texttt{.mesh} file into the VTU (Unstructured
+grid file) format which can be viewed in ParaView, for example:
+
+\begin{lyxcode}
+mesh2vtu.pl~-i~file.mesh~-o~file.vtu
+\end{lyxcode}
+\item Notice that this Perl script uses a program \texttt{mesh2vtu} in the
+\texttt{UTILS/Visualization/Paraview/}~\\
+\texttt{mesh2vtu} directory, which further uses the
+VTK \url{www.vtk.org} run-time library for its execution. Therefore,
+make sure you have them properly set in the script.
+\end{enumerate}
+\item \textbf{Copy over the source and receiver .vtk file}
+
+
+In the case of a single source and a single receiver, the simulation
+also generates the \texttt{OUTPUT\_FILES/sr.vtk} file to describe
+the source and receiver locations, which can be viewed in Paraview
+in the next step.
+
+\item \textbf{View the mesh in ParaView}
+
+
+Finally, we can view the mesh in ParaView \url{www.paraview.org}.
+
+\begin{enumerate}
+\item Open ParaView.
+\item From the top menu, \textsf{File} $\rightarrow$\textsf{ Open data},
+select \texttt{file.vtu}, and click the \textsf{Accept} button.
+
+\begin{itemize}
+\item If the mesh file is of moderate size, it shows up on the screen; otherwise,
+only the outline is shown.
+\end{itemize}
+\item Click \textsf{Display Tab} $\rightarrow$ \textsf{Display Style} $\rightarrow$
+\textsf{Representation} and select \textsf{wireframe of surface} to
+display it.
+\item To create a cross-section of the volumetric mesh, choose \textsf{Filter}
+$\rightarrow$ \textsf{cut}, and under \textsf{Parameters Tab}, choose
+\textsf{Cut Function} $\rightarrow$ \textsf{plane}.
+\item Fill in center and normal information given by the standard output
+from \texttt{global\_slice\_number.pl} script.
+\item To change the color scale, go to \textsf{Display Tab} $\rightarrow$
+\textsf{Color} $\rightarrow$ \textsf{Edit Color Map} and reselect
+lower and upper limits, or change the color scheme.
+\item Now load in the source and receiver location file by \textsf{File}
+$\rightarrow$\textsf{ Open data}, select \texttt{sr.vt}k, and click
+the \textsf{Accept} button. Choose \textsf{Filter} $\rightarrow$\textsf{
+Glyph}, and represent the points by `\textsf{spheres}'.
+\item For more information about ParaView, see the ParaView Users Guide \url{www.paraview.org/files/v1.6/ParaViewUsersGuide.PDF}.
+\end{enumerate}
+\end{enumerate}
+For illustration purposes, Figure \ref{fig:P-wave-speed-finite-frequency}
+shows the P-wave speed finite-frequency kernel for a P arrival recorded
+at an epicental distance of $60^{\circ}$ for a deep event.
+
+%
+\begin{figure}[H]
+\includegraphics[clip,scale=0.9]{figures/P_alpha_60d_18s}
+
+\caption{\label{fig:P-wave-speed-finite-frequency}P-wave speed finite-frequency
+kernel for a P arrival recorded at an epicentral distance of $60^{\circ}$.}
+
+\end{figure}
+
+
+
+\chapter{\label{cha:Running-Scheduler}Running through a Scheduler}
+
+The code is usually run on large parallel machines, often PC clusters,
+most of which use schedulers, i.e., queuing or batch management systems
+to manage the running of jobs from a large number of users. The following
+considerations need to be taken into account when running on a system
+that uses a scheduler:
+
+\begin{itemize}
+\item The processors/nodes to be used for each run are assigned dynamically
+by the scheduler, based on availability. Therefore, in order for the
+mesher and the solver (or between successive runs of the solver) to
+have access to the same database files (if they are stored on hard
+drives local to the nodes on which the code is run), they must be
+launched in sequence as a single job.
+\item On some systems, the nodes to which running jobs are assigned are
+not configured for compilation. It may therefore be necessary to pre-compile
+both the mesher and the solver. A small program provided in the distribution
+called \texttt{\small create\_header\_file.f90} can be used to directly
+create\texttt{\small{} OUTPUT\_FILES/values\_from\_mesher.h} using
+the information in the \texttt{\small DATA/Par\_file} without having
+to run the mesher (type \texttt{\small `make}{\small{} }\texttt{\small create\_header\_}~\\
+\texttt{\small file}' to compile it and `\texttt{\small xcreate\_header\_file}'
+to run it; refer to the sample scripts below). The solver can now
+be compiled as explained above.
+\item One feature of schedulers/queuing systems is that they allow submission
+of multiple jobs in a {}``launch and forget'' mode. In order to
+take advantage of this property, care needs to be taken that output
+and intermediate files from separate jobs do not overwrite each other,
+or otherwise interfere with other running jobs.
+\end{itemize}
+We describe here in some detail a job submission procedure for the
+Caltech 1024-node cluster, CITerra, under the LSF scheduling system.
+We consider the submission of a regular forward simulation. The two
+main scripts are \texttt{\small run\_lsf.bash}, which compiles the
+Fortran code and submits the job to the scheduler, and \texttt{\small go\_mesher\_solver\_lsf}~\\
+\texttt{\small .bash}, which contains the instructions that make up
+the job itself. These scripts can be found in \texttt{\small UTILS/Cluster/lsf}
+directory and can straightforwardly be modified and adapted to meet
+more specific running needs.
+
+
+\section{\texttt{run\_lsf.bash}}
+
+This script first sets the job queue to be `normal'. It then compiles
+the mesher and solver together, figures out the number of processors
+required for this simulation from \texttt{DATA/Par\_file}, and submits
+the LSF job.
+
+\begin{lyxcode}
+\#!/bin/bash
+
+\#~use~the~normal~queue~unless~otherwise~directed~queue=\char`\"{}-q~normal\char`\"{}~
+
+if~{[}~\$\#~-eq~1~];~then
+
+~~~~~~~~echo~\char`\"{}Setting~the~queue~to~\$1\char`\"{}
+
+~~~~~~~~queue=\char`\"{}-q~\$1\char`\"{}~
+
+fi~\\
+~\\
+\#~compile~the~mesher~and~the~solver~
+
+d=`date'~echo~\char`\"{}Starting~compilation~\$d\char`\"{}~
+
+make~clean~
+
+make~meshfem3D~
+
+make~create\_header\_file~
+
+xcreate\_header\_file~
+
+make~specfem3D~
+
+d=`date'~
+
+echo~\char`\"{}Finished~compilation~\$d\char`\"{}~\\
+~\\
+\#~compute~total~number~of~nodes~needed~
+
+NPROC\_XI=`grep~NPROC\_XI~DATA/Par\_file~|~cut~-c~34-~'~
+
+NPROC\_ETA=`grep~NPROC\_ETA~DATA/Par\_file~|~cut~-c~34-~'~
+
+NCHUNKS=`grep~NCHUNKS~DATA/Par\_file~|~cut~-c~34-~'~\\
+~\\
+\#~total~number~of~nodes~is~the~product~of~the~values~read~
+
+numnodes=\$((~\$NCHUNKS~{*}~\$NPROC\_XI~{*}~\$NPROC\_ETA~))~\\
+~\\
+echo~\char`\"{}Submitting~job\char`\"{}~
+
+bsub~\$queue~-n~\$numnodes~-W~60~-K~<go\_mesher\_solver\_lsf\_globe.bash~
+\end{lyxcode}
+
+\section{\texttt{go\_mesher\_solver\_lsf\_globe.bash}}
+
+This script describes the job itself, including setup steps that can
+only be done once the scheduler has assigned a job-ID and a set of
+compute nodes to the job, the \texttt{run\_lsf.bash} commands used
+to run the mesher and the solver, and calls to scripts that collect
+the output seismograms from the compute nodes and perform clean-up
+operations.
+
+\begin{enumerate}
+\item First the script directs the scheduler to save its own output and
+output from \texttt{stdout} into \texttt{\small OUTPUT\_FILES/\%J.o},
+where \texttt{\%J} is short-hand for the job-ID; it also tells the
+scheduler what version of \texttt{mpich} to use (\texttt{mpich\_gm})
+and how to name this job (\texttt{go\_mesher\_solver\_lsf}).
+\item The script then creates a list of the nodes allocated to this job
+by echoing the value of a dynamically set environment variable \texttt{LSB\_MCPU\_HOSTS}
+and parsing the output into a one-column list using the Perl script
+\texttt{UTILS/Cluster/lsf/remap\_lsf\_machines.pl}. It then creates a set of scratch
+directories on these nodes (\texttt{\small /scratch/}~\\
+\texttt{\small \$USER/DATABASES\_MPI}) to be used as the \texttt{LOCAL\_PATH}
+for temporary storage of the database files. The scratch directories
+are created using \texttt{shmux}, a shell multiplexor that can execute
+the same commands on many hosts in parallel. \texttt{shmux} is available
+from Shmux \url{web.taranis.org/shmux/}. Make sure that the \texttt{LOCAL\_PATH}
+parameter in \texttt{DATA/Par\_file} is also set properly.
+\item The next portion of the script launches the mesher and then the solver
+using \texttt{run\_lsf.bash}.
+\item The final portion of the script performs clean up on the nodes using
+the Perl script \texttt{cleanmulti.pl}
+\end{enumerate}
+\begin{lyxcode}
+\#!/bin/bash~-v
+
+\#BSUB~-o~OUTPUT\_FILES/\%J.o
+
+\#BSUB~-a~mpich\_gm
+
+\#BSUB~-J~go\_mesher\_solver\_lsf
+
+BASEMPIDIR=/scratch/\$USER/DATABASES\_MPI
+
+echo~\char`\"{}\$LSB\_MCPU\_HOSTS\char`\"{}~>~OUTPUT\_FILES/lsf\_machines
+
+echo~\char`\"{}\$LSB\_JOBID\char`\"{}~>~OUTPUT\_FILES/jobid
+
+./remap\_lsf\_machines.pl~OUTPUT\_FILES/lsf\_machines~>OUTPUT\_FILES/machines
+
+\#~Modif~:~create~a~directory~for~this~job
+
+shmux~-M50~-Sall~-c~\char`\"{}mkdir~-p~/scratch/\$USER;
+
+mkdir~-p~\$BASEMPIDIR.\$LSB\_JOBID\char`\"{}~-~<~OUTPUT\_FILES/machines~>/dev/null
+
+\#~Set~the~local~path~in~Par\_file
+
+sed~-e~\char`\"{}s:\textasciicircum{}LOCAL\_PATH~.{*}:LOCAL\_PATH~=~\$BASEMPIDIR.\$LSB\_JOBID:\char`\"{}
+
+<~DATA/Par\_file~>~DATA/Par\_file.tmp
+
+mv~DATA/Par\_file.tmp~DATA/Par\_file
+
+current\_pwd=\$PWD
+
+mpirun.lsf~~-{}-gm-no-shmem~-{}-gm-copy-env~\$current\_pwd/xmeshfem3D
+
+mpirun.lsf~-{}-gm-no-shmem~-{}-gm-copy-env~\$current\_pwd/xspecfem3D
+
+\#~clean~up
+
+cleanbase\_jobid.pl~OUTPUT\_FILES/machines~DATA/Par\_file
+\end{lyxcode}
+
+\section{\texttt{run\_lsf.kernel} and \texttt{go\_mesher\_solver\_globe.kernel}}
+
+For kernel simulations, you can use the sample run scripts \texttt{run\_lsf.kernel}
+and \texttt{go\_mesher\_solver\_globe}~\\
+\texttt{.kernel} provided in \texttt{UTILS/Cluster} directory, and modify
+the command-line arguments of \texttt{xcut\_velocity} in \texttt{go\_mesher\_}~\\
+\texttt{solver\_globe.kernel }according to the start and end time
+of the specific portion of the forward seismograms you are interested
+in.
+
+
+\chapter{{\normalsize \label{cha:-Changing-the}} Changing the Model}
+
+In this section we explain how to change the crustal, mantle, or inner
+core models. These changes involve contributing specific subroutines
+that replace existing subroutines in the \texttt{SPECFEM3D\_GLOBE}
+package.
+
+
+\section{{\normalsize \label{sec:Changing-the-Crustal}}Changing the Crustal
+Model}
+
+The 3D crustal model Crust2.0 \citep{BaLaMa00} is superimposed onto
+the mesh by the subroutine \texttt{model\_crust}~\\
+\texttt{.f90}. To accomplish this, the flag \texttt{CRUSTAL}, set
+in the subroutine \texttt{get\_model\_parameters.f90}, is used
+to indicate a 3D crustal model. When this flag is set to \texttt{.true.},
+the crust on top of the 1D reference model (PREM, IASP91, or AK135)
+is removed and replaced by extending the mantle. The 3D crustal model
+is subsequently overprinted onto the crust-less 1D reference model.
+The call to the 3D crustal routine is of the form
+
+\begin{lyxcode}
+call~model\_crust(lat,lon,r,vp,vs,rho,moho,foundcrust,CM\_V,elem\_in\_crust)
+\end{lyxcode}
+Input to this routine consists of:
+
+\begin{description}
+\item [{\texttt{lat}}] Latitude in degrees.
+\item [{\texttt{lon}}] Longitude in degrees.
+\item [{\texttt{r}}] Non-dimensionalized radius ($0<\texttt{r}<1$).
+\end{description}
+Output from the routine consists of:
+
+\begin{description}
+\item [{\texttt{vp}}] Non-dimensionalized compressional wave speed at location
+(\texttt{lat},\texttt{lon},\texttt{r}).
+\item [{\texttt{vs}}] Non-dimensionalized shear wave speed.
+\item [{\texttt{rho}}] Non-dimensionalized density.
+\item [{\texttt{moho}}] Non-dimensionalized Moho depth.
+\item [{\texttt{found\_crust}}] Logical that is set to \texttt{.true.}
+only if crust exists at location (\texttt{lat},\texttt{lon},\texttt{r}),
+i.e., \texttt{.false.} for radii \texttt{r} in the mantle. This flags
+determines whether or not a particular location is in the crust and,
+if so, what parameters to assign to the mesh at this location.
+\item [{\texttt{CM\_V}}] Fortran structure that contains the parameters,
+variables and arrays that describe the model.
+\item [{\texttt{elem\_in\_crust}}] Logical that is used to force the
+routine to return crustal values, even if the location would
+be below the crust.
+\end{description}
+All output needs to be non-dimensionalized according to the convention
+summarized in Appendix~\ref{cha:Non-Dimensionalization-Conventions}.
+You can replace this subroutine by your own routine \textit{provided
+you do not change the call structure of the routine}, i.e., the new
+routine should take exactly the same input and produce the required,
+properly non-dimensionalized output.
+
+Part of the file \texttt{model\_crust.f90} is the subroutine \texttt{model\_crust\_broadcast}.
+The call to this routine takes argument \texttt{CM\_V} and is used
+to once-and-for-all read in the databases related to Crust2.0 and
+broadcast the model to all parallel processes. If
+you replace the file \texttt{model\_crust.f90} with your own implementation,
+you \textit{must} provide a \texttt{model\_crust\_broadcast} routine,
+even if it does nothing. Model constants and variables read by the
+routine \texttt{model\_crust\_broadcast} are passed to the subroutine
+\texttt{read\_crust\_model} through the structure \texttt{CM\_V}.
+An alternative crustal model could use the same construct. Please
+feel free to contribute subroutines for new models and send them to
+us so that they can be included in future releases of the software.
+
+\begin{quote}
+\textbf{NOTE:} If you decide to create your own version of file \texttt{model\_crust.f90},
+you must add calls to \texttt{MPI\_BCAST} in the subroutine \texttt{model\_crust\_broadcast}
+after the call to the \texttt{read\_crust\_model} subroutine that
+reads the isotropic mantle model once and for all in the mesher. This
+is done in order to read the (potentially large) model data files
+on the master node (which is the processor of rank 0 in our code)
+only and then send a copy to all the other nodes using an MPI broadcast,
+rather than using an implementation in which all the nodes would read
+the same model data files from a remotely-mounted home file system,
+which could create a bottleneck on the network in the case of a large
+number of nodes. For example, in the current call to that routine
+from \texttt{model\_crust.f90,} we write:
+\end{quote}
+\begin{lyxcode}
+{\footnotesize !~the~variables~read~are~declared~and~stored~in~structure~CM\_V~~~~~}{\footnotesize \par}
+
+{\footnotesize{}~~if(myrank~==~0)~call~read\_crust\_model(CM\_V)~}{\footnotesize \par}
+
+{\footnotesize !~broadcast~the~information~read~on~the~master~to~the~nodes~~~~~}{\footnotesize \par}
+
+{\footnotesize{}~~call~MPI\_BCAST(CM\_V\%thlr,NKEYS\_CRUST{*}NLAYERS\_CRUST,MPI\_DOUBLE\_PRECISION,}{\footnotesize \par}
+
+{\footnotesize{}~~~~~~~~~~~~~~~~~~~0,MPI\_COMM\_WORLD,ier)~~~~~}{\footnotesize \par}
+
+{\footnotesize{}~~call~MPI\_BCAST(CM\_V\%velocp,NKEYS\_CRUST{*}NLAYERS\_CRUST,MPI\_DOUBLE\_PRECISION,}{\footnotesize \par}
+
+{\footnotesize{}~~~~~~~~~~~~~~~~~~~0,MPI\_COMM\_WORLD,ier)~~~~~}{\footnotesize \par}
+
+{\footnotesize{}~~call~MPI\_BCAST(CM\_V\%velocs,NKEYS\_CRUST{*}NLAYERS\_CRUST,MPI\_DOUBLE\_PRECISION,}{\footnotesize \par}
+
+{\footnotesize{}~~~~~~~~~~~~~~~~~~~0,MPI\_COMM\_WORLD,ier)~~~~~}{\footnotesize \par}
+
+{\footnotesize{}~~call~MPI\_BCAST(CM\_V\%dens,NKEYS\_CRUST{*}NLAYERS\_CRUST,MPI\_DOUBLE\_PRECISION,}{\footnotesize \par}
+
+{\footnotesize{}~~~~~~~~~~~~~~~~~~~0,MPI\_COMM\_WORLD,ier)~~~~~}{\footnotesize \par}
+
+{\footnotesize{}~~call~MPI\_BCAST(CM\_V\%abbreviation,NCAP\_CRUST{*}NCAP\_CRUST,MPI\_CHARACTER,}{\footnotesize \par}
+
+{\footnotesize{}~~~~~~~~~~~~~~~~~~~0,MPI\_COMM\_WORLD,ier)~~~~~}{\footnotesize \par}
+
+{\footnotesize{}~~call~MPI\_BCAST(CM\_V\%code,2{*}NKEYS\_CRUST,MPI\_CHARACTER,0,MPI\_COMM\_WORLD,ier)~}{\footnotesize \par}
+\end{lyxcode}
+
+\section{{\normalsize \label{sec:Changing-the-Mantle}}Changing the Mantle
+Model}
+
+This section discusses how to change isotropic and anisotropic 3D
+mantle models. Usually such changes go hand-in-hand with changing
+the 3D crustal model.
+
+
+\subsection{{\normalsize \label{sub:Isotropic-Models}Isotropic Models}}
+
+The 3D mantle model S20RTS \citep{RiVaWo99} is superimposed onto
+the mantle mesh by the subroutine \texttt{model\_s20rts.f90}. The
+call to this subroutine is of the form
+
+\begin{lyxcode}
+call~model\_s20rts(radius,theta,phi,dvs,dvp,drho,D3MM\_V)~
+\end{lyxcode}
+Input to this routine consists of:
+
+\begin{description}
+\item [{\texttt{radius}}] Non-dimensionalized radius ($\texttt{RCMB/R\_ EARTH}<\texttt{r}<\texttt{RMOHO/R\_ EARTH}$;
+for a given 1D reference model, the constants \texttt{RCMB} and \texttt{RMOHO}
+are set in the \texttt{\small get\_model\_parameters}\texttt{.f90}
+file). The code expects the isotropic mantle model to be defined between
+the Moho (with radius \texttt{RMOHO} in m) and the core-mantle boundary
+(CMB; radius \texttt{RCMB} in m) of a 1D reference model. When a 3D
+crustal model is superimposed, as will usually be the case, the 3D
+mantle model is stretched to fill any potential gap between the radius
+of the Moho in the 1D reference model and the Moho in the 3D crustal
+model. Thus, when the Moho in the 3D crustal model is shallower than
+the Moho in the reference model, e.g., typically below the oceans,
+the mantle model is extended to fill this gap.
+\item [{\texttt{theta}}] Colatitude in radians.
+\item [{\texttt{phi}}] Longitude in radians.
+\end{description}
+Output from the routine are the following non-dimensional perturbations:
+
+\begin{description}
+\item [{\texttt{dvs}}] Relative shear-wave speed perturbations $\delta\beta/\beta$
+at location (\texttt{radius},\texttt{theta},\texttt{phi}).
+\item [{\texttt{dvp}}] Relative compressional-wave speed perturbations
+$\delta\alpha/\alpha$.
+\item [{\texttt{drho}}] Relative density perturbations $\delta\rho/\rho$.
+\item [{\texttt{D3MM\_V}}] Fortran structure that contains the parameters,
+variables and arrays that describe the model.
+\end{description}
+You can replace the \texttt{model\_s20rts.f90} file with your own
+version \textit{provided you do not change the call structure of the
+routine}, i.e., the new routine should take exactly the same input
+and produce the required relative output.
+
+Part of the file \texttt{model\_s20rts.f90} is the subroutine
+\texttt{model\_s20rts\_broadcast}.
+The call to this routine takes argument\texttt{ D3MM\_V} and is used
+to once-and-for-all read in the databases related to S20RTS. If you
+replace the file \texttt{model\_s20rts.f90} with your own implementation,
+you \textit{must} provide a \texttt{model\_s20rts\_broadcast} routine,
+even if it does nothing. Model constants and variables read by the
+routine \texttt{model\_s20rts\_broadcast} are passed to the subroutine
+\texttt{read\_model\_s20rts} through the structure \texttt{D3MM\_V.}
+An alternative mantle model should use the same construct.
+
+\begin{quote}
+\textbf{NOTE:} If you decide to create your own version of file \texttt{model\_s20rts.f90},
+you must add calls to \texttt{MPI\_BCAST} in the subroutine \texttt{model\_s20rts\_broadcast}
+after the call to the \texttt{read\_model\_s20rts} subroutine that
+reads the isotropic mantle model once and for all in the mesher. This
+is done in order to read the (potentially large) model data files
+on the master node (which is the processor of rank 0 in our code)
+only and then send a copy to all the other nodes using an MPI broadcast,
+rather than using an implementation in which all the nodes would read
+the same model data files from a remotely-mounted home file system,
+which could create a bottleneck on the network in the case of a large
+number of nodes. For example, in the current call to that routine
+from \texttt{model\_s20rts.f90,} we write:
+\end{quote}
+\begin{lyxcode}
+{\footnotesize !~the~variables~read~are~declared~and~stored~in~structure~D3MM\_V}{\footnotesize \par}
+
+{\footnotesize{}~~if(myrank~==~0)~call~read\_model\_s20rts(D3MM\_V)~}{\footnotesize \par}
+
+{\footnotesize !~broadcast~the~information~read~on~the~master~to~the~nodes~~~~~~~}{\footnotesize \par}
+
+{\footnotesize{}~~call~MPI\_BCAST(D3MM\_V\%dvs\_a,(NK+1){*}(NS+1){*}(NS+1),MPI\_DOUBLE\_PRECISION,}{\footnotesize \par}
+
+{\footnotesize{}~~~~~~~~~~0,MPI\_COMM\_WORLD,ier)}{\footnotesize \par}
+
+{\footnotesize{}~~call~MPI\_BCAST(D3MM\_V\%dvs\_b,(NK+1){*}(NS+1){*}(NS+1),MPI\_DOUBLE\_PRECISION,}{\footnotesize \par}
+
+{\footnotesize{}~~~~~~~~~~0,MPI\_COMM\_WORLD,ier)}{\footnotesize \par}
+
+{\footnotesize{}~~call~MPI\_BCAST(D3MM\_V\%dvp\_a,(NK+1){*}(NS+1){*}(NS+1),MPI\_DOUBLE\_PRECISION,}{\footnotesize \par}
+
+{\footnotesize{}~~~~~~~~~~0,MPI\_COMM\_WORLD,ier)}{\footnotesize \par}
+
+{\footnotesize{}~~call~MPI\_BCAST(D3MM\_V\%dvp\_b,(NK+1){*}(NS+1){*}(NS+1),MPI\_DOUBLE\_PRECISION,}{\footnotesize \par}
+
+{\footnotesize{}~~~~~~~~~~0,MPI\_COMM\_WORLD,ier)~~~~~~}{\footnotesize \par}
+
+{\footnotesize{}~~call~MPI\_BCAST(D3MM\_V\%spknt,NK+1,MPI\_DOUBLE\_PRECISION,0,MPI\_COMM\_WORLD,ier)~~~~~~~}{\footnotesize \par}
+
+{\footnotesize{}~~call~MPI\_BCAST(D3MM\_V\%qq0,(NK+1){*}(NK+1),MPI\_DOUBLE\_PRECISION,0,MPI\_COMM\_WORLD,ier)~}{\footnotesize \par}
+
+{\footnotesize{}~~call~MPI\_BCAST(D3MM\_V\%qq,3{*}(NK+1){*}(NK+1),MPI\_DOUBLE\_PRECISION,0,MPI\_COMM\_WORLD,ier)~}{\footnotesize \par}
+\end{lyxcode}
+
+\subsection{{\normalsize \label{sub:Anisotropic-Models}Anisotropic Models}}
+
+Three-dimensional anisotropic mantle models may be superimposed on
+the mesh based upon the subroutine
+
+\begin{lyxcode}
+model\_aniso\_mantle.f90
+\end{lyxcode}
+The call to this subroutine is of the form
+
+\begin{lyxcode}
+call~model\_aniso\_mantle(r,theta,phi,rho,~\&~
+
+~~~~c11,c12,c13,c14,c15,c16,c22,c23,c24,c25,c26,~\&~
+
+~~~~c33,c34,c35,c36,c44,c45,c46,c55,c56,c66,AMM\_V)~
+\end{lyxcode}
+Input to this routine consists of:
+
+\begin{description}
+\item [{\texttt{r}}] Non-dimensionalized radius ($\texttt{RCMB/R\_ EARTH}<\texttt{r}<\texttt{RMOHO/R\_ EARTH}$;
+for a given 1D reference model, the constants \texttt{RCMB} and \texttt{RMOHO}
+are set in the \texttt{\small get\_model\_parameters}\texttt{.f90}
+file). The code expects the anisotropic mantle model to be defined
+between the Moho and the core-mantle boundary (CMB). When a 3D crustal
+model is superimposed, as will usually be the case, the 3D mantle
+model is stretched to fill any potential gap between the radius of
+the Moho in the 1D reference model and the Moho in the 3D crustal
+model. Thus, when the Moho in the 3D crustal model is shallower than
+the Moho in the reference model, e.g., typically below the oceans,
+the mantle model is extended to fill this gap.
+\item [{\texttt{theta}}] Colatitude in radians.
+\item [{\texttt{phi}}] Longitude in radians.
+\end{description}
+Output from the routine consists of the following non-dimensional
+model parameters:
+
+\begin{description}
+\item [{\texttt{rho}}] Non-dimensionalized density $\rho$.
+\item [{\texttt{c11},}] \textbf{$\cdots$,} \texttt{\textbf{c66}} 21 non-dimensionalized
+anisotropic elastic parameters.
+\item [{\texttt{AMM\_V}}] Fortran structure that contains the parameters,
+variables and arrays that describe the model.
+\end{description}
+You can replace the \texttt{model\_aniso\_mantle.f90} file by
+your own version \textit{provided you do not change the call structure
+of the routine}, i.e., the new routine should take exactly the same
+input and produce the required relative output. Part of the file \texttt{model\_aniso\_mantle.f90}
+is the subroutine \texttt{model\_aniso\_mantle\_broadcast}. The call to
+this routine takes argument \texttt{AMM\_V} and is used to once-and-for-all
+read in the static databases related to the anisotropic model. When
+you choose to replace the file \texttt{model\_aniso\_mantle.f90}
+with your own implementation you \textit{must} provide a \texttt{model\_aniso\_mantle\_broadcast}
+routine, even if it does nothing. Model constants and variables read
+by the routine \texttt{model\_aniso\_mantle\_broadcast} are passed through the
+structure \texttt{AMM\_V}. An alternative anisotropic mantle model
+should use the same construct.
+
+\begin{quote}
+\textbf{NOTE:} If you decide to create your own version of file \texttt{model\_aniso\_mantle.f90},
+you must add calls to \texttt{MPI\_BCAST} in file \texttt{model\_aniso\_mantle.f90}
+after the call to the \texttt{read\_aniso\_mantle\_model} subroutine
+that reads the anisotropic mantle model once and for all in the mesher.
+This is done in order to read the (potentially large) model data files
+on the master node (which is the processor of rank 0 in our code)
+only and then send a copy to all the other nodes using an MPI broadcast,
+rather than using an implementation in which all the nodes would read
+the same model data files from a remotely-mounted home file system,
+which could create a bottleneck on the network in the case of a large
+number of nodes. For example, in the current call to that routine
+from \texttt{model\_aniso\_mantle.f90,} we write:
+\end{quote}
+\begin{lyxcode}
+{\footnotesize !~the~variables~read~are~declared~and~stored~in~structure~AMM\_V}{\footnotesize \par}
+
+{\footnotesize{}~~if(myrank~==~0)~call~read\_aniso\_mantle\_model(AMM\_V)}{\footnotesize \par}
+
+{\footnotesize !~broadcast~the~information~read~on~the~master~to~the~nodes}{\footnotesize \par}
+
+{\footnotesize{}~~call~MPI\_BCAST(AMM\_V\%npar1,1,MPI\_INTEGER,0,MPI\_COMM\_WORLD,ier)~~~~}{\footnotesize \par}
+
+{\footnotesize{}~~call~MPI\_BCAST(AMM\_V\%beta,14{*}34{*}37{*}73,MPI\_DOUBLE\_PRECISION,0,MPI\_COMM\_WORLD,ier)}{\footnotesize \par}
+
+{\footnotesize{}~~call~MPI\_BCAST(AMM\_V\%pro,47,MPI\_DOUBLE\_PRECISION,0,MPI\_COMM\_WORLD,ier)}{\footnotesize \par}
+\end{lyxcode}
+Rotation of the anisotropic tensor elements from one coordinate system
+to another coordinate system may be accomplished based upon the subroutine
+\texttt{rotate\_aniso\_tensor}. Use of this routine requires understanding
+the coordinate system used in \texttt{SPECFEM3D\_GLOBE}, as discussed
+in Appendix~\ref{cha:Reference-Frame-Convention}.
+
+
+\section{{\normalsize \label{sec:Anelastic-Models}}Anelastic Models}
+
+Three-dimensional anelastic (attenuation) models may be superimposed
+onto the mesh based upon your subroutine .
+\texttt{model\_atten3D.f90}.
+The call to this routine would be as follows
+
+\begin{lyxcode}
+call~model\_atten3D(radius,~colatitude,~longitude,~Qmu,~QRFSI12\_Q,~idoubling)
+\end{lyxcode}
+Input to this routine consists of:
+
+\begin{description}
+\item [{\texttt{radius}}] scaled radius of the earth: $0\,(\mathrm{center})<=r\,<=1$(surface)
+\item [{\texttt{latitude}}] Colatitude in degrees: $0^{\circ}<=\theta<=180^{\circ}$
+\item [{\texttt{longitude}}] Longitude in degrees: $-180^{\circ}<=\phi<=180^{\circ}$
+\item [{\texttt{QRFSI12\_Q}}] Fortran structure that contains the parameters,
+variables and arrays that describe the model
+\item [{\texttt{idoubling}}] value of the doubling index flag in each radial region of the mesh
+\end{description}
+Output to this routine consists of:
+
+\begin{description}
+\item [{\texttt{Qmu}}] Shear wave quality factor: $0<Q_{\mu}<5000$
+\end{description}
+A 3-D attenuation model QRFSI12 \citep{DaEkDz08}
+is provided, as well as 1-D models with a PREM and a 1DREF
+attenuation structure. By default the PREM attenuation model is taken,
+using the routine \texttt{model\_attenuation\_1D\_PREM},
+found in \texttt{model\_attenuation.f90}.
+
+To create your own three-dimensional attenuation model, you add your model
+using a routine like the ~\\
+\texttt{model\_atten3D\_QRFSI12} subroutine
+and the example routine above as a guide and replace the call in file
+\texttt{meshfem3D\_models.f90} to your own subroutine.
+
+Note that the resolution and maximum value of anelastic models are
+truncated. This speeds the construction of the standard linear solids
+during the meshing stage. To change the resolution, currently at one
+significant figure following the decimal, or the maximum value (5000),
+consult \texttt{constants.h}. In order to prevent unexpected results,
+quality factors $Q_{\mu}$ should never be equal to 0 outside of the
+inner core.
+
+
+\chapter{Post-Processing Scripts}
+
+Several post-processing scripts/programs are provided in the \texttt{UTILS/seis\_process}
+directory, most of which need to be adjusted for different
+systems, for example, the path of the executable programs. Here we
+only list the available scripts and provide a brief description, and
+you can either refer to the related sections for detailed usage or,
+in many cases, type the script/program name without arguments
+to see its usage.
+
+
+\section{Clean Local Database}
+
+After all the simulations are done, you may need to clean the local
+scratch disks for the next simulation. This is especially important
+in the case of 1- or 2-chunk kernel simulations, where very large files
+are generated for the absorbing boundaries to help with the reconstruction
+of the regular forward wavefield. A sample script is provided in \texttt{UTILS/Cluster/lsf}:
+
+\begin{lyxcode}
+cleanbase.pl~machines
+\end{lyxcode}
+
+\section{Process Data and Synthetics\label{sec:Process-data-and-syn}}
+
+In many cases, the SEM synthetics are calculated and compared to observed
+seismograms recorded at seismic stations. Since the SEM synthetics
+are accurate for a certain frequency range, both the original data
+and the synthetics need to be processed before a comparison can be
+made. For such comparisons, the following steps are recommended:
+
+\begin{enumerate}
+\item Make sure that both synthetic and observed seismograms have the correct station/event and timing information.
+\item Convolve synthetic seismograms with a source time function with the half duration specified in the \texttt{CMTSOLUTION} file, provided, as recommended, you used a zero half duration in the SEM simulations.
+\item Resample both observed and synthetic seismograms to a common sampling rate.
+\item Cut the records using the same window.
+\item Remove the trend and mean from the records and taper them.
+\item Remove the instrument response from the observed seismograms (recommended) or convolve the synthetic seismograms with the instrument response.
+\item Make sure that you apply the same filters to both observed and synthetic seismograms. Preferably, avoid filtering your records more than once.
+\item Now, you are ready to compare your synthetic and observed seismograms.
+\end{enumerate}
+
+We generally use the following scripts for processing:
+
+\subsection{\texttt{process\_data.pl}}
+
+This script cuts a given portion of the original data, filters it,
+transfers the data into a displacement record, and picks the first
+P and S arrivals. For more functionality, type `\texttt{process\_data.pl}'
+without any argument. An example of the usage of the script:
+
+\begin{lyxcode}
+{\footnotesize process\_data.pl~-m~CMTSOLUTION~-s~1.0~-l~0/4000~-i~-f~-t~40/500~-p~-x~bp~DATA/1999.330{*}.LH?.SAC}{\footnotesize \par}
+\end{lyxcode}
+
+\noindent which has resampled the SAC files to a sampling rate of 1 seconds, cut them between 0 and 4000 seconds, transfered them into displacement
+records and filtered them between 40 and 500 seconds, picked the first P and S arrivals, and added suffix `\texttt{bp}' to the file names.
+
+Note that all of the scripts in this section actually use SAC,
+saclst and/or IASP91 to do the core operations; therefore make sure
+that the SAC, saclst and IASP91 packages are installed on your
+system, and that all the environment variables are set properly before
+running these scripts.
+
+
+\subsection{\texttt{\label{sub:process_syn.pl}process\_syn.pl}}
+
+This script converts the synthetic output from the SEM code from ASCII
+to SAC format, and performs similar operations as `\texttt{process\_data.pl}'.
+An example of the usage of the script:
+
+\begin{lyxcode}
+{\footnotesize process\_syn.pl~-m~CMTSOLUTION~-h~-a~STATIONS~-s~1.0~-l~0/4000~-f~-t~40/500~-p~-x~bp~SEM/{*}.LH?.sem}{\footnotesize \par}
+\end{lyxcode}
+which will convolve the synthetics with a triangular source-time function
+from the \texttt{CMTSOLUTION} file, convert the synthetics into SAC
+format, add event and station information into the SAC headers, resample the SAC files with a sampling rate of 1 seconds, cut them between 0 and 4000 seconds, filter them between 40 and
+500 seconds with the same filter used for the observed data, pick the first P and S arrivals, and add the suffix `\texttt{bp}'
+to the file names.
+
+More options are available for this script, such as adding a time shift
+to the origin time of the synthetics, convolving the synthetics with
+a triangular source time function with a given half duration, etc.
+Type \texttt{process\_syn.pl} without any argument for detailed
+usage.
+
+
+\subsection{\texttt{rotate.pl}}
+
+To rotate the horizontal components of both the data and the synthetics
+(LHN and LHE) to the transverse and radial directions (LHT and LHR),\texttt{\small{}
+}use{\small{} }\texttt{\small rotate.pl}:
+
+\begin{lyxcode}
+rotate.pl~-l~0~-L~4000~-d~DATA/{*}.LHE.SAC.bp~
+
+rotate.pl~-l~0~-L~4000~SEM/{*}.LHE.sem.sac.bp~
+\end{lyxcode}
+where the first command performs rotation on the SAC data obtained
+through IRIS (which may have timing information written in the filename),
+while the second command rotates the processed synthetics.
+
+For synthetics, another (simpler) option is to set flag \texttt{ROTATE\_SEISMOGRAMS\_RT}
+to \texttt{.true.} in the parameter file \texttt{DATA/Par\_file}.
+
+
+\subsection{\texttt{clean\_sac\_headers\_after\_crash.sh}}
+
+Note: You need to have the \texttt{sismoutil-0.9b} package installed
+on your computer if you want to run this script on binary SAC files.
+The software is available via the ORFEUS web site \url{www.orfeus-eu.org}.
+
+In case the simulation crashes during run-time without computing and
+writing all time steps, the SAC files (if flags \texttt{OUTPUT\_SEISMOS\_SAC\_ALPHANUM}
+or \texttt{OUTPUT\_SEISMOS\_SAC\_BINARY} have been set to \texttt{.true.})
+are corrupted and cannot be used in SAC. If the simulation
+ran long enough so that the synthetic data may still be of use, you
+can run the script called \texttt{clean\_sac\_headers\_after\_crash.sh}
+(located in the \texttt{UTILS} directory) on the SAC files to correct
+the header variable NPTS to the actually written number of time steps.
+The script must be called from the \texttt{SPECFEM3D} main directory,
+and the input argument to this script is simply a list of SAC seismogram
+files.
+
+
+\section{Map Local Database}
+
+A sample program \texttt{remap\_database} is provided to map the local
+database from a set of machines to another set of machines. This is
+especially useful when you want to run mesher and solver, or different
+types of solvers separately through a scheduler (refer to Chapter~\ref{cha:Running-Scheduler}).
+
+\begin{lyxcode}
+run\_lsf.bash~-{}-gm-no-shmem~-{}-gm-copy-env~remap\_database
+
+old\_machines~150~{[}old\_jobid~new\_jobid]
+\end{lyxcode}
+where \texttt{old\_machines} is the LSF machine file used in the previous
+simulation, and \texttt{150} is the number of processors in total.
+Note that you need to supply \texttt{old\_jobid} and \texttt{new\_jobid(\%J)}
+which are the LSF job-IDs for the old and new run if your databases
+are stored in a sub-directory named after the jobid on the scratch
+disk.
+
+
+\chapter*{\label{cha:Bug-Reports-and}Bug Reports and Suggestions for Improvements}
+
+To report bugs or suggest improvements to the code, please send an
+e-mail to the CIG Computational Seismology Mailing List \url{cig-seismo at geodynamics.org}
+or Jeroen Tromp \url{jtromp-AT-princeton.edu}, and/or use our online
+bug tracking system Roundup \url{www.geodynamics.org/roundup}.
+
+
+\chapter*{\label{cha:Notes-and-Acknowledgements}Notes and Acknowledgements}
+
+In order to keep the software package thread-safe in case a multithreaded
+implementation of MPI is used, developers should not add modules or
+common blocks to the source code but rather use regular subroutine
+arguments (which can be grouped in ``derived types'' if needed for
+clarity).
+
+The Gauss-Lobatto-Legendre subroutines in \texttt{gll\_library.f90}
+are based in part on software libraries from the Massachusetts Institute
+of Technology, Department of Mechanical Engineering (Cambridge, Massachusetts, USA).
+The non-structured global numbering software was provided by Paul
+F. Fischer (Brown University, Providence, Rhode Island, USA, now at Argonne National Laboratory, USA).
+
+OpenDX \url{www.opendx.org} is open-source based on IBM Data Explorer,
+AVS \url{www.avs.com} is a trademark of Advanced Visualization Systems,
+and ParaView \url{www.paraview.com} is an open-source visualization
+platform.{\small{} }{\small \par}
+
+The main developers of the \texttt{SPECFEM3D\_GLOBE} source code are
+Dimitri Komatitsch, Jeroen Tromp, Qinya Liu and David Mich\'ea. The
+following individuals (listed in alphabetical order) have also contributed
+to the development or improvement of the source code: Ebru Bozda\u g, Min Chen, Vala Hj\"orleifsd\'ottir,
+Nicolas Le Goff, Jes\'us Labarta, Matthias Meschede, Daniel Peter, Brian Savage, Bernhard Schuberth, Anne Sieminski,
+Leif Strand, Peter van Keken and Hejun Zhu. The following individuals (listed
+in alphabetical order) contributed to this manual: Ebru Bozda\u g, Min Chen, Vala
+Hj\"orleifsd\'ottir, Sue Kientz, Dimitri Komatitsch, Qinya Liu, Alessia
+Maggi, David Mich\'ea, Daniel Peter, Brian Savage, Anne Sieminski, Carl Tape, and
+Jeroen Tromp. The manual's cover graphic was created by Santiago Lombeyda
+from Caltech's Center for Advanced Computing Research (CACR) \url{www.cacr.caltech.edu}.
+Older versions of the code were initially developed by Dimitri Komatitsch at Institut de Physique du Globe (France)
+and then by Dimitri Komatitsch and Jeroen Tromp at Harvard University (USA).
+
+Please e-mail your feedback, questions, comments, and suggestions
+to Jeroen Tromp \url{jtromp-AT-princeton.edu} or to the CIG Computational Seismology Mailing List \url{cig-seismo at geodynamics.org}.
+
+
+\chapter*{\label{cha:Copyright}Copyright}
+
+Main authors: Dimitri Komatitsch and Jeroen Tromp
+
+Princeton University, USA,
+and University of Pau / CNRS / INRIA, France
+
+$\copyright$ Princeton University / California Institute of Technology and University of Pau / CNRS
+/ INRIA, March 2010
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published
+by the Free Software Foundation (see Appendix \ref{cha:License}).
+
+{\small \bibliography{bibliography}
+}{\small \par}
+
+\appendix
+
+\chapter{\label{cha:Reference-Frame-Convention}Reference Frame Convention}
+
+The code uses the following convention for the Cartesian reference
+frame:
+
+\begin{itemize}
+\item the $x$ axis points East
+\item the $y$ axis points North
+\item the $z$ axis points up
+\end{itemize}
+Note that this convention is different from both the \citet{AkRi80}
+convention and the Harvard Centroid-Moment Tensor (CMT) convention.
+The Aki \& Richards convention is
+
+\begin{itemize}
+\item the $x$ axis points North
+\item the $y$ axis points East
+\item the $z$ axis points down
+\end{itemize}
+and the Harvard CMT convention is
+
+\begin{itemize}
+\item the $x$ axis points South
+\item the $y$ axis points East
+\item the $z$ axis points up
+\end{itemize}
+
+\chapter{\label{cha:Non-Dimensionalization-Conventions}Non-Dimensionalization
+Conventions{\small{} }}
+
+All physical parameters used by the code are non-dimensionalized according
+to the conventions summarized in Table~{\small \ref{table:conventions}.
+}%
+\begin{table}[ht]
+\noindent \begin{centering}
+{\small }\begin{tabular}{|c|c|}
+\hline
+quantity (units) & non-dimensionalized with \tabularnewline
+\hline
+distance (m) & \texttt{R\_EARTH} \tabularnewline
+time (s) & $\sqrt{\texttt{PI}\times\texttt{GRAV}\times\texttt{RHOAV}}$ \tabularnewline
+density (kg/m$^{3}$) & \texttt{RHOAV} \tabularnewline
+\hline
+\end{tabular}
+\par\end{centering}{\small \par}
+
+\caption{Non-dimensionalization employed by the code. The constants \texttt{R\_EARTH}
+(the radius of the Earth), \texttt{PI} (the number $\pi$), \texttt{GRAV}
+(the universal gravitational constant), and \texttt{RHOAV} (the Earth's
+average density) are defined in the \texttt{constants.h} file. }
+
+
+{\small \label{table:conventions} }
+\end{table}
+{\small \par}
+
+
+\chapter{Benchmarks}
+
+\citet{KoTr02a,KoTr02b} carefully benchmarked the spectral-element
+simulations of global seismic waves against normal-mode seismograms.
+Version 4.0 of \texttt{SPECFEM3D\_GLOBE} has been benchmarked again
+following the same procedure.
+
+In this appendix we present two tests: a `long-period' (periods longer
+than 17~s) simulation of a shallow event in isotropic PREM \citep{DzAn81}
+without the ocean layer, without attenuation but including the effects
+of self-gravitation (in the Cowling approximation) (Figures \ref{fig:Vanuatu-with-Vertical}
+and \ref{fig:Vanuatu-with-Transverse}), and a `short-period' (periods
+longer than 9~s) simulation of a deep event in transversely isotropic
+PREM without the ocean layer and including the effects of self-gravitation
+and attenuation (Figures \ref{fig:Bolivia-with-Vertical}, \ref{fig:Bolivia-with-Transverse}
+and \ref{fig:Bolivia-PKP}).
+
+%
+\begin{figure}[ht]
+\noindent \begin{centering}
+\includegraphics[scale=0.75]{figures/vanuatu_vertical.eps}\caption{\label{fig:Vanuatu-with-Vertical} Normal-mode (blue) and SEM (red)
+vertical displacements in isotropic PREM considering the effects of
+self-gravitation but not attenuation for 13 stations at increasing
+distance from the 1999 November 26th Vanuatu event located at 15~km
+depth. The SEM computation is accurate for periods longer than 17~s.
+The seismograms have been filtered between 50~s and 500~s. The station
+names are indicated on the left. }
+
+\par\end{centering}
+\end{figure}
+%
+\begin{figure}[ht]
+\noindent \begin{centering}
+\includegraphics[scale=0.75]{figures/vanuatu_trans.eps}\caption{\label{fig:Vanuatu-with-Transverse}Same as in Figure \ref{fig:Vanuatu-with-Vertical}
+for the transverse displacements.}
+
+\par\end{centering}
+\end{figure}
+%
+\begin{figure}[ht]
+\noindent \begin{centering}
+\includegraphics[scale=0.75]{figures/bolivia_vertical.eps}\caption{\label{fig:Bolivia-with-Vertical}Normal-mode (blue) and SEM (red)
+vertical displacements in transversely isotropic PREM considering
+the effects of self-gravitation and attenuation for 12 stations at
+increasing distance from the 1994 June 9th Bolivia event located at
+647~km depth. The SEM computation is accurate for periods longer
+than 9~s. The seismograms have been filtered between 10~s and 500~s.
+The station names are indicated on the left.}
+
+\par\end{centering}
+\end{figure}
+%
+\begin{figure}[ht]
+\noindent \begin{centering}
+\includegraphics[scale=0.75]{figures/bolivia_trans.eps}\caption{\label{fig:Bolivia-with-Transverse}Same as in Figure \ref{fig:Bolivia-with-Vertical}
+for the transverse displacements.}
+
+\par\end{centering}
+\end{figure}
+
+
+%
+\begin{figure}[ht]
+\noindent \begin{centering}
+\includegraphics[scale=0.75]{figures/PKPdf_all_15s500s.eps}\caption{\label{fig:Bolivia-PKP}Seismograms recorded between 130 degrees and
+230 degrees, showing in particular the good agreement for core phases
+such as PKP. This figure is similar to Figure 24 of \citet{KoTr02a}.
+The results have been filtered between 15~s and 500~s.}
+
+\par\end{centering}
+\end{figure}
+
+
+The normal-mode synthetics are calculated with the code \texttt{QmXD}
+using mode catalogs with a shortest period of 8~s generated by the
+code \texttt{OBANI}. No free-air, tilt, or gravitational potential
+corrections were applied \citep{DaTr98}. We also turned off the effect
+of the oceans in \texttt{QmXD}.
+
+The normal-mode and SEM displacement seismograms are first calculated
+for a step source-time function, i.e., setting the parameter \texttt{half}
+\texttt{duration} in the \texttt{CMTSOLUTION} file to zero for the
+SEM simulations. Both sets of seismograms are subsequently convolved
+with a triangular source-time function using the processing script
+\texttt{UTILS/seis\_}~\\
+\texttt{process/process\_syn.pl}. They are also band-pass filtered
+and the horizontal components are rotated to the radial and transverse
+directions (with the script \texttt{UTILS/seis\_process/rotate.pl}).
+
+The match between the normal-mode and SEM seismograms is quite remarkable
+for the experiment with attenuation, considering the very different
+implementations of attenuation in the two computations (e.g., frequency
+domain versus time domain, constant Q versus absorption bands).
+
+Further tests can be found in the \texttt{EXAMPLES} directory. It
+contains the normal-mode and SEM seismograms, and the parameters (\texttt{STATIONS},
+\texttt{CMTSOLUTION} and \texttt{Par\_file}) for the SEM simulations.
+
+\chapter{\label{cha:SAC-headers}SAC Headers}
+
+Information about the simulation (i.e., event/station information, sampling rate, etc.) is written in the header of the seismograms in SAC format. The list of values and related explanation are given in Figure \ref{fig:SAC-headers}. Please check the SAC webpages \url{www.iris.edu/software/sac/} for further information. Please note that the reference time KZTIME is the centroid time ($t_\text{CMT}=t_\text{PDE}+\texttt{time shift}$) which corresponds to zero time in the synthetics. For kinematic rupture simulations, KZTIME is equal to the CMT time of the source having the minimum time-shift in the \texttt{CMTSOLUTION} file, and coordinates, depth and half-duration of the event are not provided in the headers.
+
+\begin{figure}[ht]
+\noindent \begin{centering}
+\includegraphics[scale=0.8]{figures/headers_sem_explained.pdf}\caption{\label{fig:SAC-headers}List of SAC headers and their explanations for a sample seismogram.}
+
+\par\end{centering}
+\end{figure}
+
+\chapter{\label{cha:License}License}
+
+\textbf{GNU GENERAL PUBLIC LICENSE Version 2, June 1991. Copyright
+(C) 1989, 1991 Free Software Foundation, Inc. 59 Temple Place, Suite
+330, Boston, MA 02111-1307 USA} \\
+Everyone is permitted to copy and distribute verbatim copies of this
+license document, but changing it is not allowed.
+
+
+\section*{Preamble}
+
+The licenses for most software are designed to take away your freedom
+to share and change it. By contrast, the GNU General Public License
+is intended to guarantee your freedom to share and change free software
+-- to make sure the software is free for all its users. This General
+Public License applies to most of the Free Software Foundation's software
+and to any other program whose authors commit to using it. (Some other
+Free Software Foundation software is covered by the GNU Library General
+Public License instead.) You can apply it to your programs, too.
+
+When we speak of free software, we are referring to freedom, not price.
+Our General Public Licenses are designed to make sure that you have
+the freedom to distribute copies of free software (and charge for
+this service if you wish), that you receive source code or can get
+it if you want it, that you can change the software or use pieces
+of it in new free programs; and that you know you can do these things.
+
+To protect your rights, we need to make restrictions that forbid anyone
+to deny you these rights or to ask you to surrender the rights. These
+restrictions translate to certain responsibilities for you if you
+distribute copies of the software, or if you modify it.
+
+For example, if you distribute copies of such a program, whether gratis
+or for a fee, you must give the recipients all the rights that you
+have. You must make sure that they, too, receive or can get the source
+code. And you must show them these terms so they know their rights.
+
+We protect your rights with two steps:
+
+\begin{enumerate}
+\item Copyright the software, and
+\item Offer you this license which gives you legal permission to copy, distribute
+and/or modify the software.
+\end{enumerate}
+Also, for each author's protection and ours, we want to make certain
+that everyone understands that there is no warranty for this free
+software. If the software is modified by someone else and passed on,
+we want its recipients to know that what they have is not the original,
+so that any problems introduced by others will not reflect on the
+original authors' reputations.
+
+Finally, any free program is threatened constantly by software patents.
+We wish to avoid the danger that redistributors of a free program
+will individually obtain patent licenses, in effect making the program
+proprietary. To prevent this, we have made it clear that any patent
+must be licensed for everyone's free use or not licensed at all.
+
+The precise terms and conditions for copying, distribution and modification
+follow.
+
+
+\section*{GNU GENERAL PUBLIC LICENSE TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION
+AND MODIFICATION }
+
+\begin{itemize}
+
+\item[0.]This License applies to any program or other work which
+contains a notice placed by the copyright holder saying it may be
+distributed under the terms of this General Public License. The ``Program''
+below refers to any such program or work, and a ``work based on the
+Program'' means either the Program or any derivative work under copyright
+law: that is to say, a work containing the Program or a portion of
+it, either verbatim or with modifications and/or translated into another
+language. (Hereinafter, translation is included without limitation
+in the term ``modification.'') Each licensee is addressed as ``you.''\\
+\\
+Activities other than copying, distribution and modification are not
+covered by this License; they are outside its scope. The act of running
+the Program is not restricted, and the output from the Program is
+covered only if its contents constitute a work based on the Program
+(independent of having been made by running the Program). Whether
+that is true depends on what the Program does.
+
+\end{itemize}
+
+\begin{enumerate}
+\item You may copy and distribute verbatim copies of the Program's source
+code as you receive it, in any medium, provided that you conspicuously
+and appropriately publish on each copy an appropriate copyright notice
+and disclaimer of warranty; keep intact all the notices that refer
+to this License and to the absence of any warranty; and give any other
+recipients of the Program a copy of this License along with the Program.
+
+
+You may charge a fee for the physical act of transferring a copy,
+and you may at your option offer warranty protection in exchange for
+a fee.
+
+\item You may modify your copy or copies of the Program or any portion of
+it, thus forming a work based on the Program, and copy and distribute
+such modifications or work under the terms of Section 1 above, provided
+that you also meet all of these conditions:
+
+\begin{enumerate}
+\item You must cause the modified files to carry prominent notices stating
+that you changed the files and the date of any change.
+\item You must cause any work that you distribute or publish, that in whole
+or in part contains or is derived from the Program or any part thereof,
+to be licensed as a whole at no charge to all third parties under
+the terms of this License.
+\item If the modified program normally reads commands interactively when
+run, you must cause it, when started running for such interactive
+use in the most ordinary way, to print or display an announcement
+including an appropriate copyright notice and a notice that there
+is no warranty (or else, saying that you provide a warranty) and that
+users may redistribute the program under these conditions, and telling
+the user how to view a copy of this License. (Exception: if the Program
+itself is interactive but does not normally print such an announcement,
+your work based on the Program is not required to print an announcement.)
+\end{enumerate}
+These requirements apply to the modified work as a whole. If identifiable
+sections of that work are not derived from the Program, and can be
+reasonably considered independent and separate works in themselves,
+then this License, and its terms, do not apply to those sections when
+you distribute them as separate works. But when you distribute the
+same sections as part of a whole which is a work based on the Program,
+the distribution of the whole must be on the terms of this License,
+whose permissions for other licensees extend to the entire whole,
+and thus to each and every part regardless of who wrote it.
+
+Thus, it is not the intent of this section to claim rights or contest
+your rights to work written entirely by you; rather, the intent is
+to exercise the right to control the distribution of derivative or
+collective works based on the Program.
+
+In addition, mere aggregation of another work not based on the Program
+with the Program (or with a work based on the Program) on a volume
+of a storage or distribution medium does not bring the other work
+under the scope of this License.
+
+\item You may copy and distribute the Program (or a work based on it, under
+Section 2) in object code or executable form under the terms of Sections
+1 and 2 above provided that you also do one of the following:
+
+\begin{enumerate}
+\item Accompany it with the complete corresponding machine-readable source
+code, which must be distributed under the terms of Sections 1 and
+2 above on a medium customarily used for software interchange; or,
+\item Accompany it with a written offer, valid for at least three years,
+to give any third party, for a charge no more than your cost of physically
+performing source distribution, a complete machine-readable copy of
+the corresponding source code, to be distributed under the terms of
+Sections 1 and 2 above on a medium customarily used for software interchange;
+or,
+\item Accompany it with the information you received as to the offer to
+distribute corresponding source code. (This alternative is allowed
+only for noncommercial distribution and only if you received the program
+in object code or executable form with such an offer, in accord with
+Subsection b above.)
+\end{enumerate}
+The source code for a work means the preferred form of the work for
+making modifications to it. For an executable work, complete source
+code means all the source code for all modules it contains, plus any
+associated interface definition files, plus the scripts used to control
+compilation and installation of the executable. However, as a special
+exception, the source code distributed need not include anything that
+is normally distributed (in either source or binary form) with the
+major components (compiler, kernel, and so on) of the operating system
+on which the executable runs, unless that component itself accompanies
+the executable.
+
+If distribution of executable or object code is made by offering access
+to copy from a designated place, then offering equivalent access to
+copy the source code from the same place counts as distribution of
+the source code, even though third parties are not compelled to copy
+the source along with the object code.
+
+\item You may not copy, modify, sublicense, or distribute the Program except
+as expressly provided under this License. Any attempt otherwise to
+copy, modify, sublicense or distribute the Program is void, and will
+automatically terminate your rights under this License. However, parties
+who have received copies, or rights, from you under this License will
+not have their licenses terminated so long as such parties remain
+in full compliance.
+\item You are not required to accept this License, since you have not signed
+it. However, nothing else grants you permission to modify or distribute
+the Program or its derivative works. These actions are prohibited
+by law if you do not accept this License. Therefore, by modifying
+or distributing the Program (or any work based on the Program), you
+indicate your acceptance of this License to do so, and all its terms
+and conditions for copying, distributing or modifying the Program
+or works based on it.
+\item Each time you redistribute the Program (or any work based on the Program),
+the recipient automatically receives a license from the original licensor
+to copy, distribute or modify the Program subject to these terms and
+conditions. You may not impose any further restrictions on the recipients'
+exercise of the rights granted herein. You are not responsible for
+enforcing compliance by third parties to this License.
+\item If, as a consequence of a court judgment or allegation of patent infringement
+or for any other reason (not limited to patent issues), conditions
+are imposed on you (whether by court order, agreement or otherwise)
+that contradict the conditions of this License, they do not excuse
+you from the conditions of this License. If you cannot distribute
+so as to satisfy simultaneously your obligations under this License
+and any other pertinent obligations, then as a consequence you may
+not distribute the Program at all. For example, if a patent license
+would not permit royalty-free redistribution of the Program by all
+those who receive copies directly or indirectly through you, then
+the only way you could satisfy both it and this License would be to
+refrain entirely from distribution of the Program.
+
+
+If any portion of this section is held invalid or unenforceable under
+any particular circumstance, the balance of the section is intended
+to apply and the section as a whole is intended to apply in other
+circumstances.
+
+It is not the purpose of this section to induce you to infringe any
+patents or other property right claims or to contest validity of any
+such claims; this section has the sole purpose of protecting the integrity
+of the free software distribution system, which is implemented by
+public license practices. Many people have made generous contributions
+to the wide range of software distributed through that system in reliance
+on consistent application of that system; it is up to the author/donor
+to decide if he or she is willing to distribute software through any
+other system and a licensee cannot impose that choice.
+
+This section is intended to make thoroughly clear what is believed
+to be a consequence of the rest of this License.
+
+\item If the distribution and/or use of the Program is restricted in certain
+countries either by patents or by copyrighted interfaces, the original
+copyright holder who places the Program under this License may add
+an explicit geographical distribution limitation excluding those countries,
+so that distribution is permitted only in or among countries not thus
+excluded. In such case, this License incorporates the limitation as
+if written in the body of this License.
+\item The Free Software Foundation may publish revised and/or new versions
+of the General Public License from time to time. Such new versions
+will be similar in spirit to the present version, but may differ in
+detail to address new problems or concerns.
+
+
+Each version is given a distinguishing version number. If the Program
+specifies a version number of this License which applies to it and
+``any later version,'' you have the option of following the terms
+and conditions either of that version or of any later version published
+by the Free Software Foundation. If the Program does not specify a
+version number of this License, you may choose any version ever published
+by the Free Software Foundation.
+
+\item If you wish to incorporate parts of the Program into other free programs
+whose distribution conditions are different, write to the author to
+ask for permission. For software which is copyrighted by the Free
+Software Foundation, write to the Free Software Foundation; we sometimes
+make exceptions for this. Our decision will be guided by the two goals
+of preserving the free status of all derivatives of our free software
+and of promoting the sharing and reuse of software generally.
+\end{enumerate}
+
+\subsection*{NO WARRANTY }
+
+\begin{itemize}
+
+\item[11.]BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS
+NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE
+LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS
+AND/OR OTHER PARTIES PROVIDE THE PROGRAM ``AS IS'' WITHOUT WARRANTY
+OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED
+TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE
+PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME
+THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
+
+\item[12.]IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED
+TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY
+MODIFY AND/OR REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE
+TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR
+CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE
+PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED
+INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE
+OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER
+OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
+
+\end{itemize}
+
+
+\section*{END OF TERMS AND CONDITIONS }
+
+
+\subsection*{How to Apply These Terms to Your New Programs}
+
+If you develop a new program, and you want it to be of the greatest
+possible use to the public, the best way to achieve this is to make
+it free software which everyone can redistribute and change under
+these terms.
+
+To do so, attach the following notices to the program. It is safest
+to attach them to the start of each source file to most effectively
+convey the exclusion of warranty; and each file should have at least
+the ``copyright'' line and a pointer to where the full notice is found.
+For example:
+
+\begin{quote}
+One line to give the program's name and a brief idea of what it does.
+Copyright {\footnotesize $\copyright$ (}year) (name of author)
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published
+by the Free Software Foundation; either version 2 of the License,
+or (at your option) any later version.
+
+This program is distributed in the hope that it will be useful, but
+WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
+for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software Foundation,
+Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+\end{quote}
+Also add information on how to contact you by electronic and paper
+mail.
+
+If the program is interactive, make it output a short notice like
+this when it starts in an interactive mode:
+
+\begin{quote}
+Gnomovision version 69, Copyright $\copyright$ year name of author Gnomovision
+comes with ABSOLUTELY NO WARRANTY; for details type `show w'. This
+is free software, and you are welcome to redistribute it under certain
+conditions; type `show c' for details.
+\end{quote}
+The hypothetical commands `show w' and `show c' should show the appropriate
+parts of the General Public License. Of course, the commands you use
+may be called something other than `show w' and `show c'; they could
+even be mouse-clicks or menu items -- whatever suits your program.
+
+You should also get your employer (if you work as a programmer) or
+your school, if any, to sign a ``copyright disclaimer'' for the program,
+if necessary. Here is a sample; alter the names:
+
+\begin{quote}
+Yoyodyne, Inc., hereby disclaims all copyright interest in the program
+`Gnomovision' (which makes passes at compilers) written by James Hacker.
+
+(signature of Ty Coon)\\
+1 April 1989 \\
+Ty Coon, President of Vice
+\end{quote}
+This General Public License does not permit incorporating your program
+into proprietary programs. If your program is a subroutine library,
+you may consider it more useful to permit linking proprietary applications
+with the library. If this is what you want to do, use the GNU Library
+General Public License instead of this License.
+\end{document}
More information about the CIG-COMMITS
mailing list