[cig-commits] r17238 - seismo/3D/SPECFEM3D/trunk/USER_MANUAL

pieyre at geodynamics.org pieyre at geodynamics.org
Wed Oct 6 09:21:54 PDT 2010 

Author: pieyre
Date: 2010-10-06 09:21:54 -0700 (Wed, 06 Oct 2010)
New Revision: 17238

Modified:
   seismo/3D/SPECFEM3D/trunk/USER_MANUAL/manual_SPECFEM3D.pdf
   seismo/3D/SPECFEM3D/trunk/USER_MANUAL/manual_SPECFEM3D.tex
Log:
modifided the manual, added the chapters 'Mesh Generation' and 'xgenerate_databases'


Modified: seismo/3D/SPECFEM3D/trunk/USER_MANUAL/manual_SPECFEM3D.pdf
===================================================================
(Binary files differ)

Modified: seismo/3D/SPECFEM3D/trunk/USER_MANUAL/manual_SPECFEM3D.tex
===================================================================
--- seismo/3D/SPECFEM3D/trunk/USER_MANUAL/manual_SPECFEM3D.tex	2010-10-05 13:47:02 UTC (rev 17237)
+++ seismo/3D/SPECFEM3D/trunk/USER_MANUAL/manual_SPECFEM3D.tex	2010-10-06 16:21:54 UTC (rev 17238)
@@ -168,9 +168,9 @@
 directory in which you want to install the package, type
 
 \begin{lyxcode}
-{\small tar~-zxvf~SPECFEM3D\_V1.4.4.tar.gz}{\small \par}
+{\small tar~-zxvf~SPECFEM3D\_V2.0.0.tar.gz}{\small \par}
 \end{lyxcode}
-The directory \texttt{\small SPECFEM3D\_V1.4.4} will then contain
+The directory \texttt{\small SPECFEM3D\_V2.0.0} will then contain
 the source code. To configure the software for your system, run the
 \texttt{configure} shell script. This script will attempt to guess
 the appropriate configuration values for your system. However, at
@@ -180,6 +180,13 @@
 \begin{lyxcode}
 ./configure~FC=ifort~MPIFC=mpif90
 \end{lyxcode}
+
+
+The SPECFEM3D software package relies on the Scotch library to partition the meshes created with CUBIT. Because it is not standard, the path of this library needs to be set with the option \texttt{-{}-with-scotch-lib}. Just as an example:
+\begin{lyxcode}
+./configure~FC=ifort~MPIFC=mpif90~-{}-with-scotch-lib=/opt/scotch/gcc64/lib
+\end{lyxcode} 
+
 To compile a serial version of the code for small meshes that fit
 on one compute node and can therefore be run serially, run \texttt{configure}
 with the \texttt{-{}-without-mpi} option to suppress all calls to
@@ -253,7 +260,7 @@
 
 Finally, before compiling make sure that the subdirectories \texttt{obj}
 and \texttt{OUTPUT\_FILES} exist within the directory with the source
-code (\texttt{SPECFEM3D\_V1.4.4}). The \texttt{go\_mesher} script
+code (\texttt{SPECFEM3D\_V2.0.0}). The \texttt{go\_mesher} script
 discussed in Chapter~\ref{cha:Running-the-Mesher} automatically
 takes care of creating the \texttt{OUTPUT\_FILES} directory.
 
@@ -265,8 +272,30 @@
 message.
 
 
-\chapter{\label{cha:Running-the-Mesher}Running the Mesher \texttt{xmeshfem3D}}
+\chapter{\label{cha:Mesh-Generation}Mesh Generation}
 
+\section{\label{cha:Running-the-Mesher}Meshing with \texttt{CUBIT}}
+\subsection{Exporting the Mesh}
+\subsection{Partitioning the Mesh with \texttt{xdecompose\_mesh\_SCOTCH}}
+
+The SPECFEM3D software package performs large scale simulations in a parallel 'Single Process Multiple Data' way. 
+The spectral element mesh created with CUBIT needs to be distributed on the processors. This partitioning is executed once and for all prior to
+the execution of the solver so it is referred to as a static mapping.    
+
+An efficient partitioning is important because it leverages the overall running time of the application.
+It amounts to balance the number of elements in each slice while minimizing the communication coasts 
+resulting from the placement of adjacent elements to different processors. We use the Scotch library 
+which provides efficent static mapping, graph and mesh partitioning routines.\\
+ 
+Prior to compiling \texttt{decompose\_mesh\_SCOTCH}, make sure you have correctly specified the path 
+of the Scotch library with the option \texttt{-{}-with-scotch-lib} of configure. Then you are ready to compile, 
+in the directory \texttt{decompose\_mesh\_SCOTCH} type '\texttt{make decompose\_mesh\_SCOTCH}'.
+If all paths and flags have been set correctly, the executable \texttt{xdecompose\_mesh\_SCOTCH} should be produced.
+
+
+
+\section{\label{cha:Running-the-Mesher}Meshing with \texttt{xmeshfem3D}}
+
 \noindent \begin{flushleft}
 %
 \begin{figure}[t]
@@ -286,7 +315,7 @@
 have been set correctly, the mesher should now compile and produce
 the executable \texttt{xmeshfem3D}.
 
-Input for the mesher is provided through the parameter
+Input for the mesh generation program is provided through the parameter
 file \texttt{Par\_file}, which resides in the subdirectory \texttt{meshfem3D/DATA}.
 Before running the mesher, a number of parameters need to be set in
 the \texttt{Par\_file}. This requires a basic understanding of how
@@ -573,7 +602,6 @@
 
 %%\section{\label{sec:quality}Checking the Mesh Quality (Optional)}
 
-\newpage{}
 The quality of the mesh may be inspected more precisely based upon the serial code
 \texttt{check\_mesh\_quality\_AVS\_DX.f90}. Type
 
@@ -599,14 +627,129 @@
 Running this code is optional because no information needed by the
 solver is generated.
 
+\chapter{\label{cha:Running-xgenerate-databases}Running \texttt{xgenerate\_databases}}
 
+Either you previously used  \texttt{xmeshfem3D} or \texttt{xdecompose\_mesh\_SCOTCH}, the next step in the workflow is to compile \texttt{xgenerate\_databases}. This program is going to create all the missing information needed by the SEM solver. In the main directory type '\texttt{make generate\_databases}'. Input for the program is provided through the main parameter file \texttt{Par\_file}, which resides in the subdirectory \texttt{DATA}. Before running \texttt{xgenerate\_databases}, a number of parameters need to be set in the \texttt{Par\_file}:
+
+\begin{description}
+\item [{\texttt{SIMULATION\_TYPE}}] is set to 1 for forward simulations,
+2 for adjoint simulations (see Section \ref{sec:Adjoint-simulation-finite})
+and 3 for kernel simulations (see Section \ref{sec:Finite-Frequency-Kernels}).
+\item [{\texttt{SAVE\_FORWARD}}] is only set to \texttt{.true.} for a forward
+simulation with the last frame of the simulation saved, as part of
+the finite-frequency kernel calculations (see Section \ref{sec:Finite-Frequency-Kernels}).
+For a regular forward simulation, leave \texttt{SIMULATION\_TYPE}
+and \texttt{SAVE\_FORWARD} at their default values.
+\item [{\texttt{\noun{UTM\_PROJECTION\_ZONE}}}] UTM projection zone in
+which your model resides, only valid when \texttt{SUPPRESS\_UTM\_}~\\
+\texttt{PROJECTION} is \texttt{.false.}.
+\item [{\texttt{SUPPRESS\_UTM\_PROJECTION}}] set to be \texttt{.false.}
+when your model range is specified in the geographical coordinates,
+and needs to be \texttt{.true.} when your model is specified in a
+cartesian coordinates. \noun{UTM projection zone in which your simulation
+region resides.}
+\item [{\texttt{NPROC}}] The number of MPI processors, each one is assigned one slice of the whole mesh. 
+\item [{\texttt{NSTEP}}] The number of time steps of the simulation. This controls the
+length of the numerical simulation, i.e., twice the number of time steps
+requires twice as much CPU time. This feature is not used at the time
+of meshing but is required for the solver, i.e., you may change this
+parameter after running  \texttt{xgenerate\_databases}.
+\item [{\texttt{DT}}] The length of each time step in seconds. 
+If you used xmeshfem3D to generate your mesh, you should set the value suggested in \texttt{meshfem3D/OUTPUT\_FILES/output\_mesher.txt}.
+
+\item [{\texttt{OCEANS}}] Set to \texttt{.true.} if the effect of the oceans
+on seismic wave propagation should be incorporated based upon the
+approximate treatment discussed in \citet{KoTr02b}. This feature
+is inexpensive from a numerical perspective, both in terms of memory
+requirements and CPU time. This approximation is accurate at periods
+of roughly 20~s and longer. At shorter periods the effect of water
+phases/reverberations is not taken into account, even when the flag
+is on.
+\item [{\texttt{TOPOGRAPHY}}] Set to \texttt{.true.} if topography and
+bathymetry should be incorporated based upon model ETOPO5 \citep{Etopo5}.
+This feature adds no cost to the simulation.
+\item [{\texttt{ATTENUATION}}] Set to \texttt{.true.} if attenuation should
+be incorporated. Turning this feature on increases the memory requirements
+significantly (roughly by a factor of~1.5), and is numerically fairly
+expensive. See \citet{KoTr99,KoTr02a} for a discussion on the implementation
+of attenuation based upon standard linear solids.
+\item [{\texttt{USE\_OLSEN\_ATTENUATION}}] Set to \texttt{.true.} if you
+want to use the attenuation model that scaled from the velocity model
+using Olsen's empirical relation (reference).
+\item [{\texttt{ABSORBING\_CONDITIONS}}] Set to \texttt{.true.} to turn
+on Clayton-Enquist absorbing boundary conditions (see \citet{KoTr99}).
+\item [{\texttt{RECORD\_LENGTH\_IN\_MINUTES}}] Choose the desired record
+length of the synthetic seismograms (in minutes). This controls the
+length of the numerical simulation, i.e., twice the record length
+requires twice as much CPU time. This feature is not used at the time
+of meshing but is required for the solver, i.e., you may change this
+parameter after running xgenerate\_databases.
+\item [{\texttt{MOVIE\_SURFACE}}] Set to \texttt{.false.}, unless you want
+to create a movie of seismic wave propagation on the Earth's surface.
+Turning this option on generates large output files. See Section \ref{sec:Movies}
+for a discussion on the generation of movies. This feature is only relevant for the solver.
+\item [{\texttt{MOVIE\_VOLUME}}] Set to \texttt{.false.}, unless you want
+to create a movie of seismic wave propagation in the Earth's interior.
+Turning this option on generates huge output files. See Section \ref{sec:Movies}
+for a discussion on the generation of movies. This feature is only relevant for the solver.
+\item [{\texttt{NTSTEP\_BETWEEN\_FRAMES}}] Determines the number of timesteps
+between movie frames. Typically you want to save a snapshot every
+100 timesteps. The smaller you make this number the more output will
+be generated! See Section \ref{sec:Movies} for a discussion on the
+generation of movies. This feature is only relevant for the solver.
+\item [{\texttt{CREATE\_SHAKEMAP}}] Set this flag to \texttt{.true.} to
+create a ShakeMap\textregistered, i.e., a peak ground velocity map
+of the maximum absolute value of the two horizontal components of the velocity vector.
+\item [{\texttt{SAVE\_DISPLACEMENT}}] Set this flag to \texttt{.true.}
+if you want to save the displacement instead of velocity for the movie
+frames.
+\item [{\texttt{USE\_HIGHRES\_FOR\_MOVIES}}] Set this flag to \texttt{.true.}
+if you want to save the values at all the NGLL grid points for the
+movie frames.
+\item [{\texttt{SAVE\_MESH\_FILES}}] Set this flag to \texttt{.true.} to
+save AVS \url{www.avs.com}, OpenDX \url{www.opendx.org}, or ParaView \url{www.paraview.org}
+mesh files for subsequent viewing. Turning the flag on generates large
+(distributed) files in the \texttt{LOCAL\_PATH} directory. See Section~\ref{sec:Mesh-graphics}
+for a discussion of mesh viewing features.
+\item [{\texttt{LOCAL\_PATH}}] Directory in which the databases and the synthetics 
+will be written. Generally one uses a directory on the
+local disk of the compute nodes, although on some machines these databases
+are written on a parallel (global) file system (see also the earlier
+discussion of the \texttt{LOCAL\_PATH\_IS\_ALSO\_GLOBAL} flag in Chapter~\ref{cha:Getting-Started}).
+xgenerate\_databases generates the necessary databases in parallel, one set
+for each of the $\nprocxi\times\nproceta$ slices that constitutes
+the mesh (see Figure~\ref{fig:For-parallel-computing}). After the
+mesher finishes, you can log in to one of the compute nodes and view
+the contents of the \texttt{LOCAL\_PATH} directory to see the (many)
+files generated by xgenerate\_databases.
+\item [{\texttt{NTSTEP\_BETWEEN\_OUTPUT\_INFO}}] This parameter specifies
+the interval at which basic information about a run is written to
+the file system (\texttt{timestamp{*}} files in the \texttt{OUTPUT\_FILES}
+directory). If you have access to a fast machine, set \texttt{NTSTEP\_BETWEEN\_OUTPUT\_INFO}
+to a relatively high value (e.g., at least 100, or even 1000 or more)
+to avoid writing output text files too often. This feature is not
+used at the time of meshing. One can set this parameter to a larger
+value than the number of time steps to avoid writing output during
+the run.
+\item [{\texttt{NTSTEP\_BETWEEN\_OUTPUT\_SEISMOS}}] This parameter specifies
+the interval at which synthetic seismograms are written in the \texttt{LOCAL\_PATH}
+directory. If a run crashes, you may still find usable (but shorter
+than requested) seismograms in this directory. On a fast machine set
+\texttt{NTSTEP\_BETWEEN\_OUTPUT\_SEISMOS} to a relatively high value
+to avoid writing to the seismograms too often. This feature is only relevant for the solver.
+\item [{\texttt{PRINT\_SOURCE\_TIME\_FUNCTION}}] Turn this flag on to print
+information about the source time function in the file \texttt{OUTPUT\_FILES/plot\_source\_time\_function.txt}.
+This feature is only relevant for the solver.
+\end{description}
+
+
 \chapter{\label{cha:Running-the-Solver}Running the Solver \texttt{xspecfem3D}}
 
-Now that you have successfully run the mesher, you are ready to compile
+Now that you have successfully generated the databases, you are ready to compile
 the solver. For reasons of speed, the solver uses static memory allocation.
 Therefore it needs to be recompiled (type `\texttt{make clean}' and
-`\texttt{make specfem3D}') every time one reruns the mesher. To compile
-the solver one needs a file generated by the mesher in the directory
+`\texttt{make specfem3D}') every time one reruns \texttt{xgenerate\_databases}. To compile
+the solver one needs a file created by \texttt{xgenerate\_databases} in the directory
 \texttt{OUTPUT\_FILES} called \texttt{values\_from\_mesher.h}, which
 contains parameters describing the static size of the arrays as well
 as the setting of certain flags.
@@ -616,12 +759,12 @@
 the earthquake source parameter file \texttt{CMTSOLUTION}, and the
 stations file \texttt{STATIONS}. Most parameters in the \texttt{Par\_file}
 should be set prior to running the mesher. Only the following parameters
-may be changed after running the mesher:
+may be changed after running \texttt{xgenerate\_databases}:
 
 \begin{itemize}
 \item the simulation type control parameters: \texttt{SIMULATION\_TYPE}
 and \texttt{SAVE\_FORWARD}
-\item the record length \texttt{RECORD\_LENGTH\_IN\_MINUTES}
+\item the time step parameters \texttt{NSTEP} and \texttt{DT}
 \item the movie control parameters \texttt{MOVIE\_SURFACE}, \texttt{MOVIE\_VOLUME},
 and \texttt{NTSTEPS\_BETWEEN\_FRAMES}
 \item the ShakeMap\textregistered option \texttt{CREATE\_SHAKEMAP}

More information about the CIG-COMMITS mailing list