[cig-commits] r5169 - mc/3D/CitcomS/trunk/doc/manual

[luis at geodynamics.org]

Author: luis
Date: 2006-11-01 12:47:53 -0800 (Wed, 01 Nov 2006)
New Revision: 5169

Modified:
   mc/3D/CitcomS/trunk/doc/manual/citcoms.lyx
Log:
Re-typed most of the changes from my copy of the citcoms.lyx file


Modified: mc/3D/CitcomS/trunk/doc/manual/citcoms.lyx
===================================================================
--- mc/3D/CitcomS/trunk/doc/manual/citcoms.lyx	2006-11-01 17:48:43 UTC (rev 5168)
+++ mc/3D/CitcomS/trunk/doc/manual/citcoms.lyx	2006-11-01 20:47:53 UTC (rev 5169)
@@ -1,4 +1,4 @@
-#LyX 1.4.3 created this file. For more info see http://www.lyx.org/
+#LyX 1.4.1 created this file. For more info see http://www.lyx.org/
 \lyxformat 245
 \begin_document
 \begin_header
@@ -2007,8 +2007,19 @@
  If an existing installation of the PHDF5 library is not available on your
  cluster, you can compile it from source by following the instructions in
  the file release docs/INSTALL parallel under the HDF5 source tree.
- Under Debian Linux, you may simply install the libhdf5-mpich, libhdf5-mpich-dev
- and hdf5-tools packages.
+ Under Debian Linux, you may simply install the 
+\family typewriter
+libhdf5-mpich
+\family default
+, 
+\family typewriter
+libhdf5-mpich-dev
+\family default
+ and 
+\family typewriter
+hdf5-tools
+\family default
+ packages.
 \end_layout
 
 \begin_layout Standard
@@ -2023,7 +2034,7 @@
 \end_layout
 
 \begin_layout LyX-Code
-$ ./configure --with-hdf5 --prefix=$HOME/cig [...]
+$ ./configure --with-hdf5 --prefix=$HOME/cig
 \end_layout
 
 \begin_layout Section
@@ -2042,7 +2053,7 @@
 \begin_layout Standard
 NumPy is an extension to Python which adds support for multi-dimensional
  arrays for use in scientific computing.
- You may download NumPy at the 
+ You may download NumPy from the 
 \begin_inset LatexCommand \htmlurl[NumPy home page]{numpy.scipy.org}
 
 \end_inset
@@ -2061,8 +2072,16 @@
 \end_layout
 
 \begin_layout Standard
-Alternatively, under Debian Linux you can install the python-numpy package.
- On Gentoo Linux, NumPy is available in the dev-python/numpy ebuild.
+Alternatively, under Debian Linux you can install the 
+\family typewriter
+python-numpy
+\family default
+ package.
+ On Gentoo Linux, NumPy is available in the 
+\family typewriter
+dev-python/numpy
+\family default
+ ebuild.
 \end_layout
 
 \begin_layout Subsection
@@ -2094,8 +2113,16 @@
 \end_layout
 
 \begin_layout Standard
-To install on Debian Linux, you may use the python-tables package instead.
- On Gentoo Linux, it is available in the dev-python/pytables ebuild.
+To install on Debian Linux, you may use the 
+\family typewriter
+python-tables
+\family default
+ package instead.
+ On Gentoo Linux, it is available in the 
+\family typewriter
+dev-python/pytables
+\family default
+ ebuild.
 \end_layout
 
 \begin_layout Subsection
@@ -2103,9 +2130,9 @@
 \end_layout
 
 \begin_layout Standard
-HDFView is a visual tool for browsing and editing HDF5 files.
- You may download it from 
-\begin_inset LatexCommand \htmlurl[HDF Java Products]{hdf.ncsa.uiuc.edu/hdf-java-html/hdfview}
+HDFView is a visual tool written in Java for browsing and editing HDF5 files.
+ You may download it from the 
+\begin_inset LatexCommand \htmlurl[HDFView home page]{hdf.ncsa.uiuc.edu/hdf-java-html/hdfview}
 
 \end_inset
 
@@ -3543,6 +3570,971 @@
 \end_layout
 
 \begin_layout Chapter
+Working with HDF5
+\end_layout
+
+\begin_layout Section
+Introduction
+\end_layout
+
+\begin_layout Standard
+A typical run of CitComS can create thousands, if not millions of ASCII
+ output files.
+ This situation is inefficient since it requires an extra post-processing
+ step for assembling the results from each processor (see Chapter 
+\begin_inset LatexCommand \ref{cha:Postprocessing-and-Graphics}
+
+\end_inset
+
+).
+ We can solve these issues on parallel filesystems by assembling a binary
+ HDF5 file in parallel I/O mode.
+ Since HDF5 files support 
+\end_layout
+
+\begin_layout Section
+About HDF5
+\end_layout
+
+\begin_layout Standard
+HDF (Hierarchical Data Format) is a portable file format and software developed
+ at the National Center for Supercomputing Applications (NCSA).
+ It is designed for storing, retrieving, analyzing, visualizing, and converting
+ scientific data.
+ The current and most popular version is HDF5, which stores multi-dimensional
+ arrays together with ancillary data in a portable self-describing format.
+ It uses a hierarchical structure that provides application programmers
+ with a host of options for organizing how data is stored in HDF5 files.
+\end_layout
+
+\begin_layout Standard
+HDF5 files are organized in a hierarchical structure, similar to a UNIX
+ file system.
+ Two types of primary objects, groups and datasets, are stored in this structure.
+ A group contains instances of zero or more groups or datasets, while a
+ dataset stores a multi-dimensional array of data elements.
+ Both kinds of objects are accompanied by supporting metadata.
+\end_layout
+
+\begin_layout Standard
+A dataset is physically stored in two parts: a header and a data array.
+ The header contains miscellaneous metadata describing the dataset as well
+ as information that is needed to interpret the array portion of the dataset.
+ Essentially, it includes the name, datatype, dataspace, and storage layout
+ of the dataset.
+ The name is a text string identifying the dataset.
+ The datatype describes the type of the data array elements.
+ The dataspace defines the dimensionality of the dataset, i.e., the size and
+ shape of the multi-dimensional array.
+ The dimensions of a dataset can be either fixed or unlimited (extensible).
+ The storage layout specifies how the data arrays are arranged in the file.
+\end_layout
+
+\begin_layout Standard
+The data array contains the values of the array elements and can be either
+ stored together in a contiguous file space or split into smaller 
+\shape italic
+chunks
+\shape default
+ stored at any allocated location.
+ Chunks are defined as equally-sized multi-dimensional subarrays (blocks)
+ of the whole data array and each chunk is stored in a separate contiguous
+ file space.
+ Extensible datasets whose dimensions can grow are required to be stored
+ in chunks.
+ One dimension is increased by allocating new chunks at the end of the file
+ to cover the extension.
+\end_layout
+
+\begin_layout Standard
+HDF5 also supports access to portions (or selections) of a dataset by 
+\shape italic
+hyperslabs
+\shape default
+, which consists of a subarray or strided subarray of the multi-dimensional
+ dataset.
+ The selection is performed in the file dataspace for the dataset.
+ HDF5 also supports parallel I/O.
+ Parallel access is supported through MPI-IO.
+ The file and datasets are collectively created/opened by all participating
+ processes.
+ Each process accesses part of a dataset by defining its own file dataspace
+ for that dataset.
+ When accessing data, the data transfer property specifies whether each
+ process will perform independent I/O or all processes will perform collective
+ I/O.
+\end_layout
+
+\begin_layout Section
+Data Layout
+\end_layout
+
+\begin_layout Standard
+Most of the output data from CitComS is specified at the nodes of a logically
+ cartesian grid and is therefore well represented by multi-dimensional arrays.
+ These data arrays consist of various parts, appearing in the following
+ order: First, time-dependent arrays are addressed by a time-dimension index
+ called a frame.
+ Next, a cap dimension is defined for addressing each of the CitComS caps,
+ followed by three spatial indices 
+\begin_inset Formula $(i, j, k)$
+\end_inset
+
+ in the case of 3D data, or two spatial indices 
+\begin_inset Formula $(i, j)$
+\end_inset
+
+ in the case of 2D surface data.
+ Finally, an additional dimension is provided for storing the components
+ of vector and tensor data.
+\newline
+
+\end_layout
+
+\begin_layout Standard
+\begin_inset Tabular
+<lyxtabular version="3" rows="25" columns="2">
+<features>
+<column alignment="left" valignment="top" leftline="true" width="0">
+<column alignment="left" valignment="top" leftline="true" rightline="true" width="0">
+<row topline="true" bottomline="true">
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+Dataset
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+Shape
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row topline="true">
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\family typewriter
+/coord
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\family typewriter
+(caps, nodex, nodey, nodez, 3)
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row topline="true">
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\family typewriter
+/connectivity
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\family typewriter
+(cap_elements, 8)
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row topline="true">
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row topline="true">
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\family typewriter
+/velocity
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\family typewriter
+(frames, caps, nodex, nodey, nodez, 3)
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row topline="true">
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\family typewriter
+/temperature
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\family typewriter
+(frames, caps, nodex, nodey, nodez)
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row topline="true">
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\family typewriter
+/viscosity
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\family typewriter
+(frames, caps, nodex, nodey, nodez)
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row topline="true">
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\family typewriter
+/pressure
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\family typewriter
+(frames, caps, nodex, nodey, nodez)
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row topline="true">
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\family typewriter
+/stress
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\family typewriter
+(frames, caps, nodex, nodey, nodez, 6)
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row topline="true">
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row topline="true">
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\family typewriter
+/surf/coord
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\family typewriter
+(caps, nodex, nodey, 2)
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row topline="true">
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\family typewriter
+/surf/velocity
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\family typewriter
+(frames, caps, nodex, nodey, 2)
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row topline="true">
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\family typewriter
+/surf/heatflux
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\family typewriter
+(frames, caps, nodex, nodey)
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row topline="true">
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\family typewriter
+/surf/topography
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\family typewriter
+(frames, caps, nodex, nodey)
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row topline="true">
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row topline="true">
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\family typewriter
+/botm/coord
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\family typewriter
+(caps, nodex, nodey, 2)
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row topline="true">
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\family typewriter
+/botm/velocity
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\family typewriter
+(frames, caps, nodex, nodey, 2)
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row topline="true">
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\family typewriter
+/botm/heatflux
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\family typewriter
+(frames, caps, nodex, nodey)
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row topline="true">
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\family typewriter
+/botm/topography
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\family typewriter
+(frames, caps, nodex, nodey)
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row topline="true">
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row topline="true">
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\family typewriter
+/horiz_avg/coord
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\family typewriter
+(caps, nodez)
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row topline="true">
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\family typewriter
+/horiz_avg/temperature
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\family typewriter
+(caps, nodez)
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row topline="true">
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\family typewriter
+/horiz_avg/velocity_xy
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\family typewriter
+(caps, nodez)
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row topline="true">
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\family typewriter
+/horiz_avg/velocity_z
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\family typewriter
+(caps, nodez)
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row topline="true" bottomline="true">
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Standard
+
+\end_layout
+
+\end_inset
+</cell>
+</row>
+</lyxtabular>
+
+\end_inset
+
+
+\end_layout
+
+\begin_layout Section
+Input Parameters
+\end_layout
+
+\begin_layout Standard
+To enable HDF5 output in CitComS, all you have to do is include the following
+ section in your 
+\family typewriter
+.cfg
+\family default
+ input file.
+\end_layout
+
+\begin_layout LyX-Code
+
+\family typewriter
+[CitcomS.solver.output]
+\end_layout
+
+\begin_layout LyX-Code
+
+\family typewriter
+output_format = hdf5
+\end_layout
+
+\begin_layout Standard
+Alternatively, you can specify the option 
+\family typewriter
+--solver.output.output_format=hdf5
+\family default
+ on the command line.
+ The resulting filename will simply consist of 
+\family typewriter
+.h5
+\family default
+ appended to the value of your specified 
+\family typewriter
+datafile
+\family default
+ input parameter.
+\end_layout
+
+\begin_layout Subsection
+Optimizing Parallel I/O
+\end_layout
+
+\begin_layout Standard
+There are several platform-dependent parameters that are used by the HDF5
+ library and the underlying MPI-IO routines to optimize the performance
+ of parallel I/O.
+ The optimal values for these parameters may vary from filesystem to filesystem,
+ and ideally you would hardcode these values for your installation of CitComS
+ on a particular parallel filesystem.
+ For experimental purposes we have exposed a relevant subset of these parameters
+ to the input file.
+ You can use these to tune the performance of parallel I/O on your system,
+ although to change any parameters not specified here you will have to modify
+ parts of the CitcomS source code.
+ Example values are given below:
+\end_layout
+
+\begin_layout LyX-Code
+
+\end_layout
+
+\begin_layout Standard
+To control the size of the buffer used in collective I/O
+\end_layout
+
+\begin_layout LyX-Code
+
+\family typewriter
+[CitcomS.solver.output]
+\end_layout
+
+\begin_layout LyX-Code
+cb_block_size=1048576
+\end_layout
+
+\begin_layout LyX-Code
+cb_buffer_size=4194304
+\end_layout
+
+\begin_layout Standard
+To change the size of the buffer used by the sieve algorithm
+\end_layout
+
+\begin_layout LyX-Code
+
+\family typewriter
+[CitcomS.solver.output]
+\end_layout
+
+\begin_layout LyX-Code
+sieve_buf_size=1048576
+\end_layout
+
+\begin_layout Standard
+To change the data alignment parameters
+\end_layout
+
+\begin_layout LyX-Code
+
+\family typewriter
+[CitcomS.solver.output]
+\end_layout
+
+\begin_layout LyX-Code
+output_alignment=262144
+\end_layout
+
+\begin_layout LyX-Code
+output_alignment_threshold=524288
+\end_layout
+
+\begin_layout Standard
+To change the size of the raw data chunk cache
+\end_layout
+
+\begin_layout LyX-Code
+
+\family typewriter
+[CitcomS.solver.output]
+\end_layout
+
+\begin_layout LyX-Code
+cache_rdcc_nelmts=521
+\end_layout
+
+\begin_layout LyX-Code
+cache_rdcc_nbytes=1048576
+\end_layout
+
+\begin_layout LyX-Code
+
+\end_layout
+
+\begin_layout Standard
+For more details, refer to (insert reference to chapter) in the HDF5 documentati
+on, the (insert reference to chapter) in the MPI-2 reference book, and the
+ documentation for your parallel filesystem.
+\end_layout
+
+\begin_layout Section
+Accessing Data
+\end_layout
+
+\begin_layout Standard
+As previously indicated, HDF5 is a self-describing binary file format.
+ As such we can use a variety of tools to introspect the structure of an
+ HDF5 file, as well as for retrieving data in any order.
+ To quickly inspect the structure of an HDF5 file, you can use the command
+\end_layout
+
+\begin_layout LyX-Code
+$ h5ls -r file.h5
+\end_layout
+
+\begin_layout Standard
+You can also recreate the ASCII combined cap format described in Chapter
+ (insert reference here) for specific time steps by using the command:
+\end_layout
+
+\begin_layout LyX-Code
+$ h5tocap file.h5 [step1 [step2 [...]]]
+\end_layout
+
+\begin_layout Subsection
+Python
+\end_layout
+
+\begin_layout Standard
+The small Python script 
+\family typewriter
+h5tocap.py
+\family default
+ provides a good example for using PyTables to access the data in an HDF5
+ file.
+ In PyTables, datasets can be retrieved from disk as NumPy arrays through
+ an array interface that avoids unecessary copying of data by using hyperslabs,
+ and which takes advantage of Python's powerful array slice-indexing.
+\end_layout
+
+\begin_layout Standard
+For example, obtaining the node coordinates, temperature, and topography
+ values over the entire surface of the sphere for all timesteps can be done
+ easily with the following code snippet.
+\end_layout
+
+\begin_layout LyX-Code
+import tables
+\end_layout
+
+\begin_layout LyX-Code
+h5file = tables.openFile('example.h5', 'r')
+\end_layout
+
+\begin_layout LyX-Code
+surface_coords = h5file.coord[0:12,:,:,-1,:]
+\end_layout
+
+\begin_layout LyX-Code
+surface_temperature = h5file.temperature[:,0:12,:,:,-1]
+\end_layout
+
+\begin_layout LyX-Code
+surface_topography = h5file.surf.topography[:,0:12,:,:]
+\end_layout
+
+\begin_layout Standard
+In this case, the slice 
+\family typewriter
+0:12
+\family default
+ refers to all caps explicitly, while the empty slice 
+\family typewriter
+:
+\family default
+ refers to the entire extent of the corresponding dimension.
+ The values of 
+\family typewriter
+-1
+\family default
+ above refer to the last 
+\begin_inset Formula $z$
+\end_inset
+
+-index, which corresponds to the location of the surface nodes on each of
+ the caps.
+ Finally, note how both HDF5 datasets and groups are conveniently accessible
+ as Python attributes on the PyTables file object.
+\end_layout
+
+\begin_layout Standard
+For more details, refer to the documentation for PyTables (insert link here)
+ and NumPy (insert link here).
+\end_layout
+
+\begin_layout Subsection
+HDFView
+\end_layout
+
+\begin_layout Standard
+NCSA HDFView is a visual tool written for accessing HDF files.
+ You can use it for viewing the internal file hierarchy in a tree structure,
+ creating new files, adding or deleting groups and datasets, modifying existing
+ datasets.
+ (insert image 1 here).
+ HDFView is capable of displaying 2D slices of multi-dimensional datasets,
+ with an additional slider that can range over the entire extent of a third
+ dimension.
+ (insert image 2) here.
+\end_layout
+
+\begin_layout Chapter
 \begin_inset LatexCommand \label{cha:Postprocessing-and-Graphics}
 
 \end_inset
@@ -3567,7 +4559,7 @@
 \end_inset
 
 .
- If you are uisng Linux, OpenDX is usually avaiable as package 
+ If you are uisng Linux, OpenDX is usually available as package 
 \family typewriter
 dx
 \family default