SW4: Ground Motion Simulations on the New Commodity Technology Systems (CTS-1) Platform

SW4 is being used to simulate earthquake ground motions at Lawrence Livermore National Laboratory (LLNL) on their new capacity computing systems called the Commodity Technology Systems-1 (CTS-1). One of their newest computers, Jade, is composed of over 2200 compute nodes with 36 cores per node. It is part of a major investment in capacity computing at the DOE National Labs as part of the Advanced Scientific Computing Program. 

SW4 is a fourth order time domain finite difference code developed by LLNL and distributed by the Computational Infrastructure for Geodynamics (CIG).  SW4 simulates seismic wave propagation in complex three-dimensional Earth models including anelasticity and surface topography.  Understanding near-fault earthquake strong ground motions is important for evaluating the response of engineered structures, such as nuclear power plants and other critical infrastructure.  Engineering analysis of structures requires the inclusion of high frequencies, which can cause damage but are often difficult to include in simulations because of the need for large memory to model fine grid spacing on large domains.  

To optimize the performance of SW4 on Jade, we explored various compiler and parallelization options. The simulation shown here (right) required over 40 billion grid points, ran on 2048 nodes (73,728 cores) and included frequencies up to 5 Hz. The earth model has stochastic heterogeneity to mimic the variability of wavespeeds due to geologic structure in the sedimentary basin. The source is a magnitude 6.5 vertical strike-slip earthquake generated using the Graves and Pitarka (2015) method.  While not the largest simulation we have done, it represents an evolution in our ability simulate realistic earthquake ground motions.

 

Ground motion simulation showing the modulus of ground velocity (color-coded with the USGS SHakeMap color palette) from a magnitude 6.5 earthquake for a domain with dimensions of 40x40x25 km and a 10m grid spacing. The earthquake fault (thick black line) is strike-slip and extends 20 km along strike and 13 km down dip, with hypocenter indicated (green star).  The fault is collocated with the basin edge (white dashed line).  The earth model (inset) includes 3D stochastic heterogeneity

 

Arthur Rodgers, N. Anders Petersson, Arben Pitarka and Bjorn Sjogreen, Lawrence Livermore National Laboratory

Acknowledgements

We thank Livermore Computing for access to Jade.

This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344.

Reference

Graves, R. and A. Pitarka (2015). Refinements to the Graves and Pitarka (2010) Broadband Ground-Motion Simulation Method, Seism. Res. Lett., 86, doi: 10.1785/0220140101.