SW4 is being used to simulate earthquake ground motions at Lawrence Livermore National Laboratory (LLNL) on their new capacity computing systems called the Commodity Technology Systems-1 (CTS-1). One of their newest computers, Jade, is composed of over 2200 compute nodes with 36 cores per node. It is part of a major investment in capacity computing at the DOE National Labs as part of the Advanced Scientific Computing Program.
SW4 is a fourth order time domain finite difference code developed by LLNL and distributed by the Computational Infrastructure for Geodynamics (CIG). SW4 simulates seismic wave propagation in complex three-dimensional Earth models including anelasticity and surface topography. Understanding near-fault earthquake strong ground motions is important for evaluating the response of engineered structures, such as nuclear power plants and other critical infrastructure. Engineering analysis of structures requires the inclusion of high frequencies, which can cause damage but are often difficult to include in simulations because of the need for large memory to model fine grid spacing on large domains.
To optimize the performance of SW4 on Jade, we explored various compiler and parallelization options. The simulation shown here (caption) required over 40 billion grid points, ran on 2048 nodes (73,728 cores) and included frequencies up to 5 Hz. The earth model has stochastic heterogeneity to mimic the variability of wavespeeds due to geologic structure in the sedimentary basin. The source is a magnitude 6.5 vertical strike-slip earthquake generated using the Graves and Pitarka (2015) method. While not the largest simulation we have done, it represents an evolution in our ability simulate realistic earthquake ground motions.
Arthur Rodgers, N. Anders Petersson, Arben Pitarka and Bjorn Sjogreen, Lawrence Livermore National Laboratory
We thank Livermore Computing for access to Jade.
This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344.
Graves, R. and A. Pitarka (2015). Refinements to the Graves and Pitarka (2010) Broadband Ground-Motion Simulation Method, Seism. Res. Lett., 86, doi: 10.1785/0220140101.
No Workshops are currently scheduled.