[CIG-SEISMO] SPECFEM3D: time per time step increasing during simulation
Brad Aagaard
baagaard at usgs.gov
Tue Oct 7 09:39:21 PDT 2014
SPECFEM3D users and developers,
I am finding that the average time per time step in a SPECFEM3D
simulation is increasing as the simulation progresses:
Time step # 400
Time: -1.002500 seconds
Elapsed time in seconds = 135.029711008072
Elapsed time in hh:mm:ss = 0 h 02 m 15 s
Mean elapsed time per time step in seconds = 0.337574277520180
Time step # 800
Time: -2.4999999E-03 seconds
Elapsed time in seconds = 420.503839015961
Elapsed time in hh:mm:ss = 0 h 07 m 00 s
Mean elapsed time per time step in seconds = 0.525629798769951
Time step # 1200
Time: 0.9975000 seconds
Elapsed time in seconds = 854.967207908630
Elapsed time in hh:mm:ss = 0 h 14 m 14 s
Mean elapsed time per time step in seconds = 0.712472673257192
Time step # 1600
Time: 1.997500 seconds
Elapsed time in seconds = 1439.92759609222
Elapsed time in hh:mm:ss = 0 h 23 m 59 s
Mean elapsed time per time step in seconds = 0.899954747557640
This behavior seems very odd because I would expect the work per time
step to be constant. The job is running on 4 compute nodes (32 cores
total) and easily fits in memory. I don't see any anomalous behavior on
the cluster diagnostics (CPU load, network traffic, etc) consistent with
an increasing workload. I have forked off the git master branch to add
my own seismic velocity model.
Has this behavior been observed before?
I can try turning off output to see if that isolates the problem. Does
anyone have any other suggestions?
Thanks,
Brad Aagaard
More information about the CIG-SEISMO
mailing list