[CIG-SEISMO] SPECFEM3D: time per time step increasing during simulation

[Brad Aagaard]

SPECFEM3D users and developers,

I am finding that the average time per time step in a SPECFEM3D 
simulation is increasing as the simulation progresses:

  Time step #          400
  Time:   -1.002500      seconds
  Elapsed time in seconds =    135.029711008072
  Elapsed time in hh:mm:ss =    0 h 02 m 15 s
  Mean elapsed time per time step in seconds =   0.337574277520180

  Time step #          800
  Time:  -2.4999999E-03  seconds
  Elapsed time in seconds =    420.503839015961
  Elapsed time in hh:mm:ss =    0 h 07 m 00 s
  Mean elapsed time per time step in seconds =   0.525629798769951

  Time step #         1200
  Time:   0.9975000      seconds
  Elapsed time in seconds =    854.967207908630
  Elapsed time in hh:mm:ss =    0 h 14 m 14 s
  Mean elapsed time per time step in seconds =   0.712472673257192

  Time step #         1600
  Time:    1.997500      seconds
  Elapsed time in seconds =    1439.92759609222
  Elapsed time in hh:mm:ss =    0 h 23 m 59 s
  Mean elapsed time per time step in seconds =   0.899954747557640

This behavior seems very odd because I would expect the work per time 
step to be constant. The job is running on 4 compute nodes (32 cores 
total) and easily fits in memory. I don't see any anomalous behavior on 
the cluster diagnostics (CPU load, network traffic, etc) consistent with 
an increasing workload. I have forked off the git master branch to add 
my own seismic velocity model.

Has this behavior been observed before?

I can try turning off output to see if that isolates the problem. Does 
anyone have any other suggestions?

Thanks,
Brad Aagaard