[CIG-SHORT] elapsed time concerns
Brad Aagaard
baagaard at usgs.gov
Mon Nov 21 12:58:06 PST 2016
Alberto,
The log shows that the Setup Stage is mostly spent in "ElIm init", which
is ElasticityImplicit.initialize(). This is most likely associated with
setting the initial stresses using a SimpleDB object.
The SimpleGridDB provides much faster interpolation than
SimpleDB for a logically Cartesian grid because it can find the relevant
points without a global search. The points need to conform to a grid,
but the x, y, and z coordinates do not have to be spaced uniformly.
See Appendix C.3 of the manual for an example of the SimpleGridDB.
Regards,
Brad
On 11/21/2016 12:34 PM, alberto cominelli wrote:
> Brad,
> I have included also my cfg files..
> regards,
> Alberto.
>
> 2016-11-21 19:49 GMT+01:00 Brad Aagaard <baagaard at usgs.gov
> <mailto:baagaard at usgs.gov>>:
>
> Alberto,
>
> Please send the entire output of the PETSc log (everything after
> "PETSc Performance Summary") for a representative simulation. It is
> usually easiest to simply send the entire output of stdout (gzip it
> if necessary to reduce size). The individual event logging provides
> more specifics than the summary of stages. We add custom events in
> the PETSc logging for many of the PyLith routines.
>
> If you need help understanding the format of the summary, then see
> the Profiling chapter of the PETSc manual:
> http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf
> <http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf>.
>
> Regards,
> Brad
>
>
>
>
>
> On 11/19/2016 08:09 AM, alberto cominelli wrote:
>
> Brad,
> I followed you suggestion and I also modified a bit the code to
> track
> the time spent in integrator:
>
> start_time = time.time()
> integrator.initialize(totalTime, numTimeSteps, normalizer)
> str = "--- %s seconds in integrator.initialize ---" %
> (time.time()
> - start_time)
> self._info.log(str)
> (import time at the beginning
> of lib64/python2.7/site-packages/pylith/problems/Formulation.py )
> The I run a simple case with 5733 nodes/ 4800 elements and pylith
> spent 37 seconds to run with 26.5418641567 seconds in
> integrator.initialize.
> If I look at Petsc log at the end I get this:
> Summary of Stages: ----- Time ------ ----- Flops ----- ---
> Messages
> --- -- Message Lengths -- -- Reductions --
> Avg %Total Avg %Total counts
> %Total Avg %Total counts %Total
> 0: Main Stage: 1.3829e-01 0.4% 0.0000e+00 0.0% 0.000e+00
> 0.0% 0.000e+00 0.0% 0.000e+00 0.0%
> 1: Meshing: 1.5950e-01 0.4% 1.7262e+04 0.0% 0.000e+00
> 0.0% 3.874e-02 0.0% 8.000e+00 100.0%
> 2: Setup: 2.7486e+01 77.3% 2.7133e+07 0.2% 8.000e+00
> 1.9% 2.181e+01 0.0% 0.000e+00 0.0%
> 3: Reform Jacobian: 2.8208e-01 0.8% 4.1906e+08 3.5% 0.000e+00
> 0.0% 0.000e+00 0.0% 0.000e+00 0.0%
> 4: Reform Residual: 9.8572e-02 0.3% 6.1111e+07 0.5% 8.000e+00
> 1.9% 1.967e+03 3.1% 0.000e+00 0.0%
> 5: Solve: 5.5077e+00 15.5% 1.1537e+10 95.1% 3.970e+02
> 96.1% 6.197e+04 96.9% 0.000e+00 0.0%
> 6: Prestep: 5.7586e-02 0.2% 0.0000e+00 0.0% 0.000e+00
> 0.0% 0.000e+00 0.0% 0.000e+00 0.0%
> 7: Step: 8.9577e-02 0.3% 0.0000e+00 0.0% 0.000e+00
> 0.0% 0.000e+00 0.0% 0.000e+00 0.0%
> 8: Poststep: 1.6417e+00 4.6% 8.2252e+07 0.7% 0.000e+00
> 0.0% 0.000e+00 0.0% 0.000e+00 0.0%
> 9: Finalize: 7.7139e-02 0.2% 0.0000e+00 0.0% 0.000e+00
> 0.0% 0.000e+00 0.0% 0.000e+00 0.0%
>
> As far as I understand 27 seconds are spent in setup, which I
> suppose
> includes integrators.
> I simplified the problem using a linear interpolation between
> two points
> to define the initial stress state but still setup phase takes
> 80% of
> the time.
> Is it fine this timing?
> I may send you my cfg files if you like,
> Regards,
> Alberto.
>
> P.S: I noticed that Petsc log makes my little modification into
> python
> scripts useless..I will remove.
>
>
> 2016-11-19 0:04 GMT+01:00 Brad Aagaard <baagaard at usgs.gov
> <mailto:baagaard at usgs.gov>
> <mailto:baagaard at usgs.gov <mailto:baagaard at usgs.gov>>>:
>
>
> Alberto,
>
> The PETSc log summary provides important performance
> information.
>
> Use these settings to see what is happening in the solver
> and the
> performance (as used in examples/3d/hex8/pylithapp.cfg):
>
>
> [pylithapp.petsc]
> ksp_monitor = true
> ksp_view = true
> snes_monitor = true
> snes_view = true
> log_view = true
>
> Regards,
> Brad
>
>
>
> On 11/18/16 2:24 PM, alberto cominelli wrote:
>
> Dear All,
>
> I am using pylith to make a convergence study on a 12
> core Xeon box,
> with Intel(R) Xeon(R) E5-2643 v2 cpus running at @
> 3.50GHz and
> 64 gb of
> memory.
> The problem at hand is a 3D domain consisting of two
> layers, the
> upper
> one dry, with 25000kg/m3 density and the lower on water
> saturated with a
> 20% porosity. Besides differences in saturated
> condistions, rock is
> characterised as an elastic, istropic and homogeneous
> material.
> The domain is discretised by means of hexaedral
> elements using a
> tartan type grid developed around a fault, a 20% sloping
> fault.
> Fault
> rehology is very simple, a friction model with 0.6 friction
> coefficient,
>
> To simulate a consolidation problem, fluid pressure is
> included
> in the
> model using initial stress on a cell basis assuming that
> pressure is
> constant inside each cell.
> This means I input a initial_stress.spatialdb file
> containg data for
> ncells * 8 quadrature points.
> I am a bit surprised by elapsed time values I get along my
> convergence
> study.
> For instance, one case consists of 52731 nodes and 48630
> elements. To
> properly initialise the model I give initial stress
> values in
> 386880. I
> make two steps in 48 minutes, with most of the time spent in
> integrators
> - as far as I understand.
>
> With "Integrators" I mean what is labelled by these lines in
> pylith output:
> -- Initializing integrators.
> >>
>
> /home/comi/Pylith2.1.3/lib64/python2.7/site-packages/pylith/problems/Formulation.py
> [0m:474 [0m:_initialize [0m
> I guess this step means building residuals and stiffness
> matrices, but I
> am not sure about. Notably, in the second steo I do not
> change
> anything
> and then I get very few linear/non linear iteration in the
> latter step.
>
> I wonder if this time is fine according to you
> experience and if
> it is
> worth going parallel to improve computational
> efficiency. I am
> willing
> to make much more complx cases up to some millions of
> nodes and I
> wonder how far I can go using only one core.
> Regards,
> Alberto.
>
> I am attaching a snapshot of one simulation log (not for
> the entire
> case) in case it may help.
> Regards,
> Alberto.
>
>
>
>
> _______________________________________________
> CIG-SHORT mailing list
> CIG-SHORT at geodynamics.org
> <mailto:CIG-SHORT at geodynamics.org>
> <mailto:CIG-SHORT at geodynamics.org
> <mailto:CIG-SHORT at geodynamics.org>>
>
> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
> <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
>
> <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
> <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>>
>
>
> _______________________________________________
> CIG-SHORT mailing list
> CIG-SHORT at geodynamics.org <mailto:CIG-SHORT at geodynamics.org>
> <mailto:CIG-SHORT at geodynamics.org
> <mailto:CIG-SHORT at geodynamics.org>>
>
> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
> <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
>
> <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
> <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>>
>
>
>
>
> _______________________________________________
> CIG-SHORT mailing list
> CIG-SHORT at geodynamics.org <mailto:CIG-SHORT at geodynamics.org>
> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
> <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
>
>
> _______________________________________________
> CIG-SHORT mailing list
> CIG-SHORT at geodynamics.org <mailto:CIG-SHORT at geodynamics.org>
> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
> <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
>
>
>
>
> _______________________________________________
> CIG-SHORT mailing list
> CIG-SHORT at geodynamics.org
> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>
More information about the CIG-SHORT
mailing list