[CIG-SHORT] elapsed time concerns

Brad Aagaard baagaard at usgs.gov
Wed Nov 23 12:57:23 PST 2016


Alberto,

My comments on scaling apply to BOTH setting the parameters of the 
material properties AND initial stress/strain.

If you want to understand the scaling, then I suggest starting with 
using a UniformDB EVERYWHERE you use a SimpleDB. A UniformDB should 
scale with the number of cells. Also look at the details of the log 
summary, including the time for each event (not just the stages).

Regards,
Brad


On 11/23/2016 12:38 PM, alberto cominelli wrote:
> Brad,
> In my table I was also comparing elapsed times for models where I do not
> use initial condition at all.
> The section on inital conditions is commented:
> # ----------------------------------------------------------------------
> # initial stresses
> # ----------------------------------------------------------------------
> # We must specify initial stresses for each material.
> # We provide a filename for the spatial database that gives the stresses,
> # and we change the query_type from the default 'nearest' to 'linear'.
> # alberto
> #[pylithapp.timedependent.materials.material]
> #db_initial_stress = spatialdata.spatialdb.SimpleDB
> #db_initial_stress.label = Initial stress
> #db_initial_stress.iohandler.filename = initial_stress.spatialdb
> #db_initial_stress.query_type = nearest
> #
> ## ----------------------------------------------------------------------
> # boundary conditions
> # ----------------------------------------------------------------------
>
> Still also in this case set-up takes from 87% to 94% of the elapsed time.
> I am attaching my cfg cases. I am puzzled: these cases should be eeven
> better than  SinpleGridDB initilisation.
> What do you think? Am I doing anything wrong?
> Regards,
>  Alberto.
>
>
> 2016-11-23 21:17 GMT+01:00 Brad Aagaard <baagaard at usgs.gov
> <mailto:baagaard at usgs.gov>>:
>
>     Alberto,
>
>     If n is the number of cells, the setting the material parameters
>     scales as O(n).
>
>     If you are using a SimpleDB with one or more points per cell, then
>     each SimpleDB query scales as O(n), and setting the material
>     parameters scales as O(n)*O(n). This is why we usually only use
>     SimpleDB for simple 1-D or 2-D spatial variations with a small
>     number of points.
>
>     If you are using a SimpleGridDB, then each query scales with
>     O(log(p)) where p is the number of points in that dimension, so
>     setting the material parameters scales as O(n)*O(log(p)). With p <<
>     n, setting the material parameters should scale close to O(n).
>
>     Regards,
>     Brad
>
>
>     On 11/23/2016 11:46 AM, alberto cominelli wrote:
>
>         Brad,
>         I made some tests with and w/o initial stress conditions.
>         My exctation was that set-up phase would dectrease realtive to
>         the rest
>         of the computations, Counterintuitively, this did not happen, as the
>         table below shows.
>         Model   # cells         # nodes                 Total Time
>         Main Stage      Meshing         Setup
>         Reform Jacobian         Reform Residual         Solve   Prestep
>                Step    Poststep        Finalize
>         2000x2000x100   4800    5733    No In. Cond.    22.28   2.70%
>          1.10%   80.30%
>         1.30%   0.50%   2.80%   0.70%   0.70%   9.00%   1.00%
>                                 In. Cond.       34.03   0.80%   0.90%
>          89.00%  0.90%   0.30%   1.80%
>         0.20%   0.40%   5.30%   0.40%
>         1000x1000x100   7280    8505    No In. Cond.    48.73   0.50%
>          0.50%   88.60%
>         1.10%   0.40%   2.30%   0.30%   0.60%   5.40%   0.40%
>                                 In. Cond.       86.55   0.30%   0.30%
>          91.80%  0.60%   0.20%   1.50%
>         0.30%   1.20%   3.50%   0.20%
>         500x500x50      20520   22971   No In. Cond.    317.1   0.10%
>          0.10%   94.00%
>         0.40%   0.30%   190.00%         0.00%   0.10%   3.00%   0.10%
>                                 In. Cond.       662.1   0.10%   0.30%
>          97.40%  0.20%   0.10%   0.90%
>         0.00%   0.00%   1.00%   0.00%
>
>         The time saving w/o initial conditions is negligible, and it
>         does not
>         decrease (relative to..) when I increase the number of nodes,
>         Wat am I
>         doing wrong? Is it what you expect? Consoilidation type problems are
>         set-up dominated in pylith?
>
>         Regards,
>          Alberto.
>
>         P.S: these numbers are for a case w/o explicit fault.
>
>
>         2016-11-22 21:37 GMT+01:00 alberto cominelli
>         <alberto.cominelli at gmail.com
>         <mailto:alberto.cominelli at gmail.com>
>         <mailto:alberto.cominelli at gmail.com
>         <mailto:alberto.cominelli at gmail.com>>>:
>
>             Brad,
>             I definitely agree with you on the issue we have the
>         coarsest mesh.
>             The error in any norm will be gigantic. The point is to run
>         a model
>             fine enough that i can assume it is the "truth". My fault is 20
>             degree far form vertical - my grid should be fairly smooth.
>         Now I am
>             running a very fine case 25x25x25 m^3, which should be the
>         best I
>             can probably run, the most accurate solution possible. This
>         means
>              4199000 elements and  4316895 nodes. Unfortunately the case is
>             running using SimpleDB to initiate stress  with fluid pressure
>             included and it is still running:
>
>             54172 cominelli   20   0 7393m 6.7g  11m R 94.4 10.7   5989:47
>             mpinemesis
>
>             Not sure it will survive for long. If I linearly scale with
>         respect
>             to node number from a previous case where I run with 52000
>         nodes in
>             3000 sec, this should take 64 hours serial, but it seems to
>         be scale
>             worse than linearly.
>
>             I will check if I can use SimpleGridDB. Please, apologize if I
>             bother you again on this, but could you confirm that a grid
>         as in
>             the pictures below can be filled with cellwise constant initial
>             stresses using SimpleGridDB.
>
>             regards,
>
>              Alberto,
>
>
>             Il 22/Nov/2016 19:19, "Brad Aagaard" <baagaard at usgs.gov
>         <mailto:baagaard at usgs.gov>
>             <mailto:baagaard at usgs.gov <mailto:baagaard at usgs.gov>>> ha
>         scritto:
>             >
>             > Alberto,
>             >
>             > Cells with dimensions 2km x 2km x 0.1km have a very poor
>         aspect
>             ratio. They will certainly degrade the rate of convergence and
>             affect the accuracy of the results. Skewing cells so they
>         conform to
>             the fault should be okay as long as the sides have about equal
>             dimension and the angles are greater than 45 degrees. Aspect
>         ratio
>             and/or condition number mesh quality metrics can be used in both
>             cases to assess the quality of you mesh.
>             >
>             > Regards,
>             > Brad
>             >
>             >
>             >
>             > On 11/21/2016 02:44 PM, alberto cominelli wrote:
>             >>
>             >> Dear Brad,
>             >>
>             >> my convergence studies begins with very coarse cells -
>         say 2000 x
>             2000 x
>             >> 100 m^3  cells and then I am further refining the grid by
>         a 2x2x2
>             ratio
>             >> at each stage. The fines possible grid I could run should
>         provide the
>             >> reference solution, "the truth". Actually simulation time
>         is becoming
>             >> prohibitive for 100x100x25 m^3 cells.
>             >>
>             >> This should reflect the logic of subsurface models where
>         usually
>             cells
>             >> are thin (e.g. 100m x 100m x 5m). Do you think I should
>         use "regular"
>             >> cubic cells? As regards elapsed time spent  in the set-up
>         phase, does
>             >> your suggestion apply also in the case I am working with
>         skewed cells
>             >> like the ones in my pdf, which ideally should mimic those
>         in the
>             picture?
>             >>
>             >> Regards,
>             >>
>             >>  Alberto.
>             >>
>             >> Immagine incorporata 1
>             >>
>             >> 2016-11-21 23:28 GMT+01:00 Brad Aagaard
>         <baagaard at usgs.gov <mailto:baagaard at usgs.gov>
>             <mailto:baagaard at usgs.gov <mailto:baagaard at usgs.gov>>
>             >> <mailto:baagaard at usgs.gov <mailto:baagaard at usgs.gov>
>         <mailto:baagaard at usgs.gov <mailto:baagaard at usgs.gov>>>>:
>             >>
>             >>
>             >>     Alberto,
>             >>
>             >>     If your cells have aspect ratios as shown in the
>         figure, then
>             this
>             >>     will certainly degrade the convergence. The aspect
>         ratios and
>             >>     condition number metrics should be close to 1.0. In
>             CUBIT/Trelis we
>             >>     try to get condition numbers down to less than 2.0.
>             >>
>             >>     Brad
>             >>
>             >>
>             >>     On 11/21/2016 02:20 PM, alberto cominelli wrote:
>             >>
>             >>         Thank you so much Brad.
>             >>         i will try tomorrow.
>             >>         I wonder if you suggestions do apply aslo for a
>         skewed
>             cartesian
>             >>         grid..
>             >>         Acualy my grid is skwed to follow a sloping
>         fault, hence
>             cell cross
>             >>         section paralle to y is not a square. I am
>         attaching a
>             pdf to show a
>             >>         (poor) view of the grid and some vtlk files to
>         explain
>             better my
>             >>         geometry.
>             >>         regards,
>             >>         Alberto.
>             >>
>             >>
>             >>         2016-11-21 21:58 GMT+01:00 Brad Aagaard
>             <baagaard at usgs.gov <mailto:baagaard at usgs.gov>
>         <mailto:baagaard at usgs.gov <mailto:baagaard at usgs.gov>>
>             >>         <mailto:baagaard at usgs.gov
>         <mailto:baagaard at usgs.gov> <mailto:baagaard at usgs.gov
>         <mailto:baagaard at usgs.gov>>>
>             >>         <mailto:baagaard at usgs.gov
>         <mailto:baagaard at usgs.gov> <mailto:baagaard at usgs.gov
>         <mailto:baagaard at usgs.gov>>
>             <mailto:baagaard at usgs.gov <mailto:baagaard at usgs.gov>
>         <mailto:baagaard at usgs.gov <mailto:baagaard at usgs.gov>>>>>:
>             >>
>             >>             Alberto,
>             >>
>             >>             The log shows that the Setup Stage is mostly
>         spent in
>             "ElIm
>             >>         init",
>             >>             which is ElasticityImplicit.initialize(). This is
>             most likely
>             >>             associated with setting the initial stresses
>         using a
>             >>         SimpleDB object.
>             >>
>             >>             The SimpleGridDB provides much faster
>         interpolation than
>             >>             SimpleDB for a logically Cartesian grid
>         because it
>             can find the
>             >>             relevant points without a global search. The
>         points
>             need to
>             >>         conform
>             >>             to a grid, but the x, y, and z coordinates do not
>             have to be
>             >>         spaced
>             >>             uniformly.
>             >>
>             >>             See Appendix C.3 of the manual for an example
>         of the
>             >>         SimpleGridDB.
>             >>
>             >>             Regards,
>             >>             Brad
>             >>
>             >>
>             >>             On 11/21/2016 12:34 PM, alberto cominelli wrote:
>             >>
>             >>                 Brad,
>             >>                 I have included also my cfg files..
>             >>                 regards,
>             >>                 Alberto.
>             >>
>             >>                 2016-11-21 19:49 GMT+01:00 Brad Aagaard
>             >>         <baagaard at usgs.gov <mailto:baagaard at usgs.gov>
>         <mailto:baagaard at usgs.gov <mailto:baagaard at usgs.gov>>
>             <mailto:baagaard at usgs.gov <mailto:baagaard at usgs.gov>
>         <mailto:baagaard at usgs.gov <mailto:baagaard at usgs.gov>>>
>             >>                 <mailto:baagaard at usgs.gov
>         <mailto:baagaard at usgs.gov>
>             <mailto:baagaard at usgs.gov <mailto:baagaard at usgs.gov>>
>         <mailto:baagaard at usgs.gov <mailto:baagaard at usgs.gov>
>             <mailto:baagaard at usgs.gov <mailto:baagaard at usgs.gov>>>>
>             >>                 <mailto:baagaard at usgs.gov
>         <mailto:baagaard at usgs.gov>
>             <mailto:baagaard at usgs.gov <mailto:baagaard at usgs.gov>>
>         <mailto:baagaard at usgs.gov <mailto:baagaard at usgs.gov>
>             <mailto:baagaard at usgs.gov <mailto:baagaard at usgs.gov>>>
>             >>         <mailto:baagaard at usgs.gov
>         <mailto:baagaard at usgs.gov> <mailto:baagaard at usgs.gov
>         <mailto:baagaard at usgs.gov>>
>             <mailto:baagaard at usgs.gov <mailto:baagaard at usgs.gov>
>         <mailto:baagaard at usgs.gov <mailto:baagaard at usgs.gov>>>>>>:
>             >>
>             >>                     Alberto,
>             >>
>             >>                     Please send the entire output of the
>         PETSc log
>             >>         (everything after
>             >>                     "PETSc Performance Summary") for a
>         representative
>             >>                 simulation. It is
>             >>                     usually easiest to simply send the entire
>             output of
>             >>         stdout
>             >>                 (gzip it
>             >>                     if necessary to reduce size). The
>         individual
>             event
>             >>         logging
>             >>                 provides
>             >>                     more specifics than the summary of
>         stages. We
>             add custom
>             >>                 events in
>             >>                     the PETSc logging for many of the PyLith
>             routines.
>             >>
>             >>                     If you need help understanding the
>         format of the
>             >>         summary,
>             >>                 then see
>             >>                     the Profiling chapter of the PETSc
>         manual:
>             >>
>             >>
>             http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf
>         <http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf>
>             <http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf
>         <http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf>>
>             >>
>             <http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf
>         <http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf>
>             <http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf
>         <http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf>>>
>             >>
>             >>
>             <http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf
>         <http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf>
>             <http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf
>         <http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf>>
>             >>
>             <http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf
>         <http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf>
>             <http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf
>         <http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf>>>>
>             >>
>             >>
>             <http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf
>         <http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf>
>             <http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf
>         <http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf>>
>             >>
>             <http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf
>         <http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf>
>             <http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf
>         <http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf>>>
>             >>
>             >>
>             <http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf
>         <http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf>
>             <http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf
>         <http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf>>
>             >>
>             <http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf
>         <http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf>
>             <http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf
>         <http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf>>>>>.
>             >>
>             >>
>             >>                     Regards,
>             >>                     Brad
>             >>
>             >>
>             >>
>             >>
>             >>
>             >>                     On 11/19/2016 08:09 AM, alberto
>         cominelli wrote:
>             >>
>             >>                         Brad,
>             >>                         I followed you suggestion and I also
>             modified a
>             >>         bit the
>             >>                 code to
>             >>                         track
>             >>                         the time spent in integrator:
>             >>
>             >>                               start_time = time.time()
>             >>
>          integrator.initialize(totalTime,
>             numTimeSteps,
>             >>                 normalizer)
>             >>                               str = "--- %s seconds in
>             >>         integrator.initialize ---" %
>             >>                         (time.time()
>             >>                         - start_time)
>             >>                               self._info.log(str)
>             >>                          (import time at the beginning
>             >>                         of
>             >>
>             >>
>             lib64/python2.7/site-packages/pylith/problems/Formulation.py )
>             >>                         The I run a simple case with 5733
>         nodes/ 4800
>             >>         elements
>             >>                 and  pylith
>             >>                         spent  37 seconds to run with
>         26.5418641567
>             >>         seconds in
>             >>                         integrator.initialize.
>             >>                         If I look at Petsc  log at the
>         end I get
>             this:
>             >>                         Summary of Stages:   ----- Time
>         ------
>             ----- Flops
>             >>                 -----  ---
>             >>                         Messages
>             >>                         ---  -- Message Lengths --  --
>         Reductions --
>             >>                                                 Avg
>          %Total
>              Avg
>             >>                  %Total   counts
>             >>                         %Total     Avg         %Total
>          counts
>              %Total
>             >>                          0:      Main Stage: 1.3829e-01
>          0.4%
>             0.0000e+00
>             >>                  0.0%  0.000e+00
>             >>                         0.0%  0.000e+00        0.0%
>         0.000e+00   0.0%
>             >>                          1:         Meshing: 1.5950e-01
>          0.4%
>             1.7262e+04
>             >>                  0.0%  0.000e+00
>             >>                         0.0%  3.874e-02        0.0%
>         8.000e+00 100.0%
>             >>                          2:           Setup: 2.7486e+01
>         77.3%
>             2.7133e+07
>             >>                  0.2%  8.000e+00
>             >>                         1.9%  2.181e+01        0.0%
>         0.000e+00   0.0%
>             >>                          3: Reform Jacobian: 2.8208e-01
>          0.8%
>             4.1906e+08
>             >>                  3.5%  0.000e+00
>             >>                         0.0%  0.000e+00        0.0%
>         0.000e+00   0.0%
>             >>                          4: Reform Residual: 9.8572e-02
>          0.3%
>             6.1111e+07
>             >>                  0.5%  8.000e+00
>             >>                         1.9%  1.967e+03        3.1%
>         0.000e+00   0.0%
>             >>                          5:           Solve: 5.5077e+00
>         15.5%
>             1.1537e+10
>             >>                 95.1%  3.970e+02
>             >>                          96.1%  6.197e+04       96.9%
>         0.000e+00
>              0.0%
>             >>                          6:         Prestep: 5.7586e-02
>          0.2%
>             0.0000e+00
>             >>                  0.0%  0.000e+00
>             >>                         0.0%  0.000e+00        0.0%
>         0.000e+00   0.0%
>             >>                          7:            Step: 8.9577e-02
>          0.3%
>             0.0000e+00
>             >>                  0.0%  0.000e+00
>             >>                         0.0%  0.000e+00        0.0%
>         0.000e+00   0.0%
>             >>                          8:        Poststep: 1.6417e+00
>          4.6%
>             8.2252e+07
>             >>                  0.7%  0.000e+00
>             >>                         0.0%  0.000e+00        0.0%
>         0.000e+00   0.0%
>             >>                          9:        Finalize: 7.7139e-02
>          0.2%
>             0.0000e+00
>             >>                  0.0%  0.000e+00
>             >>                         0.0%  0.000e+00        0.0%
>         0.000e+00   0.0%
>             >>
>             >>                         As far as I understand 27 seconds are
>             spent in
>             >>         setup,
>             >>                 which I
>             >>                         suppose
>             >>                         includes integrators.
>             >>                         I simplified the problem using a
>         linear
>             >>         interpolation
>             >>                 between
>             >>                         two points
>             >>                         to define the initial stress
>         state but still
>             >>         setup phase
>             >>                 takes
>             >>                         80% of
>             >>                         the time.
>             >>                         Is it fine this timing?
>             >>                         I may send you my cfg files if
>         you like,
>             >>                         Regards,
>             >>                           Alberto.
>             >>
>             >>                         P.S: I noticed that Petsc log
>         makes my little
>             >>                 modification into
>             >>                         python
>             >>                         scripts useless..I will remove.
>             >>
>             >>
>             >>                         2016-11-19 0:04 GMT+01:00 Brad
>         Aagaard
>             >>                 <baagaard at usgs.gov
>         <mailto:baagaard at usgs.gov> <mailto:baagaard at usgs.gov
>         <mailto:baagaard at usgs.gov>>
>             <mailto:baagaard at usgs.gov <mailto:baagaard at usgs.gov>
>         <mailto:baagaard at usgs.gov <mailto:baagaard at usgs.gov>>>
>             >>         <mailto:baagaard at usgs.gov
>         <mailto:baagaard at usgs.gov> <mailto:baagaard at usgs.gov
>         <mailto:baagaard at usgs.gov>>
>             <mailto:baagaard at usgs.gov <mailto:baagaard at usgs.gov>
>         <mailto:baagaard at usgs.gov <mailto:baagaard at usgs.gov>>>>
>             >>                         <mailto:baagaard at usgs.gov
>         <mailto:baagaard at usgs.gov>
>             <mailto:baagaard at usgs.gov <mailto:baagaard at usgs.gov>>
>             >>         <mailto:baagaard at usgs.gov
>         <mailto:baagaard at usgs.gov> <mailto:baagaard at usgs.gov
>         <mailto:baagaard at usgs.gov>>>
>             <mailto:baagaard at usgs.gov <mailto:baagaard at usgs.gov>
>         <mailto:baagaard at usgs.gov <mailto:baagaard at usgs.gov>>
>             >>         <mailto:baagaard at usgs.gov
>         <mailto:baagaard at usgs.gov> <mailto:baagaard at usgs.gov
>         <mailto:baagaard at usgs.gov>>>>>
>             >>                         <mailto:baagaard at usgs.gov
>         <mailto:baagaard at usgs.gov>
>             <mailto:baagaard at usgs.gov <mailto:baagaard at usgs.gov>>
>             >>         <mailto:baagaard at usgs.gov
>         <mailto:baagaard at usgs.gov> <mailto:baagaard at usgs.gov
>         <mailto:baagaard at usgs.gov>>>
>             <mailto:baagaard at usgs.gov <mailto:baagaard at usgs.gov>
>         <mailto:baagaard at usgs.gov <mailto:baagaard at usgs.gov>>
>             >>         <mailto:baagaard at usgs.gov
>         <mailto:baagaard at usgs.gov> <mailto:baagaard at usgs.gov
>         <mailto:baagaard at usgs.gov>>>>
>             >>                 <mailto:baagaard at usgs.gov
>         <mailto:baagaard at usgs.gov>
>             <mailto:baagaard at usgs.gov <mailto:baagaard at usgs.gov>>
>         <mailto:baagaard at usgs.gov <mailto:baagaard at usgs.gov>
>             <mailto:baagaard at usgs.gov <mailto:baagaard at usgs.gov>>>
>             >>         <mailto:baagaard at usgs.gov
>         <mailto:baagaard at usgs.gov> <mailto:baagaard at usgs.gov
>         <mailto:baagaard at usgs.gov>>
>             <mailto:baagaard at usgs.gov <mailto:baagaard at usgs.gov>
>         <mailto:baagaard at usgs.gov <mailto:baagaard at usgs.gov>>>>>>>:
>             >>
>             >>
>             >>
>             >>                             Alberto,
>             >>
>             >>                             The PETSc log summary
>         provides important
>             >>         performance
>             >>                         information.
>             >>
>             >>                             Use these settings to see what is
>             happening
>             >>         in the
>             >>                 solver
>             >>                         and the
>             >>                             performance (as used in
>             >>         examples/3d/hex8/pylithapp.cf
>         <http://pylithapp.cf>g):
>             >>
>             >>
>             >>                             [pylithapp.petsc]
>             >>                             ksp_monitor = true
>             >>                             ksp_view = true
>             >>                             snes_monitor = true
>             >>                             snes_view = true
>             >>                             log_view = true
>             >>
>             >>                             Regards,
>             >>                             Brad
>             >>
>             >>
>             >>
>             >>                             On 11/18/16 2:24 PM, alberto
>             cominelli wrote:
>             >>
>             >>                                 Dear All,
>             >>
>             >>                                 I am using pylith to make a
>             convergence
>             >>         study on
>             >>                 a 12
>             >>                         core Xeon box,
>             >>                                 with Intel(R) Xeon(R)
>         E5-2643 v2
>             cpus
>             >>         running at @
>             >>                         3.50GHz and
>             >>                                 64 gb of
>             >>                                 memory.
>             >>                                 The problem at hand is a
>         3D domain
>             >>         consisting of two
>             >>                         layers, the
>             >>                                 upper
>             >>                                 one dry, with 25000kg/m3
>         density
>             and the
>             >>         lower
>             >>                 on water
>             >>                                 saturated with a
>             >>                                 20% porosity.  Besides
>         differences in
>             >>         saturated
>             >>                         condistions, rock is
>             >>                                 characterised as an elastic,
>             istropic and
>             >>                 homogeneous
>             >>                         material.
>             >>                                 The domain is discretised  by
>             means of
>             >>         hexaedral
>             >>                         elements using a
>             >>                                 tartan type grid
>         developed around a
>             >>         fault, a 20%
>             >>                 sloping
>             >>                         fault.
>             >>                                 Fault
>             >>                                 rehology is very simple,
>         a friction
>             >>         model with
>             >>                 0.6 friction
>             >>                                 coefficient,
>             >>
>             >>                                 To simulate a consolidation
>             problem, fluid
>             >>                 pressure is
>             >>                         included
>             >>                                 in the
>             >>                                 model using initial
>         stress on a
>             cell basis
>             >>                 assuming that
>             >>                         pressure is
>             >>                                 constant inside each cell.
>             >>                                 This means I input a
>             >>         initial_stress.spatialdb file
>             >>                         containg data for
>             >>                                 ncells * 8 quadrature points.
>             >>                                 I am a bit surprised by
>         elapsed time
>             >>         values I
>             >>                 get along my
>             >>                                 convergence
>             >>                                 study.
>             >>                                 For instance, one case
>         consists
>             of 52731
>             >>         nodes
>             >>                 and 48630
>             >>                                 elements. To
>             >>                                 properly initialise the
>         model I give
>             >>         initial stress
>             >>                         values in
>             >>                                 386880. I
>             >>                                 make two steps in 48 minutes,
>             with most
>             >>         of the
>             >>                 time spent in
>             >>                                 integrators
>             >>                                 - as far as I understand.
>             >>
>             >>                                 With "Integrators" I mean
>         what is
>             >>         labelled by
>             >>                 these lines in
>             >>                                 pylith output:
>             >>                                 -- Initializing integrators.
>             >>                                  >>
>             >>
>             >>
>             >>
>             >>
>
>         /home/comi/Pylith2.1.3/lib64/python2.7/site-packages/pylith/problems/Formulation.py
>             >>                                 [0m:474 [0m:_initialize [0m
>             >>                                 I guess this step means
>         building
>             >>         residuals and
>             >>                 stiffness
>             >>                                 matrices, but I
>             >>                                 am not sure about.
>         Notably, in the
>             >>         second steo I
>             >>                 do not
>             >>                         change
>             >>                                 anything
>             >>                                 and then I get very few
>             linear/non linear
>             >>                 iteration in the
>             >>                                 latter step.
>             >>
>             >>                                 I wonder if this time is fine
>             according
>             >>         to you
>             >>                         experience and if
>             >>                                 it is
>             >>                                 worth going parallel to
>         improve
>             >>         computational
>             >>                         efficiency. I am
>             >>                                 willing
>             >>                                 to make much more complx
>         cases
>             up to some
>             >>                 millions of
>             >>                         nodes and I
>             >>                                 wonder how far I can go using
>             only one core.
>             >>                                 Regards,
>             >>                                  Alberto.
>             >>
>             >>                                 I am attaching a snapshot
>         of one
>             >>         simulation log
>             >>                 (not for
>             >>                         the entire
>             >>                                 case) in case it may help.
>             >>                                 Regards,
>             >>                                   Alberto.
>             >>
>             >>
>             >>
>             >>
>             >>
>             >>         _______________________________________________
>             >>                                 CIG-SHORT mailing list
>             >>                                 CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>
>             <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>>
>             >>         <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>
>             <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>>>
>             >>                 <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>
>             <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>>
>             >>         <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>
>             <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>>>>
>             >>                         <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>
>             <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>>
>             >>         <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>
>             <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>>>
>             >>                 <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>
>             <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>>
>             >>         <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>
>             <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>>>>>
>             >>                         <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>
>             <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>>
>             >>         <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>
>             <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>>>
>             >>                 <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>
>             <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>>
>             >>         <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>
>             <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>>>>
>             >>                         <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>
>             <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>>
>             >>         <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>
>             <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>>>
>             >>                 <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>
>             <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>>
>             >>         <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>
>             <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>>>>>>
>             >>
>             >>
>             >>
>             >>
>
>         http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>>
>             >>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>>>
>             >>
>             >>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>>
>             >>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>>>>
>             >>
>             >>
>             >>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>>
>             >>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>>>
>             >>
>             >>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>>
>             >>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>>>>>
>             >>
>             >>
>             >>
>             >>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>>
>             >>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>>>
>             >>
>             >>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>>
>             >>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>>>>
>             >>
>             >>
>             >>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>>
>             >>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>>>
>             >>
>             >>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>>
>             >>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>>>>>>
>             >>
>             >>
>             >>
>             _______________________________________________
>             >>                             CIG-SHORT mailing list
>             >>                             CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>
>             <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>>
>             >>         <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>
>             <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>>>
>             >>                 <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>
>             <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>>
>             >>         <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>
>             <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>>>>
>             >>                 <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>
>             <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>>
>             >>         <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>
>             <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>>>
>             >>                 <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>
>             <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>>
>             >>         <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>
>             <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>>>>>
>             >>                         <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>
>             <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>>
>             >>         <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>
>             <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>>>
>             >>                 <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>
>             <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>>
>             >>         <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>
>             <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>>>>
>             >>                         <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>
>             <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>>
>             >>         <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>
>             <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>>>
>             >>                 <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>
>             <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>>
>             >>         <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>
>             <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>>>>>>
>             >>
>             >>
>             >>
>             >>
>
>         http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>>
>             >>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>>>
>             >>
>             >>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>>
>             >>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>>>>
>             >>
>             >>
>             >>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>>
>             >>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>>>
>             >>
>             >>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>>
>             >>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>>>>>
>             >>
>             >>
>             >>
>             >>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>>
>             >>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>>>
>             >>
>             >>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>>
>             >>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>>>>
>             >>
>             >>
>             >>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>>
>             >>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>>>
>             >>
>             >>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>>
>             >>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>>>>>>
>             >>
>             >>
>             >>
>             >>
>             >>
>             _______________________________________________
>             >>                         CIG-SHORT mailing list
>             >>                         CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>
>             <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>>
>             >>         <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>
>             <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>>>
>             >>                 <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>
>             <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>>
>             >>         <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>
>             <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>>>>
>             >>                 <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>
>             <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>>
>             >>         <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>
>             <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>>>
>             >>                 <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>
>             <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>>
>             >>         <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>
>             <mailto:CIG-SHORT at geodynamics.org
>         <mailto:CIG-SHORT at geodynamics.org>>>>>
>             >>
>             >>
>             >>
>
>         http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>>
>             >>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>>>
>             >>
>             >>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
>
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>         <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>>
>
>
>
>
> _______________________________________________
> CIG-SHORT mailing list
> CIG-SHORT at geodynamics.org
> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>



More information about the CIG-SHORT mailing list