[CIG-SHORT] Change in dt means new Jacobian

[Birendra jha]

Hi Brad

Thanks for your reply. After Matt's suggestion, I did fix one issue in my setup. Now the residual norm falls below 1e-12 in 800-900 KSP iterations (using ASM). Earlier it did not, even after 1000 iterations. 

1. I will try FieldSplit with ML to see if no. of iterations goes down.

2. I need to investigate if I need 1e-12 tolerance or 1e-8 is enough.

3. How do I utilize multiple cores on a single machine? Sorry if this question is already answered somewhere. I found instructions on running parallel using a batch system but don't know how to run on 1 node and use multiple cores. Does this require re-building Pylith?  

4. For HDF5 output, is it correct that the I have to wait until finish to get the xmf file which is needed to load h5 files in Paraview? Manual says "This file is written when PyLith closes the HDF5 file at the end of the simulation". 

Thanks
Birendra

--- On Tue, 9/4/12, Brad Aagaard <baagaard at usgs.gov> wrote:

> From: Brad Aagaard <baagaard at usgs.gov>
> Subject: Re: Change in dt means new Jacobian
> To: "Birendra jha" <bjha7333 at yahoo.com>
> Cc: cig-short at geodynamics.org
> Date: Tuesday, September 4, 2012, 3:02 AM
> Birendra,
> 
> Matt's point about something odd in your problem setup is
> the first avenue I would investigate. Some comments on the
> options you proposed are below.
> 
> On 9/2/12 10:24 AM, Birendra jha wrote:
> > I am trying to run my Pylith simulation faster. Right
> now it takes about 3 min for Linear solve (using default KSP
> solver, about 960 iterations) and 3 min for integrating
> residual. I have about 190000 hex cells (mesh is very
> refined in the center and becomes coarse outward). No
> faults. Elastic materials. I used non-uniform user specified
> timesteps.
> > 
> > I thought of following options:
> > 
> > (1) Try different solver. I am thinking of using field
> split preconditioner with multigrid, as in tet4/step02.cfg.
> 
> Field split with ML for a problem without faults should be
> better than ASM for all but very small problems. This is
> especially true for large problems because the number of
> iterations with ASM grows quite rapidly with problem size,
> while AMG is much less sensitive to problem size.
> 
> > (2) Switch from vtk to hdf5 output
> 
> You can look at the log summary and see how much time is in
> the poststep stage. If it is significant, then switching to
> HDF5 output will reduce runtime. If you do any
> post-processing, HDF5 is much easier to use because Matlab
> and many other tools (we now include h5py in the binary) can
> access data in HDF5 files with just a few lines of code.
> 
> > (3) Run in parallel. I am looking into running it using
> sge. Or just running it on one multicore machine.
> 
> Running in parallel should help. Our weak-scaling
> benchmarking results show the limits of the memory bandwidth
> tend to kick in at around 4 processors on dual quad-core and
> dual six-core machines. This means you can get pretty good
> speedup by using 2-4 cores. Using a cluster with multiple
> compute nodes also gives reasonable speedup for large
> problems with millions of DOF.
> 
> > (4) Increase linear solve tolerance from 1E-12 to
> 1E-11.
> 
> What is the uncertainty in the observations you are using to
> constrain your simulations? Why 1.0e-12 and not 1.0e-8? Does
> using a larger tolerance affect your results in any
> significant way?
> 
> > (5) Fix timestep.
> > 
> > Are these good options to consider?
> > 
> > I have a question on (5). Manual says
> > Warning: Changing the time step requires recomputing
> the Jacobian of the system, which can greatly increase the
> runtime if the time-step size changes frequently.
> > 
> > Where does this happen in the program, i.e a change in
> dt setting needNewJacobian=true? I ask this because I could
> not find message on the stdout about reforming Jacobian when
> I am using non-uniform dt (it is possible that I missed it,
> the timesteps remains constant for a while then changes then
> becomes constant again).
> 
> 
> The Jacobian will only be reformed when you change the time
> step if it needs to be. This depends on the bulk
> constitutive model. If the Jacobian does need to be reformed
> there is a clear tradeoff between changing the time step and
> reforming the Jacobian and computing more time steps but not
> having to reform the Jacobian. See Charles's previous
> response for cases when this happens.
> 
> Regards,
> Brad
> 
>