[CIG-SHORT] Parallel computing problem

[Brad Aagaard]

Romain,

One more thing to note is that DataWriterHDF5* can be fragile on large 
parallel machines. Using DataWriterHDF5Ext* is much more robust, because 
the raw data values are written via MPI I/O and the datasets in the HDF5 
merely point to the external raw binary files.

Regards,
Brad


On 05/29/2012 11:09 AM, Brad Aagaard wrote:
> Romain,
>
> I found a parallel bug in FaultCohesiveDyn that could cause problems
> when running faults with friction in parallel. I pushed the fix to our
> PyLith version 1.7 code. We hope to release Fri June 1.
>
> Your problem runs for me but the nonlinear solve runs out of iterations
> before it converges. It looks like the problem is related to the absence
> of realistic fault normal tractions. You impose Dirichlet BC, which will
> create very small (in some cases negligible) fault normal compressive
> tractions. I recommend imposing initial fault tractions (compressive
> normal tractions and, optionally, shear tractions) or gravitational body
> forces consistent with the overburden. This will yield more realistic
> conditions and facilitate convergence.
>
> I used the following solver settings for your 2-D problem in addition to
> the ones you used in your pylithapp.cfg file. This accelerates the rate
> of convergence in the linear solve. These are more appropriate for your
> problem than the ones that Charles provided (which were for a 3-D
> problem and didn't use the custom constraint preconditioner).
>
> Sorry for the delayed reply. I was out of the office all last week.
>
> [pylithapp.timedependent.formulation]
> split_fields = True
> matrix_type = aij
> use_custom_constraint_pc = True
>
> [pylithapp.petsc]
> ksp_gmres_restart = 100
> fs_pc_type = fieldsplit
> fs_pc_fieldsplit_real_diagonal = True
> fs_pc_fieldsplit_type = multiplicative
> fs_fieldsplit_0_pc_type = ml
> fs_fieldsplit_1_pc_type = ml
> fs_fieldsplit_2_pc_type = jacobi
> fs_fieldsplit_0_ksp_type = preonly
> fs_fieldsplit_1_ksp_type = preonly
> fs_fieldsplit_2_ksp_type = preonly
>
> Regards,
> Brad
>
>
> On 05/22/2012 06:36 PM, Romain Jolivet wrote:
>> Hi,
>>
>> I have a problem running pylith on multiple nodes.
>>
>> My model is based on an elastic medium with a fault in it. When I use
>> the FaultCohesiveKin property and impose the displacement, everything
>> is fine.
>> However, when I use the FaultCohesiveDyn property, with
>> StaticFriction, it starts getting messy. My files are attached.
>> I run:
>> pylith step01.cfg
>> and the run is fine
>> If I run
>> pylith --nodes=10 step01.cfg
>> it crashes at the first time step. I tried with nodes=2 to nodes=12
>> (the number of cores on my machine) and I had the same result. The
>> error message comes from petsc.
>>
>> I went into the pylith examples to find a similar, but simpler,
>> problem. In example/3d/hex8/, I tried to run step10.cfg:
>> with one node, it is fine:
>> pyltih step10.cfg
>> However, when you ask for several nodes (more than 2), it crashes:
>> pylith --nodes=3 step10.cfg
>> with 2 nodes, it seems to be fine, but you have a petsc error message
>> at the end.
>>
>> I someone has an idea on this....
>> Thank you
>> Romain
>>
>>
>>
>>
>>
>>
>>
>>
>> Romain Jolivet
>> Postdoctoral Scholar
>> Geological and Planetary Sciences
>> California Institute of Technology, Pasadena, CA
>> rjolivet at caltech.edu
>> +1 (626) 560 6356
>>
>>
>>
>>
>> _______________________________________________
>> CIG-SHORT mailing list
>> CIG-SHORT at geodynamics.org
>> http://geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>