[CIG-SHORT] Pylith-running mpi on cluster

Thu Feb 8 15:12:08 PST 2018

On Fri, Feb 9, 2018 at 4:58 AM, Charles Williams <willic3 at gmail.com> wrote:

> Hi Niloufar,
>
> By problem size, what I mean is the size of your mesh (number of vertices
> and cells).  Also, the actual run log (not just the PETSc summary) would be
> helpful, as it shows us what is happening with convergence.  Also, did you
> run the problem on 24 nodes or 24 cores on the cluster?  If 24 nodes, how
> many cores per node?  If you send all of your .cfg files (including the one
> with your job submission information), that might help.
>

It is possible for 24 cores to be on a single node in a modern machine. The
best thing to do would be to run the STREAMS benchmark on your
compute machine, so we could see how much speedup we expect. However, the
output from

  --petsc.log_view

would be an acceptable substitute.

  Thanks,

     Matt

> Cheers,
> Charles
>
>
> On 8/02/2018, at 4:29 PM, Niloufar Abolfathian <
> niloufar.abolfathian at gmail.com> wrote:
>
> Hi, thanks for your replies. Here are my answers to your questions.
>
> 1.  What size of problem are you running?
> I am running a quasi-static model to simulate a vertically dipping
> strike-slip fault with static friction that is loaded by tectonic forces.
> The boundary conditions include a far-field velocity of 1 cm/yr and an
> initial displacement of 0.1 m applied normal to the fault surface to
> maintain a compressive stress on the fault. I want to run this simple model
> for thousands of years. The first issue is that the model will give
> run-time error after ~1800 years. The second problem I am encountering is
> that each run will take more than two days! That is why I am trying to use
> multicore so it may run faster. From Matt's link, I understand that I
> should not expect the program to run faster when using multicore on my own
> mac, but I have tried it on 24 nodes on the cluster and it took the same
> time as on my own mac.
>
> 2.  What solver settings are you using?
> pylith.problems.SolverNonlinear
>
> 3.  Is this a linear or nonlinear problem?
> A nonlinear problem.
>
> 4.  Is this a 2D or 3D problem?
> A 3D problem.
>
> 5.  What does the run log show?  This will include convergence information
> and a PETSc summary of calls, etc.
> I did not have my PETSc summary for the runs. I made a new run for only
> 200 years and the summary is attached as a text file. And here is my
> PETSc configuration:
>
> # Set the solver options.
> [pylithapp.petsc]
> malloc_dump =
>
> # Preconditioner settings.
> pc_type = asm
> sub_pc_factor_shift_type = nonzero
>
> # Convergence parameters.
> ksp_rtol = 1.0e-8
> ksp_atol = 1.0e-12
> ksp_max_it = 500
> ksp_gmres_restart = 50
>
> # Linear solver monitoring options.
> ksp_monitor = true
> #ksp_view = true
> ksp_converged_reason = true
> ksp_error_if_not_converged = true
>
> # Nonlinear solver monitoring options.
> snes_rtol = 1.0e-8
> snes_atol = 1.0e-12
> snes_max_it = 100
> snes_monitor = true
> snes_linesearch_monitor = true
> #snes_view = true
> snes_converged_reason = true
> snes_error_if_not_converged = true
>
>
> Hope this information can help. Please let me know if I need to provide
> you with any other information.
>
> Thanks,
> Niloufar
>
>
>
> On Wed, Feb 7, 2018 at 4:01 AM, Matthew Knepley <knepley at rice.edu> wrote:
>
>> On Wed, Feb 7, 2018 at 2:24 PM, Charles Williams <willic3 at gmail.com>
>> wrote:
>>
>>> Dear Niloufar,
>>>
>>> It is hard to diagnose your problem without more information.
>>> Information that would be helpful includes:
>>>
>>> 1.  What size of problem are you running?
>>> 2.  What solver settings are you using?
>>> 3.  Is this a linear or nonlinear problem?
>>> 4.  Is this a 2D or 3D problem?
>>> 5.  What does the run log show?  This will include convergence
>>> information and a PETSc summary of calls, etc.
>>>
>>> There are probably other things it would be good to know, but this
>>> should get us started.
>>>
>>
>> In addition to the points Charles makes, it is very useful to understand
>> how performance is affected by architecture.
>> The advantages of multiple cores are very often oversold by vendors. Here
>> is a useful reference:
>>
>>   http://www.mcs.anl.gov/petsc/documentation/faq.html#computers
>>
>> I recommend running the streams program, which can be found in the PETSc
>> installation.
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>>
>>> Cheers,
>>> Charles
>>>
>>>
>>> On 7/02/2018, at 1:06 PM, Niloufar Abolfathian <
>>> niloufar.abolfathian at gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> I am trying to run my code on the cluster but I have not gotten any
>>> improvements when using multiple cores.
>>>
>>> What I have tried:
>>>
>>> Downloaded binaries for both Mac and Linux.  Single core vs multiple
>>> cores (2 and 24 for Mac and Linux respectively) takes the same amount of
>>> time.
>>>
>>> Compiled from source, no speed up either using shared memory or mpi,
>>> even though the correct number of mpinemesis processes show up on multiple
>>> nodes.
>>>
>>> I appreciate if you can help me with running the mpi on the cluster.
>>>
>>> Thanks,
>>> Niloufar
>>> _______________________________________________
>>> CIG-SHORT mailing list
>>> CIG-SHORT at geodynamics.org
>>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>>>
>>>
>>>
>>> *Charles Williams I Geodynamic ModelerGNS Science **I** Te Pῡ Ao*
>>> 1 Fairway Drive, Avalon 5010, PO Box 30368, Lower Hutt 5040, New Zealand
>>> *Ph* 0064-4-570-4566 I *Mob* 0064-22-350-7326 I *Fax* 0064-4-570-4600
>>> *http://www.gns.cri.nz/* <http://www.gns.cri.nz/> *I* *Email: *
>>> *C.Williams at gns.cri.nz* <your.email at gns.cri.nz>
>>>
>>>
>>> _______________________________________________
>>> CIG-SHORT mailing list
>>> CIG-SHORT at geodynamics.org
>>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>>>
>>
>>
>> _______________________________________________
>> CIG-SHORT mailing list
>> CIG-SHORT at geodynamics.org
>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>>
>
> <PETSc_log_summary.txt>_______________________________________________
> CIG-SHORT mailing list
> CIG-SHORT at geodynamics.org
> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>
>
>
> *Charles Williams I Geodynamic ModelerGNS Science **I** Te Pῡ Ao*
> 1 Fairway Drive, Avalon 5010, PO Box 30368, Lower Hutt 5040, New Zealand
> *Ph* 0064-4-570-4566 I *Mob* 0064-22-350-7326 I *Fax* 0064-4-570-4600
> *http://www.gns.cri.nz/* <http://www.gns.cri.nz/> *I* *Email: *
> *C.Williams at gns.cri.nz* <your.email at gns.cri.nz>
>
>
> _______________________________________________
> CIG-SHORT mailing list
> CIG-SHORT at geodynamics.org
> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.geodynamics.org/pipermail/cig-short/attachments/20180209/7df086ca/attachment-0001.html>