[CIG-SHORT] Pylith-running mpi on cluster

Thu Feb 8 09:58:27 PST 2018

Hi Niloufar,

By problem size, what I mean is the size of your mesh (number of vertices and cells).  Also, the actual run log (not just the PETSc summary) would be helpful, as it shows us what is happening with convergence.  Also, did you run the problem on 24 nodes or 24 cores on the cluster?  If 24 nodes, how many cores per node?  If you send all of your .cfg files (including the one with your job submission information), that might help.

Cheers,
Charles

> On 8/02/2018, at 4:29 PM, Niloufar Abolfathian <niloufar.abolfathian at gmail.com> wrote:
> 
> Hi, thanks for your replies. Here are my answers to your questions. 
> 
> 1.  What size of problem are you running?
> I am running a quasi-static model to simulate a vertically dipping strike-slip fault with static friction that is loaded by tectonic forces. The boundary conditions include a far-field velocity of 1 cm/yr and an initial displacement of 0.1 m applied normal to the fault surface to maintain a compressive stress on the fault. I want to run this simple model for thousands of years. The first issue is that the model will give run-time error after ~1800 years. The second problem I am encountering is that each run will take more than two days! That is why I am trying to use multicore so it may run faster. From Matt's link, I understand that I should not expect the program to run faster when using multicore on my own mac, but I have tried it on 24 nodes on the cluster and it took the same time as on my own mac.
> 
> 2.  What solver settings are you using?
> pylith.problems.SolverNonlinear
> 
> 3.  Is this a linear or nonlinear problem?
> A nonlinear problem.
> 
> 4.  Is this a 2D or 3D problem?
> A 3D problem.
> 
> 5.  What does the run log show?  This will include convergence information and a PETSc summary of calls, etc.
> I did not have my PETSc summary for the runs. I made a new run for only 200 years and the summary is attached as a text file. And here is my PETSc configuration:
> 
> # Set the solver options.
> [pylithapp.petsc]
> malloc_dump = 
> 
> # Preconditioner settings.
> pc_type = asm
> sub_pc_factor_shift_type = nonzero
> 
> # Convergence parameters.
> ksp_rtol = 1.0e-8
> ksp_atol = 1.0e-12
> ksp_max_it = 500
> ksp_gmres_restart = 50
> 
> # Linear solver monitoring options.
> ksp_monitor = true
> #ksp_view = true
> ksp_converged_reason = true
> ksp_error_if_not_converged = true
> 
> # Nonlinear solver monitoring options.
> snes_rtol = 1.0e-8
> snes_atol = 1.0e-12
> snes_max_it = 100
> snes_monitor = true
> snes_linesearch_monitor = true
> #snes_view = true
> snes_converged_reason = true
> snes_error_if_not_converged = true
> 
> 
> Hope this information can help. Please let me know if I need to provide you with any other information.
> 
> Thanks,
> Niloufar
> 
> 
> 
> On Wed, Feb 7, 2018 at 4:01 AM, Matthew Knepley <knepley at rice.edu <mailto:knepley at rice.edu>> wrote:
> On Wed, Feb 7, 2018 at 2:24 PM, Charles Williams <willic3 at gmail.com <mailto:willic3 at gmail.com>> wrote:
> Dear Niloufar,
> 
> It is hard to diagnose your problem without more information.  Information that would be helpful includes:
> 
> 1.  What size of problem are you running?
> 2.  What solver settings are you using?
> 3.  Is this a linear or nonlinear problem?
> 4.  Is this a 2D or 3D problem?
> 5.  What does the run log show?  This will include convergence information and a PETSc summary of calls, etc.
> 
> There are probably other things it would be good to know, but this should get us started.
> 
> In addition to the points Charles makes, it is very useful to understand how performance is affected by architecture.
> The advantages of multiple cores are very often oversold by vendors. Here is a useful reference:
> 
>   http://www.mcs.anl.gov/petsc/documentation/faq.html#computers <http://www.mcs.anl.gov/petsc/documentation/faq.html#computers>
> 
> I recommend running the streams program, which can be found in the PETSc installation.
> 
>   Thanks,
> 
>      Matt
>  
> 
> Cheers,
> Charles
> 
> 
>> On 7/02/2018, at 1:06 PM, Niloufar Abolfathian <niloufar.abolfathian at gmail.com <mailto:niloufar.abolfathian at gmail.com>> wrote:
>> 
>> Hi, 
>> 
>> I am trying to run my code on the cluster but I have not gotten any improvements when using multiple cores.
>> 
>> What I have tried:
>> 
>> Downloaded binaries for both Mac and Linux.  Single core vs multiple cores (2 and 24 for Mac and Linux respectively) takes the same amount of time.
>> 
>> Compiled from source, no speed up either using shared memory or mpi, even though the correct number of mpinemesis processes show up on multiple nodes.
>> 
>> I appreciate if you can help me with running the mpi on the cluster.
>> 
>> Thanks,
>> Niloufar
>> _______________________________________________
>> CIG-SHORT mailing list
>> CIG-SHORT at geodynamics.org <mailto:CIG-SHORT at geodynamics.org>
>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
> Charles Williams I Geodynamic Modeler
> GNS Science I Te Pῡ Ao
> 1 Fairway Drive, Avalon 5010, PO Box 30368, Lower Hutt 5040, New Zealand
> Ph 0064-4-570-4566 I Mob 0064-22-350-7326 I Fax 0064-4-570-4600
> http://www.gns.cri.nz/ <http://www.gns.cri.nz/> I Email: C.Williams at gns.cri.nz <mailto:your.email at gns.cri.nz>
> 
> _______________________________________________
> CIG-SHORT mailing list
> CIG-SHORT at geodynamics.org <mailto:CIG-SHORT at geodynamics.org>
> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
> 
> 
> _______________________________________________
> CIG-SHORT mailing list
> CIG-SHORT at geodynamics.org <mailto:CIG-SHORT at geodynamics.org>
> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short <http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short>
> 
> <PETSc_log_summary.txt>_______________________________________________
> CIG-SHORT mailing list
> CIG-SHORT at geodynamics.org
> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short

Charles Williams I Geodynamic Modeler
GNS Science I Te Pῡ Ao
1 Fairway Drive, Avalon 5010, PO Box 30368, Lower Hutt 5040, New Zealand
Ph 0064-4-570-4566 I Mob 0064-22-350-7326 I Fax 0064-4-570-4600
http://www.gns.cri.nz/ <http://www.gns.cri.nz/> I Email: C.Williams at gns.cri.nz <mailto:your.email at gns.cri.nz>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.geodynamics.org/pipermail/cig-short/attachments/20180209/3f899fc8/attachment-0001.html>