[aspect-devel] convection-box-3d example hangs when more than one node is used
Robert Moucha
rmoucha at gmail.com
Fri Jul 31 13:51:14 PDT 2015
Finally got a chance to get back to this after some travel:
I tried step-32 example of deal.II and it runs without a problem on
more than one compute node.
ASPECT is the default 1.3 version, the problem occurs with both debug
and release versions.
To compile ASPECT I'm using:
gcc 4.4.7
BLAS and LAPACK 3.2.1-4
OpenMPI 1.8.4
Trilinos 12.0.1 with CXX11=OFF
p4est 1.1
pdhf5 1.8.15
deal.ii 8.2.1
no problems with compiling these as far as I could tell
When ASPECT hangs, right after initial time step, all the files
solution-00000 for all processors are written without issue as well as
other files, then it just hangs.
Thanks,
Rob
> Message: 1
> Date: Sun, 19 Jul 2015 12:25:14 +0200
> From: Rene Gassmoeller <rengas at gfz-potsdam.de>
> To: aspect-devel at geodynamics.org
> Subject: Re: [aspect-devel] convection-box-3d example hangs when more
> than one node is used
> Message-ID: <55AB7B0A.8040503 at gfz-potsdam.de>
> Content-Type: text/plain; charset=utf-8
>
> Hi Rob,
> I just checked the convection-box-3d on 2 nodes of our cluster and it
> runs fine. So it seems there is something special about your
> installation or there is a bug that is only showing in this
> configuration. Could you test the following things for us to be able to
> give you some more help:
>
> 1. Try running convection-box with an ASPECT compiled in debug mode.
> Maybe there is some error message suppressed by the release mode.
> 2. Could you try to compile the deal.II example step-32 and run that one
> on more than one node of your cluster? It should be in your deal.II
> folder /examples/step-32, and compiling should be as simple as 'cmake .
> && make'. This will give us some insight if something in aspect is
> causing the problem or if it is an issue with the deal.II code or
> configuration.
> 3. We need some more information on your deal.II configuration (your
> ASPECT is an unchanged 1.3, right?). Which version of deal.II are you
> using? Which trilinos, p4est and compiler? Were there any problems
> during compiling those?
>
> Best,
> Rene
>
> On 07/18/2015 12:23 AM, Robert Moucha wrote:
>> Hi Timo,
>>
>> Yes I still have the same problem. It occurs with the following cook
>> books (have not tried all, but it looks like anything to do with time
>> stepping is causing the hang):
>>
>> convection-box
>> convection-box-3d
>> shell_simple_2d
>> van-keken-discontinuous
>>
>> Thanks
>> Rob
>>
>>> Hey Robert,
>>>
>>> sorry for only getting back to this now. Any update on your problem?
>>> Does this happen with every .prm file (like a simple 2d problem)?
>>>
>>> On Sun, Jul 5, 2015 at 6:10 PM, Robert Moucha <rmoucha at gmail.com> wrote:
>>>> OK, it appears that I solved last-weeks issue with the files, turns
>>>> out one of the nodes did not have the correct paths (thanks).
>>>>
>>>> However, now I am still having problems when using more than one node,
>>>> this time ASPECT just hangs on time step 1, no error, the
>>>> solution-00000 files are created on each of the nodes than nothing.
>>>>
>>>> It runs fine on a single node. I should point out that the ASPECT
>>>> example stokes.prm as well as Citcoms runs on the cluster without
>>>> issues.
>>>>
>>>> Here is the log.txt for the convection-box-3d.prm -- thanks in advance Rob
>>>>
>>>> -----------------------------------------------------------------------------
>>>> -- This is ASPECT, the Advanced Solver for Problems in Earth's ConvecTion.
>>>> -- . version 1.3
>>>> -- . running in OPTIMIZED mode
>>>> -- . running with 12 MPI processes
>>>> -- . using Trilinos
>>>> -----------------------------------------------------------------------------
>>>>
>>>> Number of active cells: 512 (on 4 levels)
>>>> Number of degrees of freedom: 20381 (14739+729+4913)
>>>>
>>>> *** Timestep 0: t=0 seconds
>>>> Solving temperature system... 0 iterations.
>>>> Rebuilding Stokes preconditioner...
>>>> Solving Stokes system... 29 iterations.
>>>>
>>>> Number of active cells: 1583 (on 5 levels)
>>>> Number of degrees of freedom: 63622 (46077+2186+15359)
>>>>
>>>> *** Timestep 0: t=0 seconds
>>>> Solving temperature system... 0 iterations.
>>>> Rebuilding Stokes preconditioner...
>>>> Solving Stokes system... 30+4 iterations.
>>>>
>>>> Number of active cells: 3256 (on 5 levels)
>>>> Number of degrees of freedom: 122269 (88647+4073+29549)
>>>>
>>>> *** Timestep 0: t=0 seconds
>>>> Solving temperature system... 0 iterations.
>>>> Rebuilding Stokes preconditioner...
>>>> Solving Stokes system... 30+4 iterations.
>>>>
>>>> Number of active cells: 9010 (on 6 levels)
>>>> Number of degrees of freedom: 333145 (241677+10909+80559)
>>>>
>>>> *** Timestep 0: t=0 seconds
>>>> Solving temperature system... 0 iterations.
>>>> Rebuilding Stokes preconditioner...
>>>> Solving Stokes system... 30+4 iterations.
>>>>
>>>> Postprocessing:
>>>> RMS, max velocity: 57.6 m/s, 176 m/s
>>>> Temperature min/avg/max: 0 K, 0.5 K, 1 K
>>>> Heat fluxes through boundary parts: 7.682e-07 W, -7.682e-07 W,
>>>> 1.685e-15 W, 2.362e-15 W, -1 W, 1 W
>>>> Writing graphical output:
>>>> /state/partition1/RMOUCHA/output/solution-00000
>>>>
>>>> *** Timestep 1: t=8.87115e-05 seconds
>>>>
>>>>
>>>> ------------------------------------------------------------
>>>> Robert Moucha
>>>> Assistant Professor of Geophysics
>>>> Department of Earth Sciences
>>>> 204 Heroy Geology Lab
>>>> Syracuse University
>>>> Syracuse, NY, 13244-1070
>>>> _______________________________________________
>>>> Aspect-devel mailing list
>>>> Aspect-devel at geodynamics.org
>>>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/aspect-devel
>>>
>>>
>>> ------------------------------
>>>
>>> Subject: Digest Footer
>>>
>>> _______________________________________________
>>> Aspect-devel mailing list
>>> Aspect-devel at geodynamics.org
>>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/aspect-devel
>>>
>>> ------------------------------
>>>
>>> End of Aspect-devel Digest, Vol 44, Issue 9
>>> *******************************************
>>
>>
>>
>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> Aspect-devel mailing list
> Aspect-devel at geodynamics.org
> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/aspect-devel
>
> ------------------------------
>
> End of Aspect-devel Digest, Vol 44, Issue 11
> ********************************************
--
------------------------------------------------------------
Robert Moucha
Assistant Professor of Geophysics
Department of Earth Sciences
204 Heroy Geology Lab
Syracuse University
Syracuse, NY, 13244-1070
More information about the Aspect-devel
mailing list