[aspect-devel] problem with ASPECT - the iterative Stokes solver did not converge

[Wolfgang Bangerth]

> It works very well without adaptive refinement
>
>     set Initial global refinement = 2
>     set Initial adaptive refinement = 0
>     set Strategy = viscosity
>     set Time steps between mesh refinement = 0
>
>
> For example, I run a test using 12 CPUs and 2hrs wall time and the code
> processed ~5 Myr with no problems.
>
>     Timestep 72:  t=5.13532e+06 years
>     Solving temperature system... 15 iterations.
>     Rebuilding Stokes preconditioner...
>     Solving Stokes system... 30+14 iterations.

OK, this at least gives you a way forward until we fix the issue with the 
adaptive refinement.


> I have, however, noticed that the code crashes without error message using a
> larger level for Initial global refinement(=4). Following your comment, "I
> assume the job gets killed because it uses too much memory", I run another
> test using 48 CPUs and here are my observations (some of them are trivial, I
> hope you don't mind):
> 1. The number of active cells and the number of degrees of freedom for Initial
> global refinement=4 are greater than that for Initial global refinement=2

Of course. Every refinement replaces each cell by 8 more. So two more 
refinements lead to 64 times as many cells and roughly 64 times the number of 
processors.


> 2. The results for the first time step (Timestep=0) are quite different
>
>     Initial global refinement=2
>     RMS, max velocity: 0.185 m/year, 0.678 m/year
>     Temperature min/avg/max: 300 K, 1923 K, 3565 K
>     Heat fluxes through boundary parts: -1.685e+13 W, 1.59e+13 W
>
>     Initial global refinement=4
>     RMS, max velocity: 0.0216 m/year, 0.0846 m/year
>     Temperature min/avg/max: 300 K, 1977 K, 3565 K
>     Heat fluxes through boundary parts: -6.404e+13 W, 6.162e+13 W

This typically indicates that your mesh with refinement=2 was too coarse to 
accurately resolve the solution. You will probably be able to see this 
visually as well if you plot the solution.

Of course, this raises the question whether the mesh is sufficiently fine even 
with refinement=4. To find out, you could compare whether the solution changes 
between refinement=3 and =4, or try refinement=5.


> 3. There is a long delay in postprocessing for Initial global refinement=4
>
>       -rw-r--r-- 1 glisovic sxf-902-01 30 Aug 6 06:51 solution.visit
>
>       -rw-r--r-- 1 glisovic sxf-902-01 29928 Aug 6 07:19 depth_average.gnuplot
>
> i.e., there is a difference of ~30 min in writing between the solution file
> and depth-average file.

I think I have seen something similar in the past. It may be that computing 
the depth averages is not efficient in parallel or 3d. Are you running in 
debug or optimized mode? (See the manual if you don't know what that means.)


> 4. A simulation with Initial global refinement=4 generates ~1.7 Myr in only 2
> time steps over 2hrs wall time
>
>     Timestep 2:  t=1.70568e+06 years
>     Solving temperature system... 16 iterations.
>     Rebuilding Stokes preconditioner...
>     Solving Stokes system... 30+23 iterations.
>
> this implies a significant difference in time step between these two global
> refinements.

If you make the mesh finer by a factor of 2 (i.e., one more refinement, or 8 
times as many cells) then you also need to make the time step smaller by a 
factor of 2. Since in your case, the velocity also changes significantly, the 
time step may also be affected by this.


> Just to add, using the same viscosity profile that I'm trying to test here
> with Aspect Glisovic et al. (GJI, 2012) suggested that "a fully resolved
> solution of thermal convection dynamics in the upper mantle may require a
> horizontal expansion of the field variables up to spherical harmonic degree
> 512 (corresponding to a horizontal resolution scale of less than half a degree
> on the sphere." So, maybe it is a resolution issue, after all?

Quite possibly. If you visualize the solution, what is the resolution you have 
with refinement=4?

Best
  Wolfgang


-- 
------------------------------------------------------------------------
Wolfgang Bangerth               email:            bangerth at math.tamu.edu
                                 www: http://www.math.tamu.edu/~bangerth/