[aspect-devel] Active tracers with AMR

[Timo Heister]

Lev,

1. Your .prm was generating floating point exceptions in the "simpler"
model for me: fix https://github.com/geodynamics/aspect/pull/2025
2. during refinement the memory consumption spikes to 10+GB per
process for a few seconds even with a coarser mesh and fewer
particles. I don't know why, but this can cause your processes being
killed. At least on my workstation that was a problem.
3. I am hitting "A cell and all of its neighbors do not contain any
particles. The `cell average' interpolation scheme does not support
this case." after refinement. You might not see this error because of
https://github.com/geodynamics/aspect/pull/2018 (very new).


On Fri, Dec 1, 2017 at 1:04 AM, Lev Karatun <lev.karatun at gmail.com> wrote:
> Hi Timo,
>
> just a short update: the model works on 2 and 4 processes and crashes on 8
> and 12. Let me know if you would like me to test anything else.
>
> Best regards,
> Lev Karatun.
>
> 2017-11-30 12:01 GMT-05:00 Lev Karatun <lev.karatun at gmail.com>:
>>
>> Hi Timo,
>>
>> memory is what I thought about at first, but usually I would get some
>> "oom-killer invoked" warnings, or a segfault. This time I'm not seeing any
>> of it. 30 GB doesn't sound too bad because I was trying to run it on
>> machines with 64 and 128 GB of RAM. I also tried the same setup with 1e6
>> tracers (instead of 1e7), and it crashed the same way, so I don't think
>> memory is the reason here.
>> As for the dependence on the number of processors, I tried to run it with
>> 4 and it worked (or at least didn't crash on initial refinement). I'll try
>> other values and let you know.
>> I also attached a prm files that doesn't require my plugins (looks like
>> you already made the same one yourself, but just in case).
>>
>> Best regards,
>> Lev Karatun.
>>
>> 2017-11-29 22:08 GMT-05:00 Timo Heister <heister at clemson.edu>:
>>>
>>> Lev,
>>>
>>> I ran a modified example here successfully. Could it be that you are
>>> running out of memory? You are generating 30+GB of particle output for
>>> a single time step and this has to fit into main memory temporarily.
>>>
>>> On Wed, Nov 29, 2017 at 9:14 PM, Timo Heister <heister at clemson.edu>
>>> wrote:
>>> > Lev,
>>> >
>>> > first, is this reproducible? Does the number of processors matter?
>>> >
>>> > Also, I don't have the necessary plugins to run this here. Can you
>>> > also share the output.txt so I can see when the problem happens?
>>> >
>>> >
>>> >
>>> > On Wed, Nov 29, 2017 at 7:55 PM, Lev Karatun <lev.karatun at gmail.com>
>>> > wrote:
>>> >> Hi Timo,
>>> >>
>>> >> sure, I attached the prm file.
>>> >>
>>> >> Best regards,
>>> >> Lev Karatun.
>>> >>
>>> >> 2017-11-29 16:06 GMT-05:00 Timo Heister <heister at clemson.edu>:
>>> >>>
>>> >>> Lev,
>>> >>>
>>> >>> can you share the complete setup with us (or me privately)?
>>> >>>
>>> >>> On Nov 29, 2017 16:04, "Lev Karatun" <lev.karatun at gmail.com> wrote:
>>> >>>
>>> >>> Hi everyone,
>>> >>>
>>> >>> I was trying to run a model with active tracers and AMR, but it
>>> >>> crashes at
>>> >>> initial adaptive refinement step. This is the error that I get:
>>> >>>>
>>> >>>> [titan:42726] *** An error occurred in MPI_Irecv
>>> >>>> [titan:42726] *** reported by process [140028719661057,214748364813]
>>> >>>> [titan:42726] *** on communicator MPI COMMUNICATOR 4 DUP FROM 3
>>> >>>> [titan:42726] *** MPI_ERR_COUNT: invalid count argument
>>> >>>> [titan:42726] *** MPI_ERRORS_ARE_FATAL (processes in this
>>> >>>> communicator
>>> >>>> will now abort,
>>> >>>> [titan:42726] ***    and potentially your MPI job)
>>> >>>
>>> >>>
>>> >>> Here are the relevant model parameters:
>>> >>>
>>> >>> subsection Particles
>>> >>> set Number of particles = 1e7
>>> >>> set List of particle properties = velocity, initial composition,
>>> >>> composition, initial position, integrated strain, integrated strain
>>> >>> invariant
>>> >>> set Interpolation scheme = cell average
>>> >>> set Particle generator name = random uniform
>>> >>> set Minimum particles per cell = 10
>>> >>> set Time between data output = 1e6
>>> >>> set Particle weight = 20
>>> >>> end
>>> >>>
>>> >>> I tried the same model without AMR (refinement 5+0 instead of 4+1)
>>> >>> and it
>>> >>> works fine. If anyone has any ideas on what I can try to do to fix
>>> >>> the
>>> >>> problem, I'd appreciate it!
>>> >>>
>>> >>> Best regards,
>>> >>> Lev Karatun.
>>> >>>
>>> >>> _______________________________________________
>>> >>> Aspect-devel mailing list
>>> >>> Aspect-devel at geodynamics.org
>>> >>>
>>> >>> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.geodynamics.org_cgi-2Dbin_mailman_listinfo_aspect-2Ddevel&d=DwIBaQ&c=Ngd-ta5yRYsqeUsEDgxhcqsYYY1Xs5ogLxWPA_2Wlc4&r=c08Btfq4m9QEScXN3ZQwLZzzWQE7S8CYq1IYuzKV_Zk&m=Nzx2c0rBG2QQP1bLQIJJDWOm7wK8iucukzdJSquBDxk&s=YzWTGgS3Z3UVS6-Ka5BnsSq2o69yCsS5aZjc62XnzXA&e=
>>> >>>
>>> >>>
>>> >>>
>>> >>> _______________________________________________
>>> >>> Aspect-devel mailing list
>>> >>> Aspect-devel at geodynamics.org
>>> >>>
>>> >>> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.geodynamics.org_cgi-2Dbin_mailman_listinfo_aspect-2Ddevel&d=DwIBaQ&c=Ngd-ta5yRYsqeUsEDgxhcqsYYY1Xs5ogLxWPA_2Wlc4&r=c08Btfq4m9QEScXN3ZQwLZzzWQE7S8CYq1IYuzKV_Zk&m=Nzx2c0rBG2QQP1bLQIJJDWOm7wK8iucukzdJSquBDxk&s=YzWTGgS3Z3UVS6-Ka5BnsSq2o69yCsS5aZjc62XnzXA&e=
>>> >>
>>> >>
>>> >>
>>> >> _______________________________________________
>>> >> Aspect-devel mailing list
>>> >> Aspect-devel at geodynamics.org
>>> >>
>>> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.geodynamics.org_cgi-2Dbin_mailman_listinfo_aspect-2Ddevel&d=DwIBaQ&c=Ngd-ta5yRYsqeUsEDgxhcqsYYY1Xs5ogLxWPA_2Wlc4&r=c08Btfq4m9QEScXN3ZQwLZzzWQE7S8CYq1IYuzKV_Zk&m=Nzx2c0rBG2QQP1bLQIJJDWOm7wK8iucukzdJSquBDxk&s=YzWTGgS3Z3UVS6-Ka5BnsSq2o69yCsS5aZjc62XnzXA&e=
>>> >
>>> >
>>> >
>>> > --
>>> > Timo Heister
>>> > http://www.math.clemson.edu/~heister/
>>>
>>>
>>>
>>> --
>>> Timo Heister
>>> http://www.math.clemson.edu/~heister/
>>> _______________________________________________
>>> Aspect-devel mailing list
>>> Aspect-devel at geodynamics.org
>>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/aspect-devel
>>
>>
>
>
> _______________________________________________
> Aspect-devel mailing list
> Aspect-devel at geodynamics.org
> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/aspect-devel