[aspect-devel] Active tracers with AMR

[Lev Karatun]

Hi Timo,

just a short update: the model works on 2 and 4 processes and crashes on 8
and 12. Let me know if you would like me to test anything else.

Best regards,
Lev Karatun.

2017-11-30 12:01 GMT-05:00 Lev Karatun <lev.karatun at gmail.com>:

> Hi Timo,
>
> memory is what I thought about at first, but usually I would get some
> "oom-killer invoked" warnings, or a segfault. This time I'm not seeing any
> of it. 30 GB doesn't sound too bad because I was trying to run it on
> machines with 64 and 128 GB of RAM. I also tried the same setup with 1e6
> tracers (instead of 1e7), and it crashed the same way, so I don't think
> memory is the reason here.
> As for the dependence on the number of processors, I tried to run it with
> 4 and it worked (or at least didn't crash on initial refinement). I'll try
> other values and let you know.
> I also attached a prm files that doesn't require my plugins (looks like
> you already made the same one yourself, but just in case).
>
> Best regards,
> Lev Karatun.
>
> 2017-11-29 22:08 GMT-05:00 Timo Heister <heister at clemson.edu>:
>
>> Lev,
>>
>> I ran a modified example here successfully. Could it be that you are
>> running out of memory? You are generating 30+GB of particle output for
>> a single time step and this has to fit into main memory temporarily.
>>
>> On Wed, Nov 29, 2017 at 9:14 PM, Timo Heister <heister at clemson.edu>
>> wrote:
>> > Lev,
>> >
>> > first, is this reproducible? Does the number of processors matter?
>> >
>> > Also, I don't have the necessary plugins to run this here. Can you
>> > also share the output.txt so I can see when the problem happens?
>> >
>> >
>> >
>> > On Wed, Nov 29, 2017 at 7:55 PM, Lev Karatun <lev.karatun at gmail.com>
>> wrote:
>> >> Hi Timo,
>> >>
>> >> sure, I attached the prm file.
>> >>
>> >> Best regards,
>> >> Lev Karatun.
>> >>
>> >> 2017-11-29 16:06 GMT-05:00 Timo Heister <heister at clemson.edu>:
>> >>>
>> >>> Lev,
>> >>>
>> >>> can you share the complete setup with us (or me privately)?
>> >>>
>> >>> On Nov 29, 2017 16:04, "Lev Karatun" <lev.karatun at gmail.com> wrote:
>> >>>
>> >>> Hi everyone,
>> >>>
>> >>> I was trying to run a model with active tracers and AMR, but it
>> crashes at
>> >>> initial adaptive refinement step. This is the error that I get:
>> >>>>
>> >>>> [titan:42726] *** An error occurred in MPI_Irecv
>> >>>> [titan:42726] *** reported by process [140028719661057,214748364813]
>> >>>> [titan:42726] *** on communicator MPI COMMUNICATOR 4 DUP FROM 3
>> >>>> [titan:42726] *** MPI_ERR_COUNT: invalid count argument
>> >>>> [titan:42726] *** MPI_ERRORS_ARE_FATAL (processes in this
>> communicator
>> >>>> will now abort,
>> >>>> [titan:42726] ***    and potentially your MPI job)
>> >>>
>> >>>
>> >>> Here are the relevant model parameters:
>> >>>
>> >>> subsection Particles
>> >>> set Number of particles = 1e7
>> >>> set List of particle properties = velocity, initial composition,
>> >>> composition, initial position, integrated strain, integrated strain
>> >>> invariant
>> >>> set Interpolation scheme = cell average
>> >>> set Particle generator name = random uniform
>> >>> set Minimum particles per cell = 10
>> >>> set Time between data output = 1e6
>> >>> set Particle weight = 20
>> >>> end
>> >>>
>> >>> I tried the same model without AMR (refinement 5+0 instead of 4+1)
>> and it
>> >>> works fine. If anyone has any ideas on what I can try to do to fix the
>> >>> problem, I'd appreciate it!
>> >>>
>> >>> Best regards,
>> >>> Lev Karatun.
>> >>>
>> >>> _______________________________________________
>> >>> Aspect-devel mailing list
>> >>> Aspect-devel at geodynamics.org
>> >>> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.ge
>> odynamics.org_cgi-2Dbin_mailman_listinfo_aspect-2Ddevel&d=
>> DwIBaQ&c=Ngd-ta5yRYsqeUsEDgxhcqsYYY1Xs5ogLxWPA_2Wlc4&r=c08Bt
>> fq4m9QEScXN3ZQwLZzzWQE7S8CYq1IYuzKV_Zk&m=Nzx2c0rBG2QQP1bLQIJ
>> JDWOm7wK8iucukzdJSquBDxk&s=YzWTGgS3Z3UVS6-Ka5BnsSq2o69yCsS5aZjc62XnzXA&e=
>> >>>
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> Aspect-devel mailing list
>> >>> Aspect-devel at geodynamics.org
>> >>> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.ge
>> odynamics.org_cgi-2Dbin_mailman_listinfo_aspect-2Ddevel&d=
>> DwIBaQ&c=Ngd-ta5yRYsqeUsEDgxhcqsYYY1Xs5ogLxWPA_2Wlc4&r=c08Bt
>> fq4m9QEScXN3ZQwLZzzWQE7S8CYq1IYuzKV_Zk&m=Nzx2c0rBG2QQP1bLQIJ
>> JDWOm7wK8iucukzdJSquBDxk&s=YzWTGgS3Z3UVS6-Ka5BnsSq2o69yCsS5aZjc62XnzXA&e=
>> >>
>> >>
>> >>
>> >> _______________________________________________
>> >> Aspect-devel mailing list
>> >> Aspect-devel at geodynamics.org
>> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.ge
>> odynamics.org_cgi-2Dbin_mailman_listinfo_aspect-2Ddevel&d=
>> DwIBaQ&c=Ngd-ta5yRYsqeUsEDgxhcqsYYY1Xs5ogLxWPA_2Wlc4&r=c08Bt
>> fq4m9QEScXN3ZQwLZzzWQE7S8CYq1IYuzKV_Zk&m=Nzx2c0rBG2QQP1bLQIJ
>> JDWOm7wK8iucukzdJSquBDxk&s=YzWTGgS3Z3UVS6-Ka5BnsSq2o69yCsS5aZjc62XnzXA&e=
>> >
>> >
>> >
>> > --
>> > Timo Heister
>> > http://www.math.clemson.edu/~heister/
>>
>>
>>
>> --
>> Timo Heister
>> http://www.math.clemson.edu/~heister/
>> _______________________________________________
>> Aspect-devel mailing list
>> Aspect-devel at geodynamics.org
>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/aspect-devel
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.geodynamics.org/pipermail/aspect-devel/attachments/20171201/45e695ad/attachment.html>