[aspect-devel] Aspect 2.0 tests failure question

[Bob Myhill]

On 14 May 2018 at 19:00, Rene Gassmoeller <rene.gassmoeller at mailbox.org>
wrote:

> On 05/14/2018 05:35 AM, Wolfgang Bangerth wrote:
> > That's not quite correct: we give numdiff both a relative and an
> > absolute tolerance. Numbers are considered equal if they satisfy
> > either of these criteria. I don't quite remember what the absolute
> > tolerance is, but let's say it is 1e-6, then the numbers
> >   1.234e-9
> > and
> >   2.345e-11
> > are considered equal, even though their *relative* difference is quite
> > large.
> >
>
> Oh, you are right, we use an absolute tolerance for diffs of 1e-6 and a
> relative tolerance of 1e-8. Still there are tests that for example
> change their number of solver iterations from 6 to 7 (i.e. ~15%
> difference). We could add special cases for these columns, but since
> results will change slightly, subsequent timestep sizes will be
> different and everything will change. Maybe instead we should collect a
> list of tests that are likely to succeed on all systems (like direct
> solver, single timestep tests) and add them to the list of quick tests.
>
>
I like that idea. It might also be possible to run a double comparison to
make things more automated (and avoid creating a list which will get
outdated)... something like

ignored_tests = empty list
for test in tests:
    run test
    n_iterations_test_current = list of all iteration counts in test output
    n_iterations_test_reference = same, but grabbed from reference output

    if all(n_iterations_test_current == n_iterations_test_reference):
        do numdiff comparison
    else:
        ignored_tests.append(test_name)

maybe do something with the list of ignored tests


This still doesn't rule out the possibility that something is broken in the
ignored tests, but it might give those of us with a nervous disposition
(hi!) some confidence that we haven't messed up our installation.

Best wishes,
Bob
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.geodynamics.org/pipermail/aspect-devel/attachments/20180514/8305f57b/attachment.html>