[aspect-devel] Aspect 2.0 tests failure question

[Rene Gassmoeller]

Hi all,

let me add to this. I agree with Wolfgang that a relaxed tolerance would
probably need to be around O(1), because we have test output that should
be technically zero, but due to floating point imprecision, and due to
the nature of the iterative solvers, is around 1e-18. The slightest
change in iteration count can change this to 2e-18 or 0.5e-18, which
technically is still zero, but drastically different from the reference
result. Thus, I do not think a relaxed tolerance will be helpful. One
option we already have is to just run all of the tests and check for
crashes, but do not compare the output results. You can use this by
running cmake with the '-D ASPECT_RUN_ALL_TESTS=ON -D
ASPECT_COMPARE_TEST_RESULTS=OFF' flags. This will at least tell you if
all tests run. We can of course discuss making this the default option
if 'make tests' is run, but I also think this can be misleading, because
'running the tests' then actually does not include any comparison. What
is your opinion?

@Ravi: 35% failures is actually a pretty normal number, I have seen
clusters with more failures. To get to 100% passes you can also run the
official ASPECT tester on your local machine if you have docker
installed. Take a look at the script
'cmake/compile_and_update_tests.sh'. But this will only test the ASPECT
code, not your local installation, because it does everything inside of
the docker container.

Best,

Rene



On 05/11/2018 05:36 AM, Wolfgang Bangerth wrote:
>
> Hi Bob,
>
>> Wolfgang, do you think it might be valuable to add a relaxed
>> tolerance to the tests for end users, to suppress the non critical
>> failures? We could always keep the pure diff when testing on Timo's
>> server.
>>
>> I could play with this at the start of the Hackathon if you wanted,
>> in lieu of the annual typo hunt? My OCD always plays up when I see
>> failed tests!
> I think we all feel that way :-)
>
> We already do allow for a tolerance if you have the 'numdiff' program
> installed, which compares numbers based on their relative size. (If
> you don't have 'numdiff', it just uses 'diff' that does a stupid text
> match and doesn't know anything about numbers at all. So at the very
> least one would need to have 'numdiff' installed to make this useful.
>
> I don't know by how much one would need to relax the tolerance. I
> think it would make for an interesting experiment to find that out. It
> may be that there are some tests where small floating point
> differences lead to different refinement decisions -- in which case we
> enter the realm of O(1) differences. Either way, it would be
> interesting to play with it where cmake could set a default (loose)
> tolerance unless one passes a stricter tolerance requirement to cmake
> -- which is what one would do on the "official" tester.
>
> Nice project for sure!
>
> Cheers
>  W.
>

-- 
Rene Gassmoeller
https://gassmoeller.github.io/