|Re: [hatari-devel] The new "make test" automatic regression test suite|
[ Thread Index |
| More lists.tuxfamily.org/hatari-devel Archives
On 04/15/2018 12:02 PM, Thomas Huth wrote:
Am Sun, 15 Apr 2018 02:40:41 +0300
schrieb Eero Tamminen <oak@xxxxxxxxxxxxxx>:
I don't see why ROM would need to be in repository, or fetched
Tests needing real ROM could be executed only if there's a TOS ROM
in some specific locations, otherwise there would be just a warning
about missing the ROM, and a pointer where to get EmuTOS one.
Good point of this scheme is that everybody could be using his
own favorite TOS version for this, so there would be better
coverage. These tests might also use user's own Hatari config
for them, to increase coverage further.
If test requires some specific TOS version, it should check
that correct one is used, and skip otherwise.
Thomas, what do you think?
I don't think that this is a good idea for the "built-in" regression
test suite. You want *reproducible* tests there, and if everybody is
using a different TOS version, you can not really achieve this goal.
Of course it's reproducible. Testing just needs to record the used
TOS version, so one knows which versions is needed for reproducing.
This kind of (recorded) test-set randomizing is commonly used in
CI & QA when the full amount of testing takes too long to be run
on every commit (CI) or day (QA).
Testing with real TOS is certainly a good idea, but this should not be
part of the "make test" regression tests - thus testing with real TOS
ROM images should be done in a different project / repository instead.