Re: [hatari-devel] The new "make test" automatic regression test suite

[Eero Tamminen <oak@xxxxxxxxxxxxxx>]

Hi,

On 04/15/2018 12:02 PM, Thomas Huth wrote:
> Am Sun, 15 Apr 2018 02:40:41 +0300
> schrieb Eero Tamminen <oak@xxxxxxxxxxxxxx>:
> > I don't see why ROM would need to be in repository, or fetched
> > automatically.
> >
> > Tests needing real ROM could be executed only if there's a TOS ROM
> > in some specific locations, otherwise there would be just a warning
> > about missing the ROM, and a pointer where to get EmuTOS one.
> >
> > Good point of this scheme is that everybody could be using his
> > own favorite TOS version for this, so there would be better
> > coverage. These tests might also use user's own Hatari config
> > for them, to increase coverage further.
> >
> > If test requires some specific TOS version, it should check
> > that correct one is used, and skip otherwise.
> >
> > Thomas, what do you think?
>
> I don't think that this is a good idea for the "built-in" regression
> test suite. You want *reproducible* tests there, and if everybody is
> using a different TOS version, you can not really achieve this goal.

Of course it's reproducible.  Testing just needs to record the used
TOS version, so one knows which versions is needed for reproducing.

This kind of (recorded) test-set randomizing is commonly used in
CI & QA when the full amount of testing takes too long to be run
on every commit (CI) or day (QA).


> Testing with real TOS is certainly a good idea, but this should not be
> part of the "make test" regression tests - thus testing with real TOS
> ROM images should be done in a different project / repository instead.


	- Eero