Re: [hatari-devel] The new "make test" automatic regression test suite

[Thomas Huth <th.huth@xxxxxxxxx>]

Am Sun, 15 Apr 2018 12:47:43 +0300
schrieb Eero Tamminen <oak@xxxxxxxxxxxxxx>:

> Hi,
> 
> On 04/15/2018 12:02 PM, Thomas Huth wrote:
> > Am Sun, 15 Apr 2018 02:40:41 +0300
> > schrieb Eero Tamminen <oak@xxxxxxxxxxxxxx>:  
> >> I don't see why ROM would need to be in repository, or fetched
> >> automatically.
> >>
> >> Tests needing real ROM could be executed only if there's a TOS ROM
> >> in some specific locations, otherwise there would be just a warning
> >> about missing the ROM, and a pointer where to get EmuTOS one.
> >>
> >> Good point of this scheme is that everybody could be using his
> >> own favorite TOS version for this, so there would be better
> >> coverage. These tests might also use user's own Hatari config
> >> for them, to increase coverage further.
> >>
> >> If test requires some specific TOS version, it should check
> >> that correct one is used, and skip otherwise.
> >>
> >> Thomas, what do you think?  
> > 
> > I don't think that this is a good idea for the "built-in" regression
> > test suite. You want *reproducible* tests there, and if everybody is
> > using a different TOS version, you can not really achieve this
> > goal.  
> 
> Of course it's reproducible.  Testing just needs to record the used
> TOS version, so one knows which versions is needed for reproducing.

There are also bad and patched ROM dumps available in the internet ...
so just recording the TOS version would not be enough. You'd also need
to log/verify the md5sum of such files to make sure that you have the
right version...

 Thomas