Re: [hatari-devel] The new "make test" automatic regression test suite

[Eero Tamminen <oak@xxxxxxxxxxxxxx>]

Hi,

On 04/15/2018 01:09 PM, Thomas Huth wrote:
> Am Sun, 15 Apr 2018 12:47:43 +0300
> schrieb Eero Tamminen <oak@xxxxxxxxxxxxxx>:
> > On 04/15/2018 12:02 PM, Thomas Huth wrote:
> > > I don't think that this is a good idea for the "built-in" regression
> > > test suite. You want *reproducible* tests there, and if everybody is
> > > using a different TOS version, you can not really achieve this
> > > goal.
> >
> > Of course it's reproducible.  Testing just needs to record the used
> > TOS version, so one knows which versions is needed for reproducing.
>
> There are also bad and patched ROM dumps available in the internet ...
> so just recording the TOS version would not be enough. You'd also need
> to log/verify the md5sum of such files to make sure that you have the
> right version...

Good point.  I was thinking that we Hatari developers could split
testing of different TOS versions between us, but even for that,
it would be good to make sure that when crosschecking results,
images really are the same.


	- Eero