Re: [hatari-devel] The new "make test" automatic regression test suite

[Thomas Huth <th.huth@xxxxxxxxx>]

Am Sun, 15 Apr 2018 02:40:41 +0300
schrieb Eero Tamminen <oak@xxxxxxxxxxxxxx>:

> Hi,
> 
> On 04/14/2018 06:24 PM, Thomas Huth wrote:
> > Am Sat, 14 Apr 2018 17:02:54 +0200
> > schrieb Thorsten Otto <admin@xxxxxxxxxxx>:  
> >> On Samstag, 14. April 2018 14:05:52 CEST Thomas Huth wrote:  
> >>> The disadvantage here is that you can only use PRG files that use
> >>> the supported subset of the GEMDOS emulation layer in Hatari.  
> >>
> >> But that also means your test environment is quite different from
> >> "normal" usage. Actually, a lot of Hatari cannot be testet this way
> >> (i didn't look at your mini-ROM yet, but i doubt that it tries to
> >> do much hardware initialization, like a real TOS would do). You
> >> don't even know if it is able to boot the ROM, according to the
> >> settings.  
> > 
> > That's not the purpose of the small test PRGs that we have in the
> > tests folder: They check special stuff like bus errors, certain CPU
> > opcodes (natfeats) or emulated OS calls (xbios) - and for this you
> > certainly don't need a full-blown TOS ROM.
> > 
> > For testing the boot process of real TOS ROMs, we also have got the
> > "tosboot" tester. But for this, you need to point it to a set of TOS
> > ROM images that should be tested, so this is not includable in the
> > automatic test suite.  
> 
> Maybe Troed's system could run tosboot tester with few TOS ROMs
> e.g. once a week or month, and mail the report to Hatari devel
> (with link to directory containing the screenshots)?
> 
> 
> 
> >> PS.: once the tests run automatically, it maybe be even possible to
> >> download some "official" ROM from a known location. You will need
> >> an internet connection anyway to clone the repo, and as long as
> >> you are not distributing the ROMs, there should be no legal issues
> >> (at least i think so)  
>  >
> > First, mercurial can also be used offline as long as you don't want
> > to push/fetch stuff to/from the repository server. And believe it
> > or not, sometimes I even still work without internet, e.g. while
> > traveling. Additionally, adding a hard dependency on a third-party
> > website which is distributing legally debatable content should be
> > really avoided. So no, this idea is definitely a no-go.  
> 
> I don't see why ROM would need to be in repository, or fetched
> automatically.
> 
> Tests needing real ROM could be executed only if there's a TOS ROM
> in some specific locations, otherwise there would be just a warning
> about missing the ROM, and a pointer where to get EmuTOS one.
> 
> Good point of this scheme is that everybody could be using his
> own favorite TOS version for this, so there would be better
> coverage. These tests might also use user's own Hatari config
> for them, to increase coverage further.
> 
> If test requires some specific TOS version, it should check
> that correct one is used, and skip otherwise.
> 
> Thomas, what do you think?

I don't think that this is a good idea for the "built-in" regression
test suite. You want *reproducible* tests there, and if everybody is
using a different TOS version, you can not really achieve this goal.

Testing with real TOS is certainly a good idea, but this should not be
part of the "make test" regression tests - thus testing with real TOS
ROM images should be done in a different project / repository instead.

 Thomas