|Re: [hatari-devel] The new "make test" automatic regression test suite|
[ Thread Index |
| More lists.tuxfamily.org/hatari-devel Archives
- To: hatari-devel@xxxxxxxxxxxxxxxxxxx
- Subject: Re: [hatari-devel] The new "make test" automatic regression test suite
- From: Thomas Huth <th.huth@xxxxxxxxx>
- Date: Sun, 15 Apr 2018 11:02:02 +0200
- Dkim-signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.de; s=2017; t=1523782924; bh=vXq5+BWYjq07Rp9CA7MoBhAZIXIdOrBLPpPv7PkXzuM=; h=Date:From:To:Subject:From; b=DPRau87yJuwCIliGqYE0GVmIJhFP5yjwgKCddGdSDarNWrRq0oAKcZZEGg4PgAEij 14h7HNcdje1OIxm/bZlVTMxXZzlLRb0eGL/m/yFBGqt+h6wBcF8NUMDX8pH8GSuhXH r00P7TXDXyk9ZUqciFzP5epAdJwO3HiKQ9eO7+vJI4rBHY12bnrTd7QmdrxCE9jvWl QFNXXkTplrp7mHEPAy9nMsipGSKnXVvG/+RDe5YLtoTCkJM7tGrmYiV6zAY95fAJVy 1g3E3jC4TN4SXrc+zu7UfhlZuK2io2jT9v+vYsNbi7Fyov6LuLHsUG+GfRrW+xHZ0R cosl9rGlR16RQ==
Am Sun, 15 Apr 2018 02:40:41 +0300
schrieb Eero Tamminen <oak@xxxxxxxxxxxxxx>:
> On 04/14/2018 06:24 PM, Thomas Huth wrote:
> > Am Sat, 14 Apr 2018 17:02:54 +0200
> > schrieb Thorsten Otto <admin@xxxxxxxxxxx>:
> >> On Samstag, 14. April 2018 14:05:52 CEST Thomas Huth wrote:
> >>> The disadvantage here is that you can only use PRG files that use
> >>> the supported subset of the GEMDOS emulation layer in Hatari.
> >> But that also means your test environment is quite different from
> >> "normal" usage. Actually, a lot of Hatari cannot be testet this way
> >> (i didn't look at your mini-ROM yet, but i doubt that it tries to
> >> do much hardware initialization, like a real TOS would do). You
> >> don't even know if it is able to boot the ROM, according to the
> >> settings.
> > That's not the purpose of the small test PRGs that we have in the
> > tests folder: They check special stuff like bus errors, certain CPU
> > opcodes (natfeats) or emulated OS calls (xbios) - and for this you
> > certainly don't need a full-blown TOS ROM.
> > For testing the boot process of real TOS ROMs, we also have got the
> > "tosboot" tester. But for this, you need to point it to a set of TOS
> > ROM images that should be tested, so this is not includable in the
> > automatic test suite.
> Maybe Troed's system could run tosboot tester with few TOS ROMs
> e.g. once a week or month, and mail the report to Hatari devel
> (with link to directory containing the screenshots)?
> >> PS.: once the tests run automatically, it maybe be even possible to
> >> download some "official" ROM from a known location. You will need
> >> an internet connection anyway to clone the repo, and as long as
> >> you are not distributing the ROMs, there should be no legal issues
> >> (at least i think so)
> > First, mercurial can also be used offline as long as you don't want
> > to push/fetch stuff to/from the repository server. And believe it
> > or not, sometimes I even still work without internet, e.g. while
> > traveling. Additionally, adding a hard dependency on a third-party
> > website which is distributing legally debatable content should be
> > really avoided. So no, this idea is definitely a no-go.
> I don't see why ROM would need to be in repository, or fetched
> Tests needing real ROM could be executed only if there's a TOS ROM
> in some specific locations, otherwise there would be just a warning
> about missing the ROM, and a pointer where to get EmuTOS one.
> Good point of this scheme is that everybody could be using his
> own favorite TOS version for this, so there would be better
> coverage. These tests might also use user's own Hatari config
> for them, to increase coverage further.
> If test requires some specific TOS version, it should check
> that correct one is used, and skip otherwise.
> Thomas, what do you think?
I don't think that this is a good idea for the "built-in" regression
test suite. You want *reproducible* tests there, and if everybody is
using a different TOS version, you can not really achieve this goal.
Testing with real TOS is certainly a good idea, but this should not be
part of the "make test" regression tests - thus testing with real TOS
ROM images should be done in a different project / repository instead.