Re: [hatari-devel] Recent Hatari change problems

[Eero Tamminen <oak@xxxxxxxxxxxxxx>]

Hi,

(Sorry for the verbose reply, feel free to skip to bottom if in hurry.)

On 04/30/2018 09:59 AM, Thomas Huth wrote:
> Am Sun, 29 Apr 2018 20:16:28 +0300
> schrieb Eero Tamminen <oak@xxxxxxxxxxxxxx>:
> [...]
> > > > As all the shell files are now fine, I think best would be
> > > > to have *one* test (script) that checks all shell scripts
> > >
> > > Yes, makes sense. I was also not very happy about having to
> > > encode the file name as test name, so I've put the stuff now
> > > into a separate script as you suggested.
> >
> > It's a bit slow, compared to other tests.
>
> Oh, c'mon, it's just 2 seconds. Sure, it's slower than the other tests,
> but this test certainly does not run forever.

On my machine it's 5 seconds (4 seconds on second run), which
is 40x longer than any of the other 36 tests, and more than
half of the whole testing time.


> > Does CTest support multiple test targets, so that one could
> > split tests e.g. to sets that are run before release (all),
> > and things that should be run before commits?
>
> Not that I'm aware of. Looks like you can run tests based on applying a
> regex on the test name, but apart from that I haven't seen such a
> feature.
>
> > IMHO checkbashism is more of a release check than commit check.
>
> The point of having regression tests is that you check it regularly,
> not just right before a release.

It's normally automated, especially if it takes longer time
to complete.

-> It would be nice if e.g. antartica machine could run full
   test-set daily and mail hatari-devel if there's an error.


> Otherwise, you could also say that
> e.g. XBIOS interception hardly changes, so it should also be tested
> right before the release only etc.

XBIOS test is nearly instant, its functionality is part of Hatari
itself, and not checked otherwise.  Whereas most scripts (= tests)
aren't part of Hatari install, and test scripts not running would be
found pretty much right away, because they're run "constantly".

Main reason why it could be a release test is that errors in scripts
reported by checkbashism are trivial to fix because it already tells
on which line they are, and there's no need to bisect when they've
started to regress (like might be needed with emulation bugs).


> > Alternatively, is there some nice way to run checks in parallel
>
> Yes, simply run: ctest -j4

This reduces run-time to the time it takes to run checkbashism
i.e. only halves the total.

Now I know why you were originally splitting each script to
a separate test. ;-)


> By the way, ctest also has a nice man page in case you haven't
> found it yet ;-)

Thanks, my "problem" is solved with:
	ctest -E bash -j4


	- Eero