[hatari-devel] The new "make test" automatic regression test suite

[Thomas Huth <th.huth@xxxxxxxxx>]

 Hi all,

if you've followed the mercurial repository (or Eero's upset
e-mails ;-)) during the past weeks, you've maybe already noticed it:
There is now a way to run regression tests automatically in the Hatari
source repository. I already wanted to share some information
about this earlier, but did not have enough time yet, but here's now
finally a little bit background information about the why and how:

Each bigger software project sooner or later faces the problem that
new regression bugs creep in while you develop new features, fix other
bugs, or let's rather say change code in general. These bugs go
unnoticed until somebody runs the software in a way that triggers them.
So you need to test software to make sure that there are no new
regressions. Testing software is boring, time-consuming and cumbersome,
so the normal software developer normally only tests the bare minimum
(in the worst case, just: "it compiles, so it should work" ;-)). Thus
for bigger project, you need a way to run tests in an automated
fashion, so that it is very easy to run tests, or that you can even
build a CI (Continuous Integration) system that runs the test before or
at least after each repository check-in (we're not there yet, but the
new "make test" suite is a first step into that direction).

We already had some few test cases in the Hatari repository, but no
automated way to check them, so they were partly in a rather bad,
bit-rotten shape. Now since our build system is based on CMake, there
is a very easy way (the so-called CTest framework) to integrate tests
into your build system - this required just some few tweaks to the
CMakeLists.txt files and it was possible to run the tests from the
tests/debugger/ folder (which link directly with the debugger .c files
from Hatari to exercise the functions in there) in a nice and easy
way by typing "make test" in the root directory of the Hatari
repository. (alternatively, you can also run "ctest" directly, e.g.
with the "-v" flag to get more verbose output).

The other tests were more problematic, though: These tests (buserror,
natfeats and xbios-dbmsg) are not native code that directly links to
Hatari, but rather 68k TOS programs that need to run within the
emulator. And for this, you need a TOS ROM. So to run these tests
automatically, the most obvious solution would be to include a TOS ROM
into the Hatari repository (i.e. EmuTOS for legal reasons). But that
has a big drawbacks: The ROMs are rather big, 512k for the version
that supports all machines, so each time we have to update the ROM
image, the Hatari repository would grow by half a meg, which I really
would like to avoid. The alternative would be to add some logic to the
test suite to download the ROM from the internet automatically if it is
not locally available yet. But that's also ugly, since you then can
only initially run the regression test suite if an internet connection
is available, and you depend on something that is not part of the
repository (e.g. if the URL does not work anymore for some reason, the
tests are broken).

So I came up with another way to be able to run the 68k programs
automatically: The "--tos none" mode. The idea here is that the PRGs
are rather simple, they don't have a fancy GEM GUI and only use some
few GEMDOS OS calls to do file I/O or switch to supervisor mode etc.
Now, we already implement many GEMDOS calls for our GEMDOS hard disk
emulation, so I thought that it should be possible to run a PRG by only
using that GEMDOS emulation layer, without real TOS ROM loaded into the
emulator. And indeed, after implementing a relocation function for
the PRG files and implementing very, very small fake ROM that jumps to
the relocated PRG in memory, I was indeed able to run the PRGs in the
emulator without loading a TOS ROM image!

The disadvantage here is that you can only use PRG files that use the
supported subset of the GEMDOS emulation layer in Hatari. That's also
why I rewrote one of the test PRGs in assembly instead of using C - the
start-up code of C programs might do different things with each version
or type of C compiler, so this might break each time you upgrade your C
compiler or use a different one. With assembler code, you can exactly
say for sure which OS calls are used, so this is currently much better
suited for this "--tos none" mode.

Now how is this all glued to the "make test" framework? In each of the
tests folder, there is a little run_test.sh shell script which is
responsible for running hatari with the right parameters. Once the test
is done (e.g. the PRG file executed the PTerm GEMDOS function - Hatari
then exits automatically in the "--tos none" mode), the shell script
compares the output of Hatari (or in the case of the buserror tester
the files that are created by the test PRG) with some expected results,
and if the results match, exit the shell script with 0. If the results
do not match, the shell script is ended with a non-zero value - the
testing framework will then report a failed test.

To be able to compare the output of Hatari with expected results (with
the "diff" program for example), it is now also important that there is
a way to run Hatari in which it really only outputs the expected
strings. We already have a logging subsystem in Hatari, so
theoretically, Hatari should not print any chatty text when you run it
with "--log-level fatal" (unless you hit a fatal error, of course,
which should hardly happen). Unfortunately, the Hatari sources were
quite full with random fprintf statements and the like, which did not
care about log levels at all - and thus would have make checking the
output of Hatari way more cumbersome. So one of the things that I did
during the past weeks and months was to clean up these fprintf
statements and turn them into proper Log_Printf messages instead.

Now what does this all mean for you?

- If you write new patches for Hatari, please run "make test" before
  contributing the patches to make sure that there is no regression.

- Please make sure to use proper Log_Printf functions for printing
  stuff and remove or comment-out debugging printf/fprintf statements
  before contributing patches.

- If you have some small tests that you've written in the past to check
  the behavior of real hardware to compare it with the behavior of
  Hatari, and this test program only uses the typical GEMDOS OS calls,
  please consider to contribute the test program for the Hatari
  regression test suite, so that we could run it automatically during
  "make test" in the future.

 Thanks,
  Thomas