[hatari-devel] Automated testing of Atari software with Hatari - Some tips

[Christian Zietz <czietz@xxxxxxx>]

Hello,

recently, I was working on a low-level software routine that I wanted to
test for compatibility with as many TOS versions as possible. In order
to automate this task, I put Hatari to use. I want to give a short
summary as a reference in case someone else wants to do the same.

* Writing the test:
I wrote some test cases that exercise my routine and detect whether the
tests pass or fail. To output the status, I considered two options:
- Regular output to the screen via GEMDOS function Cconws.
- Output to the MIDI port via XBIOS function Midiws, as suggested by Eero.
In the end, I decided to keep both in. In case I want to run the test on
actual hardware, I'll still have the screen output.
The only Hatari-specific addition is at the end of my test cases: a call
to XBIOS function #255, passing a pointer to the string
"hatari-debug q\n" on the stack. This will terminate Hatari, when BIOS
intercept is on. Again, I also call GEMDOS function Pterm afterwards, in
case the test runs on real HW.

* Running the test:
Since my routine does not need any AES functions, I can run it from the
AUTO folder. Hence I created such a directory and copied my test.prg
into it. A GEM program could presumably be run with the "--auto" option
instead.

I call Hatari from the directory containing my AUTO folder with...

hatari -d . -t TOS --bios-intercept on --midi-out midi.txt --log-level
fatal --alert-level fatal --fast-forward on --run-vbls 7500 --fast-boot on
.... where "TOS" is replaced by the image of the TOS version I want to
test with. (In reality this runs in a loop over a lot of different TOS
versions.) Further options could be added to specify amount of RAM,
screen type, etc., as needed by the test cases.

The "--run-vbls 7500" option is so that if my test crashes, Hatari does
still eventually terminate. If your test runs for a long time, you need
to increase the number. Most other options are to speed Hatari up and to
remove any clutter/distraction from the output.

If I hadn't been interested in TOS 1.00, I could also have skipped the
MIDI output and redirection and I could have added "--conout 2" instead
to the command line to intercept the VT52 console output. However, due
to limitations of TOS 1.00, this does not work; which is why I used the
MIDI port as suggested by Eero. (He mentioned that for this to work
Hatari needs to be built without portmidi support. Fortunately, this is
the case for the Windows version, anyway.)

Hatari will overwrite the MIDI output file ("midi.txt" in my case), so
after each test run, I append its contents to my test log.

With all that I can automatically verify that my routine passes all test
cases under all considered TOS versions.

I hope someone will find this helpful,
Christian