Re: [hatari-devel] hatari release tester (was: problems in b/w mode)

[Eero Tamminen <oak@xxxxxxxxxxxxxx>]

Hi,

On lauantai 07 tammikuu 2012, Eero Tamminen wrote:
> > (on a more general test case, I agree that comparing selected
> > screenshots can help verifying regression in an automated way)

With GEMDOS emulation, a better way to detect TOS startup could be
to auto-run some Atari program that writes to a predefined file, which
in actuality is a FIFO that the tos-tester listens on.  If there's no
write within some long timeout, the tester would conclude boot to have
failed.

This would be more robust than timeouts and would test also reading
desktop.inf, starting a program and writing to a file (i.e. test GEMDOS
emu a bit more).  Downside is that it always requires GEMDOS emu, but
maybe a non-GEMDOS boot with otherwise same HW config could follow which
uses the time it took to start the test program as TOS boot timeout.


I could also add autorunning of the VDI testers I wrote for EmuTOS.

Output of those should be always the same as long screen is of same
size and the images are monochrome, so comparison to references images
would be easier.  Due to EmuTOS VDI implementation deficiencies, EmuTOS
would need separate reference images though.


> So, any comments on what combinations would be good to test? :-)
> 
> If e.g. "--fast-boot off" would more robust, the tester could be
> used to verify that it works fine with all combinations below.
> 
> >> The attached version will go through:
> >> * given TOS images
> 
> I would propose testing following versions to get good enough
> coverage:
>   v1.00 de, v1.02 de, v1.04 de, v1.04 us, v1.62 de, v1.62 us,
>   v2.06 de, v3.06 us, v4.04, kaostos, etos192k, etos512k[1].
> 
> [1] Just the latest version of EmuTOS.
> 
> >> * st, ste, tt, falcon machine types
> 
> EmuTOS 512k would cover all machine types, EmuTOS 256k and TOS v2
> all except Falcon[2], rest of TOSes would be limited to single
> machine type.
> 
> >> * tv, vga, rgb, mono, 1 plane vdi, 4 planes vdi
> 
> ST:
>   tv, mono, vdi-1, vdi-4
> STE:
>   rgb, mono, vdi-1, vdi-4
> TT:
>   rgb, mono, vdi-1, vdi-4
> Falcon:
>   rgb, vga, mono
> 
> Or does TT use VGA instead of RGB monitors?
> 
> VDI mode emulation doesn't work with TOS v4 and with Videl's
> programmability it's anyway unnecessary.
> 
> >> * 1, 4, 14 MB of memory
> 
> ST & STE:
>   1 & 4 MB
> TT & Falcon:
>   1 & 14 MB
> 
> Or do you think some other/additional memory configurations
> would need to be tested?  Or that one could use some base
> config with different amounts of memory instead of testing
> them with all of the other combinations?
> 
> >> * GEMDOS emulation on/off
> 
> Tested for all of above combinations.
> 
> 
> I've run all these combinations for Hatari v1.4 and they all
> worked fine.
> 
> [2]
> Testing takes >1/2h and produces well about 300 screenshots,
> about half of them for EmuTOS and TOS v2.06 as they work
> with so many machine configurations.

Thomas, would you have some comments on this?


	 - Eero