Re: [hatari-devel] Hatari screen test

[Thomas Huth <th.huth@xxxxxxxxx>]

Am Mon, 25 May 2020 17:32:55 +0300
schrieb Eero Tamminen <oak@xxxxxxxxxxxxxx>:

> Hi,
> 
> On 5/24/20 1:07 PM, Thomas Huth wrote:
> > Am Wed, 13 May 2020 19:34:09 +0200
> > schrieb Thomas Huth <th.huth@xxxxxxxxx>:
> >   
> >> Am Wed, 13 May 2020 18:59:19 +0200
> >> schrieb Nicolas Pomarède <npomarede@xxxxxxxxxxxx>:  
> > [...]  
> >>> it depends on the case, most of the time interpreting the results
> >>> means looking at the screen and  checking if some pixels are
> >>> aligned or in different colors (depending on which video test is
> >>> performed), which is not usable with natfeats.  
> >>
> >> We could maybe create a screenshot at the end and then compare if
> >> the result is the same as a reference picture?  
> > 
> > After finding an old fullscreen routine on my hard disk, I've now
> > implemented such a regression test: The program is run, then the
> > test script triggers a screenshot via the Hatari command fifo, and
> > then the screenshot is compared against a reference picture (using
> > the "identify" command from ImageMagick/GraphicsMagick). Please
> > give it a try, I hope it works in all environments (except SDL1
> > builds - screen- shots are not possible here, I only get a black
> > screen, but since we plan to get rid of SDL1 anyway in the near
> > future, I do not care too much about testing that anymore).  
> 
> Works fine for me.
> 
> NOTE: I tested the 0xFFFF820A register value
> with a breakpoint, and that can be there much
> before screen looks correct. Therefore I think
> something like VBL wait would be more robust.

I thought about that, too, but I think the current code should be
relatively safe: As far as I can see, the function that checks the
command fifo is only polled approx. once per VBL. So between the
"hatari-debug r" and the "hatari-shortcut screenshot", there should be
at least one VBL, leaving enough time to render a proper picture.
But if I'm wrong and we hit some unexpected test failures here in the
future, sure I'll rework the code in that case.

> You can actually do the whole thing with
> the debugger, now that I added "screenshot"
> command to it:
> 	$hatari --parse debug.ini ...
> 
> ----------- debugger.ini -----------
> # parse screenshot.ini after 400 VBLs
> 
> breakpoint VBL > 400 :trace :file screenshot.ini
> 
> ---------- screenshot.ini ----------
> # take screenshot and quit Hatari
> 
> screenshot verify.png
> quit
> ------------------------------------
> 
> (Quit flag is parsed in Hatari main loop.
> Breakpoint ":trace" flag use avoids Hatari
> stopping to debugger, so it gets back to
> mainloop.)

Nice trick, I'll try to remember that for the future!

> Image comparison might also be slightly shorter
> by using ImageMagick "compare" command instead
> of "identify" one.

How do you use "compare" for automatic tests? It rather seems useful
for visual inspection of the differences only?

> Btw. Would it be a lot of work to change this to
> be a reset demo, as that would test a bit more
> Hatari functionality?  :-)

:-) ... but unfortunately yes, it would be quite a bit of work, since
the "--tos none" mode does not support reset handlers yet.

 Thomas