How to visually test for bugs in the renderers

[Jeremias Maerki <de...@greenmail.ch>]

I've just found a bug in our renderers that causes nested
block-containers to be painted in the wrong position (vertically). The
area tree is fine, so it doesn't make sense to write a layout engine
test case. Now would be the time for visual comparison. With the bitmap
renderer we could do that very easily but that would only test the
Java2D renderer, not the others. In the example I attached the PDF and
Java2D renderers actually render the document differently, but both
wrong. :-)

We really need that visual comparator which consists of:
- Generic JUnit TestCase implementation that does visual checks similar
to how I implemented the layout engine tests.
- Converters from PDF, PS etc. to bitmaps (for example, calling
GhostScript externally) [1]
- Component that compares two bitmaps and creates a "diff bitmap".
- Helper application (with GUI) which makes it possible to quickly go
through many test results. For each test it would show the reference
bitmap, the actual bitmap and the diff bitmap side by side. Possibly, it
could even show a reference bitmap from a different FOP version or even
a different FO implementation.

This helper application doesn't even necessarily have to be inside FOP
because visually testing the formatting results could be interesting for
other people, too (including possibly the XSL WG). To simplify calling
different FO processors my general FO processing API could be a great
helper here. I should simply find the time to publish the latest version.

Anyone out there who would like to do such a nice little project?

[1] has the potential problem of license issues because of the GPL.
However, we would only call GhostScript via command-line but it could
still be an unproper linking. Maybe the AFPL would be less problematic.
We'd have to find out.

Jeremias Maerki

Re: How to visually test for bugs in the renderers

[Jeremias Maerki <de...@greenmail.ch>]

I forgot to mention that this application might be equally interesting
for Batik. And now that I thought about this, I remember that Batik does
image comparison. Maybe something could be borrowed from there.

On 17.06.2005 15:43:20 Jeremias Maerki wrote:
> I've just found a bug in our renderers that causes nested
> block-containers to be painted in the wrong position (vertically). The
> area tree is fine, so it doesn't make sense to write a layout engine
> test case. Now would be the time for visual comparison. With the bitmap
> renderer we could do that very easily but that would only test the
> Java2D renderer, not the others. In the example I attached the PDF and
> Java2D renderers actually render the document differently, but both
> wrong. :-)
> 
> We really need that visual comparator which consists of:
> - Generic JUnit TestCase implementation that does visual checks similar
> to how I implemented the layout engine tests.
> - Converters from PDF, PS etc. to bitmaps (for example, calling
> GhostScript externally) [1]
> - Component that compares two bitmaps and creates a "diff bitmap".
> - Helper application (with GUI) which makes it possible to quickly go
> through many test results. For each test it would show the reference
> bitmap, the actual bitmap and the diff bitmap side by side. Possibly, it
> could even show a reference bitmap from a different FOP version or even
> a different FO implementation.
> 
> This helper application doesn't even necessarily have to be inside FOP
> because visually testing the formatting results could be interesting for
> other people, too (including possibly the XSL WG). To simplify calling
> different FO processors my general FO processing API could be a great
> helper here. I should simply find the time to publish the latest version.
> 
> Anyone out there who would like to do such a nice little project?
> 
> [1] has the potential problem of license issues because of the GPL.
> However, we would only call GhostScript via command-line but it could
> still be an unproper linking. Maybe the AFPL would be less problematic.
> We'd have to find out.
> 
> Jeremias Maerki



Jeremias Maerki