Regression tests was: Re: Output from NIST test suite

["J.Pietschmann" <j3...@yahoo.de>]

John Austin wrote:
> RedHat 9.0 (my system anyhow) includes a command 'pdftopbm' that will
> convert a PDF to multiple PBM (protable Bit Map) files that might be
> comparable.
...
 >It would certainly help detect pixel-sized changes.
> That might help regression testing.

We need regression tests badly. Some problems to ponder:
a) Tests need to be automated for actually being useful.
  JUnit seems the way to go. Unfortuanately, it's still
  underutiliyed in FOP.
b) We don't have much *unit* tests. There's only the
  UtilityCodeTestSuite.java. We need much more tests for
  basic functionality. The problem seems to be however
  that an elaborated test harness needs to be written in
  order to do unt tests for, e.g. layout managers.
c) In order to test the whole engine at once, from FO input
  to generating PDF/whatever, well, a binary compare with
  a pregenerated PDF would be as sufficient as comparing
  bitmap images. Problems here:
  + The files to compare against are binary, and consume
   a lot of space. Well, take a look at GenericFOPTestCase.java
   which uses MD5 sums, one for the FO in order to detect
   accidental changes to the source, and one for the result.
  + Even small changes have potential to break the whole test
   suite, even if nothing important changed, let's say the
   order of entries in a PDF dictionary. Rendering bitmaps
   from PDF eliminates this, but then you wont find regressions
   in non-visible stuff.
All in all, if there are 143 template PDFs and a change causes
mismatches for all, what will you do? Examine everything,
comparing pixels, check whether there are visible differences
at all, and then judge whether the original or the newly
generated PDF is at fault? I don't think this will be done
often.

Ideas welcome!

J.Pietschmann

Re: AW: Regression tests was: Re: Output from NIST test suite

[John Austin <jw...@sympatico.ca>]

On Fri, 2003-12-26 at 05:29, Peter Kullmann wrote:
> J. Pietschmann wrote:
> > 
> > John Austin wrote:
> > > RedHat 9.0 (my system anyhow) includes a command 'pdftopbm' 
> > that will
> > > convert a PDF to multiple PBM (protable Bit Map) files that might be
> > > comparable.
> > ...
> >  >It would certainly help detect pixel-sized changes.
> > > That might help regression testing.

I wasn't thinking of using graphics as the primary means of 
comparing output. It was just a thought that one could use
visualization in some circumstances: 

	+ pixels that were white in both images would be
	  rendered as white
	+ pixels that were black in both images would be
	  rendered as black
	+ black pixels in the first image that were white
	  in the second could be rendered as red
	+ white pixels in the first image that were black
	  in the second could be rendered as blue

I thought of the idea of overlaying images for comparison when
I was scrolling through the side-by-side renderings of PDF's
that Finn posted yesterday (what does 'yesterday' mean in a
discussion that crosses the International Date Line ?)

Of course, this color-based scheme breaks down for test cases
that use color.

> > 
> > We need regression tests badly. Some problems to ponder:
> > a) Tests need to be automated for actually being useful.
> >   JUnit seems the way to go. Unfortuanately, it's still
> >   underutiliyed in FOP.
> > b) We don't have much *unit* tests. There's only the
> >   UtilityCodeTestSuite.java. We need much more tests for
> >   basic functionality. The problem seems to be however
> >   that an elaborated test harness needs to be written in
> >   order to do unt tests for, e.g. layout managers.
> > c) In order to test the whole engine at once, from FO input
> >   to generating PDF/whatever, well, a binary compare with
> >   a pregenerated PDF would be as sufficient as comparing
> >   bitmap images. Problems here:
> >   + The files to compare against are binary, and consume
> >    a lot of space. Well, take a look at GenericFOPTestCase.java
> >    which uses MD5 sums, one for the FO in order to detect
> >    accidental changes to the source, and one for the result.
> >   + Even small changes have potential to break the whole test
> >    suite, even if nothing important changed, let's say the
> >    order of entries in a PDF dictionary. Rendering bitmaps
> >    from PDF eliminates this, but then you wont find regressions
> >    in non-visible stuff.
> > All in all, if there are 143 template PDFs and a change causes
> > mismatches for all, what will you do? Examine everything,
> > comparing pixels, check whether there are visible differences
> > at all, and then judge whether the original or the newly
> > generated PDF is at fault? I don't think this will be done
> > often.

Use tests for binary equality to detect differences. Visualization
might be one tool, useful in following up on detected differences.

I might want to use the technique to compare the effects of changes 
to a document. For example:

What happens on page 7 when I change space-before="10pt" to
space-before="15pt" ?

A colorized visualization would give me a better idea than separate
files. Remember that our brains are all quite different. Your
rote visual memory ability is probably much better than mine. You
might learn more from a side-by-side comparison than I would.

Crap. Now I have to give an example. Perhaps it won't take that 
long.

> > 
> > Ideas welcome!
> > 
> > J.Pietschmann
> > 
> 
> As an alternative approach for c) one could create tests along 
> the following lines: Suppose you want to test left margin 
> properties of a block. For this a simple fo file is rendered as 
> a bitmap. The bitmap will not be compared to a reference bitmap
> but some elementary assertions are calculated. For instance one
> such assertion could be: "The rectangle of width 1 inch of the
> left edge is blank." I don't know of a tool that can do this
> but it should be pretty straight forward to implement. 

Probably not that hard to do once you get inside an image file
in a program. Especially if you know the colors will be black
(0,0,0) and white (255,255,255) or a small number of selected 
colors.

> So, in the test suit one has a piece of fo containing a test 
> document and some assertions in java or coded in xml that should
> be fulfilled by the rendered image of the fo. 
> 
> Assertions could contain some of the following pieces:
> - a specified rectangle is blank (or of some specific color)
> - a specified rectangle has only colors x and y (to test background
> and foreground colors of a block).
> - a given line contains the color pattern white, yellow, 
> (white, black)*, green, white. IE. a regular expression on the colors
> along a line. This could be used to test border colors along 
> a horizontal or vertical line through a bordered block.
> - along a given line the size of the first and last black region
> is at least xxx inches (to test border thickness)
> 
> The advantage of this approach seems to me that it is relatively
> easy to maintain. The test suite is small (no binaries). It can
> easily be automated in junit. 
> 
> On the other hand, the approach is limited to relatively simple
> test cases. For instance it will not be possible to test font 
> height, font style and text adjustments easily.
> 
> Peter
-- 
John Austin <jw...@sympatico.ca>

AW: Regression tests was: Re: Output from NIST test suite

["J.U. Anderegg" <ha...@bluewin.ch>]

> Peter Kullmann wrote:
>
> As an alternative approach for c) one could create tests along
> the following lines: Suppose you want to test left margin
> properties of a block. For this a simple fo file is rendered as
> a bitmap. The bitmap will not be compared to a reference bitmap
> but some elementary assertions are calculated. For instance one
> such assertion could be: "The rectangle of width 1 inch of the
> left edge is blank." I don't know of a tool that can do this
> but it should be pretty straight forward to implement.
>

There are 2 test points: renderer input and renderer output:

- Renderer input: establish the SVG renderer as reference and use an 'XML
file compare' program. The XSL Committee will hopefully prepare samples to
demonstrate and validate concepts and rules.

- Renderer output: Acrobat has a 'PDF document compare' function. This type
of the tool validates the PDF renderer. The AWT renderer can be validated
too by having the Java Printing System generate a PDF document with a PDF
printer driver. The Java Printing System can generate PCL files as well, if
better suited.

There will always be problems, because equivalent, valid results may be
achieved by different graphic objects or different sequences of graphic
calls. Therefore there will always be automated and manual/visual methods.

Hansuli Anderegg

AW: Regression tests was: Re: Output from NIST test suite

[Peter Kullmann <p....@arenae.ch>]

J. Pietschmann wrote:
> 
> John Austin wrote:
> > RedHat 9.0 (my system anyhow) includes a command 'pdftopbm' 
> that will
> > convert a PDF to multiple PBM (protable Bit Map) files that might be
> > comparable.
> ...
>  >It would certainly help detect pixel-sized changes.
> > That might help regression testing.
> 
> We need regression tests badly. Some problems to ponder:
> a) Tests need to be automated for actually being useful.
>   JUnit seems the way to go. Unfortuanately, it's still
>   underutiliyed in FOP.
> b) We don't have much *unit* tests. There's only the
>   UtilityCodeTestSuite.java. We need much more tests for
>   basic functionality. The problem seems to be however
>   that an elaborated test harness needs to be written in
>   order to do unt tests for, e.g. layout managers.
> c) In order to test the whole engine at once, from FO input
>   to generating PDF/whatever, well, a binary compare with
>   a pregenerated PDF would be as sufficient as comparing
>   bitmap images. Problems here:
>   + The files to compare against are binary, and consume
>    a lot of space. Well, take a look at GenericFOPTestCase.java
>    which uses MD5 sums, one for the FO in order to detect
>    accidental changes to the source, and one for the result.
>   + Even small changes have potential to break the whole test
>    suite, even if nothing important changed, let's say the
>    order of entries in a PDF dictionary. Rendering bitmaps
>    from PDF eliminates this, but then you wont find regressions
>    in non-visible stuff.
> All in all, if there are 143 template PDFs and a change causes
> mismatches for all, what will you do? Examine everything,
> comparing pixels, check whether there are visible differences
> at all, and then judge whether the original or the newly
> generated PDF is at fault? I don't think this will be done
> often.
> 
> Ideas welcome!
> 
> J.Pietschmann
> 

As an alternative approach for c) one could create tests along 
the following lines: Suppose you want to test left margin 
properties of a block. For this a simple fo file is rendered as 
a bitmap. The bitmap will not be compared to a reference bitmap
but some elementary assertions are calculated. For instance one
such assertion could be: "The rectangle of width 1 inch of the
left edge is blank." I don't know of a tool that can do this
but it should be pretty straight forward to implement. 

So, in the test suit one has a piece of fo containing a test 
document and some assertions in java or coded in xml that should
be fulfilled by the rendered image of the fo. 

Assertions could contain some of the following pieces:
- a specified rectangle is blank (or of some specific color)
- a specified rectangle has only colors x and y (to test background
and foreground colors of a block).
- a given line contains the color pattern white, yellow, 
(white, black)*, green, white. IE. a regular expression on the colors
along a line. This could be used to test border colors along 
a horizontal or vertical line through a bordered block.
- along a given line the size of the first and last black region
is at least xxx inches (to test border thickness)

The advantage of this approach seems to me that it is relatively
easy to maintain. The test suite is small (no binaries). It can
easily be automated in junit. 

On the other hand, the approach is limited to relatively simple
test cases. For instance it will not be possible to test font 
height, font style and text adjustments easily.

Peter