Re-organising layout engine testcases?

[Manuel Mall <mm...@arcus.com.au>]

We now have over 200 layout engine test cases in the repository which is 
great. However, with this ever growing number I wonder if we should put 
some more structure to it. There is a real chance that we get more and 
more duplication just because people wouldn't know which tests are 
doing what so one starts writing new ones which may already be covered.

I don't want to suggest some complex system with the associated 
management,  setup and on-going compliance overhead. But what about 
some simple naming system along the following lines: 

Most current tests (not all) cover a particular feature and can be 
described by the fo they target, the property they exercise and the 
particular aspect of that combination they test. Therefore giving each 
test file a name constructed like <fo name>[-<property name>]?
[-<feature>]?[<serial number>]?.xml., e.g.

table-padding-relative.xml will test relative padding values on a 
fo:table element.

Yes, this will give us some longer names but will make looking for a 
particular test much easier as simple directory search/sort/filter 
operations will do. It will also reduce the number of tests which are 
identified just by a different non descript number, that is things like 
padding1.xml, padding2.xml will be replaced by something more 
meaningful. And yes, it will not cover every case especially once we 
get into tests which deal with the interaction of multiple fos and 
properties.

If agreement is found on this I am not sure what the best way to 
actually do it with svn is. One way would be for someone (probably 
me :-)) to rename all the files and for a committer to simple delete 
everything in that directory in svn and submit all the renamed files as 
new. That would loose some history but I don't think its a big deal for 
these testcases.

Manuel

Re: Re-organising layout engine testcases?

[Jeremias Maerki <de...@greenmail.ch>]

That's certainly a good idea.

On 01.09.2005 08:49:02 Manuel Mall wrote:
<snip/>
> OK, what about me making up a big rename script like:
>   svn move padding1.xml character-padding-relative.xml
>   svn move padding2.xml basic-link-padding.xml
>  ....
> 
> and a committer can apply that?


Jeremias Maerki

Re: Re-organising layout engine testcases?

[Manuel Mall <mm...@arcus.com.au>]

On Thu, 1 Sep 2005 02:39 pm, Jeremias Maerki wrote:
> On 01.09.2005 06:04:45 Manuel Mall wrote:
> > We now have over 200 layout engine test cases in the repository
> > which is great.
>
> Wow! I just hope nobody holds a grudge against me for introducing
> that facility and pushing people to use it. ;-)
>
> > However, with this ever growing number I wonder if we should put
> > some more structure to it. There is a real chance that we get more
> > and more duplication just because people wouldn't know which tests
> > are doing what so one starts writing new ones which may already be
> > covered.
>
> I agree.
>
> > I don't want to suggest some complex system with the associated
> > management,  setup and on-going compliance overhead. But what about
> > some simple naming system along the following lines:
> >
> > Most current tests (not all) cover a particular feature and can be
> > described by the fo they target, the property they exercise and the
> > particular aspect of that combination they test. Therefore giving
> > each test file a name constructed like <fo name>[-<property name>]?
> > [-<feature>]?[<serial number>]?.xml., e.g.
> >
> > table-padding-relative.xml will test relative padding values on a
> > fo:table element.
> >
> > Yes, this will give us some longer names but will make looking for
> > a particular test much easier as simple directory
> > search/sort/filter operations will do. It will also reduce the
> > number of tests which are identified just by a different non
> > descript number, that is things like padding1.xml, padding2.xml
> > will be replaced by something more meaningful. And yes, it will not
> > cover every case especially once we get into tests which deal with
> > the interaction of multiple fos and properties.
>
> Sounds good. This was bound to produce problems when it reached a
> certain size. One additional suggestion, though:
>
> It would be good to separate feature tests from regression tests. The
> latter could, for example, contain the Bugzilla number if a Bugzilla
> issue is associated with it. I've thought about this myself a number
> of times. I wonder if we should also separate the tests into multiple
> directories.
>
> > If agreement is found on this I am not sure what the best way to
> > actually do it with svn is. One way would be for someone (probably
> > me :-)) to rename all the files and for a committer to simple
> > delete everything in that directory in svn and submit all the
> > renamed files as new. That would loose some history but I don't
> > think its a big deal for these testcases.
>
> Certainly not the best of approaches doing that via patches. I'd like
> to keep the history, so this means someone with commit access will
> have to do it. I'm sure we'll find a solution to that. It just takes
> a little more time.
>
OK, what about me making up a big rename script like:
  svn move padding1.xml character-padding-relative.xml
  svn move padding2.xml basic-link-padding.xml
 ....

and a committer can apply that?

>
> Jeremias Maerki

Manuel

Re: Re-organising layout engine testcases?

[Jeremias Maerki <de...@greenmail.ch>]

On 01.09.2005 06:04:45 Manuel Mall wrote:
> We now have over 200 layout engine test cases in the repository which is 
> great.

Wow! I just hope nobody holds a grudge against me for introducing that
facility and pushing people to use it. ;-)

> However, with this ever growing number I wonder if we should put 
> some more structure to it. There is a real chance that we get more and 
> more duplication just because people wouldn't know which tests are 
> doing what so one starts writing new ones which may already be covered.

I agree.

> I don't want to suggest some complex system with the associated 
> management,  setup and on-going compliance overhead. But what about 
> some simple naming system along the following lines: 
> 
> Most current tests (not all) cover a particular feature and can be 
> described by the fo they target, the property they exercise and the 
> particular aspect of that combination they test. Therefore giving each 
> test file a name constructed like <fo name>[-<property name>]?
> [-<feature>]?[<serial number>]?.xml., e.g.
> 
> table-padding-relative.xml will test relative padding values on a 
> fo:table element.
> 
> Yes, this will give us some longer names but will make looking for a 
> particular test much easier as simple directory search/sort/filter 
> operations will do. It will also reduce the number of tests which are 
> identified just by a different non descript number, that is things like 
> padding1.xml, padding2.xml will be replaced by something more 
> meaningful. And yes, it will not cover every case especially once we 
> get into tests which deal with the interaction of multiple fos and 
> properties.

Sounds good. This was bound to produce problems when it reached a
certain size. One additional suggestion, though:

It would be good to separate feature tests from regression tests. The
latter could, for example, contain the Bugzilla number if a Bugzilla
issue is associated with it. I've thought about this myself a number of
times. I wonder if we should also separate the tests into multiple
directories.

> If agreement is found on this I am not sure what the best way to 
> actually do it with svn is. One way would be for someone (probably 
> me :-)) to rename all the files and for a committer to simple delete 
> everything in that directory in svn and submit all the renamed files as 
> new. That would loose some history but I don't think its a big deal for 
> these testcases.

Certainly not the best of approaches doing that via patches. I'd like to
keep the history, so this means someone with commit access will have to
do it. I'm sure we'll find a solution to that. It just takes a little
more time.


Jeremias Maerki