Re: [VOTE] conf test restructuring

[David Marston/Cambridge/IBM <da...@us.ibm.com>]

Scott Boag writes:
>It's not totally clear if these would be a complete reproduction of
>the so-called gold files, or just the deltas (maybe I didn't read
>carefully enough... from paragraph 2 I think you mean just the
>deltas).

Since "deltas" implies differences over time in this context, I
don't think that's the word. Here's a more concrete description:
1. Most groups, like axes or boolean, do not need to be in the
   accept cluster at all.
2. Some groups need a few tests in accept. For example, the idkey
   group has about 63 positive tests, of which 6 give a problem
   with different versions. We can probably code around 4 and
   migrate 2 to accept.
3. The decision to migrate is made per test case. Typically, the
   test cannot be made into a universal conformance test in a
   reasonable way.
4. For those tests that go into accept, which might be 50 that
   migrate and another 20 that should be written, there would be
   multiple silver files under plan (2), or a set of tags for
   plan (1) to overwrite the gold tree as needed.

>I would feel better if we:
>1) Enumerated the differences that need to be processor dependent.

That's underway.

>2) See how many of these differences could be handled via better
>comparators, probably with configuration options.

I think that's being given full consideration. On the other hand,
if we set requirements for emitting a named entity like   under
one option and the numbered entity under another, we really want to
test that we do each of those things at the correct time.

>3) Possibly for those cases that can't be handled by (2), use a file
>naming convention, like axes01-xalanc.xml.

This breaks way more automation than the other proposals! To you,
accept-gold/axes/axes01-xalanc.xml
and
accept-silver-xalanc/axes/axes01.xml
should be close enough that you can defer to others' needs. Under
proposal (2), you will need to set a couple parameters that affect
the smoke test. (Can we make the processor setting an environment
variable, for people who only work on one?) You need to set those
parameters regardless of which naming scheme applies to the silver
files. Under proposal (1), you need to take explicit action to set
up the proper reference outputs.
.................David Marston

Re: [VOTE] conf test restructuring

[sc...@us.ibm.com]

> >3) Possibly for those cases that can't be handled by (2), use a file
> >naming convention, like axes01-xalanc.xml.

BTW, this should have been axes01-xalanc.out.  Same as Shane's proposal.

-scott

David Marston/Cambridge/IBM <da...@us.ibm.com> wrote on 12/02/2002
08:42:33 AM:

>
>
>
>
> Scott Boag writes:
> >It's not totally clear if these would be a complete reproduction of
> >the so-called gold files, or just the deltas (maybe I didn't read
> >carefully enough... from paragraph 2 I think you mean just the
> >deltas).
>
> Since "deltas" implies differences over time in this context, I
> don't think that's the word. Here's a more concrete description:
> 1. Most groups, like axes or boolean, do not need to be in the
>    accept cluster at all.
> 2. Some groups need a few tests in accept. For example, the idkey
>    group has about 63 positive tests, of which 6 give a problem
>    with different versions. We can probably code around 4 and
>    migrate 2 to accept.
> 3. The decision to migrate is made per test case. Typically, the
>    test cannot be made into a universal conformance test in a
>    reasonable way.
> 4. For those tests that go into accept, which might be 50 that
>    migrate and another 20 that should be written, there would be
>    multiple silver files under plan (2), or a set of tags for
>    plan (1) to overwrite the gold tree as needed.
>
> >I would feel better if we:
> >1) Enumerated the differences that need to be processor dependent.
>
> That's underway.
>
> >2) See how many of these differences could be handled via better
> >comparators, probably with configuration options.
>
> I think that's being given full consideration. On the other hand,
> if we set requirements for emitting a named entity like &nbsp; under
> one option and the numbered entity under another, we really want to
> test that we do each of those things at the correct time.
>
> >3) Possibly for those cases that can't be handled by (2), use a file
> >naming convention, like axes01-xalanc.xml.
>
> This breaks way more automation than the other proposals! To you,
> accept-gold/axes/axes01-xalanc.xml
> and
> accept-silver-xalanc/axes/axes01.xml
> should be close enough that you can defer to others' needs. Under
> proposal (2), you will need to set a couple parameters that affect
> the smoke test. (Can we make the processor setting an environment
> variable, for people who only work on one?) You need to set those
> parameters regardless of which naming scheme applies to the silver
> files. Under proposal (1), you need to take explicit action to set
> up the proper reference outputs.
> .................David Marston
>
>