Re: conf test restructuring

[David Marston/Cambridge/IBM <da...@us.ibm.com>]

>However, within the acceptance cluster, aren't we likely to run into
>a situation in which the way a processor handles a particular feature
>changes from one release to another?

We could. Some of our past behavior might not be the result of a
complete and consistent design rationale, and thus we might not want
to say that what we did as of that past moment was right for its
time.

>In such a situation wouldn't we need
>tagged versions of the acceptance cluster gold files that would be
>associated with particular versions of Xalan or XSLTC?

Only if we want to preserve a snapshot, and even then we would tag
sets of gold files that represent what we wanted as of past points
in time, but not a set for every release. To follow your verbiage,
we want a set for each change in the requirements, not each change
in the product. (The product might fail some.) Maybe we should archive
both tests and gold along with the code at each release point, if
someone wanted to go back and totally recreate where we stood at
the time. If we have to measure the output of a past version, I'm
more interested in knowing how that past version measures up
against today's test suite, because I think we are continually
improving our notion of what is correct behavior.

Also, note that I am agreeing about *tagging* past versions
instead of the much-more-burdensome notion of constructing a
separate tree named for that version and populating it. To
summarize, we could tag a set of tests that corresponds to
the product requirements at certain past moments where the
requirements changed. In the conformance group, there have been
no changes in the requirements, even our policies on the
discretionary choices, since the original 16-November-1999
publication of the XSLT and XPath Recs. In the acceptance
group, we probably don't have much documentation of past
changes, but we could try to do that for the future.

As for associating test snapshots with releases, I would see
that as an archival issue, which is separate from the current
discussion about ways to have various sets of sorta-gold files
available for quick retrieval.
.................David Marston