SVN test suite requirements (was CVS update ...)

[Mo DeJong <md...@cygnus.com>]

I went ahead and changed the subject for this
thread since it was getting overloaded.

Also note that I have attached an updated
goals.html file to this email.

On Tue, 27 Mar 2001, Greg Stein wrote:

> > Can folks take a look at these requirements and suggest
> > any revisions or clarifications you think might be needed?
> 
> I like it, but for a couple nits:
 
> *) automatic recovery is kind of a "3rd generation" type of thing for a test
>    harness. that can get difficult. not saying it shouldn't be there, but
>    our expectations should be set property to "probably not for a while"

Funny, I was thinking this was needed sooner rather than later.
I updated the Automatic Recovery: section to try to explain
why this is needed a bit better. Does it help?

> *) the "Platform Specific Results" section is very unclear, and I think not
>    entirely correct/appropriate. Specifically, I would think that we would
>    NOT have platform-specific results -- that all platforms should end up
>    the same. if there *are* differences, then CVS should have separate
>    copies of each platform's expected output (where that differs from the
>    "portable, expected output")

I think my description was less than clear about which "results"
I was talking about (the test results compared to the PASSED/FAILED
results one gets by comparing a test's return value to the expected
result). I have added a much more detailed example to the
"Platform Specific Results" section in hopes of addressing this.

> *) you imply that a set of information is saved from "the last run". I'm not
>    sure that I buy into saving info about the last run. Essentially, I think
>    each test suite run should be stateless. But if we *do* have a need for
>    retaining state, then the doc might want to discuss where that is
>    properly kept.

Well, yes. Each run should be stateless. I am only talking
about comparing the final results from one run to the next.
Instead of just implying, I added an "Automatic Logging of Results:"
section to try to deal with this. Does that clear things up?
I also added a section about why test named need to be unique
and how developers would interface the svntest driver.

> *) what's with the 40 column formatting? do you use really huge fonts or
>    something, and so you have fewer columns on your screen?  I'm finding it
>    is actually harder to read the text when there is a newline every five
>    words or so. (in the past, I've actually reformatted your emails before
>    reading them)

I am just strange :)
 
> I'd like to see your next draft checked into source control. I think it is
> very well done, overall.

I have my own CVS repo for the svntest code I am working on right
now. I guess I would rather just merge the whole directory in
later, if possible. If folks like my test harness, I would
like to get it added in subdirectories of a new
subversion/subversion/svntest directory. Of course, that
is still a number of weeks off (regular job, you know).
My hope is to have something I can demo at the SVLUG meeting.

Mo

Re: SVN test suite requirements (was CVS update ...)

[Greg Stein <gs...@lyra.org>]

On Tue, Mar 27, 2001 at 09:38:07PM -0800, Mo DeJong wrote:
> I went ahead and changed the subject for this
> thread since it was getting overloaded.
> 
> Also note that I have attached an updated
> goals.html file to this email.

I've checked this into source control as www/testing-goals.html. I'll do
another review after M2.

>...
> I have my own CVS repo for the svntest code I am working on right
> now. I guess I would rather just merge the whole directory in
> later, if possible. If folks like my test harness, I would
> like to get it added in subdirectories of a new
> subversion/subversion/svntest directory. Of course, that
> is still a number of weeks off (regular job, you know).
> My hope is to have something I can demo at the SVLUG meeting.

I'd prefer to see it sooner rather than later, get it checked into SVN, get
you commit access, and have you and others work on it within SVN. IMO, that
is much more preferable than something developed out of sight; collaborative
development and feedback should increase buy-in, stability, etc.

Cheers,
-g

-- 
Greg Stein, http://www.lyra.org/