regression test suite

[Vladimir Strigun <vs...@gmail.com>]

We already discussed several questions about tests. But I'd like to
open a new topic about regression test suite. Some API bugs have been
fixed (or patch suggested) and I think it will be useful to have
regression tests for all these bugs to avoid them reappearing in the
future.

I think the natural place for the regression tests should be together
with the unit tests. I am not sure, however, if the regression tests
should be marked explicitly or differ from the unit tests in any other
specific way.

What do you think about it? Any preferences?

Thank you,
Vladimir Strigun, Intel Middleware Products Division.

Re: #harmony on freenode

[John <ja...@omnis.com>]

Details (too numerous to attach with the initial email):

http://freenode.net/group_contact_form.shtml 
<http://freenode.net/group_contact_form.shtml#requesttype>

Request type: being approved (I'd have to get approval from the formal 
head of the project ... I'd imagine Geir)
Group type: Not-for-profit entity
Group name: Apache Harmony
Group URL: http://incubator.apache.org/harmony/index.html

Group Address: ?? (Need help figuring out whose address to put in this 
field)

Group Phone: (Again, I'd imagine Geir's phone)

Channel type: Primary (official)

Channels: #harmony

Project name: Apache Harmony
Project URL: http://incubator.apache.org/harmony/index.html

And then finally my own details for the second half of the form.

Anything of concern should be brought to light.

John wrote:
> I'm going to register #harmony on irc.freenode.net, I think it's 
> better to do that sooner than later.
>
> If there are any objections, let me know.
>
> Regards,
> John
>

Re: #harmony on freenode

[John <ja...@omnis.com>]

I didn't see it on this list:

http://freenode.net/primary_groups.shtml

Which led me to believe it wasn't registered with freenode as a primary 
group channel, maybe the list is not updated?

Geir Magnusson Jr wrote:
> what do you mean register?  We've had that going for a while...
>
> John wrote:
>> I'm going to register #harmony on irc.freenode.net, I think it's 
>> better to do that sooner than later.
>>
>> If there are any objections, let me know.
>>
>> Regards,
>> John
>>
>>
>

Re: #harmony on freenode

[Geir Magnusson Jr <ge...@pobox.com>]

what do you mean register?  We've had that going for a while...

John wrote:
> I'm going to register #harmony on irc.freenode.net, I think it's better 
> to do that sooner than later.
> 
> If there are any objections, let me know.
> 
> Regards,
> John
> 
> 

Re: #harmony on freenode

[Leo Simons <ma...@leosimons.com>]

Freenode has previously requested that we prefix channel names, so eg that
we se #asfharmony (like there's #asfinfra, #asfmembers, ...). I think there's
as many projects that follow that policy as there are that don't.

- LSD


On Fri, Dec 30, 2005 at 10:09:56AM -0800, John wrote:
> I'm going to register #harmony on irc.freenode.net, I think it's better 
> to do that sooner than later.
> 
> If there are any objections, let me know.
> 
> Regards,
> John

#harmony on freenode

[John <ja...@omnis.com>]

I'm going to register #harmony on irc.freenode.net, I think it's better 
to do that sooner than later.

If there are any objections, let me know.

Regards,
John

Re: regression test suite

[Geir Magnusson Jr <ge...@pobox.com>]

Tim Ellison wrote:
> Geir Magnusson Jr wrote:
> 
>>Tim Ellison wrote:
>>
>>>Aren't we getting a bit ahead of ourselves here, the existing tests
>>>hardly need organizing (all three of them!) -- but let's go mad and
>>>assume we have lots of tests housed in our repository.
>>
>>Not really, as there are quite a few in the intel security contribution.
> 
> 
> That's unfair!  It hadn't been accepted when I wrote that! :-)

LOL :)

> 
> 
>> (3?  That's pathetic!)
> 
> 
> Hey, you have commit rights everywhere -- go for it!
> 
> They are regression unit tests to match the issues fixed so far.  I
> think a 'ten test-case penalty' is appropriate for every name-calling
> incident ;-)  Besides as others have said, there are other test suites
> out there.

We'll incorporate that too.

> 
> 
>>>What is the useful distinction for regression tests being kept separate?
>>> I can see that you may distinguish unit and 'system-level' tests just
>>>because of the difference in frameworks etc. required, but why do I care
>>>if the test was written due to a JIRA issue or test-based development or
>>>someone who get's kicks out of writing tests to break the code?
>>
>>Well, it's been my experience that unit tests generally are fairly
>>straightforward and isolated, where a regression test used to
>>demonstrate a bug might be fairly complicated, incorporating more than
>>the single unit you are trying to test/fix/debug/whatever.
> 
> 
> Well like I wrote, I can see that distinction and agree -- but
> complexity is not based on whether it is a regression or not.  Put the
> test into a unit test bucket if it sits naturally there, otherwise it is
> in a suite that may require more complex set-up/environment etc. and
> maybe some scenario driving.

That works.  THat's all I really wanted.

> 
> <snip>
> 
>>>I've never used TestNG, so have no opinion at the moment.  What's so
>>>good about it?
>>
>>I'll report back.
>>
>>Since we have no current setup for tests, I'll assume some leeway to
>>take a run at it as there are no existing toes to step on.  First task
>>will be to get your 3 [pathetic! :) ] unit tests running
> 
> 
> Well now the security contribution has been accepted your task has grown.

Shouldn't be too hard.  I want to work this upwards to the root too, to 
start doing this broadly.

geir

> 
> Regards,
> Tim
> 

Re: regression test suite

[Tim Ellison <t....@gmail.com>]

Geir Magnusson Jr wrote:
> Tim Ellison wrote:
>> Aren't we getting a bit ahead of ourselves here, the existing tests
>> hardly need organizing (all three of them!) -- but let's go mad and
>> assume we have lots of tests housed in our repository.
> 
> Not really, as there are quite a few in the intel security contribution.

That's unfair!  It hadn't been accepted when I wrote that! :-)

>  (3?  That's pathetic!)

Hey, you have commit rights everywhere -- go for it!

They are regression unit tests to match the issues fixed so far.  I
think a 'ten test-case penalty' is appropriate for every name-calling
incident ;-)  Besides as others have said, there are other test suites
out there.

>> What is the useful distinction for regression tests being kept separate?
>>  I can see that you may distinguish unit and 'system-level' tests just
>> because of the difference in frameworks etc. required, but why do I care
>> if the test was written due to a JIRA issue or test-based development or
>> someone who get's kicks out of writing tests to break the code?
> 
> Well, it's been my experience that unit tests generally are fairly
> straightforward and isolated, where a regression test used to
> demonstrate a bug might be fairly complicated, incorporating more than
> the single unit you are trying to test/fix/debug/whatever.

Well like I wrote, I can see that distinction and agree -- but
complexity is not based on whether it is a regression or not.  Put the
test into a unit test bucket if it sits naturally there, otherwise it is
in a suite that may require more complex set-up/environment etc. and
maybe some scenario driving.

<snip>

>> I've never used TestNG, so have no opinion at the moment.  What's so
>> good about it?
> 
> I'll report back.
> 
> Since we have no current setup for tests, I'll assume some leeway to
> take a run at it as there are no existing toes to step on.  First task
> will be to get your 3 [pathetic! :) ] unit tests running

Well now the security contribution has been accepted your task has grown.

Regards,
Tim

-- 

Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.

Re: regression test suite

[Geir Magnusson Jr <ge...@pobox.com>]

Tim Ellison wrote:
> Geir Magnusson Jr wrote:
> 
>>Thanks for bringing this up.  This has been on my to-do list to talk about.
>>
>>inline
>>
>>Vladimir Strigun wrote:
>>
>>
>>>We already discussed several questions about tests. But I'd like to
>>>open a new topic about regression test suite. Some API bugs have been
>>>fixed (or patch suggested) and I think it will be useful to have
>>>regression tests for all these bugs to avoid them reappearing in the
>>>future.
>>>
>>>I think the natural place for the regression tests should be together
>>>with the unit tests. I am not sure, however, if the regression tests
>>>should be marked explicitly or differ from the unit tests in any other
>>>specific way.
>>>
>>>What do you think about it? Any preferences?
>>
>>
>>I think that putting them in a parallel tree is worth considering, just
>>for the sake of organization, as long as this is transparent to the
>>tools a developer/user would use to do run the tests.  I do think that
>>"regression test" is a broad term, and we might want to harvest things
>>out of regression tests for where our unit tests fell short.
> 
> 
> Aren't we getting a bit ahead of ourselves here, the existing tests
> hardly need organizing (all three of them!) -- but let's go mad and
> assume we have lots of tests housed in our repository.

Not really, as there are quite a few in the intel security contribution. 
  (3?  That's pathetic!)

> 
> What is the useful distinction for regression tests being kept separate?
>  I can see that you may distinguish unit and 'system-level' tests just
> because of the difference in frameworks etc. required, but why do I care
> if the test was written due to a JIRA issue or test-based development or
> someone who get's kicks out of writing tests to break the code?

Well, it's been my experience that unit tests generally are fairly 
straightforward and isolated, where a regression test used to 
demonstrate a bug might be fairly complicated, incorporating more than 
the single unit you are trying to test/fix/debug/whatever.

> 
> 
>>I also believe we should be encourging bug reporters to submit a test
>>that demonstrates the bug.  We'd acquire a good set of tests at that point.
> 
> 
> Agreed, a simple failing test with a description of expected results is
> the ideal way to report a bug.  In particular, people are not expected
> to suggest a patch when they raise issues, though they are of course
> welcome to do so.

s/welcome/strongly encouraged/ :)

> 
> 
>>I was going to take a hard look at the build/test framework this
>>week(end).  Might as well start now.  Currently, it's JUnit.  I'm
>>wondering if anyone would consider switching to TestNG.  I've never used
>>it, but it's written by a guy I really respect and the point was to
>>address shortcomings he found in JUnit...
> 
> 
> I've never used TestNG, so have no opinion at the moment.  What's so
> good about it?

I'll report back.

Since we have no current setup for tests, I'll assume some leeway to 
take a run at it as there are no existing toes to step on.  First task 
will be to get your 3 [pathetic! :) ] unit tests running

geir

> Regards,
> Tim
> 
> 
>>geir
>>
>>
>>>Thank you,
>>>Vladimir Strigun, Intel Middleware Products Division.
>>>
>>>
>>
> 

Re: regression test suite

[Tim Ellison <t....@gmail.com>]

Geir Magnusson Jr wrote:
> Thanks for bringing this up.  This has been on my to-do list to talk about.
> 
> inline
> 
> Vladimir Strigun wrote:
> 
>> We already discussed several questions about tests. But I'd like to
>> open a new topic about regression test suite. Some API bugs have been
>> fixed (or patch suggested) and I think it will be useful to have
>> regression tests for all these bugs to avoid them reappearing in the
>> future.
>>
>> I think the natural place for the regression tests should be together
>> with the unit tests. I am not sure, however, if the regression tests
>> should be marked explicitly or differ from the unit tests in any other
>> specific way.
>>
>> What do you think about it? Any preferences?
> 
> 
> I think that putting them in a parallel tree is worth considering, just
> for the sake of organization, as long as this is transparent to the
> tools a developer/user would use to do run the tests.  I do think that
> "regression test" is a broad term, and we might want to harvest things
> out of regression tests for where our unit tests fell short.

Aren't we getting a bit ahead of ourselves here, the existing tests
hardly need organizing (all three of them!) -- but let's go mad and
assume we have lots of tests housed in our repository.

What is the useful distinction for regression tests being kept separate?
 I can see that you may distinguish unit and 'system-level' tests just
because of the difference in frameworks etc. required, but why do I care
if the test was written due to a JIRA issue or test-based development or
someone who get's kicks out of writing tests to break the code?

> I also believe we should be encourging bug reporters to submit a test
> that demonstrates the bug.  We'd acquire a good set of tests at that point.

Agreed, a simple failing test with a description of expected results is
the ideal way to report a bug.  In particular, people are not expected
to suggest a patch when they raise issues, though they are of course
welcome to do so.

> I was going to take a hard look at the build/test framework this
> week(end).  Might as well start now.  Currently, it's JUnit.  I'm
> wondering if anyone would consider switching to TestNG.  I've never used
> it, but it's written by a guy I really respect and the point was to
> address shortcomings he found in JUnit...

I've never used TestNG, so have no opinion at the moment.  What's so
good about it?

Regards,
Tim

> geir
> 
>>
>> Thank you,
>> Vladimir Strigun, Intel Middleware Products Division.
>>
>>
> 

-- 

Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.

Re: regression test suite

[Mark Wielaard <ma...@klomp.org>]

On Fri, 2005-12-30 at 10:52 -0500, Geir Magnusson Jr wrote:
> > I think the natural place for the regression tests should be together
> > with the unit tests. I am not sure, however, if the regression tests
> > should be marked explicitly or differ from the unit tests in any other
> > specific way.
> > 
> > What do you think about it? Any preferences?
> 
> I think that putting them in a parallel tree is worth considering, just 
> for the sake of organization, as long as this is transparent to the 
> tools a developer/user would use to do run the tests.  I do think that 
> "regression test" is a broad term, and we might want to harvest things 
> out of regression tests for where our unit tests fell short.

Out of tree is probably easiest, especially if we want to share tests
across projects. Take a look at Mauve (http://sourceware.org/mauve/) it
has tens of thousands of tests for core libraries (but also for byte
code verifiers, visual tests, compiler, serialization, etc.)

The good thing about having all tests of various projects in one place
like mauve is that everybody can easily run each others unit and
regression tests. There has been a discussion on the classpath
mailinglist about importing even more tests into mauve from GNU Crypto
now that that project is going to merge with GNU classpath. There were
some suggestions for improvements in the mauve setup. You might want to
join that discussion:
http://thread.gmane.org/gmane.comp.java.classpath.devel/6846

And of course http://sourceware.org/ml/mauve-discuss/

Cheers,

Mark

-- 
Escape the Java Trap with GNU Classpath!
http://www.gnu.org/philosophy/java-trap.html

Join the community at http://planet.classpath.org/

Re: regression test suite

[Geir Magnusson Jr <ge...@pobox.com>]

Thanks for bringing this up.  This has been on my to-do list to talk about.

inline

Vladimir Strigun wrote:
> We already discussed several questions about tests. But I'd like to
> open a new topic about regression test suite. Some API bugs have been
> fixed (or patch suggested) and I think it will be useful to have
> regression tests for all these bugs to avoid them reappearing in the
> future.
> 
> I think the natural place for the regression tests should be together
> with the unit tests. I am not sure, however, if the regression tests
> should be marked explicitly or differ from the unit tests in any other
> specific way.
> 
> What do you think about it? Any preferences?

I think that putting them in a parallel tree is worth considering, just 
for the sake of organization, as long as this is transparent to the 
tools a developer/user would use to do run the tests.  I do think that 
"regression test" is a broad term, and we might want to harvest things 
out of regression tests for where our unit tests fell short.

I also believe we should be encourging bug reporters to submit a test 
that demonstrates the bug.  We'd acquire a good set of tests at that point.

I was going to take a hard look at the build/test framework this 
week(end).  Might as well start now.  Currently, it's JUnit.  I'm 
wondering if anyone would consider switching to TestNG.  I've never used 
it, but it's written by a guy I really respect and the point was to 
address shortcomings he found in JUnit...

geir

> 
> Thank you,
> Vladimir Strigun, Intel Middleware Products Division.
> 
>