how best to describe regression test bugs to enable the "automatic" tool in the sun test suite to work?

[Mike Matrigali <mi...@sbcglobal.net>]

Can anyone describe the best way to report regression test failures so
that the tool that produces the following report will pick it up:

http://www.multinet.no/~solberg/public/Apache/Failures/TinderBox-10.1/448426.html

For this issue I reported DERBY-1871 for the specific diff, but I 
probably have not described it in a way the tool can find.  I would be 
happy to update the jira description, but just looking for some hints
on what it is looking for.

Re: how best to describe regression test bugs to enable the "automatic" tool in the sun test suite to work?

[Mike Matrigali <mi...@sbcglobal.net>]

Thanks, I thought maybe you just somehow searched open bugs for the
test failure name.  The matching by diff sounds great.  If whatever
you come up with doesn't just automatically work with the info provided,
I hope that you think about making the process open so that others can
help categorize existing regression failures so that your tool can
pick them up.

I do hope at some point you can contribute the technology that matches
up failures with JIRA cases.  I would really like it if I could run my
nightly tests on my latest change and see a report similar the to great
reports provided in the tinderbox reporting.

Ole Solberg wrote:
> Mike Matrigali wrote:
> 
>>Can anyone describe the best way to report regression test failures so
>>that the tool that produces the following report will pick it up:
>>
>>http://www.multinet.no/~solberg/public/Apache/Failures/TinderBox-10.1/448426.html
>>
>>
>>For this issue I reported DERBY-1871 for the specific diff, but I
>>probably have not described it in a way the tool can find.  I would be
>>happy to update the jira description, but just looking for some hints
>>on what it is looking for.
>>
> 
> 
> Currently I only maintain a file which maps the name of the failing test
> method (in this case 'derbynet/testSecMec.java') to the possible JIRA.
> So for this case I would add 1871 as another JIRA which may be caused by
> 'derbynet/testSecMec.java' failing.
> If we could have the failing test method as part of the JIRA
> description, or in some special field, that would help keep the mapping
> file up to date.
> 
> 
> But, as there may be a many-to-many mapping between the test method
> failing and JIRA, we need a better "signature": We now often get a list
> of possible JIRAs per test method.
> 
> What I am in the process of doing is to try to map the "real" signature,
> that is e.g. the diff produced by the derby test harness, to the JIRA
> number. This should give a higher confidence in what JIRA we are
> actually seeing.
> I will experiment with setting up a Derby database to consult for these
> mappings.
> 
> If JIRA contained a field with the "signature", we might even be able to
> periodically update this "bug" database with new JIRA-"signature"
> mappings from JIRA. (This we could do even with todays test method as
> "signature".)
> 
> 

Re: how best to describe regression test bugs to enable the "automatic" tool in the sun test suite to work?

[Ole Solberg <Ol...@Sun.COM>]

Mike Matrigali wrote:
> Can anyone describe the best way to report regression test failures so
> that the tool that produces the following report will pick it up:
> 
> http://www.multinet.no/~solberg/public/Apache/Failures/TinderBox-10.1/448426.html
> 
> 
> For this issue I reported DERBY-1871 for the specific diff, but I
> probably have not described it in a way the tool can find.  I would be
> happy to update the jira description, but just looking for some hints
> on what it is looking for.
> 

Currently I only maintain a file which maps the name of the failing test
method (in this case 'derbynet/testSecMec.java') to the possible JIRA.
So for this case I would add 1871 as another JIRA which may be caused by
'derbynet/testSecMec.java' failing.
If we could have the failing test method as part of the JIRA
description, or in some special field, that would help keep the mapping
file up to date.


But, as there may be a many-to-many mapping between the test method
failing and JIRA, we need a better "signature": We now often get a list
of possible JIRAs per test method.

What I am in the process of doing is to try to map the "real" signature,
that is e.g. the diff produced by the derby test harness, to the JIRA
number. This should give a higher confidence in what JIRA we are
actually seeing.
I will experiment with setting up a Derby database to consult for these
mappings.

If JIRA contained a field with the "signature", we might even be able to
periodically update this "bug" database with new JIRA-"signature"
mappings from JIRA. (This we could do even with todays test method as
"signature".)


-- 
Ole Solberg, Database Technology Group,
Sun Microsystems, Trondheim, Norway