You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by Ole Solberg <Ol...@Sun.COM> on 2006/02/13 11:15:23 UTC
Simple "failure analysis" for regression tests. Was Re: Regression
Test Failure!
Ole.Solberg@Sun.COM wrote:
> [Auto-generated mail]
>
> *Derby* 377211/2006-02-12 19:45:57 CET
> *derbyall*
>
> Failed Tests OK Skip Duration Platform
> -------------------------------------------------------
> *Jvm: 1.5*
> 4 636 632 0 100.96% CYGWIN_NT-5.1_i686-unknown
> NA NA NA NA CYGWIN_NT-5.2_i686-unknown
> 1 636 635 0 102.48% Linux-2.4.21-27.ELsmp_i686-athlon
> 2 636 634 0 102.79% Linux-2.6.14-1.1644_FC4_i686-i686
> 2 636 634 0 93.78% SunOS-5.10_i86pc-i386
> 2 636 634 0 114.68% SunOS-5.10_sun4u-sparc
> 3 636 633 0 103.98% SunOS-5.9_sun4u-sparc
> Details in http://www.multinet.no/~solberg/public/Apache/Derby/Limited/testSummary-377211.html
*NEW:*
> Attempted failure analysis in
> http://www.multinet.no/~solberg/public/Apache/Failures/Derby/377211.html
FYI: this is trying to map the "signature" of the failure to a JIRA issue.
I keep a file with failures seen on the format
JIRA suite/test
e.g.
800 lang/ConcurrentImplicitCreateSchema.java
and match against this.
(Should probably add "Not yet reviewed by a human!" ;-) )
If there is no match I display something like:
"Create JIRA issue for failure in 'lang/autoincrement.sql'?"
> *Jvm: 1.4*
> 3 634 631 1 97.55% CYGWIN_NT-5.1_i686-unknown
> 0 634 634 1 101.83% Linux-2.4.21-27.ELsmp_i686-athlon
> NA NA NA NA Linux-2.6.14-1.1644_FC4_i686-i686
> 1 634 633 1 97.6% SunOS-5.10_i86pc-i386
> Details in http://www.multinet.no/~solberg/public/Apache/DerbyJvm1.4/Limited/testSummary-377211.html
> Attempted failure analysis in
> http://www.multinet.no/~solberg/public/Apache/Failures/DerbyJvm1.4/377211.html
>
> Changes in http://www.multinet.no/~solberg/public/Apache/Derby/UpdateInfo/377211.txt
>
> (All results in http://www.multinet.no/~solberg/public/Apache/index.html)
>
--
Ole Solberg, Database Technology Group,
Sun Microsystems, Trondheim, Norway
Re: Simple "failure analysis" for regression tests. Was Re: Regression
Test Failure!
Posted by "David W. Van Couvering" <Da...@Sun.COM>.
Wow, nice stuff, Ole!
David
Ole Solberg wrote:
> Ole.Solberg@Sun.COM wrote:
>
>> [Auto-generated mail]
>>
>> *Derby* 377211/2006-02-12 19:45:57 CET
>> *derbyall*
>>
>> Failed Tests OK Skip Duration Platform
>> -------------------------------------------------------
>> *Jvm: 1.5*
>> 4 636 632 0 100.96% CYGWIN_NT-5.1_i686-unknown
>> NA NA NA NA CYGWIN_NT-5.2_i686-unknown
>> 1 636 635 0 102.48% Linux-2.4.21-27.ELsmp_i686-athlon
>> 2 636 634 0 102.79% Linux-2.6.14-1.1644_FC4_i686-i686
>> 2 636 634 0 93.78% SunOS-5.10_i86pc-i386
>> 2 636 634 0 114.68% SunOS-5.10_sun4u-sparc
>> 3 636 633 0 103.98% SunOS-5.9_sun4u-sparc
>> Details in
>> http://www.multinet.no/~solberg/public/Apache/Derby/Limited/testSummary-377211.html
>
>
>
> *NEW:*
>
>> Attempted failure analysis in
>>
>> http://www.multinet.no/~solberg/public/Apache/Failures/Derby/377211.html
>
>
> FYI: this is trying to map the "signature" of the failure to a JIRA issue.
> I keep a file with failures seen on the format
>
> JIRA suite/test
> e.g.
> 800 lang/ConcurrentImplicitCreateSchema.java
>
> and match against this.
> (Should probably add "Not yet reviewed by a human!" ;-) )
>
> If there is no match I display something like:
> "Create JIRA issue for failure in 'lang/autoincrement.sql'?"
>
>> *Jvm: 1.4*
>> 3 634 631 1 97.55% CYGWIN_NT-5.1_i686-unknown
>> 0 634 634 1 101.83% Linux-2.4.21-27.ELsmp_i686-athlon
>> NA NA NA NA Linux-2.6.14-1.1644_FC4_i686-i686
>> 1 634 633 1 97.6% SunOS-5.10_i86pc-i386
>> Details in
>> http://www.multinet.no/~solberg/public/Apache/DerbyJvm1.4/Limited/testSummary-377211.html
>> Attempted failure analysis in
>>
>> http://www.multinet.no/~solberg/public/Apache/Failures/DerbyJvm1.4/377211.html
>>
>> Changes in
>> http://www.multinet.no/~solberg/public/Apache/Derby/UpdateInfo/377211.txt
>> (All results in http://www.multinet.no/~solberg/public/Apache/index.html)
>
>
>
Re: Simple "failure analysis" for regression tests. Was Re: Regression
Test Failure!
Posted by Øystein Grøvlen <Oy...@Sun.COM>.
Ole Solberg wrote:
> *NEW:*
>
>> Attempted failure analysis in
>>
>> http://www.multinet.no/~solberg/public/Apache/Failures/Derby/377211.html
>
>
> FYI: this is trying to map the "signature" of the failure to a JIRA issue.
> I keep a file with failures seen on the format
>
> JIRA suite/test
> e.g.
> 800 lang/ConcurrentImplicitCreateSchema.java
>
> and match against this.
> (Should probably add "Not yet reviewed by a human!" ;-) )
>
> If there is no match I display something like:
> "Create JIRA issue for failure in 'lang/autoincrement.sql'?"
Nice work. This is very useful. It makes it much easier to determine
whether tnew failures have been introduced. In fact, I think it would
be nice if this information included in the email. Especially since
currently there is at least one test failing every time, it could save
us some work if we did not have to follow a link to check whether a
regression has been introduced.
--
Øystein