You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@harmony.apache.org by Alexei Fedotov <al...@gmail.com> on 2007/10/18 13:30:29 UTC
[buildtest] pass rate definition
Hello,
I'm involved in numerous discussions on the subject, and want to make
these discussions transparent to those community members who are
interested. Imagine we have a test suite which contains five following
tests:
BuggyTest.java
The test fails due to the test bug.
FailingReferenceTest.java
The test fails on Harmony and passes on RI. The test design does
not imply that the test should pass.
IntermittentlyFailingTest.java
The test fails intermittently due to HDK bug.
UnsupportedTest.java
The test produces an expected fail due to unimplemented
functionality in HDK.
FailingTest.java
The test fails due to HDK bug.
PassingTest.java
This one prints PASSED and completes successfully.
What would be the correct formula to define a pass rate? All agree
that the rate is a number of passed tests divided to a total number of
tests. Then people start to argue what are the numerator and the
denominator.
One may say, that he counts any failures as bugs. Then she gets 16.66%
pass rate. Others get 50%, ignoring all fail reasons except the one
which produces a fixable HDK failure.
If anyone could share common sense knowledge or Apache practices on
the subject, this would be interesting.
--
With best regards,
Alexei,
ESSD, Intel
Re: [buildtest] pass rate definition
Posted by Spark Shen <sm...@gmail.com>.
2007/10/19, Alexei Fedotov <al...@gmail.com>:
>
> Sean, Leo, Spark,
>
> Thank you for your replies. I especially like the following one:
> > I understand passing rate as an indicator to show the progress of
> harmony.
>
> To my understanding this is the reason why pass rate discussions are
> sometimes so painful between people who drive the commmunity and want
> to progress in a specific direction. :-)
Hi Alexei,
Feel really happy that you like the statement :-)
If to show the status if the aim, we may redefine the pass rate into a more
fine formula. And
generate a colorful chart with it.
As far as the formula is concerned, we can assign a factor to each category
of test cases. For
example, BuggyTest has a factor of 5, IntermittentlyFailingTest of 4 etc.
And we can give a score
according to the formula. The score is the lower (or higher) the better.
And generate a pie chart, to show clearly which parts of test failures
contribute more to the total incompleteness.
On 10/19/07, Leo Li <li...@gmail.com> wrote:
> > On 10/19/07, Spark Shen <sm...@gmail.com> wrote:
> > > 2007/10/19, Leo Li <li...@gmail.com>:
> > > >
> > > > On 10/19/07, Spark Shen <sm...@gmail.com> wrote:
> > > > > 2007/10/18, Leo Li <li...@gmail.com>:
> > > > > >
> > > > > > On 10/18/07, Alexei Fedotov <al...@gmail.com> wrote:
> > > > > > > Hello,
> > > > > > >
> > > > > > > I'm involved in numerous discussions on the subject, and want
> to
> > > > make
> > > > > > > these discussions transparent to those community members who
> are
> > > > > > > interested. Imagine we have a test suite which contains five
> > > > following
> > > > > > > tests:
> > > > > > >
> > > > > > > BuggyTest.java
> > > > > > > The test fails due to the test bug.
> > > > > > >
> > > > > > > FailingReferenceTest.java
> > > > > > > The test fails on Harmony and passes on RI. The test design
> does
> > > > > > > not imply that the test should pass.
> > > > > > >
> > > > > > > IntermittentlyFailingTest.java
> > > > > > > The test fails intermittently due to HDK bug.
> > > > > > >
> > > > > > > UnsupportedTest.java
> > > > > > > The test produces an expected fail due to unimplemented
> > > > > > > functionality in HDK.
> > > > > > >
> > > > > > > FailingTest.java
> > > > > > > The test fails due to HDK bug.
> > > > > > >
> > > > > > > PassingTest.java
> > > > > > > This one prints PASSED and completes successfully.
> > > > > > >
> > > > > > > What would be the correct formula to define a pass rate? All
> agree
> > > > > > > that the rate is a number of passed tests divided to a total
> number
> > > > of
> > > > > > > tests. Then people start to argue what are the numerator and
> the
> > > > > > > denominator.
> > > > > > >
> > > > > > > One may say, that he counts any failures as bugs. Then she
> gets
> > > > 16.66%
> > > > > > > pass rate. Others get 50%, ignoring all fail reasons except
> the one
> > > > > > > which produces a fixable HDK failure.
> > > > > > >
> > > > > > > If anyone could share common sense knowledge or Apache
> practices on
> > > > > > > the subject, this would be interesting.
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > I think how to define the passing rate is not only related to
> the
> > > > > > reason of the failure tests but as the scope of the tests itself
> and
> > > > > > what the passing rate means as well.
> > > > > >
> > > > > > Current tests can be separated to two category:
> > > > > > 1. Tests provided by harmony developers.
> > > > > > 2. Tests provided by applications.
> > > > >
> > > > >
> > > > > I like your classification. :-)
> > > > >
> > > > > For 1, I do not think the passing rate can prove much. Current
> process
> > > > > > requires that till the test passes on harmony build it is not
> checked
> > > > > > in to the source code if I have not missing something. Although
> there
> > > > > > is some exceptions, we try to achieve this goal. Thus Harmony is
> > > > > > assumed to pass all these tests and if there is a functional
> missing
> > > > > > or a known bug to be fixed, the test is supposed not exist in
> > > > > > harmony's code base and cannot be calculated in the passing
> rate.
> > > > >
> > > > >
> > > > > And I have different opinion about UnsupportedTest.java and test
> fails
> > > > on
> > > > > know issue
> > > > > (Can they be categorized into FailingReferenceTest or
> > > > > IntermittentlyFailingTest
> > > > > or FailingTest according to Alexei?) with you.
> > > > >
> > > > > I understand passing rate as an indicator to show the progress of
> > > > harmony.
> > > > > And these test failures indeed
> > > > > remind us more to be improved. Why should they be excluded?
> > > > >
> > > >
> > > > But in harmony's practice,we do not first checkin and exclude the
> > > > testcase which is not supported in current code base and then
> complete
> > > > the feature.
> > >
> > >
> > > I think we have exclude list.
> > >
> > > And we have not yet provided a covering test suites for
> > > > all java features. If there is some UnsupportedTests, I do not think
> > > > they can represent the overall status of harmony's progress.
> > >
> > >
> > > And the covering test suites is on the way? For example, sean has
> integrated
> > > emma coverage
> > > report[1]. And coverage suite for java features can be committed with
> > > exclude list first?
> >
> > Excuse me if I am misleading you. I do not mean the coverage
> > report.:) What I want to cover here is not the harmony's code, but
> > the demand or function that java spec requires.
> >
> > >
> > > Correct my if I am wrong.
> > > [1] http://article.gmane.org/gmane.comp.java.harmony.devel/29585
> > >
> > > > Of cause, if there is some common sense or best practice, I will
> stick to
> > > > > them.
> > > > >
> > > > >
> > > > > For 2, the passing rate can to some degree reflect harmony's
> maturity
> > > > > > and I think normally they need not be differentiated why they do
> not
> > > > > > pass. Except for few bugs which is due to the application's
> improper
> > > > > > dependency on sun's behavior ( For example, Sean has discovered
> Jython
> > > > > > assumes the specified order of the entry stored in HashMap. It
> is
> > > > > > actually a bug but just coincides to pass on RI. ), the failure
> among
> > > > > > the majority of the tests provided by application can reveal one
> bug
> > > > > > or one incompatibility that harmony should try to resolve since
> we are
> > > > > > trying to give a compatible product as RI for user to switch
> > > > > > seamlessly.
> > > > > >
> > > > > > > --
> > > > > > > With best regards,
> > > > > > > Alexei,
> > > > > > > ESSD, Intel
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Leo Li
> > > > > > China Software Development Lab, IBM
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Spark Shen
> > > > > China Software Development Lab, IBM
> > > > >
> > > >
> > > >
> > > > --
> > > > Leo Li
> > > > China Software Development Lab, IBM
> > > >
> > >
> > >
> > >
> > > --
> > > Spark Shen
> > > China Software Development Lab, IBM
> > >
> >
> >
> > --
> > Leo Li
> > China Software Development Lab, IBM
> >
>
>
> --
> With best regards,
> Alexei,
> ESSD, Intel
>
--
Spark Shen
China Software Development Lab, IBM
Re: [buildtest] pass rate definition
Posted by Alexei Fedotov <al...@gmail.com>.
Sean, Leo, Spark,
Thank you for your replies. I especially like the following one:
> I understand passing rate as an indicator to show the progress of harmony.
To my understanding this is the reason why pass rate discussions are
sometimes so painful between people who drive the commmunity and want
to progress in a specific direction. :-)
On 10/19/07, Leo Li <li...@gmail.com> wrote:
> On 10/19/07, Spark Shen <sm...@gmail.com> wrote:
> > 2007/10/19, Leo Li <li...@gmail.com>:
> > >
> > > On 10/19/07, Spark Shen <sm...@gmail.com> wrote:
> > > > 2007/10/18, Leo Li <li...@gmail.com>:
> > > > >
> > > > > On 10/18/07, Alexei Fedotov <al...@gmail.com> wrote:
> > > > > > Hello,
> > > > > >
> > > > > > I'm involved in numerous discussions on the subject, and want to
> > > make
> > > > > > these discussions transparent to those community members who are
> > > > > > interested. Imagine we have a test suite which contains five
> > > following
> > > > > > tests:
> > > > > >
> > > > > > BuggyTest.java
> > > > > > The test fails due to the test bug.
> > > > > >
> > > > > > FailingReferenceTest.java
> > > > > > The test fails on Harmony and passes on RI. The test design does
> > > > > > not imply that the test should pass.
> > > > > >
> > > > > > IntermittentlyFailingTest.java
> > > > > > The test fails intermittently due to HDK bug.
> > > > > >
> > > > > > UnsupportedTest.java
> > > > > > The test produces an expected fail due to unimplemented
> > > > > > functionality in HDK.
> > > > > >
> > > > > > FailingTest.java
> > > > > > The test fails due to HDK bug.
> > > > > >
> > > > > > PassingTest.java
> > > > > > This one prints PASSED and completes successfully.
> > > > > >
> > > > > > What would be the correct formula to define a pass rate? All agree
> > > > > > that the rate is a number of passed tests divided to a total number
> > > of
> > > > > > tests. Then people start to argue what are the numerator and the
> > > > > > denominator.
> > > > > >
> > > > > > One may say, that he counts any failures as bugs. Then she gets
> > > 16.66%
> > > > > > pass rate. Others get 50%, ignoring all fail reasons except the one
> > > > > > which produces a fixable HDK failure.
> > > > > >
> > > > > > If anyone could share common sense knowledge or Apache practices on
> > > > > > the subject, this would be interesting.
> > > > > >
> > > > > >
> > > > >
> > > > > I think how to define the passing rate is not only related to the
> > > > > reason of the failure tests but as the scope of the tests itself and
> > > > > what the passing rate means as well.
> > > > >
> > > > > Current tests can be separated to two category:
> > > > > 1. Tests provided by harmony developers.
> > > > > 2. Tests provided by applications.
> > > >
> > > >
> > > > I like your classification. :-)
> > > >
> > > > For 1, I do not think the passing rate can prove much. Current process
> > > > > requires that till the test passes on harmony build it is not checked
> > > > > in to the source code if I have not missing something. Although there
> > > > > is some exceptions, we try to achieve this goal. Thus Harmony is
> > > > > assumed to pass all these tests and if there is a functional missing
> > > > > or a known bug to be fixed, the test is supposed not exist in
> > > > > harmony's code base and cannot be calculated in the passing rate.
> > > >
> > > >
> > > > And I have different opinion about UnsupportedTest.java and test fails
> > > on
> > > > know issue
> > > > (Can they be categorized into FailingReferenceTest or
> > > > IntermittentlyFailingTest
> > > > or FailingTest according to Alexei?) with you.
> > > >
> > > > I understand passing rate as an indicator to show the progress of
> > > harmony.
> > > > And these test failures indeed
> > > > remind us more to be improved. Why should they be excluded?
> > > >
> > >
> > > But in harmony's practice,we do not first checkin and exclude the
> > > testcase which is not supported in current code base and then complete
> > > the feature.
> >
> >
> > I think we have exclude list.
> >
> > And we have not yet provided a covering test suites for
> > > all java features. If there is some UnsupportedTests, I do not think
> > > they can represent the overall status of harmony's progress.
> >
> >
> > And the covering test suites is on the way? For example, sean has integrated
> > emma coverage
> > report[1]. And coverage suite for java features can be committed with
> > exclude list first?
>
> Excuse me if I am misleading you. I do not mean the coverage
> report.:) What I want to cover here is not the harmony's code, but
> the demand or function that java spec requires.
>
> >
> > Correct my if I am wrong.
> > [1] http://article.gmane.org/gmane.comp.java.harmony.devel/29585
> >
> > > Of cause, if there is some common sense or best practice, I will stick to
> > > > them.
> > > >
> > > >
> > > > For 2, the passing rate can to some degree reflect harmony's maturity
> > > > > and I think normally they need not be differentiated why they do not
> > > > > pass. Except for few bugs which is due to the application's improper
> > > > > dependency on sun's behavior ( For example, Sean has discovered Jython
> > > > > assumes the specified order of the entry stored in HashMap. It is
> > > > > actually a bug but just coincides to pass on RI. ), the failure among
> > > > > the majority of the tests provided by application can reveal one bug
> > > > > or one incompatibility that harmony should try to resolve since we are
> > > > > trying to give a compatible product as RI for user to switch
> > > > > seamlessly.
> > > > >
> > > > > > --
> > > > > > With best regards,
> > > > > > Alexei,
> > > > > > ESSD, Intel
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Leo Li
> > > > > China Software Development Lab, IBM
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Spark Shen
> > > > China Software Development Lab, IBM
> > > >
> > >
> > >
> > > --
> > > Leo Li
> > > China Software Development Lab, IBM
> > >
> >
> >
> >
> > --
> > Spark Shen
> > China Software Development Lab, IBM
> >
>
>
> --
> Leo Li
> China Software Development Lab, IBM
>
--
With best regards,
Alexei,
ESSD, Intel
Re: [buildtest] pass rate definition
Posted by Leo Li <li...@gmail.com>.
On 10/19/07, Spark Shen <sm...@gmail.com> wrote:
> 2007/10/19, Leo Li <li...@gmail.com>:
> >
> > On 10/19/07, Spark Shen <sm...@gmail.com> wrote:
> > > 2007/10/18, Leo Li <li...@gmail.com>:
> > > >
> > > > On 10/18/07, Alexei Fedotov <al...@gmail.com> wrote:
> > > > > Hello,
> > > > >
> > > > > I'm involved in numerous discussions on the subject, and want to
> > make
> > > > > these discussions transparent to those community members who are
> > > > > interested. Imagine we have a test suite which contains five
> > following
> > > > > tests:
> > > > >
> > > > > BuggyTest.java
> > > > > The test fails due to the test bug.
> > > > >
> > > > > FailingReferenceTest.java
> > > > > The test fails on Harmony and passes on RI. The test design does
> > > > > not imply that the test should pass.
> > > > >
> > > > > IntermittentlyFailingTest.java
> > > > > The test fails intermittently due to HDK bug.
> > > > >
> > > > > UnsupportedTest.java
> > > > > The test produces an expected fail due to unimplemented
> > > > > functionality in HDK.
> > > > >
> > > > > FailingTest.java
> > > > > The test fails due to HDK bug.
> > > > >
> > > > > PassingTest.java
> > > > > This one prints PASSED and completes successfully.
> > > > >
> > > > > What would be the correct formula to define a pass rate? All agree
> > > > > that the rate is a number of passed tests divided to a total number
> > of
> > > > > tests. Then people start to argue what are the numerator and the
> > > > > denominator.
> > > > >
> > > > > One may say, that he counts any failures as bugs. Then she gets
> > 16.66%
> > > > > pass rate. Others get 50%, ignoring all fail reasons except the one
> > > > > which produces a fixable HDK failure.
> > > > >
> > > > > If anyone could share common sense knowledge or Apache practices on
> > > > > the subject, this would be interesting.
> > > > >
> > > > >
> > > >
> > > > I think how to define the passing rate is not only related to the
> > > > reason of the failure tests but as the scope of the tests itself and
> > > > what the passing rate means as well.
> > > >
> > > > Current tests can be separated to two category:
> > > > 1. Tests provided by harmony developers.
> > > > 2. Tests provided by applications.
> > >
> > >
> > > I like your classification. :-)
> > >
> > > For 1, I do not think the passing rate can prove much. Current process
> > > > requires that till the test passes on harmony build it is not checked
> > > > in to the source code if I have not missing something. Although there
> > > > is some exceptions, we try to achieve this goal. Thus Harmony is
> > > > assumed to pass all these tests and if there is a functional missing
> > > > or a known bug to be fixed, the test is supposed not exist in
> > > > harmony's code base and cannot be calculated in the passing rate.
> > >
> > >
> > > And I have different opinion about UnsupportedTest.java and test fails
> > on
> > > know issue
> > > (Can they be categorized into FailingReferenceTest or
> > > IntermittentlyFailingTest
> > > or FailingTest according to Alexei?) with you.
> > >
> > > I understand passing rate as an indicator to show the progress of
> > harmony.
> > > And these test failures indeed
> > > remind us more to be improved. Why should they be excluded?
> > >
> >
> > But in harmony's practice,we do not first checkin and exclude the
> > testcase which is not supported in current code base and then complete
> > the feature.
>
>
> I think we have exclude list.
>
> And we have not yet provided a covering test suites for
> > all java features. If there is some UnsupportedTests, I do not think
> > they can represent the overall status of harmony's progress.
>
>
> And the covering test suites is on the way? For example, sean has integrated
> emma coverage
> report[1]. And coverage suite for java features can be committed with
> exclude list first?
Excuse me if I am misleading you. I do not mean the coverage
report.:) What I want to cover here is not the harmony's code, but
the demand or function that java spec requires.
>
> Correct my if I am wrong.
> [1] http://article.gmane.org/gmane.comp.java.harmony.devel/29585
>
> > Of cause, if there is some common sense or best practice, I will stick to
> > > them.
> > >
> > >
> > > For 2, the passing rate can to some degree reflect harmony's maturity
> > > > and I think normally they need not be differentiated why they do not
> > > > pass. Except for few bugs which is due to the application's improper
> > > > dependency on sun's behavior ( For example, Sean has discovered Jython
> > > > assumes the specified order of the entry stored in HashMap. It is
> > > > actually a bug but just coincides to pass on RI. ), the failure among
> > > > the majority of the tests provided by application can reveal one bug
> > > > or one incompatibility that harmony should try to resolve since we are
> > > > trying to give a compatible product as RI for user to switch
> > > > seamlessly.
> > > >
> > > > > --
> > > > > With best regards,
> > > > > Alexei,
> > > > > ESSD, Intel
> > > > >
> > > >
> > > >
> > > > --
> > > > Leo Li
> > > > China Software Development Lab, IBM
> > > >
> > >
> > >
> > >
> > > --
> > > Spark Shen
> > > China Software Development Lab, IBM
> > >
> >
> >
> > --
> > Leo Li
> > China Software Development Lab, IBM
> >
>
>
>
> --
> Spark Shen
> China Software Development Lab, IBM
>
--
Leo Li
China Software Development Lab, IBM
Re: [buildtest] pass rate definition
Posted by Spark Shen <sm...@gmail.com>.
2007/10/19, Leo Li <li...@gmail.com>:
>
> On 10/19/07, Spark Shen <sm...@gmail.com> wrote:
> > 2007/10/18, Leo Li <li...@gmail.com>:
> > >
> > > On 10/18/07, Alexei Fedotov <al...@gmail.com> wrote:
> > > > Hello,
> > > >
> > > > I'm involved in numerous discussions on the subject, and want to
> make
> > > > these discussions transparent to those community members who are
> > > > interested. Imagine we have a test suite which contains five
> following
> > > > tests:
> > > >
> > > > BuggyTest.java
> > > > The test fails due to the test bug.
> > > >
> > > > FailingReferenceTest.java
> > > > The test fails on Harmony and passes on RI. The test design does
> > > > not imply that the test should pass.
> > > >
> > > > IntermittentlyFailingTest.java
> > > > The test fails intermittently due to HDK bug.
> > > >
> > > > UnsupportedTest.java
> > > > The test produces an expected fail due to unimplemented
> > > > functionality in HDK.
> > > >
> > > > FailingTest.java
> > > > The test fails due to HDK bug.
> > > >
> > > > PassingTest.java
> > > > This one prints PASSED and completes successfully.
> > > >
> > > > What would be the correct formula to define a pass rate? All agree
> > > > that the rate is a number of passed tests divided to a total number
> of
> > > > tests. Then people start to argue what are the numerator and the
> > > > denominator.
> > > >
> > > > One may say, that he counts any failures as bugs. Then she gets
> 16.66%
> > > > pass rate. Others get 50%, ignoring all fail reasons except the one
> > > > which produces a fixable HDK failure.
> > > >
> > > > If anyone could share common sense knowledge or Apache practices on
> > > > the subject, this would be interesting.
> > > >
> > > >
> > >
> > > I think how to define the passing rate is not only related to the
> > > reason of the failure tests but as the scope of the tests itself and
> > > what the passing rate means as well.
> > >
> > > Current tests can be separated to two category:
> > > 1. Tests provided by harmony developers.
> > > 2. Tests provided by applications.
> >
> >
> > I like your classification. :-)
> >
> > For 1, I do not think the passing rate can prove much. Current process
> > > requires that till the test passes on harmony build it is not checked
> > > in to the source code if I have not missing something. Although there
> > > is some exceptions, we try to achieve this goal. Thus Harmony is
> > > assumed to pass all these tests and if there is a functional missing
> > > or a known bug to be fixed, the test is supposed not exist in
> > > harmony's code base and cannot be calculated in the passing rate.
> >
> >
> > And I have different opinion about UnsupportedTest.java and test fails
> on
> > know issue
> > (Can they be categorized into FailingReferenceTest or
> > IntermittentlyFailingTest
> > or FailingTest according to Alexei?) with you.
> >
> > I understand passing rate as an indicator to show the progress of
> harmony.
> > And these test failures indeed
> > remind us more to be improved. Why should they be excluded?
> >
>
> But in harmony's practice,we do not first checkin and exclude the
> testcase which is not supported in current code base and then complete
> the feature.
I think we have exclude list.
And we have not yet provided a covering test suites for
> all java features. If there is some UnsupportedTests, I do not think
> they can represent the overall status of harmony's progress.
And the covering test suites is on the way? For example, sean has integrated
emma coverage
report[1]. And coverage suite for java features can be committed with
exclude list first?
Correct my if I am wrong.
[1] http://article.gmane.org/gmane.comp.java.harmony.devel/29585
> Of cause, if there is some common sense or best practice, I will stick to
> > them.
> >
> >
> > For 2, the passing rate can to some degree reflect harmony's maturity
> > > and I think normally they need not be differentiated why they do not
> > > pass. Except for few bugs which is due to the application's improper
> > > dependency on sun's behavior ( For example, Sean has discovered Jython
> > > assumes the specified order of the entry stored in HashMap. It is
> > > actually a bug but just coincides to pass on RI. ), the failure among
> > > the majority of the tests provided by application can reveal one bug
> > > or one incompatibility that harmony should try to resolve since we are
> > > trying to give a compatible product as RI for user to switch
> > > seamlessly.
> > >
> > > > --
> > > > With best regards,
> > > > Alexei,
> > > > ESSD, Intel
> > > >
> > >
> > >
> > > --
> > > Leo Li
> > > China Software Development Lab, IBM
> > >
> >
> >
> >
> > --
> > Spark Shen
> > China Software Development Lab, IBM
> >
>
>
> --
> Leo Li
> China Software Development Lab, IBM
>
--
Spark Shen
China Software Development Lab, IBM
Re: [buildtest] pass rate definition
Posted by Leo Li <li...@gmail.com>.
On 10/19/07, Spark Shen <sm...@gmail.com> wrote:
> 2007/10/18, Leo Li <li...@gmail.com>:
> >
> > On 10/18/07, Alexei Fedotov <al...@gmail.com> wrote:
> > > Hello,
> > >
> > > I'm involved in numerous discussions on the subject, and want to make
> > > these discussions transparent to those community members who are
> > > interested. Imagine we have a test suite which contains five following
> > > tests:
> > >
> > > BuggyTest.java
> > > The test fails due to the test bug.
> > >
> > > FailingReferenceTest.java
> > > The test fails on Harmony and passes on RI. The test design does
> > > not imply that the test should pass.
> > >
> > > IntermittentlyFailingTest.java
> > > The test fails intermittently due to HDK bug.
> > >
> > > UnsupportedTest.java
> > > The test produces an expected fail due to unimplemented
> > > functionality in HDK.
> > >
> > > FailingTest.java
> > > The test fails due to HDK bug.
> > >
> > > PassingTest.java
> > > This one prints PASSED and completes successfully.
> > >
> > > What would be the correct formula to define a pass rate? All agree
> > > that the rate is a number of passed tests divided to a total number of
> > > tests. Then people start to argue what are the numerator and the
> > > denominator.
> > >
> > > One may say, that he counts any failures as bugs. Then she gets 16.66%
> > > pass rate. Others get 50%, ignoring all fail reasons except the one
> > > which produces a fixable HDK failure.
> > >
> > > If anyone could share common sense knowledge or Apache practices on
> > > the subject, this would be interesting.
> > >
> > >
> >
> > I think how to define the passing rate is not only related to the
> > reason of the failure tests but as the scope of the tests itself and
> > what the passing rate means as well.
> >
> > Current tests can be separated to two category:
> > 1. Tests provided by harmony developers.
> > 2. Tests provided by applications.
>
>
> I like your classification. :-)
>
> For 1, I do not think the passing rate can prove much. Current process
> > requires that till the test passes on harmony build it is not checked
> > in to the source code if I have not missing something. Although there
> > is some exceptions, we try to achieve this goal. Thus Harmony is
> > assumed to pass all these tests and if there is a functional missing
> > or a known bug to be fixed, the test is supposed not exist in
> > harmony's code base and cannot be calculated in the passing rate.
>
>
> And I have different opinion about UnsupportedTest.java and test fails on
> know issue
> (Can they be categorized into FailingReferenceTest or
> IntermittentlyFailingTest
> or FailingTest according to Alexei?) with you.
>
> I understand passing rate as an indicator to show the progress of harmony.
> And these test failures indeed
> remind us more to be improved. Why should they be excluded?
>
But in harmony's practice,we do not first checkin and exclude the
testcase which is not supported in current code base and then complete
the feature. And we have not yet provided a covering test suites for
all java features. If there is some UnsupportedTests, I do not think
they can represent the overall status of harmony's progress.
> Of cause, if there is some common sense or best practice, I will stick to
> them.
>
>
> For 2, the passing rate can to some degree reflect harmony's maturity
> > and I think normally they need not be differentiated why they do not
> > pass. Except for few bugs which is due to the application's improper
> > dependency on sun's behavior ( For example, Sean has discovered Jython
> > assumes the specified order of the entry stored in HashMap. It is
> > actually a bug but just coincides to pass on RI. ), the failure among
> > the majority of the tests provided by application can reveal one bug
> > or one incompatibility that harmony should try to resolve since we are
> > trying to give a compatible product as RI for user to switch
> > seamlessly.
> >
> > > --
> > > With best regards,
> > > Alexei,
> > > ESSD, Intel
> > >
> >
> >
> > --
> > Leo Li
> > China Software Development Lab, IBM
> >
>
>
>
> --
> Spark Shen
> China Software Development Lab, IBM
>
--
Leo Li
China Software Development Lab, IBM
Re: [buildtest] pass rate definition
Posted by Spark Shen <sm...@gmail.com>.
2007/10/18, Leo Li <li...@gmail.com>:
>
> On 10/18/07, Alexei Fedotov <al...@gmail.com> wrote:
> > Hello,
> >
> > I'm involved in numerous discussions on the subject, and want to make
> > these discussions transparent to those community members who are
> > interested. Imagine we have a test suite which contains five following
> > tests:
> >
> > BuggyTest.java
> > The test fails due to the test bug.
> >
> > FailingReferenceTest.java
> > The test fails on Harmony and passes on RI. The test design does
> > not imply that the test should pass.
> >
> > IntermittentlyFailingTest.java
> > The test fails intermittently due to HDK bug.
> >
> > UnsupportedTest.java
> > The test produces an expected fail due to unimplemented
> > functionality in HDK.
> >
> > FailingTest.java
> > The test fails due to HDK bug.
> >
> > PassingTest.java
> > This one prints PASSED and completes successfully.
> >
> > What would be the correct formula to define a pass rate? All agree
> > that the rate is a number of passed tests divided to a total number of
> > tests. Then people start to argue what are the numerator and the
> > denominator.
> >
> > One may say, that he counts any failures as bugs. Then she gets 16.66%
> > pass rate. Others get 50%, ignoring all fail reasons except the one
> > which produces a fixable HDK failure.
> >
> > If anyone could share common sense knowledge or Apache practices on
> > the subject, this would be interesting.
> >
> >
>
> I think how to define the passing rate is not only related to the
> reason of the failure tests but as the scope of the tests itself and
> what the passing rate means as well.
>
> Current tests can be separated to two category:
> 1. Tests provided by harmony developers.
> 2. Tests provided by applications.
I like your classification. :-)
For 1, I do not think the passing rate can prove much. Current process
> requires that till the test passes on harmony build it is not checked
> in to the source code if I have not missing something. Although there
> is some exceptions, we try to achieve this goal. Thus Harmony is
> assumed to pass all these tests and if there is a functional missing
> or a known bug to be fixed, the test is supposed not exist in
> harmony's code base and cannot be calculated in the passing rate.
And I have different opinion about UnsupportedTest.java and test fails on
know issue
(Can they be categorized into FailingReferenceTest or
IntermittentlyFailingTest
or FailingTest according to Alexei?) with you.
I understand passing rate as an indicator to show the progress of harmony.
And these test failures indeed
remind us more to be improved. Why should they be excluded?
Of cause, if there is some common sense or best practice, I will stick to
them.
For 2, the passing rate can to some degree reflect harmony's maturity
> and I think normally they need not be differentiated why they do not
> pass. Except for few bugs which is due to the application's improper
> dependency on sun's behavior ( For example, Sean has discovered Jython
> assumes the specified order of the entry stored in HashMap. It is
> actually a bug but just coincides to pass on RI. ), the failure among
> the majority of the tests provided by application can reveal one bug
> or one incompatibility that harmony should try to resolve since we are
> trying to give a compatible product as RI for user to switch
> seamlessly.
>
> > --
> > With best regards,
> > Alexei,
> > ESSD, Intel
> >
>
>
> --
> Leo Li
> China Software Development Lab, IBM
>
--
Spark Shen
China Software Development Lab, IBM
Re: [buildtest] pass rate definition
Posted by Sean Qiu <se...@gmail.com>.
2007/10/18, Leo Li <li...@gmail.com>:
>
> On 10/18/07, Alexei Fedotov <al...@gmail.com> wrote:
> > Hello,
> >
> > I'm involved in numerous discussions on the subject, and want to make
> > these discussions transparent to those community members who are
> > interested. Imagine we have a test suite which contains five following
> > tests:
> >
> > BuggyTest.java
> > The test fails due to the test bug.
> >
> > FailingReferenceTest.java
> > The test fails on Harmony and passes on RI. The test design does
> > not imply that the test should pass.
> >
> > IntermittentlyFailingTest.java
> > The test fails intermittently due to HDK bug.
> >
> > UnsupportedTest.java
> > The test produces an expected fail due to unimplemented
> > functionality in HDK.
> >
> > FailingTest.java
> > The test fails due to HDK bug.
> >
> > PassingTest.java
> > This one prints PASSED and completes successfully.
> >
> > What would be the correct formula to define a pass rate? All agree
> > that the rate is a number of passed tests divided to a total number of
> > tests. Then people start to argue what are the numerator and the
> > denominator.
> >
> > One may say, that he counts any failures as bugs. Then she gets 16.66%
> > pass rate. Others get 50%, ignoring all fail reasons except the one
> > which produces a fixable HDK failure.
> >
> > If anyone could share common sense knowledge or Apache practices on
> > the subject, this would be interesting.
> >
> >
>
> I think how to define the passing rate is not only related to the
> reason of the failure tests but as the scope of the tests itself and
> what the passing rate means as well.
>
> Current tests can be separated to two category:
> 1. Tests provided by harmony developers.
> 2. Tests provided by applications.
>
> For 1, I do not think the passing rate can prove much. Current process
> requires that till the test passes on harmony build it is not checked
> in to the source code if I have not missing something. Although there
> is some exceptions, we try to achieve this goal. Thus Harmony is
> assumed to pass all these tests and if there is a functional missing
> or a known bug to be fixed, the test is supposed not exist in
> harmony's code base and cannot be calculated in the passing rate.
+1, these tests should be suppoed to be successful all the time,
since they are designed to guarantee that our modificatons or improvements
are accurate, no regression failures are acceptabel.
For 2, the passing rate can to some degree reflect harmony's maturity
> and I think normally they need not be differentiated why they do not
> pass. Except for few bugs which is due to the application's improper
> dependency on sun's behavior ( For example, Sean has discovered Jython
> assumes the specified order of the entry stored in HashMap. It is
> actually a bug but just coincides to pass on RI. ), the failure among
> the majority of the tests provided by application can reveal one bug
> or one incompatibility that harmony should try to resolve since we are
> trying to give a compatible product as RI for user to switch
> seamlessly.
+1 again.
Why is the application test passrate important to Harmony?
Becasue it indicate that our code qualitifed and worth using.(find bugs of
couse).
This will attract more and more admission.
> --
> > With best regards,
> > Alexei,
> > ESSD, Intel
> >
>
>
> --
> Leo Li
> China Software Development Lab, IBM
>
--
Sean Qiu
China Software Development Lab, IBM
Re: [buildtest] pass rate definition
Posted by Leo Li <li...@gmail.com>.
On 10/18/07, Alexei Fedotov <al...@gmail.com> wrote:
> Hello,
>
> I'm involved in numerous discussions on the subject, and want to make
> these discussions transparent to those community members who are
> interested. Imagine we have a test suite which contains five following
> tests:
>
> BuggyTest.java
> The test fails due to the test bug.
>
> FailingReferenceTest.java
> The test fails on Harmony and passes on RI. The test design does
> not imply that the test should pass.
>
> IntermittentlyFailingTest.java
> The test fails intermittently due to HDK bug.
>
> UnsupportedTest.java
> The test produces an expected fail due to unimplemented
> functionality in HDK.
>
> FailingTest.java
> The test fails due to HDK bug.
>
> PassingTest.java
> This one prints PASSED and completes successfully.
>
> What would be the correct formula to define a pass rate? All agree
> that the rate is a number of passed tests divided to a total number of
> tests. Then people start to argue what are the numerator and the
> denominator.
>
> One may say, that he counts any failures as bugs. Then she gets 16.66%
> pass rate. Others get 50%, ignoring all fail reasons except the one
> which produces a fixable HDK failure.
>
> If anyone could share common sense knowledge or Apache practices on
> the subject, this would be interesting.
>
>
I think how to define the passing rate is not only related to the
reason of the failure tests but as the scope of the tests itself and
what the passing rate means as well.
Current tests can be separated to two category:
1. Tests provided by harmony developers.
2. Tests provided by applications.
For 1, I do not think the passing rate can prove much. Current process
requires that till the test passes on harmony build it is not checked
in to the source code if I have not missing something. Although there
is some exceptions, we try to achieve this goal. Thus Harmony is
assumed to pass all these tests and if there is a functional missing
or a known bug to be fixed, the test is supposed not exist in
harmony's code base and cannot be calculated in the passing rate.
For 2, the passing rate can to some degree reflect harmony's maturity
and I think normally they need not be differentiated why they do not
pass. Except for few bugs which is due to the application's improper
dependency on sun's behavior ( For example, Sean has discovered Jython
assumes the specified order of the entry stored in HashMap. It is
actually a bug but just coincides to pass on RI. ), the failure among
the majority of the tests provided by application can reveal one bug
or one incompatibility that harmony should try to resolve since we are
trying to give a compatible product as RI for user to switch
seamlessly.
> --
> With best regards,
> Alexei,
> ESSD, Intel
>
--
Leo Li
China Software Development Lab, IBM
Re: [buildtest] pass rate definition
Posted by Sean Qiu <se...@gmail.com>.
I am puzzle by this issues too.
For normal application, the passrate is easy to estimate.
While for harmony, we have a reference implementation to follow up.
There is no doubt that "BuggyTest", "IntermittentlyFailingTest",
"UnsupportedTest" ,"FailingTest" should be taken into our consideration.
The only question in issue is about "FailingReferenceTest" (does this mean
no-bug-difference here?).I prefer to consider them as the test's bugs rather
than our bugs. And it is accepable for most of the case.
But if there is broad consensus on this issue between developers, whose
details are not defined by reference, we still need modify our
implementation to comply with the consensus to make the test pass.
Anyway, we can take "FailingReferenceTest" as the test's bug rather than our
implementation. Since we comply with the reference anyway, this should be
the most important criteria. So my suggestion is :
Passrate = 1 - ( "BuggyTest"+ "IntermittentlyFailingTest" +
"UnsupportedTest" + "FailingTest" ) / Total
2007/10/18, Alexei Fedotov <al...@gmail.com>:
>
> Hello,
>
> I'm involved in numerous discussions on the subject, and want to make
> these discussions transparent to those community members who are
> interested. Imagine we have a test suite which contains five following
> tests:
>
> BuggyTest.java
> The test fails due to the test bug.
>
> FailingReferenceTest.java
> The test fails on Harmony and passes on RI. The test design does
> not imply that the test should pass.
>
> IntermittentlyFailingTest.java
> The test fails intermittently due to HDK bug.
>
> UnsupportedTest.java
> The test produces an expected fail due to unimplemented
> functionality in HDK.
>
> FailingTest.java
> The test fails due to HDK bug.
>
> PassingTest.java
> This one prints PASSED and completes successfully.
>
> What would be the correct formula to define a pass rate? All agree
> that the rate is a number of passed tests divided to a total number of
> tests. Then people start to argue what are the numerator and the
> denominator.
>
> One may say, that he counts any failures as bugs. Then she gets 16.66%
> pass rate. Others get 50%, ignoring all fail reasons except the one
> which produces a fixable HDK failure.
>
> If anyone could share common sense knowledge or Apache practices on
> the subject, this would be interesting.
>
>
> --
> With best regards,
> Alexei,
> ESSD, Intel
>
--
Sean Qiu
China Software Development Lab, IBM