You are viewing a plain text version of this content. The canonical link for it is here.
Posted to ojb-dev@db.apache.org by ol...@ppi.de on 2003/11/10 17:07:37 UTC
non-deterministic test failures, was: RE: junit test failures
Hello again,
I am a little bit annoyed at the moment because
I spent several hours figuring out how I corrupted
test cases until I finally discovered that
some test failures are not reproducable.
More precisely:
I checked out current CVS head into an empty
directory and then repeated
bin\build.bat junit-no-compile junit-report
several times. In between I saved the
target/test directory. Then I compared the
results. There are various results. Some
test fail non-deterministically. Besides, there
seem to be dependencies such as:
test Y succeeds iff test X runs before test Y.
Discovering that took quite a while, and I was
only doing the tests to make sure that some minor
cleanup has not done any harm.
I wonder if someone else has experienced that?
How often do you run the tests?
In order to benefit from the junit tests, every
developer should be able to rely on that all tests
that are not marked as 'allowed to fail' do not fail
unless he or she corrupts them.
> -----Original Message-----
> From: oliver.matz@ppi.de [mailto:oliver.matz@ppi.de]
> Sent: Monday, November 10, 2003 3:03 PM
> To: ojb-dev@db.apache.org
> > Can I compare these results to the 'official' ones?
> > The link http://db.apache.org/ojb/junit-report.html
> > on http://db.apache.org/ojb/maven-reports.html is broken.
>
> the tests have changed, but this issue is still open.
>
> Does someone take care of this?
>
> Can we achieve any consensus about how to deal with failing
> tests? I have stated my opinion about this in another
> thread.
This issue is really important to me.
I will have a look at some tests that fail sometimes.
Olli
---------------------------------------------------------------------
To unsubscribe, e-mail: ojb-dev-unsubscribe@db.apache.org
For additional commands, e-mail: ojb-dev-help@db.apache.org
Re: non-deterministic test failures, was: RE: junit test failures
Posted by Thomas Mahler <th...@web.de>.
Hi Olli,
That's a known problem
If you try ant junit-quick, a fresh copy of the testdatabase is
installed before executing junit-no-compile.
going this way you avoid the dependency problems.
I admit that in an ideal world there should not be such dependencies...
(But in an ideal world software won't have bugs anyway ;-))
cu,
Thomas
oliver.matz@ppi.de wrote:
> Hello again,
>
> I am a little bit annoyed at the moment because
> I spent several hours figuring out how I corrupted
> test cases until I finally discovered that
> some test failures are not reproducable.
>
> More precisely:
> I checked out current CVS head into an empty
> directory and then repeated
>
> bin\build.bat junit-no-compile junit-report
>
> several times. In between I saved the
> target/test directory. Then I compared the
> results. There are various results. Some
> test fail non-deterministically. Besides, there
> seem to be dependencies such as:
>
> test Y succeeds iff test X runs before test Y.
>
> Discovering that took quite a while, and I was
> only doing the tests to make sure that some minor
> cleanup has not done any harm.
>
> I wonder if someone else has experienced that?
> How often do you run the tests?
>
> In order to benefit from the junit tests, every
> developer should be able to rely on that all tests
> that are not marked as 'allowed to fail' do not fail
> unless he or she corrupts them.
>
>
>>-----Original Message-----
>>From: oliver.matz@ppi.de [mailto:oliver.matz@ppi.de]
>>Sent: Monday, November 10, 2003 3:03 PM
>>To: ojb-dev@db.apache.org
>
>
>>>Can I compare these results to the 'official' ones?
>>>The link http://db.apache.org/ojb/junit-report.html
>>>on http://db.apache.org/ojb/maven-reports.html is broken.
>>
>>the tests have changed, but this issue is still open.
>>
>>Does someone take care of this?
>>
>>Can we achieve any consensus about how to deal with failing
>>tests? I have stated my opinion about this in another
>>thread.
>
>
> This issue is really important to me.
>
> I will have a look at some tests that fail sometimes.
>
> Olli
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: ojb-dev-unsubscribe@db.apache.org
> For additional commands, e-mail: ojb-dev-help@db.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: ojb-dev-unsubscribe@db.apache.org
For additional commands, e-mail: ojb-dev-help@db.apache.org
Re: non-deterministic test failures, was: RE: junit test failures
Posted by Armin Waibel <ar...@code-au-lait.de>.
Hi Jakob,
Jakob Braeuchi wrote:
>
>
> Armin Waibel wrote:
>
>> Hi,
>>
>> Jakob Braeuchi wrote:
>>
>>> hi oli,
>>>
>>> the sequence of the testcases is sometimes a problem. the querytest
>>> ie. relies on a certain number of rows to be available in the
>>> database that's why it may fail if not run as first testcase.
>>
>>
>>
>> I don't know these tests in detail, but I think this should be
>> fixed if possible. Test shouldn't rely on other tests.
>
>
> the do not rely on each other. the problem is that some testcases modify
> the database and do not undo the changes they have made.
>
ahh, ok that's true. I think 90% of the test cases do not
undo work. Maybe it is possible to make the tests more
independable. E.g. use a 'flag' in objects name to find
the according object.
regards,
Armin
>>
>>> also some testcase fail if run more than once without reinitializing
>>> the database.
>>
>>
>
> same problem .
>
>>
>> This should be avoided too, because it is not a "real life test"
>> only run the api against empty "virgin" database tables
>> (e.g. maybe some side-effects only occur when re-run tests against
>> filled tables).
>>
>> just my 2 cent ;-)
>
> euro cents ?
>
>>
>> regards,
>> Armin
>
>
> jakob
>
>>
>>> i run alltests before commiting any change to the repository.
>>>
>>> jakob
>>>
>>>
>>> oliver.matz@ppi.de wrote:
>>>
>>>> Hello again,
>>>>
>>>> I am a little bit annoyed at the moment because
>>>> I spent several hours figuring out how I corrupted
>>>> test cases until I finally discovered that some test failures are
>>>> not reproducable.
>>>>
>>>> More precisely:
>>>> I checked out current CVS head into an empty directory and then
>>>> repeated
>>>>
>>>> bin\build.bat junit-no-compile junit-report
>>>>
>>>> several times. In between I saved the target/test directory. Then I
>>>> compared the results. There are various results. Some
>>>> test fail non-deterministically. Besides, there
>>>> seem to be dependencies such as:
>>>>
>>>> test Y succeeds iff test X runs before test Y.
>>>>
>>>> Discovering that took quite a while, and I was
>>>> only doing the tests to make sure that some minor
>>>> cleanup has not done any harm.
>>>>
>>>> I wonder if someone else has experienced that?
>>>> How often do you run the tests? In order to benefit from the junit
>>>> tests, every developer should be able to rely on that all tests that
>>>> are not marked as 'allowed to fail' do not fail
>>>> unless he or she corrupts them.
>>>>
>>>>
>>>>> -----Original Message-----
>>>>> From: oliver.matz@ppi.de [mailto:oliver.matz@ppi.de]
>>>>> Sent: Monday, November 10, 2003 3:03 PM
>>>>> To: ojb-dev@db.apache.org
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>>> Can I compare these results to the 'official' ones?
>>>>>> The link http://db.apache.org/ojb/junit-report.html
>>>>>> on http://db.apache.org/ojb/maven-reports.html is broken.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> the tests have changed, but this issue is still open.
>>>>>
>>>>> Does someone take care of this?
>>>>>
>>>>> Can we achieve any consensus about how to deal with failing
>>>>> tests? I have stated my opinion about this in another
>>>>> thread.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> This issue is really important to me.
>>>>
>>>> I will have a look at some tests that fail sometimes.
>>>>
>>>> Olli
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: ojb-dev-unsubscribe@db.apache.org
>>>> For additional commands, e-mail: ojb-dev-help@db.apache.org
>>>>
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: ojb-dev-unsubscribe@db.apache.org
>>> For additional commands, e-mail: ojb-dev-help@db.apache.org
>>>
>>>
>>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: ojb-dev-unsubscribe@db.apache.org
>> For additional commands, e-mail: ojb-dev-help@db.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: ojb-dev-unsubscribe@db.apache.org
> For additional commands, e-mail: ojb-dev-help@db.apache.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: ojb-dev-unsubscribe@db.apache.org
For additional commands, e-mail: ojb-dev-help@db.apache.org
Re: non-deterministic test failures, was: RE: junit test failures
Posted by Jakob Braeuchi <jb...@gmx.ch>.
Armin Waibel wrote:
> Hi,
>
> Jakob Braeuchi wrote:
>
>> hi oli,
>>
>> the sequence of the testcases is sometimes a problem. the querytest
>> ie. relies on a certain number of rows to be available in the database
>> that's why it may fail if not run as first testcase.
>
>
> I don't know these tests in detail, but I think this should be
> fixed if possible. Test shouldn't rely on other tests.
the do not rely on each other. the problem is that some testcases modify
the database and do not undo the changes they have made.
>
>> also some testcase fail if run more than once without reinitializing
>> the database.
>
same problem .
>
> This should be avoided too, because it is not a "real life test"
> only run the api against empty "virgin" database tables
> (e.g. maybe some side-effects only occur when re-run tests against
> filled tables).
>
> just my 2 cent ;-)
euro cents ?
>
> regards,
> Armin
jakob
>
>> i run alltests before commiting any change to the repository.
>>
>> jakob
>>
>>
>> oliver.matz@ppi.de wrote:
>>
>>> Hello again,
>>>
>>> I am a little bit annoyed at the moment because
>>> I spent several hours figuring out how I corrupted
>>> test cases until I finally discovered that some test failures are not
>>> reproducable.
>>>
>>> More precisely:
>>> I checked out current CVS head into an empty directory and then repeated
>>>
>>> bin\build.bat junit-no-compile junit-report
>>>
>>> several times. In between I saved the target/test directory. Then I
>>> compared the results. There are various results. Some
>>> test fail non-deterministically. Besides, there
>>> seem to be dependencies such as:
>>>
>>> test Y succeeds iff test X runs before test Y.
>>>
>>> Discovering that took quite a while, and I was
>>> only doing the tests to make sure that some minor
>>> cleanup has not done any harm.
>>>
>>> I wonder if someone else has experienced that?
>>> How often do you run the tests? In order to benefit from the junit
>>> tests, every developer should be able to rely on that all tests that
>>> are not marked as 'allowed to fail' do not fail
>>> unless he or she corrupts them.
>>>
>>>
>>>> -----Original Message-----
>>>> From: oliver.matz@ppi.de [mailto:oliver.matz@ppi.de]
>>>> Sent: Monday, November 10, 2003 3:03 PM
>>>> To: ojb-dev@db.apache.org
>>>
>>>
>>>
>>>
>>>>> Can I compare these results to the 'official' ones?
>>>>> The link http://db.apache.org/ojb/junit-report.html
>>>>> on http://db.apache.org/ojb/maven-reports.html is broken.
>>>>
>>>>
>>>>
>>>> the tests have changed, but this issue is still open.
>>>>
>>>> Does someone take care of this?
>>>>
>>>> Can we achieve any consensus about how to deal with failing
>>>> tests? I have stated my opinion about this in another
>>>> thread.
>>>
>>>
>>>
>>>
>>> This issue is really important to me.
>>>
>>> I will have a look at some tests that fail sometimes.
>>>
>>> Olli
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: ojb-dev-unsubscribe@db.apache.org
>>> For additional commands, e-mail: ojb-dev-help@db.apache.org
>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: ojb-dev-unsubscribe@db.apache.org
>> For additional commands, e-mail: ojb-dev-help@db.apache.org
>>
>>
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: ojb-dev-unsubscribe@db.apache.org
> For additional commands, e-mail: ojb-dev-help@db.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: ojb-dev-unsubscribe@db.apache.org
For additional commands, e-mail: ojb-dev-help@db.apache.org
Re: non-deterministic test failures, was: RE: junit test failures
Posted by Armin Waibel <ar...@code-au-lait.de>.
Hi,
Jakob Braeuchi wrote:
> hi oli,
>
> the sequence of the testcases is sometimes a problem. the querytest ie.
> relies on a certain number of rows to be available in the database
> that's why it may fail if not run as first testcase.
I don't know these tests in detail, but I think this should be
fixed if possible. Test shouldn't rely on other tests.
> also some testcase
> fail if run more than once without reinitializing the database.
This should be avoided too, because it is not a "real life test"
only run the api against empty "virgin" database tables
(e.g. maybe some side-effects only occur when re-run tests against
filled tables).
just my 2 cent ;-)
regards,
Armin
> i run alltests before commiting any change to the repository.
>
> jakob
>
>
> oliver.matz@ppi.de wrote:
>
>> Hello again,
>>
>> I am a little bit annoyed at the moment because
>> I spent several hours figuring out how I corrupted
>> test cases until I finally discovered that some test failures are not
>> reproducable.
>>
>> More precisely:
>> I checked out current CVS head into an empty directory and then repeated
>>
>> bin\build.bat junit-no-compile junit-report
>>
>> several times. In between I saved the target/test directory. Then I
>> compared the results. There are various results. Some
>> test fail non-deterministically. Besides, there
>> seem to be dependencies such as:
>>
>> test Y succeeds iff test X runs before test Y.
>>
>> Discovering that took quite a while, and I was
>> only doing the tests to make sure that some minor
>> cleanup has not done any harm.
>>
>> I wonder if someone else has experienced that?
>> How often do you run the tests?
>> In order to benefit from the junit tests, every developer should be
>> able to rely on that all tests that are not marked as 'allowed to
>> fail' do not fail
>> unless he or she corrupts them.
>>
>>
>>> -----Original Message-----
>>> From: oliver.matz@ppi.de [mailto:oliver.matz@ppi.de]
>>> Sent: Monday, November 10, 2003 3:03 PM
>>> To: ojb-dev@db.apache.org
>>
>>
>>
>>>> Can I compare these results to the 'official' ones?
>>>> The link http://db.apache.org/ojb/junit-report.html
>>>> on http://db.apache.org/ojb/maven-reports.html is broken.
>>>
>>>
>>> the tests have changed, but this issue is still open.
>>>
>>> Does someone take care of this?
>>>
>>> Can we achieve any consensus about how to deal with failing
>>> tests? I have stated my opinion about this in another
>>> thread.
>>
>>
>>
>> This issue is really important to me.
>>
>> I will have a look at some tests that fail sometimes.
>>
>> Olli
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: ojb-dev-unsubscribe@db.apache.org
>> For additional commands, e-mail: ojb-dev-help@db.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: ojb-dev-unsubscribe@db.apache.org
> For additional commands, e-mail: ojb-dev-help@db.apache.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: ojb-dev-unsubscribe@db.apache.org
For additional commands, e-mail: ojb-dev-help@db.apache.org
Re: non-deterministic test failures, was: RE: junit test failures
Posted by Jakob Braeuchi <jb...@gmx.ch>.
hi oli,
the sequence of the testcases is sometimes a problem. the querytest ie.
relies on a certain number of rows to be available in the database
that's why it may fail if not run as first testcase. also some testcase
fail if run more than once without reinitializing the database.
i run alltests before commiting any change to the repository.
jakob
oliver.matz@ppi.de wrote:
> Hello again,
>
> I am a little bit annoyed at the moment because
> I spent several hours figuring out how I corrupted
> test cases until I finally discovered that
> some test failures are not reproducable.
>
> More precisely:
> I checked out current CVS head into an empty
> directory and then repeated
>
> bin\build.bat junit-no-compile junit-report
>
> several times. In between I saved the
> target/test directory. Then I compared the
> results. There are various results. Some
> test fail non-deterministically. Besides, there
> seem to be dependencies such as:
>
> test Y succeeds iff test X runs before test Y.
>
> Discovering that took quite a while, and I was
> only doing the tests to make sure that some minor
> cleanup has not done any harm.
>
> I wonder if someone else has experienced that?
> How often do you run the tests?
>
> In order to benefit from the junit tests, every
> developer should be able to rely on that all tests
> that are not marked as 'allowed to fail' do not fail
> unless he or she corrupts them.
>
>
>>-----Original Message-----
>>From: oliver.matz@ppi.de [mailto:oliver.matz@ppi.de]
>>Sent: Monday, November 10, 2003 3:03 PM
>>To: ojb-dev@db.apache.org
>
>
>>>Can I compare these results to the 'official' ones?
>>>The link http://db.apache.org/ojb/junit-report.html
>>>on http://db.apache.org/ojb/maven-reports.html is broken.
>>
>>the tests have changed, but this issue is still open.
>>
>>Does someone take care of this?
>>
>>Can we achieve any consensus about how to deal with failing
>>tests? I have stated my opinion about this in another
>>thread.
>
>
> This issue is really important to me.
>
> I will have a look at some tests that fail sometimes.
>
> Olli
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: ojb-dev-unsubscribe@db.apache.org
> For additional commands, e-mail: ojb-dev-help@db.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: ojb-dev-unsubscribe@db.apache.org
For additional commands, e-mail: ojb-dev-help@db.apache.org