You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Michael McCandless <lu...@mikemccandless.com> on 2009/02/03 21:15:31 UTC
failure in TestTrieRangeQuery
I just had this failure happen:
[junit] Testcase:
testRangeSplit_4bit(org.apache.lucene.search.trie.TestTrieRangeQuery):
FAILED
[junit] Returned count of range query must be equal to exclusive
range length expected:<0> but was:<-1>
[junit] junit.framework.AssertionFailedError: Returned count of
range query must be equal to exclusive range length expected:<0> but
was:<-1>
[junit] at
org
.apache
.lucene
.search.trie.TestTrieRangeQuery.testRangeSplit(TestTrieRangeQuery.java:
203)
[junit] at
org
.apache
.lucene
.search
.trie.TestTrieRangeQuery.testRangeSplit_4bit(TestTrieRangeQuery.java:
220)
It's not repeatable, which is fine (because the test has randomness,
which we should leave in there).
I think it's a false failure; it happened because upper and lower were
the same value.
Uwe does that sound right? If so maybe we can just add this:
if (upper == lower) {
upper = 1+lower;
}
?
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
RE: failure in TestTrieRangeQuery
Posted by Uwe Schindler <uw...@thetaphi.de>.
I look into it shortly, this is my fault when I updated the test the last
time I think.
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Michael McCandless [mailto:lucene@mikemccandless.com]
> Sent: Tuesday, February 03, 2009 9:16 PM
> To: java-dev@lucene.apache.org
> Subject: failure in TestTrieRangeQuery
>
>
> I just had this failure happen:
>
> [junit] Testcase:
> testRangeSplit_4bit(org.apache.lucene.search.trie.TestTrieRangeQuery):
> FAILED
> [junit] Returned count of range query must be equal to exclusive
> range length expected:<0> but was:<-1>
> [junit] junit.framework.AssertionFailedError: Returned count of
> range query must be equal to exclusive range length expected:<0> but
> was:<-1>
> [junit] at
> org
> .apache
> .lucene
> .search.trie.TestTrieRangeQuery.testRangeSplit(TestTrieRangeQuery.java:
> 203)
> [junit] at
> org
> .apache
> .lucene
> .search
> .trie.TestTrieRangeQuery.testRangeSplit_4bit(TestTrieRangeQuery.java:
> 220)
>
> It's not repeatable, which is fine (because the test has randomness,
> which we should leave in there).
>
> I think it's a false failure; it happened because upper and lower were
> the same value.
>
> Uwe does that sound right? If so maybe we can just add this:
>
> if (upper == lower) {
> upper = 1+lower;
> }
>
> ?
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: failure in TestTrieRangeQuery
Posted by Stefan Trcek <wz...@abas.de>.
On Wednesday 04 February 2009 12:01:54 Michael McCandless wrote:
> Though: I thought JUnit invokes tests in the sequential order as they
> are defined in your class? (I'm not sure about this... it's just
> what seems to be the case).
Just looking at one of my unit tests: It reorders in some cases. If you
need order or setup / cleanup, the FAQ
http://junit.sourceforge.net/doc/faq/faq.htm#tests_2 is a good starting
point. It also states:
> The ordering of test-method invocations is not guaranteed, ...
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: failure in TestTrieRangeQuery
Posted by Michael McCandless <lu...@mikemccandless.com>.
Great!!
Yes please open JIRA.
Mike
Uwe Schindler wrote:
> Hi,
>
> atached is a patch for LuceneTestCase and a implementation in
> TestTrieRangeQuery. It overrides the protected method runTest() and
> inserts
> a try-catch around the super.runTest() call. Two new methods
> newRandom() and
> newRandom(long) are available for the test case. As each test case
> is run in
> an own TestCase object instance (so 5 test methods in a class
> instantiate 5
> instances each method working in separate), the random seed is saved
> on
> newRandom() and when the test fails with any Throwable, a message
> with the
> seed (if not null) is printed out. If newRandom was never called no
> message
> will be printed.
>
> This patch has only one problem: If a single test method calls
> newRandom()
> more than once, only the last seed is saved and printed out. But
> each test
> method in a Testcase should call newRandom() exactly once for usage
> during
> the execution of this test method. And it is not thread save (no
> sync, no
> volatile), but for tests it's unimportant.
>
> Maybe we open a JIRA issue?
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>> -----Original Message-----
>> From: Michael McCandless [mailto:lucene@mikemccandless.com]
>> Sent: Wednesday, February 04, 2009 12:02 PM
>> To: java-dev@lucene.apache.org
>> Subject: Re: failure in TestTrieRangeQuery
>>
>>
>> Sweet!
>>
>> I was wondering (but didn't dig) whether we could extend
>> LuceneTestCase to expose a getRandom() method (which'd record the
>> seed), and then override invocation of a test (which I'm not sure
>> JUnit allows you to do) to add a try/finally that prints out the
>> seeds.
>>
>> Though: I thought JUnit invokes tests in the sequential order as they
>> are defined in your class? (I'm not sure about this... it's just
>> what
>> seems to be the case). And even if it is the case, it's not clear
>> that's guaranteed as part of JUnit's "contract". If it isn't, we
>> could have getRandom take a String name and then on exception we
>> print
>> out the full name -> seed for all getRandom calls for that test?
>>
>> I'd like to to find a simple common API, if we can, so that we can
>> fix
>> all tests that use Random to use it... though really the mods you had
>> to make are fairly minimal, so we could simply adopt that per test
>> too.
>>
>> Mike
>>
>> Uwe Schindler wrote:
>>
>>> Hi,
>>>
>>>>> : By allowing Random to randomly seed itself, we effectively
>>>>> test a
>>>>> much
>>>>> : much larger space, ie every time we all run the test, it's
>>>>> different. We can
>>>>> : potentially cast a much larger net than a fixed seed.
>>>>>
>>>>> i guess i'm just in favor of less randomness and more iterations.
>>>>>
>>>>> : Fixing the bug is the "easy" part; discovering a bug is
>>>>> present is
>>>>> where
>>>>> : we need all the help we can get ;)
>>>>>
>>>>> yes, but knowing a bug is there w/o having any idea what it is or
>>>>> how to
>>>>> trigger it can be very frustrating.
>>>>
>>>> I agree, it's frustrating. But I'd prefer to know the bug is there
>>>> and then
>>>> writhe in frustration at not being able to reproduce it very
>>>> easily,
>>>> then let
>>>> the bug go undetected. I guess ignorance is not bliss, for me ;)
>>>>
>>>>> it would be enough for tests to pick a random number, log it, and
>>>>> then use
>>>>> it as the seed ... that way if you get a failure you at least know
>>>>> what
>>>>> seed was used and you can then hardcode it temporarily to
>>>>> reproduce/
>>>>> debug
>>>>
>>>> +1! I like this approach. We could record the seed up front, and
>>>> then in
>>>> a try/finally if the test failed, print the seed.
>>>
>>> I implemented this for TestTrieRangeQuery (see patch). I catch
>>> java.lang.Error and print the random seed recorded before (The
>>> seed is
>>> generated by a static Random instance for each test method in
>>> separate:
>>> Because you cannot predict the order of tests, each test method
>>> should have
>>> its own Random instance). As both the Java 1.4 AssertionError and
>>> the jUnit
>>> AssertionFailedError are subclasses of Error, they can be catched
>>> and
>>> rethrown easily.
>>>
>>> Uwe
>>> <random-trie-
>>> test
>>> .patch
>>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
> <LuceneTestCase-
> random
> .patch
> >---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
RE: failure in TestTrieRangeQuery
Posted by Uwe Schindler <uw...@thetaphi.de>.
I forgot to mention: If a test fails, the message using the seed is printed
to stdout. The developer can then change the test temporarily:
LuceneTestCase.newRandom() -> LuceneTestCase.newRandom(long)
using the seed from the failed test printout.
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Uwe Schindler [mailto:uwe@thetaphi.de]
> Sent: Wednesday, February 04, 2009 2:22 PM
> To: java-dev@lucene.apache.org
> Subject: RE: failure in TestTrieRangeQuery
>
> Hi,
>
> atached is a patch for LuceneTestCase and a implementation in
> TestTrieRangeQuery. It overrides the protected method runTest() and
> inserts
> a try-catch around the super.runTest() call. Two new methods newRandom()
> and
> newRandom(long) are available for the test case. As each test case is run
> in
> an own TestCase object instance (so 5 test methods in a class instantiate
> 5
> instances each method working in separate), the random seed is saved on
> newRandom() and when the test fails with any Throwable, a message with the
> seed (if not null) is printed out. If newRandom was never called no
> message
> will be printed.
>
> This patch has only one problem: If a single test method calls newRandom()
> more than once, only the last seed is saved and printed out. But each test
> method in a Testcase should call newRandom() exactly once for usage during
> the execution of this test method. And it is not thread save (no sync, no
> volatile), but for tests it's unimportant.
>
> Maybe we open a JIRA issue?
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
> > -----Original Message-----
> > From: Michael McCandless [mailto:lucene@mikemccandless.com]
> > Sent: Wednesday, February 04, 2009 12:02 PM
> > To: java-dev@lucene.apache.org
> > Subject: Re: failure in TestTrieRangeQuery
> >
> >
> > Sweet!
> >
> > I was wondering (but didn't dig) whether we could extend
> > LuceneTestCase to expose a getRandom() method (which'd record the
> > seed), and then override invocation of a test (which I'm not sure
> > JUnit allows you to do) to add a try/finally that prints out the seeds.
> >
> > Though: I thought JUnit invokes tests in the sequential order as they
> > are defined in your class? (I'm not sure about this... it's just what
> > seems to be the case). And even if it is the case, it's not clear
> > that's guaranteed as part of JUnit's "contract". If it isn't, we
> > could have getRandom take a String name and then on exception we print
> > out the full name -> seed for all getRandom calls for that test?
> >
> > I'd like to to find a simple common API, if we can, so that we can fix
> > all tests that use Random to use it... though really the mods you had
> > to make are fairly minimal, so we could simply adopt that per test too.
> >
> > Mike
> >
> > Uwe Schindler wrote:
> >
> > > Hi,
> > >
> > >>> : By allowing Random to randomly seed itself, we effectively test a
> > >>> much
> > >>> : much larger space, ie every time we all run the test, it's
> > >>> different. We can
> > >>> : potentially cast a much larger net than a fixed seed.
> > >>>
> > >>> i guess i'm just in favor of less randomness and more iterations.
> > >>>
> > >>> : Fixing the bug is the "easy" part; discovering a bug is present is
> > >>> where
> > >>> : we need all the help we can get ;)
> > >>>
> > >>> yes, but knowing a bug is there w/o having any idea what it is or
> > >>> how to
> > >>> trigger it can be very frustrating.
> > >>
> > >> I agree, it's frustrating. But I'd prefer to know the bug is there
> > >> and then
> > >> writhe in frustration at not being able to reproduce it very easily,
> > >> then let
> > >> the bug go undetected. I guess ignorance is not bliss, for me ;)
> > >>
> > >>> it would be enough for tests to pick a random number, log it, and
> > >>> then use
> > >>> it as the seed ... that way if you get a failure you at least know
> > >>> what
> > >>> seed was used and you can then hardcode it temporarily to reproduce/
> > >>> debug
> > >>
> > >> +1! I like this approach. We could record the seed up front, and
> > >> then in
> > >> a try/finally if the test failed, print the seed.
> > >
> > > I implemented this for TestTrieRangeQuery (see patch). I catch
> > > java.lang.Error and print the random seed recorded before (The seed is
> > > generated by a static Random instance for each test method in
> > > separate:
> > > Because you cannot predict the order of tests, each test method
> > > should have
> > > its own Random instance). As both the Java 1.4 AssertionError and
> > > the jUnit
> > > AssertionFailedError are subclasses of Error, they can be catched and
> > > rethrown easily.
> > >
> > > Uwe
> > > <random-trie-
> > > test
> > > .patch
> > > >---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-dev-help@lucene.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
RE: failure in TestTrieRangeQuery
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
atached is a patch for LuceneTestCase and a implementation in
TestTrieRangeQuery. It overrides the protected method runTest() and inserts
a try-catch around the super.runTest() call. Two new methods newRandom() and
newRandom(long) are available for the test case. As each test case is run in
an own TestCase object instance (so 5 test methods in a class instantiate 5
instances each method working in separate), the random seed is saved on
newRandom() and when the test fails with any Throwable, a message with the
seed (if not null) is printed out. If newRandom was never called no message
will be printed.
This patch has only one problem: If a single test method calls newRandom()
more than once, only the last seed is saved and printed out. But each test
method in a Testcase should call newRandom() exactly once for usage during
the execution of this test method. And it is not thread save (no sync, no
volatile), but for tests it's unimportant.
Maybe we open a JIRA issue?
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Michael McCandless [mailto:lucene@mikemccandless.com]
> Sent: Wednesday, February 04, 2009 12:02 PM
> To: java-dev@lucene.apache.org
> Subject: Re: failure in TestTrieRangeQuery
>
>
> Sweet!
>
> I was wondering (but didn't dig) whether we could extend
> LuceneTestCase to expose a getRandom() method (which'd record the
> seed), and then override invocation of a test (which I'm not sure
> JUnit allows you to do) to add a try/finally that prints out the seeds.
>
> Though: I thought JUnit invokes tests in the sequential order as they
> are defined in your class? (I'm not sure about this... it's just what
> seems to be the case). And even if it is the case, it's not clear
> that's guaranteed as part of JUnit's "contract". If it isn't, we
> could have getRandom take a String name and then on exception we print
> out the full name -> seed for all getRandom calls for that test?
>
> I'd like to to find a simple common API, if we can, so that we can fix
> all tests that use Random to use it... though really the mods you had
> to make are fairly minimal, so we could simply adopt that per test too.
>
> Mike
>
> Uwe Schindler wrote:
>
> > Hi,
> >
> >>> : By allowing Random to randomly seed itself, we effectively test a
> >>> much
> >>> : much larger space, ie every time we all run the test, it's
> >>> different. We can
> >>> : potentially cast a much larger net than a fixed seed.
> >>>
> >>> i guess i'm just in favor of less randomness and more iterations.
> >>>
> >>> : Fixing the bug is the "easy" part; discovering a bug is present is
> >>> where
> >>> : we need all the help we can get ;)
> >>>
> >>> yes, but knowing a bug is there w/o having any idea what it is or
> >>> how to
> >>> trigger it can be very frustrating.
> >>
> >> I agree, it's frustrating. But I'd prefer to know the bug is there
> >> and then
> >> writhe in frustration at not being able to reproduce it very easily,
> >> then let
> >> the bug go undetected. I guess ignorance is not bliss, for me ;)
> >>
> >>> it would be enough for tests to pick a random number, log it, and
> >>> then use
> >>> it as the seed ... that way if you get a failure you at least know
> >>> what
> >>> seed was used and you can then hardcode it temporarily to reproduce/
> >>> debug
> >>
> >> +1! I like this approach. We could record the seed up front, and
> >> then in
> >> a try/finally if the test failed, print the seed.
> >
> > I implemented this for TestTrieRangeQuery (see patch). I catch
> > java.lang.Error and print the random seed recorded before (The seed is
> > generated by a static Random instance for each test method in
> > separate:
> > Because you cannot predict the order of tests, each test method
> > should have
> > its own Random instance). As both the Java 1.4 AssertionError and
> > the jUnit
> > AssertionFailedError are subclasses of Error, they can be catched and
> > rethrown easily.
> >
> > Uwe
> > <random-trie-
> > test
> > .patch
> > >---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: failure in TestTrieRangeQuery
Posted by Michael McCandless <lu...@mikemccandless.com>.
Sweet!
I was wondering (but didn't dig) whether we could extend
LuceneTestCase to expose a getRandom() method (which'd record the
seed), and then override invocation of a test (which I'm not sure
JUnit allows you to do) to add a try/finally that prints out the seeds.
Though: I thought JUnit invokes tests in the sequential order as they
are defined in your class? (I'm not sure about this... it's just what
seems to be the case). And even if it is the case, it's not clear
that's guaranteed as part of JUnit's "contract". If it isn't, we
could have getRandom take a String name and then on exception we print
out the full name -> seed for all getRandom calls for that test?
I'd like to to find a simple common API, if we can, so that we can fix
all tests that use Random to use it... though really the mods you had
to make are fairly minimal, so we could simply adopt that per test too.
Mike
Uwe Schindler wrote:
> Hi,
>
>>> : By allowing Random to randomly seed itself, we effectively test a
>>> much
>>> : much larger space, ie every time we all run the test, it's
>>> different. We can
>>> : potentially cast a much larger net than a fixed seed.
>>>
>>> i guess i'm just in favor of less randomness and more iterations.
>>>
>>> : Fixing the bug is the "easy" part; discovering a bug is present is
>>> where
>>> : we need all the help we can get ;)
>>>
>>> yes, but knowing a bug is there w/o having any idea what it is or
>>> how to
>>> trigger it can be very frustrating.
>>
>> I agree, it's frustrating. But I'd prefer to know the bug is there
>> and then
>> writhe in frustration at not being able to reproduce it very easily,
>> then let
>> the bug go undetected. I guess ignorance is not bliss, for me ;)
>>
>>> it would be enough for tests to pick a random number, log it, and
>>> then use
>>> it as the seed ... that way if you get a failure you at least know
>>> what
>>> seed was used and you can then hardcode it temporarily to reproduce/
>>> debug
>>
>> +1! I like this approach. We could record the seed up front, and
>> then in
>> a try/finally if the test failed, print the seed.
>
> I implemented this for TestTrieRangeQuery (see patch). I catch
> java.lang.Error and print the random seed recorded before (The seed is
> generated by a static Random instance for each test method in
> separate:
> Because you cannot predict the order of tests, each test method
> should have
> its own Random instance). As both the Java 1.4 AssertionError and
> the jUnit
> AssertionFailedError are subclasses of Error, they can be catched and
> rethrown easily.
>
> Uwe
> <random-trie-
> test
> .patch
> >---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
RE: failure in TestTrieRangeQuery
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
> > : By allowing Random to randomly seed itself, we effectively test a
> > much
> > : much larger space, ie every time we all run the test, it's
> > different. We can
> > : potentially cast a much larger net than a fixed seed.
> >
> > i guess i'm just in favor of less randomness and more iterations.
> >
> > : Fixing the bug is the "easy" part; discovering a bug is present is
> > where
> > : we need all the help we can get ;)
> >
> > yes, but knowing a bug is there w/o having any idea what it is or
> > how to
> > trigger it can be very frustrating.
>
> I agree, it's frustrating. But I'd prefer to know the bug is there
> and then
> writhe in frustration at not being able to reproduce it very easily,
> then let
> the bug go undetected. I guess ignorance is not bliss, for me ;)
>
> > it would be enough for tests to pick a random number, log it, and
> > then use
> > it as the seed ... that way if you get a failure you at least know
> > what
> > seed was used and you can then hardcode it temporarily to reproduce/
> > debug
>
> +1! I like this approach. We could record the seed up front, and
> then in
> a try/finally if the test failed, print the seed.
I implemented this for TestTrieRangeQuery (see patch). I catch
java.lang.Error and print the random seed recorded before (The seed is
generated by a static Random instance for each test method in separate:
Because you cannot predict the order of tests, each test method should have
its own Random instance). As both the Java 1.4 AssertionError and the jUnit
AssertionFailedError are subclasses of Error, they can be catched and
rethrown easily.
Uwe
Re: failure in TestTrieRangeQuery
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Feb 3, 2009, at 7:17 PM, Chris Hostetter wrote:
>
> : By allowing Random to randomly seed itself, we effectively test a
> much
> : much larger space, ie every time we all run the test, it's
> different. We can
> : potentially cast a much larger net than a fixed seed.
>
> i guess i'm just in favor of less randomness and more iterations.
>
> : Fixing the bug is the "easy" part; discovering a bug is present is
> where
> : we need all the help we can get ;)
>
> yes, but knowing a bug is there w/o having any idea what it is or
> how to
> trigger it can be very frustrating.
I agree, it's frustrating. But I'd prefer to know the bug is there
and then
writhe in frustration at not being able to reproduce it very easily,
then let
the bug go undetected. I guess ignorance is not bliss, for me ;)
> it would be enough for tests to pick a random number, log it, and
> then use
> it as the seed ... that way if you get a failure you at least know
> what
> seed was used and you can then hardcode it temporarily to reproduce/
> debug
+1! I like this approach. We could record the seed up front, and
then in
a try/finally if the test failed, print the seed.
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: failure in TestTrieRangeQuery
Posted by Chris Hostetter <ho...@fucit.org>.
: By allowing Random to randomly seed itself, we effectively test a much
: much larger space, ie every time we all run the test, it's different. We can
: potentially cast a much larger net than a fixed seed.
i guess i'm just in favor of less randomness and more iterations.
: Fixing the bug is the "easy" part; discovering a bug is present is where
: we need all the help we can get ;)
yes, but knowing a bug is there w/o having any idea what it is or how to
trigger it can be very frustrating.
it would be enough for tests to pick a random number, log it, and then use
it as the seed ... that way if you get a failure you at least know what
seed was used and you can then hardcode it temporarily to reproduce/debug
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: failure in TestTrieRangeQuery
Posted by Michael McCandless <lu...@mikemccandless.com>.
I used to favor determinism/repeatability (fixed seed to Random) too,
but I recently changed my mind.
By allowing Random to randomly seed itself, we effectively test a much
much larger space, ie every time we all run the test, it's different.
We can
potentially cast a much larger net than a fixed seed.
This way, if there is a bug lurking, it will eventually be seen by
someone.
Once the bug is discovered (which is 90% of the "challenge"), we can
always run many iterations, test different seeds, etc., to track down
the cause & fix it.
Fixing the bug is the "easy" part; discovering a bug is present is where
we need all the help we can get ;)
Mike
Chris Hostetter wrote:
>
> : It's not repeatable, which is fine (because the test has
> randomness, which we
> : should leave in there).
>
> Side note: while i agree that test with randomness (ie: do lots of
> iterations over randomly selected data) are good to help find weird
> edge
> casees you might not otherwise think to explicitly code tests for,
> i've
> found by painful experience that generating the random data using a
> Random
> object with a hard coded seed is a good practice -- that way you get
> the
> benefits of hte Random data, but the test is also reproducable.
>
> otherwise you get test failures that that you *think* you understand
> the
> cause of based on the stack trace, but without knowing exactly what
> input
> was used, you can't be sure you're fixing the "right" bug.
>
>
>
> -Hoss
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: failure in TestTrieRangeQuery
Posted by Chris Hostetter <ho...@fucit.org>.
: It's not repeatable, which is fine (because the test has randomness, which we
: should leave in there).
Side note: while i agree that test with randomness (ie: do lots of
iterations over randomly selected data) are good to help find weird edge
casees you might not otherwise think to explicitly code tests for, i've
found by painful experience that generating the random data using a Random
object with a hard coded seed is a good practice -- that way you get the
benefits of hte Random data, but the test is also reproducable.
otherwise you get test failures that that you *think* you understand the
cause of based on the stack trace, but without knowing exactly what input
was used, you can't be sure you're fixing the "right" bug.
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: failure in TestTrieRangeQuery
Posted by Michael McCandless <lu...@mikemccandless.com>.
OK, no problem, thanks for fixing so quickly!
Mike
Uwe Schindler wrote:
> Hi Mike,
>
> I fixed the test. The problem was as you noted because of the both-
> sides
> exclusive range:
>
> assertEquals("Returned count of range query must be equal to
> exclusive
> range length", tTopDocs.totalHits, Math.max(upper-lower-1, 0) );
>
> Your fix was not enough, because the test then would fail, if
> upper==lower==9999, because 10000 cannot be found in index.
>
> Sorry for two commits, my first fix was one Math.max() too much :)
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
>> -----Original Message-----
>> From: Michael McCandless [mailto:lucene@mikemccandless.com]
>> Sent: Tuesday, February 03, 2009 9:16 PM
>> To: java-dev@lucene.apache.org
>> Subject: failure in TestTrieRangeQuery
>>
>>
>> I just had this failure happen:
>>
>> [junit] Testcase:
>> testRangeSplit_4bit
>> (org.apache.lucene.search.trie.TestTrieRangeQuery):
>> FAILED
>> [junit] Returned count of range query must be equal to exclusive
>> range length expected:<0> but was:<-1>
>> [junit] junit.framework.AssertionFailedError: Returned count of
>> range query must be equal to exclusive range length expected:<0> but
>> was:<-1>
>> [junit] at
>> org
>> .apache
>> .lucene
>> .search
>> .trie.TestTrieRangeQuery.testRangeSplit(TestTrieRangeQuery.java:
>> 203)
>> [junit] at
>> org
>> .apache
>> .lucene
>> .search
>> .trie.TestTrieRangeQuery.testRangeSplit_4bit(TestTrieRangeQuery.java:
>> 220)
>>
>> It's not repeatable, which is fine (because the test has randomness,
>> which we should leave in there).
>>
>> I think it's a false failure; it happened because upper and lower
>> were
>> the same value.
>>
>> Uwe does that sound right? If so maybe we can just add this:
>>
>> if (upper == lower) {
>> upper = 1+lower;
>> }
>>
>> ?
>>
>> Mike
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
RE: failure in TestTrieRangeQuery
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi Mike,
I fixed the test. The problem was as you noted because of the both-sides
exclusive range:
assertEquals("Returned count of range query must be equal to exclusive
range length", tTopDocs.totalHits, Math.max(upper-lower-1, 0) );
Your fix was not enough, because the test then would fail, if
upper==lower==9999, because 10000 cannot be found in index.
Sorry for two commits, my first fix was one Math.max() too much :)
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Michael McCandless [mailto:lucene@mikemccandless.com]
> Sent: Tuesday, February 03, 2009 9:16 PM
> To: java-dev@lucene.apache.org
> Subject: failure in TestTrieRangeQuery
>
>
> I just had this failure happen:
>
> [junit] Testcase:
> testRangeSplit_4bit(org.apache.lucene.search.trie.TestTrieRangeQuery):
> FAILED
> [junit] Returned count of range query must be equal to exclusive
> range length expected:<0> but was:<-1>
> [junit] junit.framework.AssertionFailedError: Returned count of
> range query must be equal to exclusive range length expected:<0> but
> was:<-1>
> [junit] at
> org
> .apache
> .lucene
> .search.trie.TestTrieRangeQuery.testRangeSplit(TestTrieRangeQuery.java:
> 203)
> [junit] at
> org
> .apache
> .lucene
> .search
> .trie.TestTrieRangeQuery.testRangeSplit_4bit(TestTrieRangeQuery.java:
> 220)
>
> It's not repeatable, which is fine (because the test has randomness,
> which we should leave in there).
>
> I think it's a false failure; it happened because upper and lower were
> the same value.
>
> Uwe does that sound right? If so maybe we can just add this:
>
> if (upper == lower) {
> upper = 1+lower;
> }
>
> ?
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org