You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Benson Margulies <bi...@gmail.com> on 2012/04/02 23:28:49 UTC
Repeatability of results
We've observed something that, in some ways, is not surprising.
If you take a set of documents that are close in 'score' to some query,
and shuffle them in different orders
and then see what results you get in what order from the reference query,
the scores will vary according to the insertion order.
I can't see any way to argue that it's wrong, but we find it
inconvenient when we are testing something and we want to multithread
the test to speed it up, thus making the insertion order
nondeterministic.
It occurred to me that perhaps you all have some similar concerns in
testing lucene itself, and might have some advice about how to get
around it, thus this email.
We currently observe this with 2.9.1 and 3.5.0.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Repeatability of results
Posted by Chris Hostetter <ho...@fucit.org>.
: OK this could make sense (floating point math is frustrating!).
:
: But, Lucene generally scores one document at a time, so in theory just
: changing its docid shouldn't alter the order of float operations.
i haven't thought this through, but couldn't scorer re-ordering in
BooleanScorer2 possibly? tickle weird little floating point math
excentricities?
if the documents are in diff orders, then the skipTo calls (or are they
called advance() now?) would result in the subScoresr being in a diff
order right? ... so the floats from each subscorer would be added in a
diff order?
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Repeatability of results
Posted by Marvin Humphrey <ma...@rectangular.com>.
On Wed, Apr 4, 2012 at 4:18 PM, Michael McCandless
<lu...@mikemccandless.com> wrote:
> On Wed, Apr 4, 2012 at 6:15 PM, Alan Bawden <al...@basistech.com> wrote:
>> The key observation is that the differences in scores we see are always
>> down around the sixth decimal place -- down where 32-bit floating point
>> loses precision.
---->8 snip 8<----
> But, Lucene generally scores one document at a time, so in theory just
> changing its docid shouldn't alter the order of float operations.
Could this be an excess floating point precision problem?
http://www.rectangular.com/cgi-bin/viewvc.cgi?view=rev&root=kinosearch&revision=6277
Fix excess precision problem affecting SortCollector on x86 platforms.
Before this patch, scores which should have tied did not because the float
score which came from the method call Matcher_Score() would have too much
precision compared to the score which had been stored and then retrieved
from the MatchDoc. This could manifest as unpredictable sort orders for
multi-level sorts incorporating scores.
Also:
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=323#c92
I'd like to welcome the newest members of the bug 323 community, where all
x87 floating point errors in gcc come to die!
Marvin Humphrey
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Repeatability of results
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Wed, Apr 4, 2012 at 6:15 PM, Alan Bawden <al...@basistech.com> wrote:
> So I sat down to try to make a small test case that exhibited this
> behavior, and while I was working on that I thought of a possible
> explanation for what we are seeing. If you agree that my explanation is
> what's going on here, then Benson and I can stop working on making a test
> case, and move on to figuring out how we can live with what may be
> unavoidable behavior.
>
> The key observation is that the differences in scores we see are always
> down around the sixth decimal place -- down where 32-bit floating point
> loses precision. So what we're seeing seems likely to simply be the result
> of the fact that floating point addition isn't associative.
>
> In theory, the order of the documents in an index doesn't matter when
> computing a score, but if the documents are stored in a different order,
> any quantity that is computed using floating point by iterating over the
> set of documents may come out differently due to changes in the order in
> which the documents are processed.
>
> So could something like this cause what we are seeing?
OK this could make sense (floating point math is frustrating!).
But, Lucene generally scores one document at a time, so in theory just
changing its docid shouldn't alter the order of float operations.
Or... is it possible the query is different? EG a BooleanQuery w/
SHOULD clauses in a different order?
Maybe try explain() in the two cases and compare?
Hmm, another idea: are you doing deletions in the test? Or only adds?
Mike McCandless
http://blog.mikemccandless.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Repeatability of results
Posted by Alan Bawden <al...@basistech.com>.
So I sat down to try to make a small test case that exhibited this
behavior, and while I was working on that I thought of a possible
explanation for what we are seeing. If you agree that my explanation is
what's going on here, then Benson and I can stop working on making a test
case, and move on to figuring out how we can live with what may be
unavoidable behavior.
The key observation is that the differences in scores we see are always
down around the sixth decimal place -- down where 32-bit floating point
loses precision. So what we're seeing seems likely to simply be the result
of the fact that floating point addition isn't associative.
In theory, the order of the documents in an index doesn't matter when
computing a score, but if the documents are stored in a different order,
any quantity that is computed using floating point by iterating over the
set of documents may come out differently due to changes in the order in
which the documents are processed.
So could something like this cause what we are seeing?
On Mon, Apr 2, 2012 at 17:41, Benson Margulies <bi...@gmail.com> wrote:
> On Mon, Apr 2, 2012 at 5:33 PM, Michael McCandless
> <lu...@mikemccandless.com> wrote:
>> Hmm that's odd.
>>
>> If the scores were identical I'd expect different sort order, since we
>> tie-break by internal docID.
>>
>> But if the scores are different... the insertion order shouldn't
>> matter. And, the score should not change as a function of insertion
>> order...
>
> Well, I assumed that TF-IDF would wiggle.
>
>>
>> Do you have a small test case?
>
> SInce this surprises you, I will build a test case.
>
>
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>> On Mon, Apr 2, 2012 at 5:28 PM, Benson Margulies <bi...@gmail.com> wrote:
>>> We've observed something that, in some ways, is not surprising.
>>>
>>> If you take a set of documents that are close in 'score' to some query,
>>>
>>> and shuffle them in different orders
>>>
>>> and then see what results you get in what order from the reference query,
>>>
>>> the scores will vary according to the insertion order.
>>>
>>> I can't see any way to argue that it's wrong, but we find it
>>> inconvenient when we are testing something and we want to multithread
>>> the test to speed it up, thus making the insertion order
>>> nondeterministic.
>>>
>>> It occurred to me that perhaps you all have some similar concerns in
>>> testing lucene itself, and might have some advice about how to get
>>> around it, thus this email.
>>>
>>> We currently observe this with 2.9.1 and 3.5.0.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Repeatability of results
Posted by Benson Margulies <bi...@gmail.com>.
On Mon, Apr 2, 2012 at 5:33 PM, Michael McCandless
<lu...@mikemccandless.com> wrote:
> Hmm that's odd.
>
> If the scores were identical I'd expect different sort order, since we
> tie-break by internal docID.
>
> But if the scores are different... the insertion order shouldn't
> matter. And, the score should not change as a function of insertion
> order...
Well, I assumed that TF-IDF would wiggle.
>
> Do you have a small test case?
SInce this surprises you, I will build a test case.
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Mon, Apr 2, 2012 at 5:28 PM, Benson Margulies <bi...@gmail.com> wrote:
>> We've observed something that, in some ways, is not surprising.
>>
>> If you take a set of documents that are close in 'score' to some query,
>>
>> and shuffle them in different orders
>>
>> and then see what results you get in what order from the reference query,
>>
>> the scores will vary according to the insertion order.
>>
>> I can't see any way to argue that it's wrong, but we find it
>> inconvenient when we are testing something and we want to multithread
>> the test to speed it up, thus making the insertion order
>> nondeterministic.
>>
>> It occurred to me that perhaps you all have some similar concerns in
>> testing lucene itself, and might have some advice about how to get
>> around it, thus this email.
>>
>> We currently observe this with 2.9.1 and 3.5.0.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Repeatability of results
Posted by Michael McCandless <lu...@mikemccandless.com>.
Hmm that's odd.
If the scores were identical I'd expect different sort order, since we
tie-break by internal docID.
But if the scores are different... the insertion order shouldn't
matter. And, the score should not change as a function of insertion
order...
Do you have a small test case?
Mike McCandless
http://blog.mikemccandless.com
On Mon, Apr 2, 2012 at 5:28 PM, Benson Margulies <bi...@gmail.com> wrote:
> We've observed something that, in some ways, is not surprising.
>
> If you take a set of documents that are close in 'score' to some query,
>
> and shuffle them in different orders
>
> and then see what results you get in what order from the reference query,
>
> the scores will vary according to the insertion order.
>
> I can't see any way to argue that it's wrong, but we find it
> inconvenient when we are testing something and we want to multithread
> the test to speed it up, thus making the insertion order
> nondeterministic.
>
> It occurred to me that perhaps you all have some similar concerns in
> testing lucene itself, and might have some advice about how to get
> around it, thus this email.
>
> We currently observe this with 2.9.1 and 3.5.0.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org