You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Brian Lamb <br...@journalexperts.com> on 2011/07/22 22:11:01 UTC
Rounding errors in solr
Hi all,
I've noticed some peculiar scoring issues going on in my application. For
example, I have a field that is multivalued and has several records that
have the same value. For example,
<arr name="references">
<str>National Society of Animal Lovers</str>
<str>Nat. Soc. of Ani. Lov.</str>
</arr>
I have about 300 records with that exact value.
Now, when I do a search for references:(national society animal lovers), I
get the following results:
<id>252</id>
<id>159</id>
<id>82</id>
<id>452</id>
<id>105</id>
When I do a search for references:(nat soc ani lov), I get the results
ordered differently:
<id>510</id>
<id>122</id>
<id>501</id>
<id>82</id>
<id>252</id>
When I load all the records that match, I notice that at some point, the
scores aren't the same but differ by only a little:
1.471928 in one and the one before it was 1.471929
I turned on debugQuery=on and the scores for each of those two records are
exactly the same. Therefore, I think there is some kind of rounding error
going on.
Is there a way I can fix this?
Alternatively, can I sort by a rounded version of the score? I tried
sort=round(score,5) but I get the following message:
Can't determine Sort Order: 'round(score,5) ', pos=5
I also tried sort=sum(score,1) just to see if I was using round
incorrectly but I get an error message there too saying score is not a
recognized field.
Please help!
Thanks,
Brian Lamb
Re: Rounding errors in solr
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Mon, Jul 25, 2011 at 10:12 AM, Brian Lamb
<br...@journalexperts.com> wrote:
> Yes and that's causing some problems in my application. Is there a way to
> truncate the 7th decimal place in regards to sorting by the score?
Not built in.
With some Java coding, you could create a post filter that manipulates scores.
http://wiki.apache.org/solr/CommonQueryParameters#Caching_of_filters
-Yonik
http://www.lucidimagination.com
> On Fri, Jul 22, 2011 at 4:27 PM, Yonik Seeley <yo...@lucidimagination.com>
> wrote:
>>
>> On Fri, Jul 22, 2011 at 4:11 PM, Brian Lamb
>> <br...@journalexperts.com> wrote:
>> > I've noticed some peculiar scoring issues going on in my application.
>> > For
>> > example, I have a field that is multivalued and has several records that
>> > have the same value. For example,
>> >
>> > <arr name="references">
>> > <str>National Society of Animal Lovers</str>
>> > <str>Nat. Soc. of Ani. Lov.</str>
>> > </arr>
>> >
>> > I have about 300 records with that exact value.
>> >
>> > Now, when I do a search for references:(national society animal lovers),
>> > I
>> > get the following results:
>> >
>> > <id>252</id>
>> > <id>159</id>
>> > <id>82</id>
>> > <id>452</id>
>> > <id>105</id>
>> >
>> > When I do a search for references:(nat soc ani lov), I get the results
>> > ordered differently:
>> >
>> > <id>510</id>
>> > <id>122</id>
>> > <id>501</id>
>> > <id>82</id>
>> > <id>252</id>
>> >
>> > When I load all the records that match, I notice that at some point, the
>> > scores aren't the same but differ by only a little:
>> >
>> > 1.471928 in one and the one before it was 1.471929
>>
>> 32 bit floats only have 7 decimal digits of precision, and in floating
>> point land (a+b+c) can be slightly different than (c+b+a)
>>
>> -Yonik
>> http://www.lucidimagination.com
>
>
Re: Rounding errors in solr
Posted by Brian Lamb <br...@journalexperts.com>.
Is this possible to do? If so, how?
On 7/25/11, Brian Lamb <br...@journalexperts.com> wrote:
> Yes and that's causing some problems in my application. Is there a way to
> truncate the 7th decimal place in regards to sorting by the score?
>
> On Fri, Jul 22, 2011 at 4:27 PM, Yonik Seeley
> <yo...@lucidimagination.com>wrote:
>
>> On Fri, Jul 22, 2011 at 4:11 PM, Brian Lamb
>> <br...@journalexperts.com> wrote:
>> > I've noticed some peculiar scoring issues going on in my application.
>> > For
>> > example, I have a field that is multivalued and has several records
>> > that
>> > have the same value. For example,
>> >
>> > <arr name="references">
>> > <str>National Society of Animal Lovers</str>
>> > <str>Nat. Soc. of Ani. Lov.</str>
>> > </arr>
>> >
>> > I have about 300 records with that exact value.
>> >
>> > Now, when I do a search for references:(national society animal
>> > lovers),
>> I
>> > get the following results:
>> >
>> > <id>252</id>
>> > <id>159</id>
>> > <id>82</id>
>> > <id>452</id>
>> > <id>105</id>
>> >
>> > When I do a search for references:(nat soc ani lov), I get the results
>> > ordered differently:
>> >
>> > <id>510</id>
>> > <id>122</id>
>> > <id>501</id>
>> > <id>82</id>
>> > <id>252</id>
>> >
>> > When I load all the records that match, I notice that at some point,
>> > the
>> > scores aren't the same but differ by only a little:
>> >
>> > 1.471928 in one and the one before it was 1.471929
>>
>> 32 bit floats only have 7 decimal digits of precision, and in floating
>> point land (a+b+c) can be slightly different than (c+b+a)
>>
>> -Yonik
>> http://www.lucidimagination.com
>>
>
Re: Rounding errors in solr
Posted by Brian Lamb <br...@journalexperts.com>.
Yes and that's causing some problems in my application. Is there a way to
truncate the 7th decimal place in regards to sorting by the score?
On Fri, Jul 22, 2011 at 4:27 PM, Yonik Seeley <yo...@lucidimagination.com>wrote:
> On Fri, Jul 22, 2011 at 4:11 PM, Brian Lamb
> <br...@journalexperts.com> wrote:
> > I've noticed some peculiar scoring issues going on in my application. For
> > example, I have a field that is multivalued and has several records that
> > have the same value. For example,
> >
> > <arr name="references">
> > <str>National Society of Animal Lovers</str>
> > <str>Nat. Soc. of Ani. Lov.</str>
> > </arr>
> >
> > I have about 300 records with that exact value.
> >
> > Now, when I do a search for references:(national society animal lovers),
> I
> > get the following results:
> >
> > <id>252</id>
> > <id>159</id>
> > <id>82</id>
> > <id>452</id>
> > <id>105</id>
> >
> > When I do a search for references:(nat soc ani lov), I get the results
> > ordered differently:
> >
> > <id>510</id>
> > <id>122</id>
> > <id>501</id>
> > <id>82</id>
> > <id>252</id>
> >
> > When I load all the records that match, I notice that at some point, the
> > scores aren't the same but differ by only a little:
> >
> > 1.471928 in one and the one before it was 1.471929
>
> 32 bit floats only have 7 decimal digits of precision, and in floating
> point land (a+b+c) can be slightly different than (c+b+a)
>
> -Yonik
> http://www.lucidimagination.com
>
Re: Rounding errors in solr
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Fri, Jul 22, 2011 at 4:11 PM, Brian Lamb
<br...@journalexperts.com> wrote:
> I've noticed some peculiar scoring issues going on in my application. For
> example, I have a field that is multivalued and has several records that
> have the same value. For example,
>
> <arr name="references">
> <str>National Society of Animal Lovers</str>
> <str>Nat. Soc. of Ani. Lov.</str>
> </arr>
>
> I have about 300 records with that exact value.
>
> Now, when I do a search for references:(national society animal lovers), I
> get the following results:
>
> <id>252</id>
> <id>159</id>
> <id>82</id>
> <id>452</id>
> <id>105</id>
>
> When I do a search for references:(nat soc ani lov), I get the results
> ordered differently:
>
> <id>510</id>
> <id>122</id>
> <id>501</id>
> <id>82</id>
> <id>252</id>
>
> When I load all the records that match, I notice that at some point, the
> scores aren't the same but differ by only a little:
>
> 1.471928 in one and the one before it was 1.471929
32 bit floats only have 7 decimal digits of precision, and in floating
point land (a+b+c) can be slightly different than (c+b+a)
-Yonik
http://www.lucidimagination.com