You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Nick Vasilyev <ni...@gmail.com> on 2016/07/21 17:48:44 UTC
Solr Rounding Issue On Float fields.
Hi, I am running into a weird rounding issue on Solr 5.2.1. I have a float
field (also tried tfloat), I am indexing 154035.26 into it (confirmed in
the data), but at query time, I get back 154035.27 (.01 more).
Additionally when I query for the document and include this number in the q
parameter, it comes up with both values, .26 and .27.
I've fed the values through the analyzer and I get this bizarre behavior
per the screenshot below. The field is a single value float or tfloat
field.
Any help would be much appreciated, thanks in advance
[image: Inline image 1]
Re: Solr Rounding Issue On Float fields.
Posted by Nick Vasilyev <ni...@gmail.com>.
I did a bit more investigating here is something that may help
troubleshooting:
- Seems that numbers above 131071 - are impacted. 131071.26 is fine,
but 131072.26 is not. 131071 is a large prime and also a Mersenne prime.
- 131072.24 gets rounded down to 131072.23. While 131072.26 gets rounded up
to 131072.27. Similarily, 131072.76 gets rounded up to 131072.77
and 131072.74 gets rounded down to 131072.73. 131072.49 gets rounded down
to 131072.48 and 131072.51 gets rounded up to 131072.52.
I haven't validated this code, just doing some manual digging.
On Thu, Jul 21, 2016 at 1:48 PM, Nick Vasilyev <ni...@gmail.com>
wrote:
> Hi, I am running into a weird rounding issue on Solr 5.2.1. I have a float
> field (also tried tfloat), I am indexing 154035.26 into it (confirmed in
> the data), but at query time, I get back 154035.27 (.01 more).
> Additionally when I query for the document and include this number in the q
> parameter, it comes up with both values, .26 and .27.
>
> I've fed the values through the analyzer and I get this bizarre behavior
> per the screenshot below. The field is a single value float or tfloat
> field.
>
> Any help would be much appreciated, thanks in advance
>
> [image: Inline image 1]
>
Re: Solr Rounding Issue On Float fields.
Posted by Ray Niu <ne...@gmail.com>.
I met simliar issue before，suggest to use double as field type for this case
Ray
2016年7月21日星期四，Nick Vasilyev <ni...@gmail.com> 写道：
> Thanks Chris.
>
> Searching for both values and retrieving the documents would be alright as
> long as the data was correct. In this case, the data that I am indexing
> into Solr is not the same data that I am pulling out at query time. That is
> the real impact here.
>
> On Thu, Jul 21, 2016 at 6:12 PM, Chris Hostetter <hossman_lucene@fucit.org
> <javascript:;>>
> wrote:
>
> >
> > : Hi, I am running into a weird rounding issue on Solr 5.2.1. I have a
> > float
> > : field (also tried tfloat), I am indexing 154035.26 into it (confirmed
> in
> > : the data), but at query time, I get back 154035.27 (.01 more).
> > : Additionally when I query for the document and include this number in
> > the q
> > : parameter, it comes up with both values, .26 and .27.
> >
> > Pretty sure what you are observing is just the normal consequences of
> IEEE
> > floats (as used by java) being base2 -- not every base10 decimal value
> > has a precise base2 representation.
> >
> > Quering for 154035.27 and 154035.26 will both match the same docs,
> because
> > the String->Float parsing in both cases will produce the closest *legal*
> > float value, which is identical for both inputs.
> >
> > If need precise decimal values in solr, you need to either use 2
> > ints/longs (ie num_base="154035", num_decimal="26") or use one int/long
> > and multiply/divide by a power of 10 corisponding to the number of
> > significant digits you want in the client (ie: "15403526" divide by 100)
> >
> >
> > Some good reading linked to from here...
> >
> > http://perlmonks.org/?node_id=203257
> >
> > And of course, if you really want to bang java against your head,
> > this is a classic (all of which is still appliable i believe) ...
> >
> > https://people.eecs.berkeley.edu/~wkahan/JAVAhurt.pdf
> >
> >
> >
> >
> >
> > -Hoss
> > http://www.lucidworks.com/
> >
>
Re: Solr Rounding Issue On Float fields.
Posted by Nick Vasilyev <ni...@gmail.com>.
Thanks Chris.
Searching for both values and retrieving the documents would be alright as
long as the data was correct. In this case, the data that I am indexing
into Solr is not the same data that I am pulling out at query time. That is
the real impact here.
On Thu, Jul 21, 2016 at 6:12 PM, Chris Hostetter <ho...@fucit.org>
wrote:
>
> : Hi, I am running into a weird rounding issue on Solr 5.2.1. I have a
> float
> : field (also tried tfloat), I am indexing 154035.26 into it (confirmed in
> : the data), but at query time, I get back 154035.27 (.01 more).
> : Additionally when I query for the document and include this number in
> the q
> : parameter, it comes up with both values, .26 and .27.
>
> Pretty sure what you are observing is just the normal consequences of IEEE
> floats (as used by java) being base2 -- not every base10 decimal value
> has a precise base2 representation.
>
> Quering for 154035.27 and 154035.26 will both match the same docs, because
> the String->Float parsing in both cases will produce the closest *legal*
> float value, which is identical for both inputs.
>
> If need precise decimal values in solr, you need to either use 2
> ints/longs (ie num_base="154035", num_decimal="26") or use one int/long
> and multiply/divide by a power of 10 corisponding to the number of
> significant digits you want in the client (ie: "15403526" divide by 100)
>
>
> Some good reading linked to from here...
>
> http://perlmonks.org/?node_id=203257
>
> And of course, if you really want to bang java against your head,
> this is a classic (all of which is still appliable i believe) ...
>
> https://people.eecs.berkeley.edu/~wkahan/JAVAhurt.pdf
>
>
>
>
>
> -Hoss
> http://www.lucidworks.com/
>
Re: Solr Rounding Issue On Float fields.
Posted by Chris Hostetter <ho...@fucit.org>.
: Hi, I am running into a weird rounding issue on Solr 5.2.1. I have a float
: field (also tried tfloat), I am indexing 154035.26 into it (confirmed in
: the data), but at query time, I get back 154035.27 (.01 more).
: Additionally when I query for the document and include this number in the q
: parameter, it comes up with both values, .26 and .27.
Pretty sure what you are observing is just the normal consequences of IEEE
floats (as used by java) being base2 -- not every base10 decimal value
has a precise base2 representation.
Quering for 154035.27 and 154035.26 will both match the same docs, because
the String->Float parsing in both cases will produce the closest *legal*
float value, which is identical for both inputs.
If need precise decimal values in solr, you need to either use 2
ints/longs (ie num_base="154035", num_decimal="26") or use one int/long
and multiply/divide by a power of 10 corisponding to the number of
significant digits you want in the client (ie: "15403526" divide by 100)
Some good reading linked to from here...
http://perlmonks.org/?node_id=203257
And of course, if you really want to bang java against your head,
this is a classic (all of which is still appliable i believe) ...
https://people.eecs.berkeley.edu/~wkahan/JAVAhurt.pdf
-Hoss
http://www.lucidworks.com/