You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Arjun Dhar <dh...@yahoo.com> on 2013/06/03 10:00:03 UTC
Why Numeric (IntField) tokenized true by Default?
This post was updated on Jun 03, 2013; 3:48am.
If One refers to the JavaDoc for Sort it states that INDEX and and the field
should NOT be TOKENIZED.
Its a common use case for Numbers to be Sorted.
Am curious to find the default values of say IntField as :
TYPE_NOT_STORED.setTokenized(true);
Any reason why its tokenized by default?
The constructors suggest you should play with STORE or NO STORE @ best. But
I feel there is sufficient reason to have to define your own FieldType for
sorting ?
--
View this message in context: http://lucene.472066.n3.nabble.com/Why-Numeric-IntField-tokenized-true-by-Default-tp4067738.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Why Numeric (IntField) tokenized true by Default?
Posted by Arjun Dhar <dh...@yahoo.com>.
Ah .. Sorry i think i got it.
It is still tokenized for purposes u mentioned.
Only for sort there is an exception to the rule.
ok cool
thanks !! :)
--
View this message in context: http://lucene.472066.n3.nabble.com/Why-Numeric-IntField-tokenized-true-by-Default-tp4067738p4067790.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Why Numeric (IntField) tokenized true by Default?
Posted by Arjun Dhar <dh...@yahoo.com>.
Thanks
In other words does that imply "tokenized" property is ignored for Numbers?
..if so, would it be technically correct (atleast from a correctness
standpoint), to have it set as "false" in the FieldType definitions then ?
--
View this message in context: http://lucene.472066.n3.nabble.com/Why-Numeric-IntField-tokenized-true-by-Default-tp4067738p4067789.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Why Numeric (IntField) tokenized true by Default?
Posted by Michael McCandless <lu...@mikemccandless.com>.
This is because, under the hood, a single number is indexed into
several "hierarchical" tokens, to make NumericRangeFilter/Query very
fast.
When this field is sorted, there is careful logic to use only the
original number for sorting purposes
So, it's true that in general you should not tokenize a field that you
will sort on, but numeric fields are an exception to this rule, and
sorting will work correctly.
Mike McCandless
http://blog.mikemccandless.com
On Mon, Jun 3, 2013 at 4:00 AM, Arjun Dhar <dh...@yahoo.com> wrote:
> This post was updated on Jun 03, 2013; 3:48am.
> If One refers to the JavaDoc for Sort it states that INDEX and and the field
> should NOT be TOKENIZED.
> Its a common use case for Numbers to be Sorted.
>
> Am curious to find the default values of say IntField as :
> TYPE_NOT_STORED.setTokenized(true);
>
> Any reason why its tokenized by default?
> The constructors suggest you should play with STORE or NO STORE @ best. But
> I feel there is sufficient reason to have to define your own FieldType for
> sorting ?
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Why-Numeric-IntField-tokenized-true-by-Default-tp4067738.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org