You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Ganesh <em...@yahoo.co.in> on 2012/02/28 11:15:09 UTC
Date time as String or Numeric field
Hello all,
I was using DateTime as String and now i am using NumericField. Using NumericField takes more heap and storage space then the earlier String version. Is it good to move to NumericField or stick with String. I am using this field for search and sort.
Regards
Ganesh
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: [Bulk] RE: RE: Date time as String or Numeric field
Posted by Ganesh <em...@yahoo.co.in>.
Thanks. I use this field for Rangequery and sort. I think it is best to use Int to gain some heap.
Regards
Ganesh
----- Original Message -----
From: "Uwe Schindler" <uw...@thetaphi.de>
To: <ja...@lucene.apache.org>
Sent: Tuesday, February 28, 2012 5:08 PM
Subject: [Bulk] RE: RE: Date time as String or Numeric field
> Hi,
>
> The long or int size mostly only affects the size of e.g. FieldCache during
> sorting (which doubles its size). The term dictionary's size depends on the
> number of unique terms and that does not really change by the data type. The
> size of the values is of minor importance because how the data is encoded on
> disk.
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
>> -----Original Message-----
>> From: Ganesh [mailto:emailgane@yahoo.co.in]
>> Sent: Tuesday, February 28, 2012 12:17 PM
>> To: java-user@lucene.apache.org
>> Subject: Re: RE: Date time as String or Numeric field
>>
>> I tried NumericField with Integer value and Long value. There is no
> difference
>> in space and heap utilization. Will it be? Are both are same?
>>
>> Regards
>> Ganesh
>>
>> ----- Original Message -----
>> From: "Uwe Schindler" <uw...@thetaphi.de>
>> To: <ja...@lucene.apache.org>
>> Sent: Tuesday, February 28, 2012 3:52 PM
>> Subject: [Bulk] RE: Date time as String or Numeric field
>>
>>
>> > Hi,
>> >
>> > NumericField takes more space on disk and (possibly more heap because
>> term
>> > dictionary is larger), but is much faster on RANGE searches
>> > (NumericRangeQuery). Depending on index size this can be hundreds of
> times
>> > faster.
>> >
>> > If you don't want to do numeric searches (like range from...to) but only
>> > sort against the numeric field, use precisionStep=Integer.MAX_VALUE,
> then it
>> > takes even less space on disk or heap. The additional disk space is
> needed
>> > for optimizing range searches (which is wanted in most cases). If you
> use
>> > the infinite precision step, each numeric value only generates one
> token,
>> > but search with NumericRangeQuery is then as slow as a TermRangeQuery.
>> >
>> > Uwe
>> >
>> > -----
>> > Uwe Schindler
>> > H.-H.-Meier-Allee 63, D-28213 Bremen
>> > http://www.thetaphi.de
>> > eMail: uwe@thetaphi.de
>> >
>> >
>> >> -----Original Message-----
>> >> From: Ganesh [mailto:emailgane@yahoo.co.in]
>> >> Sent: Tuesday, February 28, 2012 11:15 AM
>> >> To: java-user@lucene.apache.org
>> >> Subject: Date time as String or Numeric field
>> >>
>> >> Hello all,
>> >>
>> >> I was using DateTime as String and now i am using NumericField. Using
>> >> NumericField takes more heap and storage space then the earlier String
>> >> version. Is it good to move to NumericField or stick with String. I am
>> > using this
>> >> field for search and sort.
>> >>
>> >> Regards
>> >> Ganesh
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> >> For additional commands, e-mail: java-user-help@lucene.apache.org
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> > For additional commands, e-mail: java-user-help@lucene.apache.org
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: RE: Date time as String or Numeric field
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
The long or int size mostly only affects the size of e.g. FieldCache during
sorting (which doubles its size). The term dictionary's size depends on the
number of unique terms and that does not really change by the data type. The
size of the values is of minor importance because how the data is encoded on
disk.
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Ganesh [mailto:emailgane@yahoo.co.in]
> Sent: Tuesday, February 28, 2012 12:17 PM
> To: java-user@lucene.apache.org
> Subject: Re: RE: Date time as String or Numeric field
>
> I tried NumericField with Integer value and Long value. There is no
difference
> in space and heap utilization. Will it be? Are both are same?
>
> Regards
> Ganesh
>
> ----- Original Message -----
> From: "Uwe Schindler" <uw...@thetaphi.de>
> To: <ja...@lucene.apache.org>
> Sent: Tuesday, February 28, 2012 3:52 PM
> Subject: [Bulk] RE: Date time as String or Numeric field
>
>
> > Hi,
> >
> > NumericField takes more space on disk and (possibly more heap because
> term
> > dictionary is larger), but is much faster on RANGE searches
> > (NumericRangeQuery). Depending on index size this can be hundreds of
times
> > faster.
> >
> > If you don't want to do numeric searches (like range from...to) but only
> > sort against the numeric field, use precisionStep=Integer.MAX_VALUE,
then it
> > takes even less space on disk or heap. The additional disk space is
needed
> > for optimizing range searches (which is wanted in most cases). If you
use
> > the infinite precision step, each numeric value only generates one
token,
> > but search with NumericRangeQuery is then as slow as a TermRangeQuery.
> >
> > Uwe
> >
> > -----
> > Uwe Schindler
> > H.-H.-Meier-Allee 63, D-28213 Bremen
> > http://www.thetaphi.de
> > eMail: uwe@thetaphi.de
> >
> >
> >> -----Original Message-----
> >> From: Ganesh [mailto:emailgane@yahoo.co.in]
> >> Sent: Tuesday, February 28, 2012 11:15 AM
> >> To: java-user@lucene.apache.org
> >> Subject: Date time as String or Numeric field
> >>
> >> Hello all,
> >>
> >> I was using DateTime as String and now i am using NumericField. Using
> >> NumericField takes more heap and storage space then the earlier String
> >> version. Is it good to move to NumericField or stick with String. I am
> > using this
> >> field for search and sort.
> >>
> >> Regards
> >> Ganesh
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: RE: Date time as String or Numeric field
Posted by Ganesh <em...@yahoo.co.in>.
I tried NumericField with Integer value and Long value. There is no difference in space and heap utilization. Will it be? Are both are same?
Regards
Ganesh
----- Original Message -----
From: "Uwe Schindler" <uw...@thetaphi.de>
To: <ja...@lucene.apache.org>
Sent: Tuesday, February 28, 2012 3:52 PM
Subject: [Bulk] RE: Date time as String or Numeric field
> Hi,
>
> NumericField takes more space on disk and (possibly more heap because term
> dictionary is larger), but is much faster on RANGE searches
> (NumericRangeQuery). Depending on index size this can be hundreds of times
> faster.
>
> If you don't want to do numeric searches (like range from...to) but only
> sort against the numeric field, use precisionStep=Integer.MAX_VALUE, then it
> takes even less space on disk or heap. The additional disk space is needed
> for optimizing range searches (which is wanted in most cases). If you use
> the infinite precision step, each numeric value only generates one token,
> but search with NumericRangeQuery is then as slow as a TermRangeQuery.
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
>> -----Original Message-----
>> From: Ganesh [mailto:emailgane@yahoo.co.in]
>> Sent: Tuesday, February 28, 2012 11:15 AM
>> To: java-user@lucene.apache.org
>> Subject: Date time as String or Numeric field
>>
>> Hello all,
>>
>> I was using DateTime as String and now i am using NumericField. Using
>> NumericField takes more heap and storage space then the earlier String
>> version. Is it good to move to NumericField or stick with String. I am
> using this
>> field for search and sort.
>>
>> Regards
>> Ganesh
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Date time as String or Numeric field
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
NumericField takes more space on disk and (possibly more heap because term
dictionary is larger), but is much faster on RANGE searches
(NumericRangeQuery). Depending on index size this can be hundreds of times
faster.
If you don't want to do numeric searches (like range from...to) but only
sort against the numeric field, use precisionStep=Integer.MAX_VALUE, then it
takes even less space on disk or heap. The additional disk space is needed
for optimizing range searches (which is wanted in most cases). If you use
the infinite precision step, each numeric value only generates one token,
but search with NumericRangeQuery is then as slow as a TermRangeQuery.
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Ganesh [mailto:emailgane@yahoo.co.in]
> Sent: Tuesday, February 28, 2012 11:15 AM
> To: java-user@lucene.apache.org
> Subject: Date time as String or Numeric field
>
> Hello all,
>
> I was using DateTime as String and now i am using NumericField. Using
> NumericField takes more heap and storage space then the earlier String
> version. Is it good to move to NumericField or stick with String. I am
using this
> field for search and sort.
>
> Regards
> Ganesh
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org