You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Wei Wang <we...@gmail.com> on 2013/04/04 10:30:28 UTC
DocValues questions
A few quick questions about DocValues:
1. If only small number of documents have a ShortDocValueField defined,
should each document in the index has this field filled with some value?
The add() function of Document seems not enforce a DocValues field is
always added to each document.
2. Is there any examples to show how DocValues are stored and retrieved? It
seems JavaDoc only shows how to add it, and no complete examples are out
there.
Thanks in advance,
Wei
Re: DocValues questions
Posted by Adrien Grand <jp...@gmail.com>.
On Fri, Apr 5, 2013 at 4:05 AM, Wei Wang <we...@gmail.com> wrote:
> Do we need to use setLongValue() all the time?
Yes.
--
Adrien
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: DocValues questions
Posted by Wei Wang <we...@gmail.com>.
We now start using NumericDocValuesField. The Javadoc says the old types
such as ShortDocValuesField is deprecated. So even the values of a field is
short, we still use NumericDocValuesField as the Javadoc suggests.
However, when we call function SetIntValue() of NumericDocValuesField, we
got an error:
Exception in thread "main" java.lang.IllegalArgumentException: cannot
change value type from Long to Integer
Do we need to use setLongValue() all the time?
Thanks.
On Thu, Apr 4, 2013 at 3:58 PM, Wei Wang <we...@gmail.com> wrote:
> Thanks! Good to know the codec uses variable length encoding mechanism
> here.
>
>
> On Thu, Apr 4, 2013 at 3:36 PM, Adrien Grand <jp...@gmail.com> wrote:
>
>> On Thu, Apr 4, 2013 at 11:03 PM, Wei Wang <we...@gmail.com> wrote:
>> > Given the new Lucene 4.2 DocValues API, it seems no matter it is byte,
>> > short, int, or long, they are all stored as NumericDocValuesField. Does
>> > this mean "long" values are always stored regardless of the initial
>> type?
>> > If so, do we still save space if the value range is small? Do we need to
>> > give some hint to NumericDocValuesField to save space?
>>
>> Space savings are codec-dependent, but the default codecs use bit
>> packing to save space. For example:
>> - if all your values are between 0 and 255, Lucene will only use 8
>> bits per value on average,
>> - if your documents only have three distinct values 1, 100 and 10000,
>> Lucene will detect that this is a low-cardinality field and only use 2
>> bits per value on average.
>>
>> This makes doc values storage-efficient, and much more
>> memory-efficient than FieldCache, that people had to use unti Lucene
>> 4.0.
>>
>> --
>> Adrien
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
Re: DocValues questions
Posted by Wei Wang <we...@gmail.com>.
Thanks! Good to know the codec uses variable length encoding mechanism here.
On Thu, Apr 4, 2013 at 3:36 PM, Adrien Grand <jp...@gmail.com> wrote:
> On Thu, Apr 4, 2013 at 11:03 PM, Wei Wang <we...@gmail.com> wrote:
> > Given the new Lucene 4.2 DocValues API, it seems no matter it is byte,
> > short, int, or long, they are all stored as NumericDocValuesField. Does
> > this mean "long" values are always stored regardless of the initial type?
> > If so, do we still save space if the value range is small? Do we need to
> > give some hint to NumericDocValuesField to save space?
>
> Space savings are codec-dependent, but the default codecs use bit
> packing to save space. For example:
> - if all your values are between 0 and 255, Lucene will only use 8
> bits per value on average,
> - if your documents only have three distinct values 1, 100 and 10000,
> Lucene will detect that this is a low-cardinality field and only use 2
> bits per value on average.
>
> This makes doc values storage-efficient, and much more
> memory-efficient than FieldCache, that people had to use unti Lucene
> 4.0.
>
> --
> Adrien
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: DocValues questions
Posted by Adrien Grand <jp...@gmail.com>.
On Thu, Apr 4, 2013 at 11:03 PM, Wei Wang <we...@gmail.com> wrote:
> Given the new Lucene 4.2 DocValues API, it seems no matter it is byte,
> short, int, or long, they are all stored as NumericDocValuesField. Does
> this mean "long" values are always stored regardless of the initial type?
> If so, do we still save space if the value range is small? Do we need to
> give some hint to NumericDocValuesField to save space?
Space savings are codec-dependent, but the default codecs use bit
packing to save space. For example:
- if all your values are between 0 and 255, Lucene will only use 8
bits per value on average,
- if your documents only have three distinct values 1, 100 and 10000,
Lucene will detect that this is a low-cardinality field and only use 2
bits per value on average.
This makes doc values storage-efficient, and much more
memory-efficient than FieldCache, that people had to use unti Lucene
4.0.
--
Adrien
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: DocValues questions
Posted by Wei Wang <we...@gmail.com>.
Given the new Lucene 4.2 DocValues API, it seems no matter it is byte,
short, int, or long, they are all stored as NumericDocValuesField. Does
this mean "long" values are always stored regardless of the initial type?
If so, do we still save space if the value range is small? Do we need to
give some hint to NumericDocValuesField to save space?
On Thu, Apr 4, 2013 at 11:53 AM, Wei Wang <we...@gmail.com> wrote:
> Hi Adrien,
>
> Thanks for the clarification. It is very helpful. Will try Lucene 4.2 and
> AtomicReader API.
>
> Wei
>
>
> On Thu, Apr 4, 2013 at 11:22 AM, Adrien Grand <jp...@gmail.com> wrote:
>
>> Hi,
>>
>> On Thu, Apr 4, 2013 at 10:30 AM, Wei Wang <we...@gmail.com> wrote:
>> > A few quick questions about DocValues:
>> >
>> > 1. If only small number of documents have a ShortDocValueField defined,
>> > should each document in the index has this field filled with some value?
>> > The add() function of Document seems not enforce a DocValues field is
>> > always added to each document.
>>
>> Given the name of the fied you are referring to, I assume that you are
>> using Lucene 4.0 or 4.1. I would highly recommend to upgrade to Lucene
>> 4.2 since the API has been completely refactored (but the disk format
>> is compatible) and should hopefully be a little clearer.
>>
>> You are right that there is nothing that enforces that every document
>> has a value : Lucene will give a default value to documents: 0 for
>> numeric doc values and an empty byte array for binary doc values.
>>
>> > 2. Is there any examples to show how DocValues are stored and
>> retrieved? It
>> > seems JavaDoc only shows how to add it, and no complete examples are out
>> > there.
>>
>> This should be transparent if you use doc values for eg. sorting.
>> Otherwise, just call getNumericDocValues(field), getBinaryDocValues or
>> getSortedDocValues on an AtomicReader.
>>
>> I hope this helps.
>>
>> --
>> Adrien
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
Re: DocValues questions
Posted by Wei Wang <we...@gmail.com>.
Hi Adrien,
Thanks for the clarification. It is very helpful. Will try Lucene 4.2 and
AtomicReader API.
Wei
On Thu, Apr 4, 2013 at 11:22 AM, Adrien Grand <jp...@gmail.com> wrote:
> Hi,
>
> On Thu, Apr 4, 2013 at 10:30 AM, Wei Wang <we...@gmail.com> wrote:
> > A few quick questions about DocValues:
> >
> > 1. If only small number of documents have a ShortDocValueField defined,
> > should each document in the index has this field filled with some value?
> > The add() function of Document seems not enforce a DocValues field is
> > always added to each document.
>
> Given the name of the fied you are referring to, I assume that you are
> using Lucene 4.0 or 4.1. I would highly recommend to upgrade to Lucene
> 4.2 since the API has been completely refactored (but the disk format
> is compatible) and should hopefully be a little clearer.
>
> You are right that there is nothing that enforces that every document
> has a value : Lucene will give a default value to documents: 0 for
> numeric doc values and an empty byte array for binary doc values.
>
> > 2. Is there any examples to show how DocValues are stored and retrieved?
> It
> > seems JavaDoc only shows how to add it, and no complete examples are out
> > there.
>
> This should be transparent if you use doc values for eg. sorting.
> Otherwise, just call getNumericDocValues(field), getBinaryDocValues or
> getSortedDocValues on an AtomicReader.
>
> I hope this helps.
>
> --
> Adrien
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: DocValues questions
Posted by Adrien Grand <jp...@gmail.com>.
Hi,
On Thu, Apr 4, 2013 at 10:30 AM, Wei Wang <we...@gmail.com> wrote:
> A few quick questions about DocValues:
>
> 1. If only small number of documents have a ShortDocValueField defined,
> should each document in the index has this field filled with some value?
> The add() function of Document seems not enforce a DocValues field is
> always added to each document.
Given the name of the fied you are referring to, I assume that you are
using Lucene 4.0 or 4.1. I would highly recommend to upgrade to Lucene
4.2 since the API has been completely refactored (but the disk format
is compatible) and should hopefully be a little clearer.
You are right that there is nothing that enforces that every document
has a value : Lucene will give a default value to documents: 0 for
numeric doc values and an empty byte array for binary doc values.
> 2. Is there any examples to show how DocValues are stored and retrieved? It
> seems JavaDoc only shows how to add it, and no complete examples are out
> there.
This should be transparent if you use doc values for eg. sorting.
Otherwise, just call getNumericDocValues(field), getBinaryDocValues or
getSortedDocValues on an AtomicReader.
I hope this helps.
--
Adrien
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org