You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Peter Keegan <pe...@gmail.com> on 2013/03/26 15:59:43 UTC
DocValues memory usage
Inspired by this presentation of DocValues:
http://www.slideshare.net/lucenerevolution/willnauer-simon-doc-values-column-stride-fields-in-lucene
I decided to try them out in 4.2. I created a 1M document index with one
DocValues field:
BinaryDocValuesField conceptsDV = new BinaryDocValuesField("concepts",new
BytesRef(byteArray(4000)));
d.add(conceptsDV);
writer.addDocument(d);
I searched the index and fetched the DocValues field:
TopDocs docs = searcher.search(new TermQuery(new Term("guid", val)), 1);
int docId = docs.scoreDocs[0].doc;
BinaryDocValues conceptValues =
MultiDocValues.getBinaryValues(r,"concepts");
BytesRef result = new BytesRef();
conceptValues.get(docId,result);
However, the first call to MultiDocValues.getBinaryValues reads in the
values for the entire index:
Lucene42DocValuesProducer.loadBinary // loads DocValues for entire index
My hope was to take advantage of faster disk access than stored fields and
less RAM than FieldCache, but this is using too much memory. Are my
assumptions and my usage correct?
Thanks,
Peter
Re: DocValues memory usage
Posted by Peter Keegan <pe...@gmail.com>.
This is wierd. I indexed using DiskDocValuesFormat as the default codec and
observed 16K qps with BinaryDocValuesField. But with a simple StoredField,
I observed a much higher 30K qps. When I added both fields
(BinaryDocValuesField and StoredField) to the index, I observed only 100
qps on each field.
Peter
On Tue, Mar 26, 2013 at 12:30 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:
> DiskDocValuesFormat is the right thing to use: it loads certain things
> into RAM, eg the compressed bits that tell it the addresses of the
> bytes on disk, but then leaves the actual bytes on disk.
>
> I believe the old DirectSource was more extreme as it left the
> addresses on disk too, so there were 2 seeks to load a value.
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Tue, Mar 26, 2013 at 11:55 AM, Duke <du...@gmail.com> wrote:
> > I made the same experiment and got same result. Then I used per-field
> codec with DiskDocValuesFormat, it works like DirectSource in 4.0.0, but
> I'm not feeling confident with this usage. Anyone can say more about
> removing DirectSource API?
> >
> >
> >
> > On 2013-3-26, at 22:59, Peter Keegan <pe...@gmail.com> wrote:
> >
> >> Inspired by this presentation of DocValues:
> >>
> http://www.slideshare.net/lucenerevolution/willnauer-simon-doc-values-column-stride-fields-in-lucene
> >> I decided to try them out in 4.2. I created a 1M document index with one
> >> DocValues field:
> >>
> >> BinaryDocValuesField conceptsDV = new
> BinaryDocValuesField("concepts",new
> >> BytesRef(byteArray(4000)));
> >> d.add(conceptsDV);
> >> writer.addDocument(d);
> >>
> >> I searched the index and fetched the DocValues field:
> >>
> >> TopDocs docs = searcher.search(new TermQuery(new Term("guid", val)), 1);
> >> int docId = docs.scoreDocs[0].doc;
> >> BinaryDocValues conceptValues =
> >> MultiDocValues.getBinaryValues(r,"concepts");
> >> BytesRef result = new BytesRef();
> >> conceptValues.get(docId,result);
> >>
> >> However, the first call to MultiDocValues.getBinaryValues reads in the
> >> values for the entire index:
> >>
> >> Lucene42DocValuesProducer.loadBinary // loads DocValues for entire index
> >>
> >> My hope was to take advantage of faster disk access than stored fields
> and
> >> less RAM than FieldCache, but this is using too much memory. Are my
> >> assumptions and my usage correct?
> >>
> >> Thanks,
> >> Peter
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: DocValues memory usage
Posted by Michael McCandless <lu...@mikemccandless.com>.
DiskDocValuesFormat is the right thing to use: it loads certain things
into RAM, eg the compressed bits that tell it the addresses of the
bytes on disk, but then leaves the actual bytes on disk.
I believe the old DirectSource was more extreme as it left the
addresses on disk too, so there were 2 seeks to load a value.
Mike McCandless
http://blog.mikemccandless.com
On Tue, Mar 26, 2013 at 11:55 AM, Duke <du...@gmail.com> wrote:
> I made the same experiment and got same result. Then I used per-field codec with DiskDocValuesFormat, it works like DirectSource in 4.0.0, but I'm not feeling confident with this usage. Anyone can say more about removing DirectSource API?
>
>
>
> On 2013-3-26, at 22:59, Peter Keegan <pe...@gmail.com> wrote:
>
>> Inspired by this presentation of DocValues:
>> http://www.slideshare.net/lucenerevolution/willnauer-simon-doc-values-column-stride-fields-in-lucene
>> I decided to try them out in 4.2. I created a 1M document index with one
>> DocValues field:
>>
>> BinaryDocValuesField conceptsDV = new BinaryDocValuesField("concepts",new
>> BytesRef(byteArray(4000)));
>> d.add(conceptsDV);
>> writer.addDocument(d);
>>
>> I searched the index and fetched the DocValues field:
>>
>> TopDocs docs = searcher.search(new TermQuery(new Term("guid", val)), 1);
>> int docId = docs.scoreDocs[0].doc;
>> BinaryDocValues conceptValues =
>> MultiDocValues.getBinaryValues(r,"concepts");
>> BytesRef result = new BytesRef();
>> conceptValues.get(docId,result);
>>
>> However, the first call to MultiDocValues.getBinaryValues reads in the
>> values for the entire index:
>>
>> Lucene42DocValuesProducer.loadBinary // loads DocValues for entire index
>>
>> My hope was to take advantage of faster disk access than stored fields and
>> less RAM than FieldCache, but this is using too much memory. Are my
>> assumptions and my usage correct?
>>
>> Thanks,
>> Peter
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: DocValues memory usage
Posted by Duke <du...@gmail.com>.
I made the same experiment and got same result. Then I used per-field codec with DiskDocValuesFormat, it works like DirectSource in 4.0.0, but I'm not feeling confident with this usage. Anyone can say more about removing DirectSource API?
On 2013-3-26, at 22:59, Peter Keegan <pe...@gmail.com> wrote:
> Inspired by this presentation of DocValues:
> http://www.slideshare.net/lucenerevolution/willnauer-simon-doc-values-column-stride-fields-in-lucene
> I decided to try them out in 4.2. I created a 1M document index with one
> DocValues field:
>
> BinaryDocValuesField conceptsDV = new BinaryDocValuesField("concepts",new
> BytesRef(byteArray(4000)));
> d.add(conceptsDV);
> writer.addDocument(d);
>
> I searched the index and fetched the DocValues field:
>
> TopDocs docs = searcher.search(new TermQuery(new Term("guid", val)), 1);
> int docId = docs.scoreDocs[0].doc;
> BinaryDocValues conceptValues =
> MultiDocValues.getBinaryValues(r,"concepts");
> BytesRef result = new BytesRef();
> conceptValues.get(docId,result);
>
> However, the first call to MultiDocValues.getBinaryValues reads in the
> values for the entire index:
>
> Lucene42DocValuesProducer.loadBinary // loads DocValues for entire index
>
> My hope was to take advantage of faster disk access than stored fields and
> less RAM than FieldCache, but this is using too much memory. Are my
> assumptions and my usage correct?
>
> Thanks,
> Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org