You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Arun Kumar K <ar...@gmail.com> on 2013/06/04 09:16:15 UTC
Lucene 4.2 Doc Vals
Hi Guys,
I am trying to get hands on Lucene 4.2 Doc Values (RAM Based Which is by
default).
I have a 1GB index with 540000 documents.
When retrieving the DocVals for matched docs i am able to retrieve vals
only upto some limit around 45000 docvals only.
for (AtomicReaderContext context : reader.leaves())
{
NumericDocValues vals = context.reader().getNumericDocValues("ID");
int leafDocBase = context.docBase;
int leafMaxDoc = context.reader().maxDoc();
int realdocId = docId - leafDocBase;
if (realdocId >= 0 && realdocId < leafMaxDoc)
{
vals.get(docId) ;
}
}
vals.get(docid) throws java.lang.ArrayIndexOutOfBoundsException: 73
at o.a.lucene.util.packed.BlockPackedReader.get(BlockPackedReader.java:86)
at
o.a.lucene.codecs.lucene42.Lucene42DocValuesProducer$2.get(Lucene42DocValuesProducer.java:171)
at lucene42.DocValSearch.search(DocValSearch.java:147)
at lucene42.DocValSearch.main(DocValSearch.java:241)
I am using defaults for java and lucene.
It seems that NumericDocValues doesn't have all the DocVals ? Is it because
of some RAM constraint ?
Also i didn't understand NumeriDocValues structure. Is it some HashMap
containing all vals or some Data Structure which loads data when required ?
I don't see any .length() for it.
Can someone explain ? Will it differ in case of Disk Based DocVals ?
Arun
Re: Lucene 4.2 Doc Vals
Posted by Arun Kumar K <ar...@gmail.com>.
My bad ! Taking time to get used to 4.X ways :)
Thanks Uwe for spotting the problem and for the useful pointers!
Arun
On Tue, Jun 4, 2013 at 2:05 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
> Hi.
>
> Your code is wrong: You must use the realdocid variable on vals.get ().
>
> Your code might be inefficient for random lookup. You schould do a bibary
> search to find the right leaf. ComposuteReader and ReaderUtil have utility
> methods to do this.
>
> Uwe
>
>
>
> Arun Kumar K <ar...@gmail.com> schrieb:
> >Hi Guys,
> >
> >I am trying to get hands on Lucene 4.2 Doc Values (RAM Based Which is
> >by
> >default).
> >
> >I have a 1GB index with 540000 documents.
> >
> >When retrieving the DocVals for matched docs i am able to retrieve vals
> >only upto some limit around 45000 docvals only.
> >
> >for (AtomicReaderContext context : reader.leaves())
> > {
> > NumericDocValues vals = context.reader().getNumericDocValues("ID");
> > int leafDocBase = context.docBase;
> > int leafMaxDoc = context.reader().maxDoc();
> > int realdocId = docId - leafDocBase;
> >
> > if (realdocId >= 0 && realdocId < leafMaxDoc)
> > {
> > vals.get(docId) ;
> > }
> > }
> >
> >vals.get(docid) throws java.lang.ArrayIndexOutOfBoundsException: 73
> >at
> >o.a.lucene.util.packed.BlockPackedReader.get(BlockPackedReader.java:86)
> >at
>
> >o.a.lucene.codecs.lucene42.Lucene42DocValuesProducer$2.get(Lucene42DocValuesProducer.java:171)
> >at lucene42.DocValSearch.search(DocValSearch.java:147)
> >at lucene42.DocValSearch.main(DocValSearch.java:241)
> >
> >I am using defaults for java and lucene.
> >
> >It seems that NumericDocValues doesn't have all the DocVals ? Is it
> >because
> >of some RAM constraint ?
> >Also i didn't understand NumeriDocValues structure. Is it some HashMap
> >containing all vals or some Data Structure which loads data when
> >required ?
> >I don't see any .length() for it.
> >
> >Can someone explain ? Will it differ in case of Disk Based DocVals ?
> >
> >Arun
>
> --
> Uwe Schindler
> H.-H.-Meier-Allee 63, 28213 Bremen
> http://www.thetaphi.de
Re: Lucene 4.2 Doc Vals
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi.
Your code is wrong: You must use the realdocid variable on vals.get ().
Your code might be inefficient for random lookup. You schould do a bibary search to find the right leaf. ComposuteReader and ReaderUtil have utility methods to do this.
Uwe
Arun Kumar K <ar...@gmail.com> schrieb:
>Hi Guys,
>
>I am trying to get hands on Lucene 4.2 Doc Values (RAM Based Which is
>by
>default).
>
>I have a 1GB index with 540000 documents.
>
>When retrieving the DocVals for matched docs i am able to retrieve vals
>only upto some limit around 45000 docvals only.
>
>for (AtomicReaderContext context : reader.leaves())
> {
> NumericDocValues vals = context.reader().getNumericDocValues("ID");
> int leafDocBase = context.docBase;
> int leafMaxDoc = context.reader().maxDoc();
> int realdocId = docId - leafDocBase;
>
> if (realdocId >= 0 && realdocId < leafMaxDoc)
> {
> vals.get(docId) ;
> }
> }
>
>vals.get(docid) throws java.lang.ArrayIndexOutOfBoundsException: 73
>at
>o.a.lucene.util.packed.BlockPackedReader.get(BlockPackedReader.java:86)
>at
>o.a.lucene.codecs.lucene42.Lucene42DocValuesProducer$2.get(Lucene42DocValuesProducer.java:171)
>at lucene42.DocValSearch.search(DocValSearch.java:147)
>at lucene42.DocValSearch.main(DocValSearch.java:241)
>
>I am using defaults for java and lucene.
>
>It seems that NumericDocValues doesn't have all the DocVals ? Is it
>because
>of some RAM constraint ?
>Also i didn't understand NumeriDocValues structure. Is it some HashMap
>containing all vals or some Data Structure which loads data when
>required ?
>I don't see any .length() for it.
>
>Can someone explain ? Will it differ in case of Disk Based DocVals ?
>
>Arun
--
Uwe Schindler
H.-H.-Meier-Allee 63, 28213 Bremen
http://www.thetaphi.de