You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Rohit Banga <ia...@gmail.com> on 2014/03/21 21:35:12 UTC
Question about Payloads in Lucene 4.5
Hi everyone
When I query a lucene index, I get back a list of document ids. This index
search is fast. Now for all documents matching the result I need a unique
String field called "id" which is stored in the document. From the
documentation I gather that document ids are internal and I should not use
them for referencing my own data structures. Currently I iterate over all
the hits matching the document and then for each one I get the document to
read the field using IndexReader.document().
http://lucene.apache.org/core/4_5_0/core/org/apache/lucene/index/IndexReader.html
I read the "id" field from the document and then use it further in my
processing logic.
The problem is that reading all documents to get all "id"'s is turning out
to be very slow. It is the bottleneck in my application. It would be nice
to have a way if lucene could return some metadata along with the internal
document id when I did a search. I do not want to read all documents just
to retrieve this metadata.
The best solution I have come across searching on the net is to use
payloads which will be returned by the fast index search query along with
the document ids.
Is my understanding correct that using payloads I can get "id" string field
for all my documents faster than reading my entire document?
I am not able to find a good example of how to store and retrieve payloads?
Can you please point me to a good resource to learn how to use payloads and
how they will impact performance?
I am using Lucene 4.5.
Thanks
Rohit Banga
http://iamrohitbanga.com/
Re: Question about Payloads in Lucene 4.5
Posted by Rohit Banga <ia...@gmail.com>.
Awesome works well for me!
Thanks
Rohit Banga
http://iamrohitbanga.com/
On Sun, Mar 23, 2014 at 10:06 PM, Manuel Le Normand <
manuel.lenormand@gmail.com> wrote:
> Hello Rohit,
> We had a similar query time bottleneck when attempting to map lucene's
> internal id's to the uniqueKey, especially as we generally return only the
> uniqueKey to the user we had no other use of the stored field. As you
> noted, every internal id --> uniqueKey id requires a disk seek and as in
> our use case the documents did not get cached most of these reads were
> faulting (random read miss).
> There are two patches (for Solr) at SOLR-5478, both make use of docValues
> for this need. First one caches the above mapping and a second is more
> general and retrieves that mapping (or any other field value) if exists in
> fieldCache.
> Since implementing the first patch (although not efficiently) we see a
> significant performance improvement.
>
> I don't know if these patches can be easily ported to Lucene but I hope
> this use case helps you,
> Manuel
>
>
>
> On Sat, Mar 22, 2014 at 11:45 AM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
> > On Sat, Mar 22, 2014 at 5:18 AM, Rohit Banga <ia...@gmail.com>
> > wrote:
> >
> > > Awesome BinaryDocValues sounds nice!
> > > I saw that NumericDocValues did not inherit from a base class hence I
> > > thought there is no StringDocValues :).
> > >
> > > Can I expect that a searcher manager will invoke
> > > searcherfactory.newSearcher at most once between searcher manager
> > > refreshes? I believe IndexSearcher is threadsafe. Is my assumption that
> > > newSearcher is invoked only once correct?
> >
> > It's invoked once, for each refresh.
> >
> > > If BinaryDocValues didn't exist I was thinking of using a custom
> searcher
> > > factory which would return an instance of a custom subclass of
> > > IndexSearcher.This subclass could encapsulate a map from numeric doc
> > value
> > > to string. I was thinking SearcherManager.acquire could then be used to
> > > fetch the instance of this subclass while permitting concurrent updates
> > and
> > > reads to index and HashMap.
> > > Is using SearcherManager in this way appropriate? Just want to make
> sure
> > my
> > > understanding of how SearcherManager works is correct.
> >
> > I think in theory that would be fine (custom IndexSearcher subclass,
> > custom map), but hopefully BDV solves this in a simpler way!
> >
> > Mike McCandless
> >
> > http://blog.mikemccandless.com
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
Re: Question about Payloads in Lucene 4.5
Posted by Manuel Le Normand <ma...@gmail.com>.
Hello Rohit,
We had a similar query time bottleneck when attempting to map lucene's
internal id's to the uniqueKey, especially as we generally return only the
uniqueKey to the user we had no other use of the stored field. As you
noted, every internal id --> uniqueKey id requires a disk seek and as in
our use case the documents did not get cached most of these reads were
faulting (random read miss).
There are two patches (for Solr) at SOLR-5478, both make use of docValues
for this need. First one caches the above mapping and a second is more
general and retrieves that mapping (or any other field value) if exists in
fieldCache.
Since implementing the first patch (although not efficiently) we see a
significant performance improvement.
I don't know if these patches can be easily ported to Lucene but I hope
this use case helps you,
Manuel
On Sat, Mar 22, 2014 at 11:45 AM, Michael McCandless <
lucene@mikemccandless.com> wrote:
> On Sat, Mar 22, 2014 at 5:18 AM, Rohit Banga <ia...@gmail.com>
> wrote:
>
> > Awesome BinaryDocValues sounds nice!
> > I saw that NumericDocValues did not inherit from a base class hence I
> > thought there is no StringDocValues :).
> >
> > Can I expect that a searcher manager will invoke
> > searcherfactory.newSearcher at most once between searcher manager
> > refreshes? I believe IndexSearcher is threadsafe. Is my assumption that
> > newSearcher is invoked only once correct?
>
> It's invoked once, for each refresh.
>
> > If BinaryDocValues didn't exist I was thinking of using a custom searcher
> > factory which would return an instance of a custom subclass of
> > IndexSearcher.This subclass could encapsulate a map from numeric doc
> value
> > to string. I was thinking SearcherManager.acquire could then be used to
> > fetch the instance of this subclass while permitting concurrent updates
> and
> > reads to index and HashMap.
> > Is using SearcherManager in this way appropriate? Just want to make sure
> my
> > understanding of how SearcherManager works is correct.
>
> I think in theory that would be fine (custom IndexSearcher subclass,
> custom map), but hopefully BDV solves this in a simpler way!
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Question about Payloads in Lucene 4.5
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Sat, Mar 22, 2014 at 5:18 AM, Rohit Banga <ia...@gmail.com> wrote:
> Awesome BinaryDocValues sounds nice!
> I saw that NumericDocValues did not inherit from a base class hence I
> thought there is no StringDocValues :).
>
> Can I expect that a searcher manager will invoke
> searcherfactory.newSearcher at most once between searcher manager
> refreshes? I believe IndexSearcher is threadsafe. Is my assumption that
> newSearcher is invoked only once correct?
It's invoked once, for each refresh.
> If BinaryDocValues didn't exist I was thinking of using a custom searcher
> factory which would return an instance of a custom subclass of
> IndexSearcher.This subclass could encapsulate a map from numeric doc value
> to string. I was thinking SearcherManager.acquire could then be used to
> fetch the instance of this subclass while permitting concurrent updates and
> reads to index and HashMap.
> Is using SearcherManager in this way appropriate? Just want to make sure my
> understanding of how SearcherManager works is correct.
I think in theory that would be fine (custom IndexSearcher subclass,
custom map), but hopefully BDV solves this in a simpler way!
Mike McCandless
http://blog.mikemccandless.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Question about Payloads in Lucene 4.5
Posted by Rohit Banga <ia...@gmail.com>.
Awesome BinaryDocValues sounds nice!
I saw that NumericDocValues did not inherit from a base class hence I
thought there is no StringDocValues :).
Can I expect that a searcher manager will invoke
searcherfactory.newSearcher at most once between searcher manager
refreshes? I believe IndexSearcher is threadsafe. Is my assumption that
newSearcher is invoked only once correct?
If BinaryDocValues didn't exist I was thinking of using a custom searcher
factory which would return an instance of a custom subclass of
IndexSearcher.This subclass could encapsulate a map from numeric doc value
to string. I was thinking SearcherManager.acquire could then be used to
fetch the instance of this subclass while permitting concurrent updates and
reads to index and HashMap.
Is using SearcherManager in this way appropriate? Just want to make sure my
understanding of how SearcherManager works is correct.
Thanks
Rohit
On Mar 22, 2014 1:29 AM, "Michael McCandless" <lu...@mikemccandless.com>
wrote:
> On Fri, Mar 21, 2014 at 10:25 PM, Rohit Banga <ia...@gmail.com>
> wrote:
> > Thanks Michael for your response.
>
> You're welcome!
>
> > Few questions:
> >
> > 1. Can I expect better performance when retrieving a single
> NumericDocValue
> > for all hits vs when I retrieve documents for all hits to fetch the field
> > value? As far as I understand retrieving n documents from the index
> > requires n disk reads. How many disk reads to I do when using
> > NumericDocValues? How are they stored?
>
> It should be faster; doc values are stored "column stride", where all
> values across all docs for that one field are stored together, vs "row
> stride" of a stored document, where all fields for each document are
> stored together.
>
> The default DV format is Lucene45DocValuesFormat; it tries to compress
> the values, and then leaves the compressed form on disk and seeks for
> each lookup, but often the OS will cache those pages in RAM, if your
> application keeps them hot.
>
> You should test that first; if it's still too slow, and you're willing
> to use RAM, then swap in a different DVFormat for your field, e.g.
> DirectDocValuesFormat is the most RAM consuming (stores native java
> array under the hood) but should be the fastest.
>
> Swapping in a custom DVFormat for a field is easy: just make your own
> codec by subclassing the default Lucene46Codec, and override the
> method getDocValuesFormatForField.
>
> > 2. I tried looking for examples on how to use numeric doc values. I found
> > that in new versions of lucene we have to use "AtomicReader".
> > Found this:
> http://www.gossamer-threads.com/lists/lucene/java-user/182641
> >
> > So is this the code I am looking for:
> > long getNumericDocValueForDocument(IndexSearcher searcher, int docId) {
> > IndexReader reader = searcher.getIndexReader();
> > long docVal = 0;
> > for (AtomicReaderContext rc : reader.leaves()) {
> > AtomicReader ar = rc.reader();
> > docVal = ar.getNumericDocValues().get(*docID*);
> > }
> > return docVal;
> > }
> >
> > How do I know which docVal to return? It appears that each AtomicReader
> > (every iteration of the loop) may return a docVal?
>
> Looks like you solved this already ...
>
> > 3. Can I only store NumericDocValues? Can I get something like
> > StringDocValues? I have a string "id". I guess I could keep a mapping
> from
> > numeric doc value (Long) to String but I want to avoid keeping two
> sources
> > of information (Lucene Index and a HashMap). I can use SearcherManager to
> > deal with concurrent searches and index updates (
> >
> http://blog.mikemccandless.com/2011/09/lucenes-searchermanager-simplifies.html
> ),
> > but how about managing two data sources Lucene index and HashMap<Long,
> > String> with SearcherManager? Is there a way to achieve this using a
> custom
> > SearcherFactory?
>
> There are also binary doc values, maybe that helps?
>
> You may also want LiveFieldValues, if you need precise (real-time)
> lookup of the id for all docs, including just indexed ones.
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Question about Payloads in Lucene 4.5
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Fri, Mar 21, 2014 at 10:25 PM, Rohit Banga <ia...@gmail.com> wrote:
> Thanks Michael for your response.
You're welcome!
> Few questions:
>
> 1. Can I expect better performance when retrieving a single NumericDocValue
> for all hits vs when I retrieve documents for all hits to fetch the field
> value? As far as I understand retrieving n documents from the index
> requires n disk reads. How many disk reads to I do when using
> NumericDocValues? How are they stored?
It should be faster; doc values are stored "column stride", where all
values across all docs for that one field are stored together, vs "row
stride" of a stored document, where all fields for each document are
stored together.
The default DV format is Lucene45DocValuesFormat; it tries to compress
the values, and then leaves the compressed form on disk and seeks for
each lookup, but often the OS will cache those pages in RAM, if your
application keeps them hot.
You should test that first; if it's still too slow, and you're willing
to use RAM, then swap in a different DVFormat for your field, e.g.
DirectDocValuesFormat is the most RAM consuming (stores native java
array under the hood) but should be the fastest.
Swapping in a custom DVFormat for a field is easy: just make your own
codec by subclassing the default Lucene46Codec, and override the
method getDocValuesFormatForField.
> 2. I tried looking for examples on how to use numeric doc values. I found
> that in new versions of lucene we have to use "AtomicReader".
> Found this: http://www.gossamer-threads.com/lists/lucene/java-user/182641
>
> So is this the code I am looking for:
> long getNumericDocValueForDocument(IndexSearcher searcher, int docId) {
> IndexReader reader = searcher.getIndexReader();
> long docVal = 0;
> for (AtomicReaderContext rc : reader.leaves()) {
> AtomicReader ar = rc.reader();
> docVal = ar.getNumericDocValues().get(*docID*);
> }
> return docVal;
> }
>
> How do I know which docVal to return? It appears that each AtomicReader
> (every iteration of the loop) may return a docVal?
Looks like you solved this already ...
> 3. Can I only store NumericDocValues? Can I get something like
> StringDocValues? I have a string "id". I guess I could keep a mapping from
> numeric doc value (Long) to String but I want to avoid keeping two sources
> of information (Lucene Index and a HashMap). I can use SearcherManager to
> deal with concurrent searches and index updates (
> http://blog.mikemccandless.com/2011/09/lucenes-searchermanager-simplifies.html),
> but how about managing two data sources Lucene index and HashMap<Long,
> String> with SearcherManager? Is there a way to achieve this using a custom
> SearcherFactory?
There are also binary doc values, maybe that helps?
You may also want LiveFieldValues, if you need precise (real-time)
lookup of the id for all docs, including just indexed ones.
Mike McCandless
http://blog.mikemccandless.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Question about Payloads in Lucene 4.5
Posted by Rohit Banga <ia...@gmail.com>.
Just saw the implementation of MultiDocValues.getNumericValues(). It uses
sort of returns an anonymous inner classes to get the doc value from the
appropriate index reader. Very cool impleentation!
I guess that answers my question on how to get docVal from multiple
atomic readers.
It would be nice if you could help me with the other two questions though.
Thanks
Rohit Banga
http://iamrohitbanga.com/
On Fri, Mar 21, 2014 at 7:25 PM, Rohit Banga <ia...@gmail.com>wrote:
> Thanks Michael for your response.
>
> Few questions:
>
> 1. Can I expect better performance when retrieving a single
> NumericDocValue for all hits vs when I retrieve documents for all hits to
> fetch the field value? As far as I understand retrieving n documents from
> the index requires n disk reads. How many disk reads to I do when using
> NumericDocValues? How are they stored?
>
> 2. I tried looking for examples on how to use numeric doc values. I found
> that in new versions of lucene we have to use "AtomicReader".
> Found this: http://www.gossamer-threads.com/lists/lucene/java-user/182641
>
> So is this the code I am looking for:
> long getNumericDocValueForDocument(IndexSearcher searcher, int docId) {
> IndexReader reader = searcher.getIndexReader();
> long docVal = 0;
> for (AtomicReaderContext rc : reader.leaves()) {
> AtomicReader ar = rc.reader();
> docVal = ar.getNumericDocValues().get(*docID*);
> }
> return docVal;
> }
>
> How do I know which docVal to return? It appears that each AtomicReader
> (every iteration of the loop) may return a docVal?
>
> 3. Can I only store NumericDocValues? Can I get something like
> StringDocValues? I have a string "id". I guess I could keep a mapping from
> numeric doc value (Long) to String but I want to avoid keeping two sources
> of information (Lucene Index and a HashMap). I can use SearcherManager to
> deal with concurrent searches and index updates (
> http://blog.mikemccandless.com/2011/09/lucenes-searchermanager-simplifies.html),
> but how about managing two data sources Lucene index and HashMap<Long,
> String> with SearcherManager? Is there a way to achieve this using a custom
> SearcherFactory?
>
>
> Thanks
> Rohit Banga
> http://iamrohitbanga.com/
>
>
> On Fri, Mar 21, 2014 at 3:26 PM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>> DocValues are better than payloads.
>>
>> E.g. index a NumericDocValuesField with each doc, holding your id.
>>
>> Then at search time you can use MultiDocValues.getNumericValues.
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>>
>> On Fri, Mar 21, 2014 at 4:35 PM, Rohit Banga <ia...@gmail.com>
>> wrote:
>> > Hi everyone
>> >
>> > When I query a lucene index, I get back a list of document ids. This
>> index
>> > search is fast. Now for all documents matching the result I need a
>> unique
>> > String field called "id" which is stored in the document. From the
>> > documentation I gather that document ids are internal and I should not
>> use
>> > them for referencing my own data structures. Currently I iterate over
>> all
>> > the hits matching the document and then for each one I get the document
>> to
>> > read the field using IndexReader.document().
>> >
>> http://lucene.apache.org/core/4_5_0/core/org/apache/lucene/index/IndexReader.html
>> >
>> > I read the "id" field from the document and then use it further in my
>> > processing logic.
>> > The problem is that reading all documents to get all "id"'s is turning
>> out
>> > to be very slow. It is the bottleneck in my application. It would be
>> nice
>> > to have a way if lucene could return some metadata along with the
>> internal
>> > document id when I did a search. I do not want to read all documents
>> just
>> > to retrieve this metadata.
>> >
>> > The best solution I have come across searching on the net is to use
>> > payloads which will be returned by the fast index search query along
>> with
>> > the document ids.
>> >
>> > Is my understanding correct that using payloads I can get "id" string
>> field
>> > for all my documents faster than reading my entire document?
>> >
>> > I am not able to find a good example of how to store and retrieve
>> payloads?
>> > Can you please point me to a good resource to learn how to use payloads
>> and
>> > how they will impact performance?
>> > I am using Lucene 4.5.
>> >
>> > Thanks
>> > Rohit Banga
>> > http://iamrohitbanga.com/
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
Re: Question about Payloads in Lucene 4.5
Posted by Rohit Banga <ia...@gmail.com>.
Thanks Michael for your response.
Few questions:
1. Can I expect better performance when retrieving a single NumericDocValue
for all hits vs when I retrieve documents for all hits to fetch the field
value? As far as I understand retrieving n documents from the index
requires n disk reads. How many disk reads to I do when using
NumericDocValues? How are they stored?
2. I tried looking for examples on how to use numeric doc values. I found
that in new versions of lucene we have to use "AtomicReader".
Found this: http://www.gossamer-threads.com/lists/lucene/java-user/182641
So is this the code I am looking for:
long getNumericDocValueForDocument(IndexSearcher searcher, int docId) {
IndexReader reader = searcher.getIndexReader();
long docVal = 0;
for (AtomicReaderContext rc : reader.leaves()) {
AtomicReader ar = rc.reader();
docVal = ar.getNumericDocValues().get(*docID*);
}
return docVal;
}
How do I know which docVal to return? It appears that each AtomicReader
(every iteration of the loop) may return a docVal?
3. Can I only store NumericDocValues? Can I get something like
StringDocValues? I have a string "id". I guess I could keep a mapping from
numeric doc value (Long) to String but I want to avoid keeping two sources
of information (Lucene Index and a HashMap). I can use SearcherManager to
deal with concurrent searches and index updates (
http://blog.mikemccandless.com/2011/09/lucenes-searchermanager-simplifies.html),
but how about managing two data sources Lucene index and HashMap<Long,
String> with SearcherManager? Is there a way to achieve this using a custom
SearcherFactory?
Thanks
Rohit Banga
http://iamrohitbanga.com/
On Fri, Mar 21, 2014 at 3:26 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:
> DocValues are better than payloads.
>
> E.g. index a NumericDocValuesField with each doc, holding your id.
>
> Then at search time you can use MultiDocValues.getNumericValues.
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Fri, Mar 21, 2014 at 4:35 PM, Rohit Banga <ia...@gmail.com>
> wrote:
> > Hi everyone
> >
> > When I query a lucene index, I get back a list of document ids. This
> index
> > search is fast. Now for all documents matching the result I need a unique
> > String field called "id" which is stored in the document. From the
> > documentation I gather that document ids are internal and I should not
> use
> > them for referencing my own data structures. Currently I iterate over all
> > the hits matching the document and then for each one I get the document
> to
> > read the field using IndexReader.document().
> >
> http://lucene.apache.org/core/4_5_0/core/org/apache/lucene/index/IndexReader.html
> >
> > I read the "id" field from the document and then use it further in my
> > processing logic.
> > The problem is that reading all documents to get all "id"'s is turning
> out
> > to be very slow. It is the bottleneck in my application. It would be nice
> > to have a way if lucene could return some metadata along with the
> internal
> > document id when I did a search. I do not want to read all documents just
> > to retrieve this metadata.
> >
> > The best solution I have come across searching on the net is to use
> > payloads which will be returned by the fast index search query along with
> > the document ids.
> >
> > Is my understanding correct that using payloads I can get "id" string
> field
> > for all my documents faster than reading my entire document?
> >
> > I am not able to find a good example of how to store and retrieve
> payloads?
> > Can you please point me to a good resource to learn how to use payloads
> and
> > how they will impact performance?
> > I am using Lucene 4.5.
> >
> > Thanks
> > Rohit Banga
> > http://iamrohitbanga.com/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Question about Payloads in Lucene 4.5
Posted by Michael McCandless <lu...@mikemccandless.com>.
DocValues are better than payloads.
E.g. index a NumericDocValuesField with each doc, holding your id.
Then at search time you can use MultiDocValues.getNumericValues.
Mike McCandless
http://blog.mikemccandless.com
On Fri, Mar 21, 2014 at 4:35 PM, Rohit Banga <ia...@gmail.com> wrote:
> Hi everyone
>
> When I query a lucene index, I get back a list of document ids. This index
> search is fast. Now for all documents matching the result I need a unique
> String field called "id" which is stored in the document. From the
> documentation I gather that document ids are internal and I should not use
> them for referencing my own data structures. Currently I iterate over all
> the hits matching the document and then for each one I get the document to
> read the field using IndexReader.document().
> http://lucene.apache.org/core/4_5_0/core/org/apache/lucene/index/IndexReader.html
>
> I read the "id" field from the document and then use it further in my
> processing logic.
> The problem is that reading all documents to get all "id"'s is turning out
> to be very slow. It is the bottleneck in my application. It would be nice
> to have a way if lucene could return some metadata along with the internal
> document id when I did a search. I do not want to read all documents just
> to retrieve this metadata.
>
> The best solution I have come across searching on the net is to use
> payloads which will be returned by the fast index search query along with
> the document ids.
>
> Is my understanding correct that using payloads I can get "id" string field
> for all my documents faster than reading my entire document?
>
> I am not able to find a good example of how to store and retrieve payloads?
> Can you please point me to a good resource to learn how to use payloads and
> how they will impact performance?
> I am using Lucene 4.5.
>
> Thanks
> Rohit Banga
> http://iamrohitbanga.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org