You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by jamie <ja...@stimulussoft.com> on 2015/08/08 14:32:34 UTC
Lucene TermsFilter lookup slow
Greetings
Our app primarily uses Lucene for its intended purpose i.e. to search
across large amounts of unstructured text. However, recently our
requirement expanded to perform look-ups on specific documents in the
index based on associated custom defined unique keys. For our purposes,
a unique key is the string representation of a 128 bit murmur hash,
stored in a Lucene field named uid. We are currently using the
TermsFilter to lookup Documents in the Lucene index as follows:
List<Term> terms = new LinkedList<>();
for (String id : ids) {
terms.add(new Term("uid", id));
}
TermsFilter idFilter = new TermsFilter(terms);
... search logic...
At any time we may need to lookup say a couple of thousand documents.
Our problem is one of performance. On very large indexes with 30 million
records or more, the lookup can be excruciatingly slow. At this stage,
its not practical for us to move the data over to fit for purpose
database, nor change the uid field to a numeric type. I fully appreciate
the fact that Lucene is not designed to be a database, however, is there
anything we can do to improve the performance of these look-ups?
Much appreciate
Jamie
Re: Compressing docValues with variable length bytes[] by block of
16k ?
Posted by Olivier Binda <ol...@wanadoo.fr>.
On 08/09/2015 06:29 PM, Uwe Schindler wrote:
> Hi,
>
>> My values are unique and equal to the number of documents, They have
>> varying sizes, say at least 10 bytes and may be a lot bigger (say 4kbytes)
>>
>> I don't share, index or sort them.
>> I don't do grouping/faceting either
>>
>>
>> I only want to store, retrieve and traverse those values
> Then use stored fields.
ok, I'll try.
Thanks !
>
> Uwe
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Compressing docValues with variable length bytes[] by block of 16k ?
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
> My values are unique and equal to the number of documents, They have
> varying sizes, say at least 10 bytes and may be a lot bigger (say 4kbytes)
>
> I don't share, index or sort them.
> I don't do grouping/faceting either
>
>
> I only want to store, retrieve and traverse those values
Then use stored fields.
Uwe
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Compressing docValues with variable length bytes[] by block of
16k ?
Posted by Olivier Binda <ol...@wanadoo.fr>.
On 08/09/2015 04:55 PM, Arjen van der Meijden wrote:
>
> On 9-8-2015 16:22, Toke Eskildsen wrote:
>> Robert Muir <rc...@gmail.com> wrote:
>>> I am tired of repeating this:
>>> Don't use BINARY docvalues
>>> Don't use BINARY docvalues
>>> Don't use BINARY docvalues
>>> Use types like SORTED/SORTED_SET which will compress the term
>>> dictionary and make use of ordinals in your application instead.
>> This seems contrary to
>> http://lucene.apache.org/core/5_2_0/core/org/apache/lucene/document/BinaryDocValuesField.html
>>
>> Maybe you could update the JavaDoc for that field to warn against using it?
> It (probably) depends on the contents of the values. If the number of
> distinct values is roughly equal to the number of documents the javadoc
> suggest the binary docvalues are a valid choice.
My values are unique and equal to the number of documents,
They have varying sizes, say at least 10 bytes and may be a lot bigger
(say 4kbytes)
I don't share, index or sort them.
I don't do grouping/faceting either
I only want to store, retrieve and traverse those values
>
> That's this part:
> "The values are stored directly with no sharing, which is a good fit
> when the fields don't share (many) values, such as a title field."
>
> If there are (much) less distinct values than documents, Robert's reply
> and the documentation suggest the same:
> " If values may be shared and sorted it's better to use
> SortedDocValuesField."
>
> So as soon as compression of smallish values starts making sense due to
> repetition amongst documents, it may be time to move away from the
> BinaryDocValuesField towards another variant.
>
> If only parts of the values are repeated (for instance something like
> e-mail addresses where many will end with 'gmail.com' and 'outlook.com')
> it becomes more complicated.
At the moment, there are some repeated parts inside but a lot of
repeated parts across docIds like "Expression", "Reading"
Also, I'm stuck with using Lucene 4.7.0 (or 4.7.2) because starting with
version 4.8, lucene uses "try with resource" and this isn't supported on
Android before Android 4.4
SortedDocValuesField stores a per-document|BytesRef|
<http://lucene.apache.org/core/5_2_0/core/org/apache/lucene/util/BytesRef.html>value,
indexed for sorting.
If you also need to store the value, you should add a
separate|StoredField|
<http://lucene.apache.org/core/5_2_0/core/org/apache/lucene/document/StoredField.html>instance.
I actually went with the binaryDocValues because I thought that
DocValues were way more efficient than the pre 4.0 fields to store stuff
(like only using 1 seek/read ...with mmap...), especially with traversal.
In my app, I traverse all binaryDocValues in increading docId order,
unserializes my docValues (lightning fast with FlatBuffers, no object
creation -> complex objects) and do some stats....
Would I be able to do that as efficiently with a StoredField ?
Apparently, only StoredField are compressed
CompressingStoredFieldsFormat
Maybee I should use that (and ditch the useless docValue or make it
store a bytesRef) to get compression ?
Many thanks for all the insights, :)
Olivier
> Best regards,
>
> Arjen
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Compressing docValues with variable length bytes[] by block of
16k ?
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
Arjen van der Meijden <ac...@tweakers.net> wrote:
> On 9-8-2015 16:22, Toke Eskildsen wrote:
> > Maybe you could update the JavaDoc for that field to warn against using it?
> It (probably) depends on the contents of the values.
That was my impression too, but we both seem to be second-guessing Robert's very non-nuanced and clearly oft-repeated recommendation. I hope Robert can shed some light on this and tell us if he finds the JavaDocs to be in order or if binary DocValues should not be used at all.
- Toke Eskildsen
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Compressing docValues with variable length bytes[] by block of
16k ?
Posted by Arjen van der Meijden <ac...@tweakers.net>.
On 9-8-2015 16:22, Toke Eskildsen wrote:
> Robert Muir <rc...@gmail.com> wrote:
>> I am tired of repeating this:
>> Don't use BINARY docvalues
>> Don't use BINARY docvalues
>> Don't use BINARY docvalues
>> Use types like SORTED/SORTED_SET which will compress the term
>> dictionary and make use of ordinals in your application instead.
> This seems contrary to
> http://lucene.apache.org/core/5_2_0/core/org/apache/lucene/document/BinaryDocValuesField.html
>
> Maybe you could update the JavaDoc for that field to warn against using it?
It (probably) depends on the contents of the values. If the number of
distinct values is roughly equal to the number of documents the javadoc
suggest the binary docvalues are a valid choice.
That's this part:
"The values are stored directly with no sharing, which is a good fit
when the fields don't share (many) values, such as a title field."
If there are (much) less distinct values than documents, Robert's reply
and the documentation suggest the same:
" If values may be shared and sorted it's better to use
SortedDocValuesField."
So as soon as compression of smallish values starts making sense due to
repetition amongst documents, it may be time to move away from the
BinaryDocValuesField towards another variant.
If only parts of the values are repeated (for instance something like
e-mail addresses where many will end with 'gmail.com' and 'outlook.com')
it becomes more complicated.
Best regards,
Arjen
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Compressing docValues with variable length bytes[] by block of
16k ?
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
Robert Muir <rc...@gmail.com> wrote:
> I am tired of repeating this:
> Don't use BINARY docvalues
> Don't use BINARY docvalues
> Don't use BINARY docvalues
> Use types like SORTED/SORTED_SET which will compress the term
> dictionary and make use of ordinals in your application instead.
This seems contrary to
http://lucene.apache.org/core/5_2_0/core/org/apache/lucene/document/BinaryDocValuesField.html
Maybe you could update the JavaDoc for that field to warn against using it?
- Toke Eskildsen
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Compressing docValues with variable length bytes[] by block of
16k ?
Posted by Robert Muir <rc...@gmail.com>.
That makes no sense at all, it would make it slow as shit.
I am tired of repeating this:
Don't use BINARY docvalues
Don't use BINARY docvalues
Don't use BINARY docvalues
Use types like SORTED/SORTED_SET which will compress the term
dictionary and make use of ordinals in your application instead.
On Sat, Aug 8, 2015 at 10:19 AM, Olivier Binda <ol...@wanadoo.fr> wrote:
> Greetings
>
> are there any plans to implement compression of the variable length bites[]
> binary doc Values,
> say in blocks of 16k like for stored values ?
>
> my .cfs file goes from 2MB to like 400k when I zip it
>
> Best regards,
> Olivier
>
>
>
> On 08/08/2015 02:32 PM, jamie wrote:
>>
>> Greetings
>>
>> Our app primarily uses Lucene for its intended purpose i.e. to search
>> across large amounts of unstructured text. However, recently our requirement
>> expanded to perform look-ups on specific documents in the index based on
>> associated custom defined unique keys. For our purposes, a unique key is the
>> string representation of a 128 bit murmur hash, stored in a Lucene field
>> named uid. We are currently using the TermsFilter to lookup Documents in
>> the Lucene index as follows:
>>
>> List<Term> terms = new LinkedList<>();
>> for (String id : ids) {
>> terms.add(new Term("uid", id));
>> }
>> TermsFilter idFilter = new TermsFilter(terms);
>> ... search logic...
>>
>> At any time we may need to lookup say a couple of thousand documents. Our
>> problem is one of performance. On very large indexes with 30 million records
>> or more, the lookup can be excruciatingly slow. At this stage, its not
>> practical for us to move the data over to fit for purpose database, nor
>> change the uid field to a numeric type. I fully appreciate the fact that
>> Lucene is not designed to be a database, however, is there anything we can
>> do to improve the performance of these look-ups?
>>
>> Much appreciate
>>
>> Jamie
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Compressing docValues with variable length bytes[] by block of 16k
?
Posted by Olivier Binda <ol...@wanadoo.fr>.
Greetings
are there any plans to implement compression of the variable length
bites[] binary doc Values,
say in blocks of 16k like for stored values ?
my .cfs file goes from 2MB to like 400k when I zip it
Best regards,
Olivier
On 08/08/2015 02:32 PM, jamie wrote:
> Greetings
>
> Our app primarily uses Lucene for its intended purpose i.e. to search
> across large amounts of unstructured text. However, recently our
> requirement expanded to perform look-ups on specific documents in the
> index based on associated custom defined unique keys. For our
> purposes, a unique key is the string representation of a 128 bit
> murmur hash, stored in a Lucene field named uid. We are currently
> using the TermsFilter to lookup Documents in the Lucene index as follows:
>
> List<Term> terms = new LinkedList<>();
> for (String id : ids) {
> terms.add(new Term("uid", id));
> }
> TermsFilter idFilter = new TermsFilter(terms);
> ... search logic...
>
> At any time we may need to lookup say a couple of thousand documents.
> Our problem is one of performance. On very large indexes with 30
> million records or more, the lookup can be excruciatingly slow. At
> this stage, its not practical for us to move the data over to fit for
> purpose database, nor change the uid field to a numeric type. I fully
> appreciate the fact that Lucene is not designed to be a database,
> however, is there anything we can do to improve the performance of
> these look-ups?
>
> Much appreciate
>
> Jamie
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Lucene TermsFilter lookup slow
Posted by jamie <ja...@stimulussoft.com>.
Michael
Forgive me, I am not familiar with Lucene internal code. Can you verify
whether these suggested changes are indeed correct.
I am changing line 210 of TermsFilter.
if (result == null) {
if (docs.nextDoc() != DocIdSetIterator.NO_MORE_DOCS) {
result = new FixedBitSet(reader.maxDoc());
// lazy init but don't do it in the hot loop since we could
read many docs
result.set(docs.docID());
}
}
// below commented out
// while (docs.nextDoc() != DocIdSetIterator.NO_MORE_DOCS) {
// result.set(docs.docID());
//}
This change seems to have very little impact on performance.
It is taking around 25 second to look up documents associated with
murmur hash string id's on an index size of 10m records.
Thanks in advance
Jamie
On 2015/08/10 2:46 PM, Michael McCandless wrote:
> OK, indeed, that version has the changes I was thinking of,
> specifically optimizing the case when only a single doc contains a
> term by inlining that docID into the terms dict.
>
> You should be able to improve on TermsFilter a bit because you know
> only 1 doc matches each ID, so after the first segment finds a given
> ID you should stop testing other segments. Also, since you are doing
> bulk lookup, you should pre-sort the IDs so it's a sequential scan
> through the terms dict.
>
> There is another thread right now, subject "Mapping doc values back to
> doc ID (in decent time)", also talking about how to do faster PK
> lookups.
>
> Mike McCandless
>
> http://blog.mikemccandless.com
Re: Lucene TermsFilter lookup slow
Posted by Michael McCandless <lu...@mikemccandless.com>.
OK, indeed, that version has the changes I was thinking of,
specifically optimizing the case when only a single doc contains a
term by inlining that docID into the terms dict.
You should be able to improve on TermsFilter a bit because you know
only 1 doc matches each ID, so after the first segment finds a given
ID you should stop testing other segments. Also, since you are doing
bulk lookup, you should pre-sort the IDs so it's a sequential scan
through the terms dict.
There is another thread right now, subject "Mapping doc values back to
doc ID (in decent time)", also talking about how to do faster PK
lookups.
Mike McCandless
http://blog.mikemccandless.com
On Sun, Aug 9, 2015 at 3:17 AM, jamie <ja...@stimulussoft.com> wrote:
> Mike
>
> Thank you kindly for the reply. I am using Lucene v4.10.4. Are the
> optimization you refer to, available in this version?
>
> We haven't yet upgraded to Lucene 5 as there appear to be many API changes.
>
> Jamie
>
>
> On 2015/08/08 5:13 PM, Michael McCandless wrote:
>>
>> Which version of Lucene are you using? Newer versions have optimized
>> the "primary key" use case somewhat...
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>>
>> On Sat, Aug 8, 2015 at 8:32 AM, jamie <ja...@stimulussoft.com> wrote:
>>>
>>> Greetings
>>>
>>> Our app primarily uses Lucene for its intended purpose i.e. to search
>>> across
>>> large amounts of unstructured text. However, recently our requirement
>>> expanded to perform look-ups on specific documents in the index based on
>>> associated custom defined unique keys. For our purposes, a unique key is
>>> the
>>> string representation of a 128 bit murmur hash, stored in a Lucene field
>>> named uid. We are currently using the TermsFilter to lookup Documents in
>>> the Lucene index as follows:
>>>
>>> List<Term> terms = new LinkedList<>();
>>> for (String id : ids) {
>>> terms.add(new Term("uid", id));
>>> }
>>> TermsFilter idFilter = new TermsFilter(terms);
>>> ... search logic...
>>>
>>> At any time we may need to lookup say a couple of thousand documents. Our
>>> problem is one of performance. On very large indexes with 30 million
>>> records
>>> or more, the lookup can be excruciatingly slow. At this stage, its not
>>> practical for us to move the data over to fit for purpose database, nor
>>> change the uid field to a numeric type. I fully appreciate the fact that
>>> Lucene is not designed to be a database, however, is there anything we
>>> can
>>> do to improve the performance of these look-ups?
>>>
>>> Much appreciate
>>>
>>> Jamie
>>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Lucene TermsFilter lookup slow
Posted by jamie <ja...@stimulussoft.com>.
Mike
Thank you kindly for the reply. I am using Lucene v4.10.4. Are the
optimization you refer to, available in this version?
We haven't yet upgraded to Lucene 5 as there appear to be many API changes.
Jamie
On 2015/08/08 5:13 PM, Michael McCandless wrote:
> Which version of Lucene are you using? Newer versions have optimized
> the "primary key" use case somewhat...
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Sat, Aug 8, 2015 at 8:32 AM, jamie <ja...@stimulussoft.com> wrote:
>> Greetings
>>
>> Our app primarily uses Lucene for its intended purpose i.e. to search across
>> large amounts of unstructured text. However, recently our requirement
>> expanded to perform look-ups on specific documents in the index based on
>> associated custom defined unique keys. For our purposes, a unique key is the
>> string representation of a 128 bit murmur hash, stored in a Lucene field
>> named uid. We are currently using the TermsFilter to lookup Documents in
>> the Lucene index as follows:
>>
>> List<Term> terms = new LinkedList<>();
>> for (String id : ids) {
>> terms.add(new Term("uid", id));
>> }
>> TermsFilter idFilter = new TermsFilter(terms);
>> ... search logic...
>>
>> At any time we may need to lookup say a couple of thousand documents. Our
>> problem is one of performance. On very large indexes with 30 million records
>> or more, the lookup can be excruciatingly slow. At this stage, its not
>> practical for us to move the data over to fit for purpose database, nor
>> change the uid field to a numeric type. I fully appreciate the fact that
>> Lucene is not designed to be a database, however, is there anything we can
>> do to improve the performance of these look-ups?
>>
>> Much appreciate
>>
>> Jamie
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Lucene TermsFilter lookup slow
Posted by Michael McCandless <lu...@mikemccandless.com>.
Which version of Lucene are you using? Newer versions have optimized
the "primary key" use case somewhat...
Mike McCandless
http://blog.mikemccandless.com
On Sat, Aug 8, 2015 at 8:32 AM, jamie <ja...@stimulussoft.com> wrote:
> Greetings
>
> Our app primarily uses Lucene for its intended purpose i.e. to search across
> large amounts of unstructured text. However, recently our requirement
> expanded to perform look-ups on specific documents in the index based on
> associated custom defined unique keys. For our purposes, a unique key is the
> string representation of a 128 bit murmur hash, stored in a Lucene field
> named uid. We are currently using the TermsFilter to lookup Documents in
> the Lucene index as follows:
>
> List<Term> terms = new LinkedList<>();
> for (String id : ids) {
> terms.add(new Term("uid", id));
> }
> TermsFilter idFilter = new TermsFilter(terms);
> ... search logic...
>
> At any time we may need to lookup say a couple of thousand documents. Our
> problem is one of performance. On very large indexes with 30 million records
> or more, the lookup can be excruciatingly slow. At this stage, its not
> practical for us to move the data over to fit for purpose database, nor
> change the uid field to a numeric type. I fully appreciate the fact that
> Lucene is not designed to be a database, however, is there anything we can
> do to improve the performance of these look-ups?
>
> Much appreciate
>
> Jamie
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org