You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by jichi <ji...@gmail.com> on 2015/11/20 22:50:45 UTC
Search with very large boolean filter
Hi,
I am using Solr 4.7.0 to search text with an id filter, like this:
id:(100 OR 2 OR 5 OR 81 OR 10 ...)
The number of IDs in the boolean filter are usually less than 100, but
could sometimes be very large (around 30k IDs).
We currently set maxBooleanClauses to 1024, partitioned the IDs by every
1000, and batched the solr queries, which worked but became slow when the
total number of IDs is larger than 10k.
I am wondering what would be the best strategy to handle this kind of
problem?
Can we increase the maxBooleanClauses to reduce the number of batches?
And if possible, we prefer not to create additionally large indexes.
Thanks!
Re: Search with very large boolean filter
Posted by jichi <ji...@gmail.com>.
Hi Shawn,
We have already switched the request method to POST.
I am going to try the term query parser soon. I will post the performance
difference against the IN syntax here later.
Thanks!
2015-11-20 15:23 GMT-08:00 Shawn Heisey <ap...@elyograg.org>:
> On 11/20/2015 4:09 PM, jichi wrote:
> > Thanks for the quick replies, Alex and Jack!
> >
> >> definitely can improve on the ORing the ids with
> > Going to try that! But I guess it would still hit the
> maxBooleanClauses=1024
> > threshold.
>
> The terms query parser does not have a limit like boolean queries do.
> This query parser was added in version 4.10, so be aware of that.
> Querying for a large number of terms with the terms query parser will
> scale a lot better than a boolean query -- better performance.
>
> The number of terms you query will affect the size of the query text.
> The query size is constrained by either the max HTTP header size if the
> request is a GET, or the max form size if it's a POST. The max HTTP
> header size is configurable in the servlet container (jetty, tomcat,
> etc) and I would not recommend going over about 32K with it. The max
> form size is configurable in solrconfig.xml with the
> formdataUploadLimitInKB attribute on the requestParsers element. That
> attribute defaults to 2048, which yields a default size of 2MB.
> Switching your queries to POST requests is advisable.
>
> Thanks,
> Shawn
>
>
--
jichi
Re: Search with very large boolean filter
Posted by Shawn Heisey <ap...@elyograg.org>.
On 11/20/2015 4:09 PM, jichi wrote:
> Thanks for the quick replies, Alex and Jack!
>
>> definitely can improve on the ORing the ids with
> Going to try that! But I guess it would still hit the maxBooleanClauses=1024
> threshold.
The terms query parser does not have a limit like boolean queries do.
This query parser was added in version 4.10, so be aware of that.
Querying for a large number of terms with the terms query parser will
scale a lot better than a boolean query -- better performance.
The number of terms you query will affect the size of the query text.
The query size is constrained by either the max HTTP header size if the
request is a GET, or the max form size if it's a POST. The max HTTP
header size is configurable in the servlet container (jetty, tomcat,
etc) and I would not recommend going over about 32K with it. The max
form size is configurable in solrconfig.xml with the
formdataUploadLimitInKB attribute on the requestParsers element. That
attribute defaults to 2048, which yields a default size of 2MB.
Switching your queries to POST requests is advisable.
Thanks,
Shawn
Re: Search with very large boolean filter
Posted by jichi <ji...@gmail.com>.
Thanks for the quick replies, Alex and Jack!
> definitely can improve on the ORing the ids with
Going to try that! But I guess it would still hit the maxBooleanClauses=1024
threshold.
> 1. Are you trying to retrieve a large number of documents, or simply
perform queries against a subset of the index?
We would like to perform queries against a subset of the index.
> 2. How many unique queries are you expecting to perform against each
specific filter set of IDs?
There are usually only a couple (around 10) of unique queries for the same
set of IDs for a short period of time (around 1min).
> 3. How often does the set of IDs change?
The IDs are almost different for each query.
btw., the total number would be 99% be less than 1k.
But in 1% rare cases it could be more than 10k.
> 4. Is there more than one filter set of IDs in use during a particular
interval of time?
No. The ID set will be the only filter applied to "id".
Thanks!
2015-11-20 14:26 GMT-08:00 Jack Krupansky <ja...@gmail.com>:
> 1. Are you trying to retrieve a large number of documents, or simply
> perform queries against a subset of the index?
>
> 2. How many unique queries are you expecting to perform against each
> specific filter set of IDs?
>
> 3. How often does the set of IDs change?
>
> 4. Is there more than one filter set of IDs in use during a particular
> interval of time?
>
>
>
> -- Jack Krupansky
>
> On Fri, Nov 20, 2015 at 4:50 PM, jichi <ji...@gmail.com> wrote:
>
>> Hi,
>>
>> I am using Solr 4.7.0 to search text with an id filter, like this:
>>
>> id:(100 OR 2 OR 5 OR 81 OR 10 ...)
>>
>> The number of IDs in the boolean filter are usually less than 100, but
>> could sometimes be very large (around 30k IDs).
>>
>> We currently set maxBooleanClauses to 1024, partitioned the IDs by every
>> 1000, and batched the solr queries, which worked but became slow when the
>> total number of IDs is larger than 10k.
>>
>> I am wondering what would be the best strategy to handle this kind of
>> problem?
>> Can we increase the maxBooleanClauses to reduce the number of batches?
>> And if possible, we prefer not to create additionally large indexes.
>>
>> Thanks!
>>
>
>
--
jichi
Re: Search with very large boolean filter
Posted by Jack Krupansky <ja...@gmail.com>.
1. Are you trying to retrieve a large number of documents, or simply
perform queries against a subset of the index?
2. How many unique queries are you expecting to perform against each
specific filter set of IDs?
3. How often does the set of IDs change?
4. Is there more than one filter set of IDs in use during a particular
interval of time?
-- Jack Krupansky
On Fri, Nov 20, 2015 at 4:50 PM, jichi <ji...@gmail.com> wrote:
> Hi,
>
> I am using Solr 4.7.0 to search text with an id filter, like this:
>
> id:(100 OR 2 OR 5 OR 81 OR 10 ...)
>
> The number of IDs in the boolean filter are usually less than 100, but
> could sometimes be very large (around 30k IDs).
>
> We currently set maxBooleanClauses to 1024, partitioned the IDs by every
> 1000, and batched the solr queries, which worked but became slow when the
> total number of IDs is larger than 10k.
>
> I am wondering what would be the best strategy to handle this kind of
> problem?
> Can we increase the maxBooleanClauses to reduce the number of batches?
> And if possible, we prefer not to create additionally large indexes.
>
> Thanks!
>
Re: Search with very large boolean filter
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
I don't know what to do about 30K ids, but you definitely can improve
on the ORing the ids with
https://cwiki.apache.org/confluence/display/solr/Other+Parsers#OtherParsers-TermsQueryParser
Regards,
Alex.
----
Newsletter and resources for Solr beginners and intermediates:
http://www.solr-start.com/
On 20 November 2015 at 16:50, jichi <ji...@gmail.com> wrote:
> Hi,
>
> I am using Solr 4.7.0 to search text with an id filter, like this:
>
> id:(100 OR 2 OR 5 OR 81 OR 10 ...)
>
> The number of IDs in the boolean filter are usually less than 100, but
> could sometimes be very large (around 30k IDs).
>
> We currently set maxBooleanClauses to 1024, partitioned the IDs by every
> 1000, and batched the solr queries, which worked but became slow when the
> total number of IDs is larger than 10k.
>
> I am wondering what would be the best strategy to handle this kind of
> problem?
> Can we increase the maxBooleanClauses to reduce the number of batches?
> And if possible, we prefer not to create additionally large indexes.
>
> Thanks!