Is there any performance cost of using lots of OR in the solr query

[roz dev <ro...@gmail.com>]

Hi All,

I am working on an application which makes few solr calls to get the data.

On the high level, We have a requirement like this


   - Make first call to Solr, to get the list of products which are
   children of a given category
   - Make 2nd solr call to get product documents based on a list of product
   ids

2nd query will look like

q=document_type:SKU&fq=product_id:(34 OR 45 OR 56 OR 77)

We can have close to 100 product ids in fq.

is there a performance cost of doing these solr calls which have lots of OR?

As per Slide # 41 of Presentation "The Seven Deadly Sins of Solr", it is a
bad idea to have these kind of queries.

http://www.slideshare.net/lucenerevolution/hill-jay-7-sins-of-solrpdf

But, It does not become clear the reason it is bad.

Any inputs will be welcome.

Thanks

Saroj

Re: Is there any performance cost of using lots of OR in the solr query

[Erick Erickson <er...@gmail.com>]

Shawn:

Ahhh, so *that* was what your JIRA was about....

Consider https://issues.apache.org/jira/browse/SOLR-2429
for your ACL calculations, that's what this was developed
for.

The basic idea is that you can write a custom filter that returns
whether the document should be included in the results set that's
only called _after_ all other clauses (search and FQs) have been
satisfied.

Here's the issue. Normally, fqs are calculated across the entire
document set. That's what allows them to be cached and
re-used. But, as you've found, doing ACL calculations
for the entire document set is expensive. So this is an attempt
to make a lower-cost alternative. The downside is that it is NOT
cached, so it must be calculated anew each time. But it's only
calculated for a subset of documents.

Best
Erick

On Fri, Apr 6, 2012 at 9:00 AM, Shawn Heisey <so...@elyograg.org> wrote:
> On 4/5/2012 3:49 PM, Erick Erickson wrote:
>>
>> Of course putting more clauses in an OR query will
>> have a performance cost, there's more work to do....
>>
>> OK, being a smart-alec aside you will probably
>> be fine with a few hundred clauses. The question
>> is simply whether the performance hit is acceptable.
>> I'm afraid that question can't be answered in the
>> abstract, you'll have to test...
>>
>> Since you're putting them in an fq, there's also some chance
>> that they'll be re-used from the cache, at least if there
>> are common patterns.
>
>
> Roz,
>
> I have a similar situation going on in my index.  Because employees have
> access to far more than real users, they get filter queries constructed that
> have HUGE number of clauses in them.  We have implemented a new field for a
> feature that we call "search groups" but it has not penetrated all aspects
> of the application yet.  Also, until we can make those groups use a
> hierarchy, which is not a trivial undertaking, we may be stuck with large
> filter queries.
>
> These complex filters have led to a problem that you have probably not
> considered - really long filterCache autowarm times.  I have reduced the
> autoWarm value on my filterCache to FOUR, and there are still times that the
> autowarm takes up to 60 seconds.  Most of the time it is only a few seconds,
> with up to 30 seconds being relatively common.
>
> I just thought of a new localparam feature for this situation and filed
> SOLR-3333.  I will talk to our developers about using the existing
> localparam that skips filterCache entirely.
>
> Thanks,
> Shawn
>

Re: Is there any performance cost of using lots of OR in the solr query

[Shawn Heisey <so...@elyograg.org>]

On 4/5/2012 3:49 PM, Erick Erickson wrote:
> Of course putting more clauses in an OR query will
> have a performance cost, there's more work to do....
>
> OK, being a smart-alec aside you will probably
> be fine with a few hundred clauses. The question
> is simply whether the performance hit is acceptable.
> I'm afraid that question can't be answered in the
> abstract, you'll have to test...
>
> Since you're putting them in an fq, there's also some chance
> that they'll be re-used from the cache, at least if there
> are common patterns.

Roz,

I have a similar situation going on in my index.  Because employees have 
access to far more than real users, they get filter queries constructed 
that have HUGE number of clauses in them.  We have implemented a new 
field for a feature that we call "search groups" but it has not 
penetrated all aspects of the application yet.  Also, until we can make 
those groups use a hierarchy, which is not a trivial undertaking, we may 
be stuck with large filter queries.

These complex filters have led to a problem that you have probably not 
considered - really long filterCache autowarm times.  I have reduced the 
autoWarm value on my filterCache to FOUR, and there are still times that 
the autowarm takes up to 60 seconds.  Most of the time it is only a few 
seconds, with up to 30 seconds being relatively common.

I just thought of a new localparam feature for this situation and filed 
SOLR-3333.  I will talk to our developers about using the existing 
localparam that skips filterCache entirely.

Thanks,
Shawn

Re: Is there any performance cost of using lots of OR in the solr query

[Erick Erickson <er...@gmail.com>]

Of course putting more clauses in an OR query will
have a performance cost, there's more work to do....

OK, being a smart-alec aside you will probably
be fine with a few hundred clauses. The question
is simply whether the performance hit is acceptable.
I'm afraid that question can't be answered in the
abstract, you'll have to test...

Since you're putting them in an fq, there's also some chance
that they'll be re-used from the cache, at least if there
are common patterns.

Best
Erick

On Wed, Apr 4, 2012 at 8:05 PM, roz dev <ro...@gmail.com> wrote:
> Hi All,
>
> I am working on an application which makes few solr calls to get the data.
>
> On the high level, We have a requirement like this
>
>
>   - Make first call to Solr, to get the list of products which are
>   children of a given category
>   - Make 2nd solr call to get product documents based on a list of product
>   ids
>
> 2nd query will look like
>
> q=document_type:SKU&fq=product_id:(34 OR 45 OR 56 OR 77)
>
> We can have close to 100 product ids in fq.
>
> is there a performance cost of doing these solr calls which have lots of OR?
>
> As per Slide # 41 of Presentation "The Seven Deadly Sins of Solr", it is a
> bad idea to have these kind of queries.
>
> http://www.slideshare.net/lucenerevolution/hill-jay-7-sins-of-solrpdf
>
> But, It does not become clear the reason it is bad.
>
> Any inputs will be welcome.
>
> Thanks
>
> Saroj