Filter query question

[LOPEZ-CORTES Mariano-ext <ma...@pole-emploi.fr>]

Hi

In our search application we have one facet filter (Status)

Each status value corresponds to multiple values in the Solr database

Example : Status : Initialized --> status in solr =  11I, 12I, 13I, 14I, ...

On status value click, search is re-fired with fq filter:

fq: status:(11I OR 12I OR 13I ....)

This was very very inefficient. Filter query response time was longer than same search without filter!

We have changed status value in Solr database for corresponding to visual filter values. In consequence, there is no OR in the fq filter.
The performance is better now.

What is the reason?

Thanks!

Re: Filter query question

[Emir Arnautović <em...@sematext.com>]

Hi,
What is the number of these status indicators? It is expected to have slower query if you have more clauses since Solr/Lucene has to load postings for each term and then OR them. The real question is why it is constantly slow since you are using fq and it should be cached. Did you disable filter cache? Or you maybe commit frequently and do not have warmup query that would load that filter in cache?

Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 12 Apr 2018, at 09:46, LOPEZ-CORTES Mariano-ext <ma...@pole-emploi.fr> wrote:
> 
> Hi
> 
> In our search application we have one facet filter (Status)
> 
> Each status value corresponds to multiple values in the Solr database
> 
> Example : Status : Initialized --> status in solr =  11I, 12I, 13I, 14I, ...
> 
> On status value click, search is re-fired with fq filter:
> 
> fq: status:(11I OR 12I OR 13I ....)
> 
> This was very very inefficient. Filter query response time was longer than same search without filter!
> 
> We have changed status value in Solr database for corresponding to visual filter values. In consequence, there is no OR in the fq filter.
> The performance is better now.
> 
> What is the reason?
> 
> Thanks!
> 

Re: Filter query question

[Shawn Heisey <ap...@elyograg.org>]

On 4/12/2018 1:46 AM, LOPEZ-CORTES Mariano-ext wrote:
> In our search application we have one facet filter (Status)
>
> Each status value corresponds to multiple values in the Solr database
>
> Example : Status : Initialized --> status in solr =  11I, 12I, 13I, 14I, ...
>
> On status value click, search is re-fired with fq filter:
>
> fq: status:(11I OR 12I OR 13I ....)
>
> This was very very inefficient. Filter query response time was longer than same search without filter!

How many different status values are you including in that query?  And 
how many unique values are there in the status field for the entire index?

Echoing what Emir said:  If the numbers are large, it's going to be 
slow.  But once that filter is run successfully, it should be cached 
until the next time you commit changes to your index and open a new 
searcher, or there are enough different filters executed that this entry 
is pushed out of the cache.

Having sufficient memory for your operating system to effectively cache 
your index data can speed up ALL queries.  How much memory do you have 
in the server?  How much memory have you allocated to Solr on that 
system?  Is there software other than Solr installed on the server?  How 
much disk space do all the Solr indexes on that server consume?  How 
many documents are represented by that disk space?

Back in late February, I discussed this with you when you were wanting 
query times below one second.

> We have changed status value in Solr database for corresponding to visual filter values. In consequence, there is no OR in the fq filter.
> The performance is better now.

This seems to say that you've made a change that improved your 
situation.  But reading what you wrote, I cannot tell what that change was.

If you're running at least version 4.10, you could use the terms query 
parser instead of a search clause with OR separators.  In addition to 
better performance, this query parser doesn't have the maxBooleanClauses 
limit.

https://lucene.apache.org/solr/guide/7_3/other-parsers.html

Thanks,
Shawn