Solr and query abortion

[Aleksey Vorona <av...@ea.com>]

Hi, we are running Solr 3.6.1 and see an issue in our load tests. Some 
of the queries our load test script produces result in huge number of 
hits. It may go as high as 90% of all documents we have (2.5M). Those 
are all range queries. I see in the log that those queries take much 
more time to execute.

Since such a query does not make any sense from the end user 
perspective, I would like to limit its performance impact.

Is it possible to abort the query after certain number of document hits 
or certain time elapsed and return a error? I would render that error as 
"Please refine your search" message to the end user in my application. I 
know that many sites on the web do that, and I guess most of them do 
that with Solr.

I tried setting timeAllowed limit, but, for some reason, I did not see 
those query times to go down. I suspect that most of the time is spent 
not in Search phase (which is the only one respecting timeAllowed, as 
far as I know), but in the sorting phase. And still, I want to abort any 
longer running query. Otherwise they accumulate over time, pushing 
server's load average sky high and killing performance even for regular 
queries.

-- Aleksey

Re: Solr and query abortion

[Aleksey Vorona <av...@ea.com>]

We are working on optimizing query performance. My concern was to ensure 
some stable QoS. Given our API and UI layout, user may generate an 
expensive query. Given the nature of the service, user may want to 
"hack" it. Currently, our Search API is a good point to try to inflict 
DoS on our server. And even though search outage will not cause any real 
security concern, it would be not nice.

That is why I wanted to put a hard limit on the query complexity. Thank 
you for a hint on how to do it.

As a side note, search performance with Solr is great. It is only during 
a good load test I am able to see those long running queries. When there 
is no load, even the most expensive query I have takes less than 100ms 
to be processed. As you said, 2.5M docs is not a very big index.

Thanks again for the reply. I am not sure if we are going to implement 
custom component for Solr or put query complexity estimation code in our 
application. But in any case your response was greatly appreciated, 
because I was thinking that I am missing something.

-- Aleksey

On 12-08-30 05:51 AM, Erick Erickson wrote:
> The first thing I'd do is run your query with &debguQuery=on and look
> at the "timings" section. That'll tell you what component is taking all
> the time and should help you figure out where the problem is....
>
> But worst-case you could implement a custom component to stop
> processing after some set number of responses..
>
> 2.5M docs isn't a very big index. So I'd look at the rest of the
> tuning knobs before jumping to a solution. Also be aware that
> the first time, for instance, a sort gets performed there's a lengthy
> hit for warming the caches so you should disregard the first few
> queries, or do appropriate autowarming.
>
> Best
> Erick
>
> On Wed, Aug 29, 2012 at 1:26 PM, Aleksey Vorona <av...@ea.com> wrote:
>> Hi, we are running Solr 3.6.1 and see an issue in our load tests. Some of
>> the queries our load test script produces result in huge number of hits. It
>> may go as high as 90% of all documents we have (2.5M). Those are all range
>> queries. I see in the log that those queries take much more time to execute.
>>
>> Since such a query does not make any sense from the end user perspective, I
>> would like to limit its performance impact.
>>
>> Is it possible to abort the query after certain number of document hits or
>> certain time elapsed and return a error? I would render that error as
>> "Please refine your search" message to the end user in my application. I
>> know that many sites on the web do that, and I guess most of them do that
>> with Solr.
>>
>> I tried setting timeAllowed limit, but, for some reason, I did not see those
>> query times to go down. I suspect that most of the time is spent not in
>> Search phase (which is the only one respecting timeAllowed, as far as I
>> know), but in the sorting phase. And still, I want to abort any longer
>> running query. Otherwise they accumulate over time, pushing server's load
>> average sky high and killing performance even for regular queries.
>>
>> -- Aleksey

Re: Solr and query abortion

[Erick Erickson <er...@gmail.com>]

The first thing I'd do is run your query with &debguQuery=on and look
at the "timings" section. That'll tell you what component is taking all
the time and should help you figure out where the problem is....

But worst-case you could implement a custom component to stop
processing after some set number of responses..

2.5M docs isn't a very big index. So I'd look at the rest of the
tuning knobs before jumping to a solution. Also be aware that
the first time, for instance, a sort gets performed there's a lengthy
hit for warming the caches so you should disregard the first few
queries, or do appropriate autowarming.

Best
Erick

On Wed, Aug 29, 2012 at 1:26 PM, Aleksey Vorona <av...@ea.com> wrote:
> Hi, we are running Solr 3.6.1 and see an issue in our load tests. Some of
> the queries our load test script produces result in huge number of hits. It
> may go as high as 90% of all documents we have (2.5M). Those are all range
> queries. I see in the log that those queries take much more time to execute.
>
> Since such a query does not make any sense from the end user perspective, I
> would like to limit its performance impact.
>
> Is it possible to abort the query after certain number of document hits or
> certain time elapsed and return a error? I would render that error as
> "Please refine your search" message to the end user in my application. I
> know that many sites on the web do that, and I guess most of them do that
> with Solr.
>
> I tried setting timeAllowed limit, but, for some reason, I did not see those
> query times to go down. I suspect that most of the time is spent not in
> Search phase (which is the only one respecting timeAllowed, as far as I
> know), but in the sorting phase. And still, I want to abort any longer
> running query. Otherwise they accumulate over time, pushing server's load
> average sky high and killing performance even for regular queries.
>
> -- Aleksey