Predicting query execution time.

[Modassar Ather <mo...@gmail.com>]

Hi,

We see queries executing in less than a second and taking minutes to
execute as well. We need to predict the approximate time a query might take
to execute.
Need your help in finding the factors to be considered and calculating an
approximate execution time.

Thanks,
Modassar

Re: Predicting query execution time.

[Modassar Ather <mo...@gmail.com>]

Thanks Shawn for your suggestions.

Best,
Modassar

On Wed, Oct 12, 2016 at 11:44 AM, Shawn Heisey <ap...@elyograg.org> wrote:

> On 10/11/2016 11:46 PM, Modassar Ather wrote:
> > We see queries executing in less than a second and taking minutes to
> > execute as well. We need to predict the approximate time a query might
> > take to execute. Need your help in finding the factors to be
> > considered and calculating an approximate execution time.
>
> If a query or filter query has been cached successfully by Solr, then
> running it again while on the same searcher will happen very quickly.
>
> I think you can look at the number of fields in the query and the number
> of terms being searched in each of those fields as a general guide for
> query complexity.  Wildcard queries can expand to a large number of
> terms, so those can be quite slow.
>
> Deep paging (setting the start parameter to a high number) can make
> queries slow, particularly on multi-shard indexes.
>
> Using complex queries as filters (fq parameter) will tend to run slower
> than the same thing in the q parameter, at least the first time the
> filter is executed on an index searcher.  Once a given filter query is
> in the filterCache, it typically will execute VERY quickly.  Be aware
> that using NOW in date queries without rounding will change the query on
> every execution, so it cannot be cached.
>
> Complex features like faceting, grouping, and the stats component will
> tend to take longer with fields that have a very large number of terms,
> and might also take longer with queries that have many matches.
>
> Other people might have a better idea of what kinds of queries are slow,
> but hopefully this is a decent guide.
>
> If you truly cannot see any sort of common qualities in your slow
> queries, then you may be running into a situation where you don't have
> enough memory for your index to be effectively cached.  This situation
> can cause a query that would normally be fast to be very slow, if the
> index data the query needs has been pushed out of the operating system
> disk cache by other queries.  Information about effective disk caching
> is discussed here:
>
> https://wiki.apache.org/solr/SolrPerformanceProblems
>
> Thanks,
> Shawn
>
>

Re: Predicting query execution time.

[Shawn Heisey <ap...@elyograg.org>]

On 10/11/2016 11:46 PM, Modassar Ather wrote:
> We see queries executing in less than a second and taking minutes to
> execute as well. We need to predict the approximate time a query might
> take to execute. Need your help in finding the factors to be
> considered and calculating an approximate execution time. 

If a query or filter query has been cached successfully by Solr, then
running it again while on the same searcher will happen very quickly.

I think you can look at the number of fields in the query and the number
of terms being searched in each of those fields as a general guide for
query complexity.  Wildcard queries can expand to a large number of
terms, so those can be quite slow.

Deep paging (setting the start parameter to a high number) can make
queries slow, particularly on multi-shard indexes.

Using complex queries as filters (fq parameter) will tend to run slower
than the same thing in the q parameter, at least the first time the
filter is executed on an index searcher.  Once a given filter query is
in the filterCache, it typically will execute VERY quickly.  Be aware
that using NOW in date queries without rounding will change the query on
every execution, so it cannot be cached.

Complex features like faceting, grouping, and the stats component will
tend to take longer with fields that have a very large number of terms,
and might also take longer with queries that have many matches.

Other people might have a better idea of what kinds of queries are slow,
but hopefully this is a decent guide.

If you truly cannot see any sort of common qualities in your slow
queries, then you may be running into a situation where you don't have
enough memory for your index to be effectively cached.  This situation
can cause a query that would normally be fast to be very slow, if the
index data the query needs has been pushed out of the operating system
disk cache by other queries.  Information about effective disk caching
is discussed here:

https://wiki.apache.org/solr/SolrPerformanceProblems

Thanks,
Shawn