Monitoring Solr for currently running queries

[ufuk yılmaz <uy...@vivaldi.net.INVALID>]

Hello All,

Is there a way to see currently executing queries in a SolrCloud? Or a general strategy to detect a query using absurd amount or resources?

We are using Solr for not only simple querying, but running complex streaming expressions, facets with large data etc. Sometimes, randomly, CPU usage gets so high that it starts to respond very slowly to even simple queries, or don’t respond at all. I’m trying to determine if it’s a result of simple overloading of the system by many “normal” queries, or someone sends Solr an unreasonably compute-heavy request.

A few days ago when this occured, I stopped every service that can send Solr a query. After that, for about an hour, nodes were reading from the disk at 1GB/s which is the maximum of our disks. Then everything went back to the normal as I started the other services.

One (bad) idea I had is to build a proxy service which proxies every request to our SolrCloud and monitors current running requests, but scaling this to the size of SolrCloud may be reinventing the wheel.

For now all I can detect is that Solr is struggling, but I have no idea what causes that and when.

-Chees and happy new year

Re: Monitoring Solr for currently running queries

[Markus Jelsma <ma...@openindex.io>]

Hello Ufuk,

You can log slow queries [1].

If you would want to see currently running queries you would have to extend
SearchHandler and build the custom logic yourself. Watch out for SolrCloud
because the main query as well as the per-shard queries can pass through
that same SearchHandler. You can distinguish between then reading the
shard=true parameter.

Regards,
Markus

[1] https://lucene.apache.org/solr/guide/6_6/configuring-logging.html

Op di 29 dec. 2020 om 16:49 schreef ufuk yılmaz <uyilmaz@vivaldi.net.invalid
>:

> Hello All,
>
> Is there a way to see currently executing queries in a SolrCloud? Or a
> general strategy to detect a query using absurd amount or resources?
>
> We are using Solr for not only simple querying, but running complex
> streaming expressions, facets with large data etc. Sometimes, randomly, CPU
> usage gets so high that it starts to respond very slowly to even simple
> queries, or don’t respond at all. I’m trying to determine if it’s a result
> of simple overloading of the system by many “normal” queries, or someone
> sends Solr an unreasonably compute-heavy request.
>
> A few days ago when this occured, I stopped every service that can send
> Solr a query. After that, for about an hour, nodes were reading from the
> disk at 1GB/s which is the maximum of our disks. Then everything went back
> to the normal as I started the other services.
>
> One (bad) idea I had is to build a proxy service which proxies every
> request to our SolrCloud and monitors current running requests, but scaling
> this to the size of SolrCloud may be reinventing the wheel.
>
> For now all I can detect is that Solr is struggling, but I have no idea
> what causes that and when.
>
> -Chees and happy new year
>