You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Rajani Maski <ra...@gmail.com> on 2023/05/04 14:44:47 UTC
Re: Debug time spent in aggregating the search results
Hi Hoss,
Do I need to enable something to retrieve this metric? I tried to query it
http://localhost:8983/solr/admin/metrics?wt=json&prefix=QUERY./select.distrib.requestTimes
but there is no "distrib" in the response, there are only select
QUERY./select.requestTimes
and others
[image: image.png]
On Wed, Apr 19, 2023 at 5:26 PM Chris Hostetter <ho...@fucit.org>
wrote:
> : Hi Solr Users,
> :
> : Is there a metric endpoint or a debug/explain type query param that
> : returns average time spent in aggregating the search results from shards?
>
> Sort of?
>
> Metrics like "QUERY./select.distrib.requestTimes" tell you the stats on
> handling a "distributed" request -- which is when a core is responsible to
> sending out "per-shard" requests and merging the responses.
>
> But it doesn't *only* include the "time spent in aggregating the search
> results from shards" ... it also includes the time spent determining which
> requests to send to which shards, and waiting for the responses to those
> (frequently concurrent) requests"
>
>
> -Hoss
> http://www.lucidworks.com/
>
Re: Debug time spent in aggregating the search results
Posted by Chris Hostetter <ho...@fucit.org>.
Ok, my mistake -- aparently this was all heavily changed in 9.0 and I
didn't notice until you asked about it...
https://issues.apache.org/jira/browse/SOLR-14401
https://solr.apache.org/guide/solr/9_1/deployment-guide/performance-statistics-reference.html
So going back to my previous comment...
> Metrics like "QUERY./select.distrib.requestTimes" tell you the stats on
> handling a "distributed" request -- which is when a core is responsible to
> sending out "per-shard" requests and merging the responses.
> ...
...IIUC in SolrCloud multishard deployments, 9.x's
"QUERY./select.requestTimes" is the equivilent of 8.x's
"QUERY./select.distrib.requestTimes", while
"QUERY./select[shard].requestTimes" is the equivilent of 8.x's
"QUERY./select.local.requestTimes"
But this has really just attempted to simplify the metric naming -- so
that monitoring systems can look at the same info, regardless of wether
there are multishard collection/requests or not -- my caveats about how
exactly this applies to your original question are still true...
> ...
> But it doesn't *only* include the "time spent in aggregating the search
> results from shards" ... it also includes the time spent determining
> which requests to send to which shards, and waiting for the responses to
> those (frequently concurrent) requests"
-Hoss
http://www.lucidworks.com/
Re: Debug time spent in aggregating the search results
Posted by Rajani Maski <ra...@gmail.com>.
What could be the reason for the missing metric? Where else can I look for
that datapoint? I am trying to figure out the time spent in aggregating the
query results.
On Thu, May 4, 2023 at 9:37 PM Rajani Maski <ra...@gmail.com> wrote:
>
> Solr version* 9.1.1*
>
>
> Query -
>
> solr/admin/metrics?nodes=all&wt=json&prefix=QUERY./select.distrib.requestTimes
>
> {
> "responseHeader":{
> "status":0,
> "QTime":7},
> "ip:8983_solr":{
> "responseHeader":{
> "status":0,
> "QTime":2},
> "metrics":{}},
> "ip:8983_solr":{
> "responseHeader":{
> "status":0,
> "QTime":2},
> "metrics":{}},
> "ip:8983_solr":{
> "responseHeader":{
> "status":0,
> "QTime":2},
> "metrics":{}}}
>
>
> Query without "distrib" solr/admin/metrics?nodes=all&wt=json&prefix=QUERY./select.requestTimes
>
>
> {
> "responseHeader":{
> "status":0,
> "QTime":10},
> "10.146.38.84:8983_solr":{
> "responseHeader":{
> "status":0,
> "QTime":5},
> "metrics":{
> "solr.core.test.shard1.replica_n1":{
> "QUERY./select.requestTimes":{
> "count":403,
> "meanRate":4.906764808325113E-4,
> "1minRate":2.964393875E-314,
> "5minRate":1.4821969375E-313,
> "15minRate":4.44659081257E-313,
> "min_ms":0.0,
> "max_ms":0.0,
> "mean_ms":0.0,
> "median_ms":0.0,
> "stddev_ms":0.0,
> "p75_ms":0.0,
> "p95_ms":0.0,
> "p99_ms":0.0,
> "p999_ms":0.0}},
>
>
> On Thu, May 4, 2023 at 2:51 PM Chris Hostetter <ho...@fucit.org>
> wrote:
>
>>
>> : Do I need to enable something to retrieve this metric? I tried to
>> query it
>> :
>> http://localhost:8983/solr/admin/metrics?wt=json&prefix=QUERY./select.distrib.requestTimes
>> :
>> : but there is no "distrib" in the response, there are only select
>>
>> Hrm.... what version of solr are you running?
>>
>>
>> Here's 8.11 ...
>>
>> $ ./bin/solr -e cloud -noprompt
>> ...
>> $ curl -sS '
>> http://localhost:8983/solr/admin/metrics?nodes=all&wt=json&prefix=QUERY./select.distrib.requestTimes
>> '
>> {
>> "responseHeader":{
>> "status":0,
>> "QTime":35},
>> "127.0.1.1:7574_solr":{
>> "responseHeader":{
>> "status":0,
>> "QTime":23},
>> "metrics":{
>> "solr.core.gettingstarted.shard2.replica_n6":{
>> "QUERY./select.distrib.requestTimes":{
>> "count":0,
>> "meanRate":0.0,
>> "1minRate":0.0,
>> "5minRate":0.0,
>> "15minRate":0.0,
>> "min_ms":0.0,
>> "max_ms":0.0,
>> "mean_ms":0.0,
>> "median_ms":0.0,
>> "stddev_ms":0.0,
>> "p75_ms":0.0,
>> "p95_ms":0.0,
>> "p99_ms":0.0,
>> "p999_ms":0.0}},
>> "solr.core.gettingstarted.shard1.replica_n2":{
>> "QUERY./select.distrib.requestTimes":{
>> "count":0,
>> "meanRate":0.0,
>> "1minRate":0.0,
>> "5minRate":0.0,
>> "15minRate":0.0,
>> "min_ms":0.0,
>> "max_ms":0.0,
>> "mean_ms":0.0,
>> "median_ms":0.0,
>> "stddev_ms":0.0,
>> "p75_ms":0.0,
>> "p95_ms":0.0,
>> "p99_ms":0.0,
>> "p999_ms":0.0}}}},
>> "127.0.1.1:8983_solr":{
>> "responseHeader":{
>> "status":0,
>> "QTime":2},
>> "metrics":{
>> "solr.core.gettingstarted.shard2.replica_n4":{
>> "QUERY./select.distrib.requestTimes":{
>> "count":0,
>> "meanRate":0.0,
>> "1minRate":0.0,
>> "5minRate":0.0,
>> "15minRate":0.0,
>> "min_ms":0.0,
>> "max_ms":0.0,
>> "mean_ms":0.0,
>> "median_ms":0.0,
>> "stddev_ms":0.0,
>> "p75_ms":0.0,
>> "p95_ms":0.0,
>> "p99_ms":0.0,
>> "p999_ms":0.0}},
>> "solr.core.gettingstarted.shard1.replica_n1":{
>> "QUERY./select.distrib.requestTimes":{
>> "count":0,
>> "meanRate":0.0,
>> "1minRate":0.0,
>> "5minRate":0.0,
>> "15minRate":0.0,
>> "min_ms":0.0,
>> "max_ms":0.0,
>> "mean_ms":0.0,
>> "median_ms":0.0,
>> "stddev_ms":0.0,
>> "p75_ms":0.0,
>> "p95_ms":0.0,
>> "p99_ms":0.0,
>> "p999_ms":0.0}}}}}
>> $ curl -sS '
>> http://localhost:8983/solr/gettingstarted/select?omitHeader=true&q=*:*'
>> {
>>
>>
>> "response":{"numFound":0,"start":0,"maxScore":0.0,"numFoundExact":true,"docs":[]
>> }}
>> $ curl -sS '
>> http://localhost:8983/solr/admin/metrics?nodes=all&wt=json&prefix=QUERY./select.distrib.requestTimes
>> '
>> {
>> "responseHeader":{
>> "status":0,
>> "QTime":25},
>> "127.0.1.1:7574_solr":{
>> "responseHeader":{
>> "status":0,
>> "QTime":8},
>> "metrics":{
>> "solr.core.gettingstarted.shard2.replica_n6":{
>> "QUERY./select.distrib.requestTimes":{
>> "count":0,
>> "meanRate":0.0,
>> "1minRate":0.0,
>> "5minRate":0.0,
>> "15minRate":0.0,
>> "min_ms":0.0,
>> "max_ms":0.0,
>> "mean_ms":0.0,
>> "median_ms":0.0,
>> "stddev_ms":0.0,
>> "p75_ms":0.0,
>> "p95_ms":0.0,
>> "p99_ms":0.0,
>> "p999_ms":0.0}},
>> "solr.core.gettingstarted.shard1.replica_n2":{
>> "QUERY./select.distrib.requestTimes":{
>> "count":0,
>> "meanRate":0.0,
>> "1minRate":0.0,
>> "5minRate":0.0,
>> "15minRate":0.0,
>> "min_ms":0.0,
>> "max_ms":0.0,
>> "mean_ms":0.0,
>> "median_ms":0.0,
>> "stddev_ms":0.0,
>> "p75_ms":0.0,
>> "p95_ms":0.0,
>> "p99_ms":0.0,
>> "p999_ms":0.0}}}},
>> "127.0.1.1:8983_solr":{
>> "responseHeader":{
>> "status":0,
>> "QTime":6},
>> "metrics":{
>> "solr.core.gettingstarted.shard2.replica_n4":{
>> "QUERY./select.distrib.requestTimes":{
>> "count":0,
>> "meanRate":0.0,
>> "1minRate":0.0,
>> "5minRate":0.0,
>> "15minRate":0.0,
>> "min_ms":0.0,
>> "max_ms":0.0,
>> "mean_ms":0.0,
>> "median_ms":0.0,
>> "stddev_ms":0.0,
>> "p75_ms":0.0,
>> "p95_ms":0.0,
>> "p99_ms":0.0,
>> "p999_ms":0.0}},
>> "solr.core.gettingstarted.shard1.replica_n1":{
>> "QUERY./select.distrib.requestTimes":{
>> "count":1,
>> "meanRate":0.004266352892634931,
>> "1minRate":0.013536188363841833,
>> "5minRate":0.0031973351962583784,
>> "15minRate":0.001095787094460976,
>> "min_ms":383.362809,
>> "max_ms":383.362809,
>> "mean_ms":383.362809,
>> "median_ms":383.362809,
>> "stddev_ms":0.0,
>> "p75_ms":383.362809,
>> "p95_ms":383.362809,
>> "p99_ms":383.362809,
>> "p999_ms":383.362809}}}}}
>>
>>
>>
>>
>> -Hoss
>> http://www.lucidworks.com/
>>
>
Re: Debug time spent in aggregating the search results
Posted by Rajani Maski <ra...@gmail.com>.
Solr version* 9.1.1*
Query -
solr/admin/metrics?nodes=all&wt=json&prefix=QUERY./select.distrib.requestTimes
{
"responseHeader":{
"status":0,
"QTime":7},
"ip:8983_solr":{
"responseHeader":{
"status":0,
"QTime":2},
"metrics":{}},
"ip:8983_solr":{
"responseHeader":{
"status":0,
"QTime":2},
"metrics":{}},
"ip:8983_solr":{
"responseHeader":{
"status":0,
"QTime":2},
"metrics":{}}}
Query without "distrib"
solr/admin/metrics?nodes=all&wt=json&prefix=QUERY./select.requestTimes
{
"responseHeader":{
"status":0,
"QTime":10},
"10.146.38.84:8983_solr":{
"responseHeader":{
"status":0,
"QTime":5},
"metrics":{
"solr.core.test.shard1.replica_n1":{
"QUERY./select.requestTimes":{
"count":403,
"meanRate":4.906764808325113E-4,
"1minRate":2.964393875E-314,
"5minRate":1.4821969375E-313,
"15minRate":4.44659081257E-313,
"min_ms":0.0,
"max_ms":0.0,
"mean_ms":0.0,
"median_ms":0.0,
"stddev_ms":0.0,
"p75_ms":0.0,
"p95_ms":0.0,
"p99_ms":0.0,
"p999_ms":0.0}},
On Thu, May 4, 2023 at 2:51 PM Chris Hostetter <ho...@fucit.org>
wrote:
>
> : Do I need to enable something to retrieve this metric? I tried to query
> it
> :
> http://localhost:8983/solr/admin/metrics?wt=json&prefix=QUERY./select.distrib.requestTimes
> :
> : but there is no "distrib" in the response, there are only select
>
> Hrm.... what version of solr are you running?
>
>
> Here's 8.11 ...
>
> $ ./bin/solr -e cloud -noprompt
> ...
> $ curl -sS '
> http://localhost:8983/solr/admin/metrics?nodes=all&wt=json&prefix=QUERY./select.distrib.requestTimes
> '
> {
> "responseHeader":{
> "status":0,
> "QTime":35},
> "127.0.1.1:7574_solr":{
> "responseHeader":{
> "status":0,
> "QTime":23},
> "metrics":{
> "solr.core.gettingstarted.shard2.replica_n6":{
> "QUERY./select.distrib.requestTimes":{
> "count":0,
> "meanRate":0.0,
> "1minRate":0.0,
> "5minRate":0.0,
> "15minRate":0.0,
> "min_ms":0.0,
> "max_ms":0.0,
> "mean_ms":0.0,
> "median_ms":0.0,
> "stddev_ms":0.0,
> "p75_ms":0.0,
> "p95_ms":0.0,
> "p99_ms":0.0,
> "p999_ms":0.0}},
> "solr.core.gettingstarted.shard1.replica_n2":{
> "QUERY./select.distrib.requestTimes":{
> "count":0,
> "meanRate":0.0,
> "1minRate":0.0,
> "5minRate":0.0,
> "15minRate":0.0,
> "min_ms":0.0,
> "max_ms":0.0,
> "mean_ms":0.0,
> "median_ms":0.0,
> "stddev_ms":0.0,
> "p75_ms":0.0,
> "p95_ms":0.0,
> "p99_ms":0.0,
> "p999_ms":0.0}}}},
> "127.0.1.1:8983_solr":{
> "responseHeader":{
> "status":0,
> "QTime":2},
> "metrics":{
> "solr.core.gettingstarted.shard2.replica_n4":{
> "QUERY./select.distrib.requestTimes":{
> "count":0,
> "meanRate":0.0,
> "1minRate":0.0,
> "5minRate":0.0,
> "15minRate":0.0,
> "min_ms":0.0,
> "max_ms":0.0,
> "mean_ms":0.0,
> "median_ms":0.0,
> "stddev_ms":0.0,
> "p75_ms":0.0,
> "p95_ms":0.0,
> "p99_ms":0.0,
> "p999_ms":0.0}},
> "solr.core.gettingstarted.shard1.replica_n1":{
> "QUERY./select.distrib.requestTimes":{
> "count":0,
> "meanRate":0.0,
> "1minRate":0.0,
> "5minRate":0.0,
> "15minRate":0.0,
> "min_ms":0.0,
> "max_ms":0.0,
> "mean_ms":0.0,
> "median_ms":0.0,
> "stddev_ms":0.0,
> "p75_ms":0.0,
> "p95_ms":0.0,
> "p99_ms":0.0,
> "p999_ms":0.0}}}}}
> $ curl -sS '
> http://localhost:8983/solr/gettingstarted/select?omitHeader=true&q=*:*'
> {
>
>
> "response":{"numFound":0,"start":0,"maxScore":0.0,"numFoundExact":true,"docs":[]
> }}
> $ curl -sS '
> http://localhost:8983/solr/admin/metrics?nodes=all&wt=json&prefix=QUERY./select.distrib.requestTimes
> '
> {
> "responseHeader":{
> "status":0,
> "QTime":25},
> "127.0.1.1:7574_solr":{
> "responseHeader":{
> "status":0,
> "QTime":8},
> "metrics":{
> "solr.core.gettingstarted.shard2.replica_n6":{
> "QUERY./select.distrib.requestTimes":{
> "count":0,
> "meanRate":0.0,
> "1minRate":0.0,
> "5minRate":0.0,
> "15minRate":0.0,
> "min_ms":0.0,
> "max_ms":0.0,
> "mean_ms":0.0,
> "median_ms":0.0,
> "stddev_ms":0.0,
> "p75_ms":0.0,
> "p95_ms":0.0,
> "p99_ms":0.0,
> "p999_ms":0.0}},
> "solr.core.gettingstarted.shard1.replica_n2":{
> "QUERY./select.distrib.requestTimes":{
> "count":0,
> "meanRate":0.0,
> "1minRate":0.0,
> "5minRate":0.0,
> "15minRate":0.0,
> "min_ms":0.0,
> "max_ms":0.0,
> "mean_ms":0.0,
> "median_ms":0.0,
> "stddev_ms":0.0,
> "p75_ms":0.0,
> "p95_ms":0.0,
> "p99_ms":0.0,
> "p999_ms":0.0}}}},
> "127.0.1.1:8983_solr":{
> "responseHeader":{
> "status":0,
> "QTime":6},
> "metrics":{
> "solr.core.gettingstarted.shard2.replica_n4":{
> "QUERY./select.distrib.requestTimes":{
> "count":0,
> "meanRate":0.0,
> "1minRate":0.0,
> "5minRate":0.0,
> "15minRate":0.0,
> "min_ms":0.0,
> "max_ms":0.0,
> "mean_ms":0.0,
> "median_ms":0.0,
> "stddev_ms":0.0,
> "p75_ms":0.0,
> "p95_ms":0.0,
> "p99_ms":0.0,
> "p999_ms":0.0}},
> "solr.core.gettingstarted.shard1.replica_n1":{
> "QUERY./select.distrib.requestTimes":{
> "count":1,
> "meanRate":0.004266352892634931,
> "1minRate":0.013536188363841833,
> "5minRate":0.0031973351962583784,
> "15minRate":0.001095787094460976,
> "min_ms":383.362809,
> "max_ms":383.362809,
> "mean_ms":383.362809,
> "median_ms":383.362809,
> "stddev_ms":0.0,
> "p75_ms":383.362809,
> "p95_ms":383.362809,
> "p99_ms":383.362809,
> "p999_ms":383.362809}}}}}
>
>
>
>
> -Hoss
> http://www.lucidworks.com/
>
Re: Debug time spent in aggregating the search results
Posted by Chris Hostetter <ho...@fucit.org>.
: Do I need to enable something to retrieve this metric? I tried to query it
: http://localhost:8983/solr/admin/metrics?wt=json&prefix=QUERY./select.distrib.requestTimes
:
: but there is no "distrib" in the response, there are only select
Hrm.... what version of solr are you running?
Here's 8.11 ...
$ ./bin/solr -e cloud -noprompt
...
$ curl -sS 'http://localhost:8983/solr/admin/metrics?nodes=all&wt=json&prefix=QUERY./select.distrib.requestTimes'
{
"responseHeader":{
"status":0,
"QTime":35},
"127.0.1.1:7574_solr":{
"responseHeader":{
"status":0,
"QTime":23},
"metrics":{
"solr.core.gettingstarted.shard2.replica_n6":{
"QUERY./select.distrib.requestTimes":{
"count":0,
"meanRate":0.0,
"1minRate":0.0,
"5minRate":0.0,
"15minRate":0.0,
"min_ms":0.0,
"max_ms":0.0,
"mean_ms":0.0,
"median_ms":0.0,
"stddev_ms":0.0,
"p75_ms":0.0,
"p95_ms":0.0,
"p99_ms":0.0,
"p999_ms":0.0}},
"solr.core.gettingstarted.shard1.replica_n2":{
"QUERY./select.distrib.requestTimes":{
"count":0,
"meanRate":0.0,
"1minRate":0.0,
"5minRate":0.0,
"15minRate":0.0,
"min_ms":0.0,
"max_ms":0.0,
"mean_ms":0.0,
"median_ms":0.0,
"stddev_ms":0.0,
"p75_ms":0.0,
"p95_ms":0.0,
"p99_ms":0.0,
"p999_ms":0.0}}}},
"127.0.1.1:8983_solr":{
"responseHeader":{
"status":0,
"QTime":2},
"metrics":{
"solr.core.gettingstarted.shard2.replica_n4":{
"QUERY./select.distrib.requestTimes":{
"count":0,
"meanRate":0.0,
"1minRate":0.0,
"5minRate":0.0,
"15minRate":0.0,
"min_ms":0.0,
"max_ms":0.0,
"mean_ms":0.0,
"median_ms":0.0,
"stddev_ms":0.0,
"p75_ms":0.0,
"p95_ms":0.0,
"p99_ms":0.0,
"p999_ms":0.0}},
"solr.core.gettingstarted.shard1.replica_n1":{
"QUERY./select.distrib.requestTimes":{
"count":0,
"meanRate":0.0,
"1minRate":0.0,
"5minRate":0.0,
"15minRate":0.0,
"min_ms":0.0,
"max_ms":0.0,
"mean_ms":0.0,
"median_ms":0.0,
"stddev_ms":0.0,
"p75_ms":0.0,
"p95_ms":0.0,
"p99_ms":0.0,
"p999_ms":0.0}}}}}
$ curl -sS 'http://localhost:8983/solr/gettingstarted/select?omitHeader=true&q=*:*'
{
"response":{"numFound":0,"start":0,"maxScore":0.0,"numFoundExact":true,"docs":[]
}}
$ curl -sS 'http://localhost:8983/solr/admin/metrics?nodes=all&wt=json&prefix=QUERY./select.distrib.requestTimes'
{
"responseHeader":{
"status":0,
"QTime":25},
"127.0.1.1:7574_solr":{
"responseHeader":{
"status":0,
"QTime":8},
"metrics":{
"solr.core.gettingstarted.shard2.replica_n6":{
"QUERY./select.distrib.requestTimes":{
"count":0,
"meanRate":0.0,
"1minRate":0.0,
"5minRate":0.0,
"15minRate":0.0,
"min_ms":0.0,
"max_ms":0.0,
"mean_ms":0.0,
"median_ms":0.0,
"stddev_ms":0.0,
"p75_ms":0.0,
"p95_ms":0.0,
"p99_ms":0.0,
"p999_ms":0.0}},
"solr.core.gettingstarted.shard1.replica_n2":{
"QUERY./select.distrib.requestTimes":{
"count":0,
"meanRate":0.0,
"1minRate":0.0,
"5minRate":0.0,
"15minRate":0.0,
"min_ms":0.0,
"max_ms":0.0,
"mean_ms":0.0,
"median_ms":0.0,
"stddev_ms":0.0,
"p75_ms":0.0,
"p95_ms":0.0,
"p99_ms":0.0,
"p999_ms":0.0}}}},
"127.0.1.1:8983_solr":{
"responseHeader":{
"status":0,
"QTime":6},
"metrics":{
"solr.core.gettingstarted.shard2.replica_n4":{
"QUERY./select.distrib.requestTimes":{
"count":0,
"meanRate":0.0,
"1minRate":0.0,
"5minRate":0.0,
"15minRate":0.0,
"min_ms":0.0,
"max_ms":0.0,
"mean_ms":0.0,
"median_ms":0.0,
"stddev_ms":0.0,
"p75_ms":0.0,
"p95_ms":0.0,
"p99_ms":0.0,
"p999_ms":0.0}},
"solr.core.gettingstarted.shard1.replica_n1":{
"QUERY./select.distrib.requestTimes":{
"count":1,
"meanRate":0.004266352892634931,
"1minRate":0.013536188363841833,
"5minRate":0.0031973351962583784,
"15minRate":0.001095787094460976,
"min_ms":383.362809,
"max_ms":383.362809,
"mean_ms":383.362809,
"median_ms":383.362809,
"stddev_ms":0.0,
"p75_ms":383.362809,
"p95_ms":383.362809,
"p99_ms":383.362809,
"p999_ms":383.362809}}}}}
-Hoss
http://www.lucidworks.com/