You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Rajani Maski <ra...@gmail.com> on 2023/04/19 16:09:52 UTC
Debug time spent in aggregating the search results
Hi Solr Users,
Is there a metric endpoint or a debug/explain type query param that
returns average time spent in aggregating the search results from shards?
Thanks,
Rajani
Re: Debug time spent in aggregating the search results
Posted by Chris Hostetter <ho...@fucit.org>.
Ok, my mistake -- aparently this was all heavily changed in 9.0 and I
didn't notice until you asked about it...
https://issues.apache.org/jira/browse/SOLR-14401
https://solr.apache.org/guide/solr/9_1/deployment-guide/performance-statistics-reference.html
So going back to my previous comment...
> Metrics like "QUERY./select.distrib.requestTimes" tell you the stats on
> handling a "distributed" request -- which is when a core is responsible to
> sending out "per-shard" requests and merging the responses.
> ...
...IIUC in SolrCloud multishard deployments, 9.x's
"QUERY./select.requestTimes" is the equivilent of 8.x's
"QUERY./select.distrib.requestTimes", while
"QUERY./select[shard].requestTimes" is the equivilent of 8.x's
"QUERY./select.local.requestTimes"
But this has really just attempted to simplify the metric naming -- so
that monitoring systems can look at the same info, regardless of wether
there are multishard collection/requests or not -- my caveats about how
exactly this applies to your original question are still true...
> ...
> But it doesn't *only* include the "time spent in aggregating the search
> results from shards" ... it also includes the time spent determining
> which requests to send to which shards, and waiting for the responses to
> those (frequently concurrent) requests"
-Hoss
http://www.lucidworks.com/
Re: Debug time spent in aggregating the search results
Posted by Rajani Maski <ra...@gmail.com>.
What could be the reason for the missing metric? Where else can I look for
that datapoint? I am trying to figure out the time spent in aggregating the
query results.
On Thu, May 4, 2023 at 9:37 PM Rajani Maski <ra...@gmail.com> wrote:
>
> Solr version* 9.1.1*
>
>
> Query -
>
> solr/admin/metrics?nodes=all&wt=json&prefix=QUERY./select.distrib.requestTimes
>
> {
> "responseHeader":{
> "status":0,
> "QTime":7},
> "ip:8983_solr":{
> "responseHeader":{
> "status":0,
> "QTime":2},
> "metrics":{}},
> "ip:8983_solr":{
> "responseHeader":{
> "status":0,
> "QTime":2},
> "metrics":{}},
> "ip:8983_solr":{
> "responseHeader":{
> "status":0,
> "QTime":2},
> "metrics":{}}}
>
>
> Query without "distrib" solr/admin/metrics?nodes=all&wt=json&prefix=QUERY./select.requestTimes
>
>
> {
> "responseHeader":{
> "status":0,
> "QTime":10},
> "10.146.38.84:8983_solr":{
> "responseHeader":{
> "status":0,
> "QTime":5},
> "metrics":{
> "solr.core.test.shard1.replica_n1":{
> "QUERY./select.requestTimes":{
> "count":403,
> "meanRate":4.906764808325113E-4,
> "1minRate":2.964393875E-314,
> "5minRate":1.4821969375E-313,
> "15minRate":4.44659081257E-313,
> "min_ms":0.0,
> "max_ms":0.0,
> "mean_ms":0.0,
> "median_ms":0.0,
> "stddev_ms":0.0,
> "p75_ms":0.0,
> "p95_ms":0.0,
> "p99_ms":0.0,
> "p999_ms":0.0}},
>
>
> On Thu, May 4, 2023 at 2:51 PM Chris Hostetter <ho...@fucit.org>
> wrote:
>
>>
>> : Do I need to enable something to retrieve this metric? I tried to
>> query it
>> :
>> http://localhost:8983/solr/admin/metrics?wt=json&prefix=QUERY./select.distrib.requestTimes
>> :
>> : but there is no "distrib" in the response, there are only select
>>
>> Hrm.... what version of solr are you running?
>>
>>
>> Here's 8.11 ...
>>
>> $ ./bin/solr -e cloud -noprompt
>> ...
>> $ curl -sS '
>> http://localhost:8983/solr/admin/metrics?nodes=all&wt=json&prefix=QUERY./select.distrib.requestTimes
>> '
>> {
>> "responseHeader":{
>> "status":0,
>> "QTime":35},
>> "127.0.1.1:7574_solr":{
>> "responseHeader":{
>> "status":0,
>> "QTime":23},
>> "metrics":{
>> "solr.core.gettingstarted.shard2.replica_n6":{
>> "QUERY./select.distrib.requestTimes":{
>> "count":0,
>> "meanRate":0.0,
>> "1minRate":0.0,
>> "5minRate":0.0,
>> "15minRate":0.0,
>> "min_ms":0.0,
>> "max_ms":0.0,
>> "mean_ms":0.0,
>> "median_ms":0.0,
>> "stddev_ms":0.0,
>> "p75_ms":0.0,
>> "p95_ms":0.0,
>> "p99_ms":0.0,
>> "p999_ms":0.0}},
>> "solr.core.gettingstarted.shard1.replica_n2":{
>> "QUERY./select.distrib.requestTimes":{
>> "count":0,
>> "meanRate":0.0,
>> "1minRate":0.0,
>> "5minRate":0.0,
>> "15minRate":0.0,
>> "min_ms":0.0,
>> "max_ms":0.0,
>> "mean_ms":0.0,
>> "median_ms":0.0,
>> "stddev_ms":0.0,
>> "p75_ms":0.0,
>> "p95_ms":0.0,
>> "p99_ms":0.0,
>> "p999_ms":0.0}}}},
>> "127.0.1.1:8983_solr":{
>> "responseHeader":{
>> "status":0,
>> "QTime":2},
>> "metrics":{
>> "solr.core.gettingstarted.shard2.replica_n4":{
>> "QUERY./select.distrib.requestTimes":{
>> "count":0,
>> "meanRate":0.0,
>> "1minRate":0.0,
>> "5minRate":0.0,
>> "15minRate":0.0,
>> "min_ms":0.0,
>> "max_ms":0.0,
>> "mean_ms":0.0,
>> "median_ms":0.0,
>> "stddev_ms":0.0,
>> "p75_ms":0.0,
>> "p95_ms":0.0,
>> "p99_ms":0.0,
>> "p999_ms":0.0}},
>> "solr.core.gettingstarted.shard1.replica_n1":{
>> "QUERY./select.distrib.requestTimes":{
>> "count":0,
>> "meanRate":0.0,
>> "1minRate":0.0,
>> "5minRate":0.0,
>> "15minRate":0.0,
>> "min_ms":0.0,
>> "max_ms":0.0,
>> "mean_ms":0.0,
>> "median_ms":0.0,
>> "stddev_ms":0.0,
>> "p75_ms":0.0,
>> "p95_ms":0.0,
>> "p99_ms":0.0,
>> "p999_ms":0.0}}}}}
>> $ curl -sS '
>> http://localhost:8983/solr/gettingstarted/select?omitHeader=true&q=*:*'
>> {
>>
>>
>> "response":{"numFound":0,"start":0,"maxScore":0.0,"numFoundExact":true,"docs":[]
>> }}
>> $ curl -sS '
>> http://localhost:8983/solr/admin/metrics?nodes=all&wt=json&prefix=QUERY./select.distrib.requestTimes
>> '
>> {
>> "responseHeader":{
>> "status":0,
>> "QTime":25},
>> "127.0.1.1:7574_solr":{
>> "responseHeader":{
>> "status":0,
>> "QTime":8},
>> "metrics":{
>> "solr.core.gettingstarted.shard2.replica_n6":{
>> "QUERY./select.distrib.requestTimes":{
>> "count":0,
>> "meanRate":0.0,
>> "1minRate":0.0,
>> "5minRate":0.0,
>> "15minRate":0.0,
>> "min_ms":0.0,
>> "max_ms":0.0,
>> "mean_ms":0.0,
>> "median_ms":0.0,
>> "stddev_ms":0.0,
>> "p75_ms":0.0,
>> "p95_ms":0.0,
>> "p99_ms":0.0,
>> "p999_ms":0.0}},
>> "solr.core.gettingstarted.shard1.replica_n2":{
>> "QUERY./select.distrib.requestTimes":{
>> "count":0,
>> "meanRate":0.0,
>> "1minRate":0.0,
>> "5minRate":0.0,
>> "15minRate":0.0,
>> "min_ms":0.0,
>> "max_ms":0.0,
>> "mean_ms":0.0,
>> "median_ms":0.0,
>> "stddev_ms":0.0,
>> "p75_ms":0.0,
>> "p95_ms":0.0,
>> "p99_ms":0.0,
>> "p999_ms":0.0}}}},
>> "127.0.1.1:8983_solr":{
>> "responseHeader":{
>> "status":0,
>> "QTime":6},
>> "metrics":{
>> "solr.core.gettingstarted.shard2.replica_n4":{
>> "QUERY./select.distrib.requestTimes":{
>> "count":0,
>> "meanRate":0.0,
>> "1minRate":0.0,
>> "5minRate":0.0,
>> "15minRate":0.0,
>> "min_ms":0.0,
>> "max_ms":0.0,
>> "mean_ms":0.0,
>> "median_ms":0.0,
>> "stddev_ms":0.0,
>> "p75_ms":0.0,
>> "p95_ms":0.0,
>> "p99_ms":0.0,
>> "p999_ms":0.0}},
>> "solr.core.gettingstarted.shard1.replica_n1":{
>> "QUERY./select.distrib.requestTimes":{
>> "count":1,
>> "meanRate":0.004266352892634931,
>> "1minRate":0.013536188363841833,
>> "5minRate":0.0031973351962583784,
>> "15minRate":0.001095787094460976,
>> "min_ms":383.362809,
>> "max_ms":383.362809,
>> "mean_ms":383.362809,
>> "median_ms":383.362809,
>> "stddev_ms":0.0,
>> "p75_ms":383.362809,
>> "p95_ms":383.362809,
>> "p99_ms":383.362809,
>> "p999_ms":383.362809}}}}}
>>
>>
>>
>>
>> -Hoss
>> http://www.lucidworks.com/
>>
>
Re: Debug time spent in aggregating the search results
Posted by Rajani Maski <ra...@gmail.com>.
Solr version* 9.1.1*
Query -
solr/admin/metrics?nodes=all&wt=json&prefix=QUERY./select.distrib.requestTimes
{
"responseHeader":{
"status":0,
"QTime":7},
"ip:8983_solr":{
"responseHeader":{
"status":0,
"QTime":2},
"metrics":{}},
"ip:8983_solr":{
"responseHeader":{
"status":0,
"QTime":2},
"metrics":{}},
"ip:8983_solr":{
"responseHeader":{
"status":0,
"QTime":2},
"metrics":{}}}
Query without "distrib"
solr/admin/metrics?nodes=all&wt=json&prefix=QUERY./select.requestTimes
{
"responseHeader":{
"status":0,
"QTime":10},
"10.146.38.84:8983_solr":{
"responseHeader":{
"status":0,
"QTime":5},
"metrics":{
"solr.core.test.shard1.replica_n1":{
"QUERY./select.requestTimes":{
"count":403,
"meanRate":4.906764808325113E-4,
"1minRate":2.964393875E-314,
"5minRate":1.4821969375E-313,
"15minRate":4.44659081257E-313,
"min_ms":0.0,
"max_ms":0.0,
"mean_ms":0.0,
"median_ms":0.0,
"stddev_ms":0.0,
"p75_ms":0.0,
"p95_ms":0.0,
"p99_ms":0.0,
"p999_ms":0.0}},
On Thu, May 4, 2023 at 2:51 PM Chris Hostetter <ho...@fucit.org>
wrote:
>
> : Do I need to enable something to retrieve this metric? I tried to query
> it
> :
> http://localhost:8983/solr/admin/metrics?wt=json&prefix=QUERY./select.distrib.requestTimes
> :
> : but there is no "distrib" in the response, there are only select
>
> Hrm.... what version of solr are you running?
>
>
> Here's 8.11 ...
>
> $ ./bin/solr -e cloud -noprompt
> ...
> $ curl -sS '
> http://localhost:8983/solr/admin/metrics?nodes=all&wt=json&prefix=QUERY./select.distrib.requestTimes
> '
> {
> "responseHeader":{
> "status":0,
> "QTime":35},
> "127.0.1.1:7574_solr":{
> "responseHeader":{
> "status":0,
> "QTime":23},
> "metrics":{
> "solr.core.gettingstarted.shard2.replica_n6":{
> "QUERY./select.distrib.requestTimes":{
> "count":0,
> "meanRate":0.0,
> "1minRate":0.0,
> "5minRate":0.0,
> "15minRate":0.0,
> "min_ms":0.0,
> "max_ms":0.0,
> "mean_ms":0.0,
> "median_ms":0.0,
> "stddev_ms":0.0,
> "p75_ms":0.0,
> "p95_ms":0.0,
> "p99_ms":0.0,
> "p999_ms":0.0}},
> "solr.core.gettingstarted.shard1.replica_n2":{
> "QUERY./select.distrib.requestTimes":{
> "count":0,
> "meanRate":0.0,
> "1minRate":0.0,
> "5minRate":0.0,
> "15minRate":0.0,
> "min_ms":0.0,
> "max_ms":0.0,
> "mean_ms":0.0,
> "median_ms":0.0,
> "stddev_ms":0.0,
> "p75_ms":0.0,
> "p95_ms":0.0,
> "p99_ms":0.0,
> "p999_ms":0.0}}}},
> "127.0.1.1:8983_solr":{
> "responseHeader":{
> "status":0,
> "QTime":2},
> "metrics":{
> "solr.core.gettingstarted.shard2.replica_n4":{
> "QUERY./select.distrib.requestTimes":{
> "count":0,
> "meanRate":0.0,
> "1minRate":0.0,
> "5minRate":0.0,
> "15minRate":0.0,
> "min_ms":0.0,
> "max_ms":0.0,
> "mean_ms":0.0,
> "median_ms":0.0,
> "stddev_ms":0.0,
> "p75_ms":0.0,
> "p95_ms":0.0,
> "p99_ms":0.0,
> "p999_ms":0.0}},
> "solr.core.gettingstarted.shard1.replica_n1":{
> "QUERY./select.distrib.requestTimes":{
> "count":0,
> "meanRate":0.0,
> "1minRate":0.0,
> "5minRate":0.0,
> "15minRate":0.0,
> "min_ms":0.0,
> "max_ms":0.0,
> "mean_ms":0.0,
> "median_ms":0.0,
> "stddev_ms":0.0,
> "p75_ms":0.0,
> "p95_ms":0.0,
> "p99_ms":0.0,
> "p999_ms":0.0}}}}}
> $ curl -sS '
> http://localhost:8983/solr/gettingstarted/select?omitHeader=true&q=*:*'
> {
>
>
> "response":{"numFound":0,"start":0,"maxScore":0.0,"numFoundExact":true,"docs":[]
> }}
> $ curl -sS '
> http://localhost:8983/solr/admin/metrics?nodes=all&wt=json&prefix=QUERY./select.distrib.requestTimes
> '
> {
> "responseHeader":{
> "status":0,
> "QTime":25},
> "127.0.1.1:7574_solr":{
> "responseHeader":{
> "status":0,
> "QTime":8},
> "metrics":{
> "solr.core.gettingstarted.shard2.replica_n6":{
> "QUERY./select.distrib.requestTimes":{
> "count":0,
> "meanRate":0.0,
> "1minRate":0.0,
> "5minRate":0.0,
> "15minRate":0.0,
> "min_ms":0.0,
> "max_ms":0.0,
> "mean_ms":0.0,
> "median_ms":0.0,
> "stddev_ms":0.0,
> "p75_ms":0.0,
> "p95_ms":0.0,
> "p99_ms":0.0,
> "p999_ms":0.0}},
> "solr.core.gettingstarted.shard1.replica_n2":{
> "QUERY./select.distrib.requestTimes":{
> "count":0,
> "meanRate":0.0,
> "1minRate":0.0,
> "5minRate":0.0,
> "15minRate":0.0,
> "min_ms":0.0,
> "max_ms":0.0,
> "mean_ms":0.0,
> "median_ms":0.0,
> "stddev_ms":0.0,
> "p75_ms":0.0,
> "p95_ms":0.0,
> "p99_ms":0.0,
> "p999_ms":0.0}}}},
> "127.0.1.1:8983_solr":{
> "responseHeader":{
> "status":0,
> "QTime":6},
> "metrics":{
> "solr.core.gettingstarted.shard2.replica_n4":{
> "QUERY./select.distrib.requestTimes":{
> "count":0,
> "meanRate":0.0,
> "1minRate":0.0,
> "5minRate":0.0,
> "15minRate":0.0,
> "min_ms":0.0,
> "max_ms":0.0,
> "mean_ms":0.0,
> "median_ms":0.0,
> "stddev_ms":0.0,
> "p75_ms":0.0,
> "p95_ms":0.0,
> "p99_ms":0.0,
> "p999_ms":0.0}},
> "solr.core.gettingstarted.shard1.replica_n1":{
> "QUERY./select.distrib.requestTimes":{
> "count":1,
> "meanRate":0.004266352892634931,
> "1minRate":0.013536188363841833,
> "5minRate":0.0031973351962583784,
> "15minRate":0.001095787094460976,
> "min_ms":383.362809,
> "max_ms":383.362809,
> "mean_ms":383.362809,
> "median_ms":383.362809,
> "stddev_ms":0.0,
> "p75_ms":383.362809,
> "p95_ms":383.362809,
> "p99_ms":383.362809,
> "p999_ms":383.362809}}}}}
>
>
>
>
> -Hoss
> http://www.lucidworks.com/
>
Re: Debug time spent in aggregating the search results
Posted by Chris Hostetter <ho...@fucit.org>.
: Do I need to enable something to retrieve this metric? I tried to query it
: http://localhost:8983/solr/admin/metrics?wt=json&prefix=QUERY./select.distrib.requestTimes
:
: but there is no "distrib" in the response, there are only select
Hrm.... what version of solr are you running?
Here's 8.11 ...
$ ./bin/solr -e cloud -noprompt
...
$ curl -sS 'http://localhost:8983/solr/admin/metrics?nodes=all&wt=json&prefix=QUERY./select.distrib.requestTimes'
{
"responseHeader":{
"status":0,
"QTime":35},
"127.0.1.1:7574_solr":{
"responseHeader":{
"status":0,
"QTime":23},
"metrics":{
"solr.core.gettingstarted.shard2.replica_n6":{
"QUERY./select.distrib.requestTimes":{
"count":0,
"meanRate":0.0,
"1minRate":0.0,
"5minRate":0.0,
"15minRate":0.0,
"min_ms":0.0,
"max_ms":0.0,
"mean_ms":0.0,
"median_ms":0.0,
"stddev_ms":0.0,
"p75_ms":0.0,
"p95_ms":0.0,
"p99_ms":0.0,
"p999_ms":0.0}},
"solr.core.gettingstarted.shard1.replica_n2":{
"QUERY./select.distrib.requestTimes":{
"count":0,
"meanRate":0.0,
"1minRate":0.0,
"5minRate":0.0,
"15minRate":0.0,
"min_ms":0.0,
"max_ms":0.0,
"mean_ms":0.0,
"median_ms":0.0,
"stddev_ms":0.0,
"p75_ms":0.0,
"p95_ms":0.0,
"p99_ms":0.0,
"p999_ms":0.0}}}},
"127.0.1.1:8983_solr":{
"responseHeader":{
"status":0,
"QTime":2},
"metrics":{
"solr.core.gettingstarted.shard2.replica_n4":{
"QUERY./select.distrib.requestTimes":{
"count":0,
"meanRate":0.0,
"1minRate":0.0,
"5minRate":0.0,
"15minRate":0.0,
"min_ms":0.0,
"max_ms":0.0,
"mean_ms":0.0,
"median_ms":0.0,
"stddev_ms":0.0,
"p75_ms":0.0,
"p95_ms":0.0,
"p99_ms":0.0,
"p999_ms":0.0}},
"solr.core.gettingstarted.shard1.replica_n1":{
"QUERY./select.distrib.requestTimes":{
"count":0,
"meanRate":0.0,
"1minRate":0.0,
"5minRate":0.0,
"15minRate":0.0,
"min_ms":0.0,
"max_ms":0.0,
"mean_ms":0.0,
"median_ms":0.0,
"stddev_ms":0.0,
"p75_ms":0.0,
"p95_ms":0.0,
"p99_ms":0.0,
"p999_ms":0.0}}}}}
$ curl -sS 'http://localhost:8983/solr/gettingstarted/select?omitHeader=true&q=*:*'
{
"response":{"numFound":0,"start":0,"maxScore":0.0,"numFoundExact":true,"docs":[]
}}
$ curl -sS 'http://localhost:8983/solr/admin/metrics?nodes=all&wt=json&prefix=QUERY./select.distrib.requestTimes'
{
"responseHeader":{
"status":0,
"QTime":25},
"127.0.1.1:7574_solr":{
"responseHeader":{
"status":0,
"QTime":8},
"metrics":{
"solr.core.gettingstarted.shard2.replica_n6":{
"QUERY./select.distrib.requestTimes":{
"count":0,
"meanRate":0.0,
"1minRate":0.0,
"5minRate":0.0,
"15minRate":0.0,
"min_ms":0.0,
"max_ms":0.0,
"mean_ms":0.0,
"median_ms":0.0,
"stddev_ms":0.0,
"p75_ms":0.0,
"p95_ms":0.0,
"p99_ms":0.0,
"p999_ms":0.0}},
"solr.core.gettingstarted.shard1.replica_n2":{
"QUERY./select.distrib.requestTimes":{
"count":0,
"meanRate":0.0,
"1minRate":0.0,
"5minRate":0.0,
"15minRate":0.0,
"min_ms":0.0,
"max_ms":0.0,
"mean_ms":0.0,
"median_ms":0.0,
"stddev_ms":0.0,
"p75_ms":0.0,
"p95_ms":0.0,
"p99_ms":0.0,
"p999_ms":0.0}}}},
"127.0.1.1:8983_solr":{
"responseHeader":{
"status":0,
"QTime":6},
"metrics":{
"solr.core.gettingstarted.shard2.replica_n4":{
"QUERY./select.distrib.requestTimes":{
"count":0,
"meanRate":0.0,
"1minRate":0.0,
"5minRate":0.0,
"15minRate":0.0,
"min_ms":0.0,
"max_ms":0.0,
"mean_ms":0.0,
"median_ms":0.0,
"stddev_ms":0.0,
"p75_ms":0.0,
"p95_ms":0.0,
"p99_ms":0.0,
"p999_ms":0.0}},
"solr.core.gettingstarted.shard1.replica_n1":{
"QUERY./select.distrib.requestTimes":{
"count":1,
"meanRate":0.004266352892634931,
"1minRate":0.013536188363841833,
"5minRate":0.0031973351962583784,
"15minRate":0.001095787094460976,
"min_ms":383.362809,
"max_ms":383.362809,
"mean_ms":383.362809,
"median_ms":383.362809,
"stddev_ms":0.0,
"p75_ms":383.362809,
"p95_ms":383.362809,
"p99_ms":383.362809,
"p999_ms":383.362809}}}}}
-Hoss
http://www.lucidworks.com/
Re: Debug time spent in aggregating the search results
Posted by Rajani Maski <ra...@gmail.com>.
Hi Hoss,
Do I need to enable something to retrieve this metric? I tried to query it
http://localhost:8983/solr/admin/metrics?wt=json&prefix=QUERY./select.distrib.requestTimes
but there is no "distrib" in the response, there are only select
QUERY./select.requestTimes
and others
[image: image.png]
On Wed, Apr 19, 2023 at 5:26 PM Chris Hostetter <ho...@fucit.org>
wrote:
> : Hi Solr Users,
> :
> : Is there a metric endpoint or a debug/explain type query param that
> : returns average time spent in aggregating the search results from shards?
>
> Sort of?
>
> Metrics like "QUERY./select.distrib.requestTimes" tell you the stats on
> handling a "distributed" request -- which is when a core is responsible to
> sending out "per-shard" requests and merging the responses.
>
> But it doesn't *only* include the "time spent in aggregating the search
> results from shards" ... it also includes the time spent determining which
> requests to send to which shards, and waiting for the responses to those
> (frequently concurrent) requests"
>
>
> -Hoss
> http://www.lucidworks.com/
>
Re: Debug time spent in aggregating the search results
Posted by Chris Hostetter <ho...@fucit.org>.
: Hi Solr Users,
:
: Is there a metric endpoint or a debug/explain type query param that
: returns average time spent in aggregating the search results from shards?
Sort of?
Metrics like "QUERY./select.distrib.requestTimes" tell you the stats on
handling a "distributed" request -- which is when a core is responsible to
sending out "per-shard" requests and merging the responses.
But it doesn't *only* include the "time spent in aggregating the search
results from shards" ... it also includes the time spent determining which
requests to send to which shards, and waiting for the responses to those
(frequently concurrent) requests"
-Hoss
http://www.lucidworks.com/