You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-dev@lucene.apache.org by "Jason Rutherglen (JIRA)" <ji...@apache.org> on 2009/08/13 22:07:14 UTC
[jira] Created: (SOLR-1364) Distributed search return Solr shard
header information (like qtime)
Distributed search return Solr shard header information (like qtime)
--------------------------------------------------------------------
Key: SOLR-1364
URL: https://issues.apache.org/jira/browse/SOLR-1364
Project: Solr
Issue Type: Improvement
Components: search
Affects Versions: 1.4
Reporter: Jason Rutherglen
Priority: Minor
Fix For: 1.5
Distributed queries can expose the Solr shard query information
such as qtime. The aggregate qtime can be broken up into the
time required for each stage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1364) Distributed search return Solr shard
header information (like qtime)
Posted by "Otis Gospodnetic (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12838521#action_12838521 ]
Otis Gospodnetic commented on SOLR-1364:
----------------------------------------
Ian - please copy/paste an example of your own stats gathered with your changes, so it's easy for people to see what the output is, thus making it easy to understand and evaluate.
> Distributed search return Solr shard header information (like qtime)
> --------------------------------------------------------------------
>
> Key: SOLR-1364
> URL: https://issues.apache.org/jira/browse/SOLR-1364
> Project: Solr
> Issue Type: Improvement
> Components: search
> Affects Versions: 1.4
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 1.5
>
> Attachments: SOLR-1364.patch
>
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> Distributed queries can expose the Solr shard query information
> such as qtime. The aggregate qtime can be broken up into the
> time required for each stage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1364) Distributed search return Solr shard
header information (like qtime)
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12838439#action_12838439 ]
Jason Rutherglen commented on SOLR-1364:
----------------------------------------
Thanks Ian, to clarify, will this patch display the qtime of the response of each shard?
> Distributed search return Solr shard header information (like qtime)
> --------------------------------------------------------------------
>
> Key: SOLR-1364
> URL: https://issues.apache.org/jira/browse/SOLR-1364
> Project: Solr
> Issue Type: Improvement
> Components: search
> Affects Versions: 1.4
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 1.5
>
> Attachments: SOLR-1364.patch
>
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> Distributed queries can expose the Solr shard query information
> such as qtime. The aggregate qtime can be broken up into the
> time required for each stage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1364) Distributed search return Solr shard
header information (like qtime)
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755832#action_12755832 ]
Jason Rutherglen commented on SOLR-1364:
----------------------------------------
I'm adding debugging to this issue, where if debugQuery=true or
debugShards=true then the results will basically list the
returned response headers from each shard. Also needed, is which
shard each debug entry is from. Otherwise distributed queries
become rather difficult to regulate.
> Distributed search return Solr shard header information (like qtime)
> --------------------------------------------------------------------
>
> Key: SOLR-1364
> URL: https://issues.apache.org/jira/browse/SOLR-1364
> Project: Solr
> Issue Type: Improvement
> Components: search
> Affects Versions: 1.4
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 1.5
>
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> Distributed queries can expose the Solr shard query information
> such as qtime. The aggregate qtime can be broken up into the
> time required for each stage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-1364) Distributed search return Solr shard
header information (like qtime)
Posted by "ian connor (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ian connor updated SOLR-1364:
-----------------------------
Attachment: SOLR-1364.patch
This patch will track average speeds from the shards it finds in the request. I am hoping this will show patterns of slow shards over time but ideally, I would like to see the shard times also come back on the request as well per query. My hope is that this hack will show where this information is found and if debug is passed in, it could be written to the response.
> Distributed search return Solr shard header information (like qtime)
> --------------------------------------------------------------------
>
> Key: SOLR-1364
> URL: https://issues.apache.org/jira/browse/SOLR-1364
> Project: Solr
> Issue Type: Improvement
> Components: search
> Affects Versions: 1.4
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 1.5
>
> Attachments: SOLR-1364.patch
>
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> Distributed queries can expose the Solr shard query information
> such as qtime. The aggregate qtime can be broken up into the
> time required for each stage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1364) Distributed search return Solr shard
header information (like qtime)
Posted by "ian connor (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12838557#action_12838557 ]
ian connor commented on SOLR-1364:
----------------------------------
Good idea. Here is some sample out put on a solr stats page.
description: The standard Solr request handler
stats:
handlerStart : 1267037173182
requests : 3099817
errors : 2221
timeouts : 0
totalTime : 52082310
avgTimePerRequest : 16.801737
standardDeviation : 304.06168
avgRequestsPerSecond : 31.80935
10.0.16.181:8895/solr_numRequests : 12869
10.0.16.181:8884/solr_averageQTime : 9.651404
10.0.16.181:8896/solr_queryTime : 314763
10.0.16.181:8896/solr_elapsedTime : 681581
10.0.16.181:8885/solr_elapsedTime : 193555
10.0.16.181:8882/solr_elapsedTime : 329673
10.0.16.181:8898/solr_elapsedTime : 519454
10.0.16.181:8896/solr_queryQTime : 0
10.0.16.181:8897/solr_numRequests : 15344
10.0.16.181:8885/solr_queryTime : 62374
10.0.16.181:8891/solr_elapsedTime : 549124
10.0.16.181:8884/solr_elapsedTime : 367643
10.0.16.181:8898/solr_queryTime : 183239
10.0.16.181:8885/solr_averageQTime : 11.983478
10.0.16.181:8888/solr_averageQTime : 52.41645
10.0.16.181:8892/solr_averageQTime : 25.302937
10.0.16.181:8887/solr_queryQTime : 101
etc.....
>From this sample, we can see 10.0.16.181:8892 average is only 25ms where 10.0.16.181:8888 is at 52ms (twice as much) so we might consider rebalancing the shards to give 10.0.16.181:8888 less work.
Ideally, it would be good to get individual times to also go back on the request to the client so that these could be tracked and to see if any patterns emerge (certain queries hurt certain shards or the time might correspond to a replication event). If someone wants to take these times and put on them on the request - that would be brilliant (I did not figure that part out yet).
> Distributed search return Solr shard header information (like qtime)
> --------------------------------------------------------------------
>
> Key: SOLR-1364
> URL: https://issues.apache.org/jira/browse/SOLR-1364
> Project: Solr
> Issue Type: Improvement
> Components: search
> Affects Versions: 1.4
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 1.5
>
> Attachments: SOLR-1364.patch
>
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> Distributed queries can expose the Solr shard query information
> such as qtime. The aggregate qtime can be broken up into the
> time required for each stage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1364) Distributed search return Solr shard
header information (like qtime)
Posted by "ian connor (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12838473#action_12838473 ]
ian connor commented on SOLR-1364:
----------------------------------
Hi Jason, this patch is not that good. It does expose where you can capture that information - but does not report it on each request. Instead, it adds it to the statistics page and calculates a running average, total and count per shard. It will at least help you see if you have a hot shard that is on average taking a long time.
> Distributed search return Solr shard header information (like qtime)
> --------------------------------------------------------------------
>
> Key: SOLR-1364
> URL: https://issues.apache.org/jira/browse/SOLR-1364
> Project: Solr
> Issue Type: Improvement
> Components: search
> Affects Versions: 1.4
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 1.5
>
> Attachments: SOLR-1364.patch
>
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> Distributed queries can expose the Solr shard query information
> such as qtime. The aggregate qtime can be broken up into the
> time required for each stage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1364) Distributed search return Solr shard
header information (like qtime)
Posted by "ian connor (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12829803#action_12829803 ]
ian connor commented on SOLR-1364:
----------------------------------
I agree, to find hot shards and better debug a complex distributed system - this is essential.
If the shard and elapsedTime were made public off the request, you could pull them from the srsp in SearchHandler.java
long elapsedTime = srsp.getElapsedTime();
String shard = srsp.getShard();
some_hashmap_of_shard_times.set(shard, elapsedTime);
then you could report them back I think with code like:
// add the timing info
if( rb.getDebugInfo() == null ) {
rb.setDebugInfo( new SimpleOrderedMap<Object>() );
}
rb.getDebugInfo().add( "shardtimes", some_hashmap_of_shard_times );
However, I am not sure if debugInfo takes a HashMap or if a different object would be better suited.
> Distributed search return Solr shard header information (like qtime)
> --------------------------------------------------------------------
>
> Key: SOLR-1364
> URL: https://issues.apache.org/jira/browse/SOLR-1364
> Project: Solr
> Issue Type: Improvement
> Components: search
> Affects Versions: 1.4
> Reporter: Jason Rutherglen
> Priority: Minor
> Fix For: 1.5
>
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> Distributed queries can expose the Solr shard query information
> such as qtime. The aggregate qtime can be broken up into the
> time required for each stage.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.