You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Erick Erickson <er...@gmail.com> on 2013/07/10 22:46:00 UTC
Additional timing/debug info
I've been waving my hands for a while with "QTime is just the query
time, it doesn't count network latency, assembling the response blah
blah blah".
It seems like we could at least provide the time it takes to write out
the docs that would include decompression time, disk latency, all that
stuff. Still wouldn't deal with network latency, but it'd be progress.
I'm thinking that TextResponsWriter.writeDocuments and
BinaryResponseWriter.writeResultsBody each have a loop that could
pretty easily be modified to collect this bit of data and it could be
returned in another tag in the "timings" section.
Does this seem do-able? What about valuable? I'm assuming that just
_adding_ a section wouldn't break back-compat. What do people think?
Should I raise a JIRA?
Not quite sure what it would mean in a cloud setup, possibly start out
as an element in the debug/timing/process section of the response?
(not sure what about the binary response writer yet, haven't thought
it through much).
<lst name="assemblyTimes">
<str name="shard1">43</str>
<str name="shard2">54</str>
</lst>
Although the above would help pinpoint laggard shards.
But _much_ lower fruit would be to just log this, which should be
really easy unless I'm missing something.
Let me know and I'll raise a JIRA if there's interest.,
Erick
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Additional timing/debug info
Posted by Yonik Seeley <yo...@lucidworks.com>.
Can't you turn on jetty logging and get full response time + size of
response, etc?
-Yonik
http://lucidworks.com
On Wed, Jul 10, 2013 at 4:46 PM, Erick Erickson <er...@gmail.com> wrote:
> I've been waving my hands for a while with "QTime is just the query
> time, it doesn't count network latency, assembling the response blah
> blah blah".
>
> It seems like we could at least provide the time it takes to write out
> the docs that would include decompression time, disk latency, all that
> stuff. Still wouldn't deal with network latency, but it'd be progress.
>
> I'm thinking that TextResponsWriter.writeDocuments and
> BinaryResponseWriter.writeResultsBody each have a loop that could
> pretty easily be modified to collect this bit of data and it could be
> returned in another tag in the "timings" section.
>
> Does this seem do-able? What about valuable? I'm assuming that just
> _adding_ a section wouldn't break back-compat. What do people think?
> Should I raise a JIRA?
>
> Not quite sure what it would mean in a cloud setup, possibly start out
> as an element in the debug/timing/process section of the response?
> (not sure what about the binary response writer yet, haven't thought
> it through much).
>
> <lst name="assemblyTimes">
> <str name="shard1">43</str>
> <str name="shard2">54</str>
> </lst>
>
> Although the above would help pinpoint laggard shards.
>
> But _much_ lower fruit would be to just log this, which should be
> really easy unless I'm missing something.
>
> Let me know and I'll raise a JIRA if there's interest.,
> Erick
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Additional timing/debug info
Posted by Grant Ingersoll <gs...@apache.org>.
On Jul 10, 2013, at 6:04 PM, Shawn Heisey <so...@elyograg.org> wrote:
> and any calculations that do slow things down could be part of debugQuery output.
FWIW, debugQuery has a timings option in it already, which may or may not help.
Re: Additional timing/debug info
Posted by Alan Woodward <al...@flax.co.uk>.
That'll be https://issues.apache.org/jira/browse/SOLR-4735, which I made a start on, but then got diverted into a bunch of other CoreContainer cleanup bits and pieces. Too many JIRAs, too little time...
It's not quite as simple as just plugging the metrics library in, unfortunately, as it has a fairly fixed idea of how to report things via JMX which doesn't fit in with how Solr does things now. I do hope to get back to that soon, though.
Alan Woodward
www.flax.co.uk
On 11 Jul 2013, at 03:47, Walter Underwood wrote:
> Sorry, I misunderstood your short response as "JMX already does it so we don't need something else".
>
> wunder
>
> On Jul 10, 2013, at 7:28 PM, Otis Gospodnetic wrote:
>
>> Sorry, I don't follow what you mean by "another server" and "get it
>> *into* CM" and fear we're getting OT. Just meant to say that if
>> somebody is working on tracking more stats with Coda's metrics
>> package, those stats should/could be exposed via JMX like all other
>> Solr stats and that CM has the ability to expose whatever it collects
>> via JMX, which makes me think that's the easiest and most useful thing
>> to do. But maybe I'm missing something here...
>>
>> Otis
>>
>>
>>
>> On Wed, Jul 10, 2013 at 10:20 PM, Walter Underwood
>> <wu...@wunderwood.org> wrote:
>>> So?
>>>
>>> JMX-only requires another server to get it into Codahale Metrics.
>>>
>>> wunder
>>>
>>> On Jul 10, 2013, at 7:19 PM, Otis Gospodnetic wrote:
>>>
>>> Coda's stuff exports to JMX and Solr exports to JMX.
>>>
>>> Otis
>>> --
>>> Solr Performance Monitoring -- http://sematext.com/spm
>>>
>>>
>>>
>>> On Wed, Jul 10, 2013 at 10:09 PM, Walter Underwood
>>> <wu...@wunderwood.org> wrote:
>>>
>>> There is some work on reporting this through the Codahale Graphics system.
>>>
>>> For us, that is way way better than a Solr-specific metrics interface.
>>>
>>>
>>> wunder
>>>
>>>
>>> On Jul 10, 2013, at 7:06 PM, Erick Erickson wrote:
>>>
>>>
>>> Yonik:
>>>
>>>
>>> Yes, but correlating these is a bit awkward. My notion is it would be
>>>
>>> useful to have this in a debug response and avoid having to
>>>
>>> reconcile things from log files.... Perhaps Shawn's idea
>>>
>>> would be a good thing to put in a (new?) debug section rather than
>>>
>>> re-purpose the QTime which we all know and love.
>>>
>>>
>>> On Wed, Jul 10, 2013 at 10:01 PM, Otis Gospodnetic
>>>
>>> <ot...@gmail.com> wrote:
>>>
>>>
>>> This sounds attractive to me. What other times are you thinking about,
>>>
>>> Shawn?
>>>
>>>
>>>
>>> I think this type of info should be owned by Solr and one should not
>>>
>>>
>>> rely on Jetty. Plus the plan is to ditch the servlet container
>>>
>>>
>>> anyway.
>>>
>>>
>>>
>>> Otis
>>>
>>>
>>> --
>>>
>>>
>>> Solr Performance Monitoring -- http://sematext.com/spm
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Jul 10, 2013 at 6:04 PM, Shawn Heisey <so...@elyograg.org> wrote:
>>>
>>>
>>> On 7/10/2013 2:46 PM, Erick Erickson wrote:
>>>
>>>
>>>
>>> I've been waving my hands for a while with "QTime is just the query
>>>
>>>
>>> time, it doesn't count network latency, assembling the response blah
>>>
>>>
>>> blah blah".
>>>
>>>
>>>
>>> It seems like we could at least provide the time it takes to write out
>>>
>>>
>>> the docs that would include decompression time, disk latency, all that
>>>
>>>
>>> stuff. Still wouldn't deal with network latency, but it'd be progress.
>>>
>>>
>>>
>>>
>>> <snip>
>>>
>>>
>>>
>>>
>>> Does this seem do-able? What about valuable? I'm assuming that just
>>>
>>>
>>> _adding_ a section wouldn't break back-compat. What do people think?
>>>
>>>
>>> Should I raise a JIRA?
>>>
>>>
>>>
>>>
>>> +1 on raising a JIRA. Here's my radical notion:
>>>
>>>
>>>
>>> IMHO we should add all available timing information up and display that as
>>>
>>>
>>> QTime. Having that QTime further broken down into additional information
>>>
>>>
>>> would be very good. Any simple calculations (which shouldn't really slow
>>>
>>>
>>> down a request) should be included by default, and any calculations that do
>>>
>>>
>>> slow things down could be part of debugQuery output.
>>>
>>>
>>>
>>> My preference would be to make these changes in branch_4x, but if we do
>>>
>>>
>>> that, we'll suddenly be dealing with people who think that a minor version
>>>
>>>
>>> upgrade has incredibly worse performance just based on QTime numbers, even
>>>
>>>
>>> though nothing has really changed.
>>>
>>>
>>>
>>> If we just make the additional information available in 4.x and then update
>>>
>>>
>>> QTime to include everything in 5.0, that seems like a reasonable path. It's
>>>
>>>
>>> easier to manage expectations on a major version bump.
>>>
>>>
>>>
>>> Thanks,
>>>
>>>
>>> Shawn
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>
> --
> Walter Underwood
> wunder@wunderwood.org
>
>
>
Re: Additional timing/debug info
Posted by Walter Underwood <wu...@wunderwood.org>.
Sorry, I misunderstood your short response as "JMX already does it so we don't need something else".
wunder
On Jul 10, 2013, at 7:28 PM, Otis Gospodnetic wrote:
> Sorry, I don't follow what you mean by "another server" and "get it
> *into* CM" and fear we're getting OT. Just meant to say that if
> somebody is working on tracking more stats with Coda's metrics
> package, those stats should/could be exposed via JMX like all other
> Solr stats and that CM has the ability to expose whatever it collects
> via JMX, which makes me think that's the easiest and most useful thing
> to do. But maybe I'm missing something here...
>
> Otis
>
>
>
> On Wed, Jul 10, 2013 at 10:20 PM, Walter Underwood
> <wu...@wunderwood.org> wrote:
>> So?
>>
>> JMX-only requires another server to get it into Codahale Metrics.
>>
>> wunder
>>
>> On Jul 10, 2013, at 7:19 PM, Otis Gospodnetic wrote:
>>
>> Coda's stuff exports to JMX and Solr exports to JMX.
>>
>> Otis
>> --
>> Solr Performance Monitoring -- http://sematext.com/spm
>>
>>
>>
>> On Wed, Jul 10, 2013 at 10:09 PM, Walter Underwood
>> <wu...@wunderwood.org> wrote:
>>
>> There is some work on reporting this through the Codahale Graphics system.
>>
>> For us, that is way way better than a Solr-specific metrics interface.
>>
>>
>> wunder
>>
>>
>> On Jul 10, 2013, at 7:06 PM, Erick Erickson wrote:
>>
>>
>> Yonik:
>>
>>
>> Yes, but correlating these is a bit awkward. My notion is it would be
>>
>> useful to have this in a debug response and avoid having to
>>
>> reconcile things from log files.... Perhaps Shawn's idea
>>
>> would be a good thing to put in a (new?) debug section rather than
>>
>> re-purpose the QTime which we all know and love.
>>
>>
>> On Wed, Jul 10, 2013 at 10:01 PM, Otis Gospodnetic
>>
>> <ot...@gmail.com> wrote:
>>
>>
>> This sounds attractive to me. What other times are you thinking about,
>>
>> Shawn?
>>
>>
>>
>> I think this type of info should be owned by Solr and one should not
>>
>>
>> rely on Jetty. Plus the plan is to ditch the servlet container
>>
>>
>> anyway.
>>
>>
>>
>> Otis
>>
>>
>> --
>>
>>
>> Solr Performance Monitoring -- http://sematext.com/spm
>>
>>
>>
>>
>>
>> On Wed, Jul 10, 2013 at 6:04 PM, Shawn Heisey <so...@elyograg.org> wrote:
>>
>>
>> On 7/10/2013 2:46 PM, Erick Erickson wrote:
>>
>>
>>
>> I've been waving my hands for a while with "QTime is just the query
>>
>>
>> time, it doesn't count network latency, assembling the response blah
>>
>>
>> blah blah".
>>
>>
>>
>> It seems like we could at least provide the time it takes to write out
>>
>>
>> the docs that would include decompression time, disk latency, all that
>>
>>
>> stuff. Still wouldn't deal with network latency, but it'd be progress.
>>
>>
>>
>>
>> <snip>
>>
>>
>>
>>
>> Does this seem do-able? What about valuable? I'm assuming that just
>>
>>
>> _adding_ a section wouldn't break back-compat. What do people think?
>>
>>
>> Should I raise a JIRA?
>>
>>
>>
>>
>> +1 on raising a JIRA. Here's my radical notion:
>>
>>
>>
>> IMHO we should add all available timing information up and display that as
>>
>>
>> QTime. Having that QTime further broken down into additional information
>>
>>
>> would be very good. Any simple calculations (which shouldn't really slow
>>
>>
>> down a request) should be included by default, and any calculations that do
>>
>>
>> slow things down could be part of debugQuery output.
>>
>>
>>
>> My preference would be to make these changes in branch_4x, but if we do
>>
>>
>> that, we'll suddenly be dealing with people who think that a minor version
>>
>>
>> upgrade has incredibly worse performance just based on QTime numbers, even
>>
>>
>> though nothing has really changed.
>>
>>
>>
>> If we just make the additional information available in 4.x and then update
>>
>>
>> QTime to include everything in 5.0, that seems like a reasonable path. It's
>>
>>
>> easier to manage expectations on a major version bump.
>>
>>
>>
>> Thanks,
>>
>>
>> Shawn
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
--
Walter Underwood
wunder@wunderwood.org
Re: Additional timing/debug info
Posted by Otis Gospodnetic <ot...@gmail.com>.
Sorry, I don't follow what you mean by "another server" and "get it
*into* CM" and fear we're getting OT. Just meant to say that if
somebody is working on tracking more stats with Coda's metrics
package, those stats should/could be exposed via JMX like all other
Solr stats and that CM has the ability to expose whatever it collects
via JMX, which makes me think that's the easiest and most useful thing
to do. But maybe I'm missing something here...
Otis
On Wed, Jul 10, 2013 at 10:20 PM, Walter Underwood
<wu...@wunderwood.org> wrote:
> So?
>
> JMX-only requires another server to get it into Codahale Metrics.
>
> wunder
>
> On Jul 10, 2013, at 7:19 PM, Otis Gospodnetic wrote:
>
> Coda's stuff exports to JMX and Solr exports to JMX.
>
> Otis
> --
> Solr Performance Monitoring -- http://sematext.com/spm
>
>
>
> On Wed, Jul 10, 2013 at 10:09 PM, Walter Underwood
> <wu...@wunderwood.org> wrote:
>
> There is some work on reporting this through the Codahale Graphics system.
>
> For us, that is way way better than a Solr-specific metrics interface.
>
>
> wunder
>
>
> On Jul 10, 2013, at 7:06 PM, Erick Erickson wrote:
>
>
> Yonik:
>
>
> Yes, but correlating these is a bit awkward. My notion is it would be
>
> useful to have this in a debug response and avoid having to
>
> reconcile things from log files.... Perhaps Shawn's idea
>
> would be a good thing to put in a (new?) debug section rather than
>
> re-purpose the QTime which we all know and love.
>
>
> On Wed, Jul 10, 2013 at 10:01 PM, Otis Gospodnetic
>
> <ot...@gmail.com> wrote:
>
>
> This sounds attractive to me. What other times are you thinking about,
>
> Shawn?
>
>
>
> I think this type of info should be owned by Solr and one should not
>
>
> rely on Jetty. Plus the plan is to ditch the servlet container
>
>
> anyway.
>
>
>
> Otis
>
>
> --
>
>
> Solr Performance Monitoring -- http://sematext.com/spm
>
>
>
>
>
> On Wed, Jul 10, 2013 at 6:04 PM, Shawn Heisey <so...@elyograg.org> wrote:
>
>
> On 7/10/2013 2:46 PM, Erick Erickson wrote:
>
>
>
> I've been waving my hands for a while with "QTime is just the query
>
>
> time, it doesn't count network latency, assembling the response blah
>
>
> blah blah".
>
>
>
> It seems like we could at least provide the time it takes to write out
>
>
> the docs that would include decompression time, disk latency, all that
>
>
> stuff. Still wouldn't deal with network latency, but it'd be progress.
>
>
>
>
> <snip>
>
>
>
>
> Does this seem do-able? What about valuable? I'm assuming that just
>
>
> _adding_ a section wouldn't break back-compat. What do people think?
>
>
> Should I raise a JIRA?
>
>
>
>
> +1 on raising a JIRA. Here's my radical notion:
>
>
>
> IMHO we should add all available timing information up and display that as
>
>
> QTime. Having that QTime further broken down into additional information
>
>
> would be very good. Any simple calculations (which shouldn't really slow
>
>
> down a request) should be included by default, and any calculations that do
>
>
> slow things down could be part of debugQuery output.
>
>
>
> My preference would be to make these changes in branch_4x, but if we do
>
>
> that, we'll suddenly be dealing with people who think that a minor version
>
>
> upgrade has incredibly worse performance just based on QTime numbers, even
>
>
> though nothing has really changed.
>
>
>
> If we just make the additional information available in 4.x and then update
>
>
> QTime to include everything in 5.0, that seems like a reasonable path. It's
>
>
> easier to manage expectations on a major version bump.
>
>
>
> Thanks,
>
>
> Shawn
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Additional timing/debug info
Posted by Walter Underwood <wu...@wunderwood.org>.
So?
JMX-only requires another server to get it into Codahale Metrics.
wunder
On Jul 10, 2013, at 7:19 PM, Otis Gospodnetic wrote:
> Coda's stuff exports to JMX and Solr exports to JMX.
>
> Otis
> --
> Solr Performance Monitoring -- http://sematext.com/spm
>
>
>
> On Wed, Jul 10, 2013 at 10:09 PM, Walter Underwood
> <wu...@wunderwood.org> wrote:
>> There is some work on reporting this through the Codahale Graphics system.
>> For us, that is way way better than a Solr-specific metrics interface.
>>
>> wunder
>>
>> On Jul 10, 2013, at 7:06 PM, Erick Erickson wrote:
>>
>> Yonik:
>>
>> Yes, but correlating these is a bit awkward. My notion is it would be
>> useful to have this in a debug response and avoid having to
>> reconcile things from log files.... Perhaps Shawn's idea
>> would be a good thing to put in a (new?) debug section rather than
>> re-purpose the QTime which we all know and love.
>>
>> On Wed, Jul 10, 2013 at 10:01 PM, Otis Gospodnetic
>> <ot...@gmail.com> wrote:
>>
>> This sounds attractive to me. What other times are you thinking about,
>> Shawn?
>>
>>
>> I think this type of info should be owned by Solr and one should not
>>
>> rely on Jetty. Plus the plan is to ditch the servlet container
>>
>> anyway.
>>
>>
>> Otis
>>
>> --
>>
>> Solr Performance Monitoring -- http://sematext.com/spm
>>
>>
>>
>>
>> On Wed, Jul 10, 2013 at 6:04 PM, Shawn Heisey <so...@elyograg.org> wrote:
>>
>> On 7/10/2013 2:46 PM, Erick Erickson wrote:
>>
>>
>> I've been waving my hands for a while with "QTime is just the query
>>
>> time, it doesn't count network latency, assembling the response blah
>>
>> blah blah".
>>
>>
>> It seems like we could at least provide the time it takes to write out
>>
>> the docs that would include decompression time, disk latency, all that
>>
>> stuff. Still wouldn't deal with network latency, but it'd be progress.
>>
>>
>>
>> <snip>
>>
>>
>>
>> Does this seem do-able? What about valuable? I'm assuming that just
>>
>> _adding_ a section wouldn't break back-compat. What do people think?
>>
>> Should I raise a JIRA?
>>
>>
>>
>> +1 on raising a JIRA. Here's my radical notion:
>>
>>
>> IMHO we should add all available timing information up and display that as
>>
>> QTime. Having that QTime further broken down into additional information
>>
>> would be very good. Any simple calculations (which shouldn't really slow
>>
>> down a request) should be included by default, and any calculations that do
>>
>> slow things down could be part of debugQuery output.
>>
>>
>> My preference would be to make these changes in branch_4x, but if we do
>>
>> that, we'll suddenly be dealing with people who think that a minor version
>>
>> upgrade has incredibly worse performance just based on QTime numbers, even
>>
>> though nothing has really changed.
>>
>>
>> If we just make the additional information available in 4.x and then update
>>
>> QTime to include everything in 5.0, that seems like a reasonable path. It's
>>
>> easier to manage expectations on a major version bump.
>>
>>
>> Thanks,
>>
>> Shawn
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>>
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>
>>
>> ---------------------------------------------------------------------
>>
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>
>> --
>> Walter Underwood
>> wunder@wunderwood.org
>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
--
Walter Underwood
wunder@wunderwood.org
Re: Additional timing/debug info
Posted by Otis Gospodnetic <ot...@gmail.com>.
Coda's stuff exports to JMX and Solr exports to JMX.
Otis
--
Solr Performance Monitoring -- http://sematext.com/spm
On Wed, Jul 10, 2013 at 10:09 PM, Walter Underwood
<wu...@wunderwood.org> wrote:
> There is some work on reporting this through the Codahale Graphics system.
> For us, that is way way better than a Solr-specific metrics interface.
>
> wunder
>
> On Jul 10, 2013, at 7:06 PM, Erick Erickson wrote:
>
> Yonik:
>
> Yes, but correlating these is a bit awkward. My notion is it would be
> useful to have this in a debug response and avoid having to
> reconcile things from log files.... Perhaps Shawn's idea
> would be a good thing to put in a (new?) debug section rather than
> re-purpose the QTime which we all know and love.
>
> On Wed, Jul 10, 2013 at 10:01 PM, Otis Gospodnetic
> <ot...@gmail.com> wrote:
>
> This sounds attractive to me. What other times are you thinking about,
> Shawn?
>
>
> I think this type of info should be owned by Solr and one should not
>
> rely on Jetty. Plus the plan is to ditch the servlet container
>
> anyway.
>
>
> Otis
>
> --
>
> Solr Performance Monitoring -- http://sematext.com/spm
>
>
>
>
> On Wed, Jul 10, 2013 at 6:04 PM, Shawn Heisey <so...@elyograg.org> wrote:
>
> On 7/10/2013 2:46 PM, Erick Erickson wrote:
>
>
> I've been waving my hands for a while with "QTime is just the query
>
> time, it doesn't count network latency, assembling the response blah
>
> blah blah".
>
>
> It seems like we could at least provide the time it takes to write out
>
> the docs that would include decompression time, disk latency, all that
>
> stuff. Still wouldn't deal with network latency, but it'd be progress.
>
>
>
> <snip>
>
>
>
> Does this seem do-able? What about valuable? I'm assuming that just
>
> _adding_ a section wouldn't break back-compat. What do people think?
>
> Should I raise a JIRA?
>
>
>
> +1 on raising a JIRA. Here's my radical notion:
>
>
> IMHO we should add all available timing information up and display that as
>
> QTime. Having that QTime further broken down into additional information
>
> would be very good. Any simple calculations (which shouldn't really slow
>
> down a request) should be included by default, and any calculations that do
>
> slow things down could be part of debugQuery output.
>
>
> My preference would be to make these changes in branch_4x, but if we do
>
> that, we'll suddenly be dealing with people who think that a minor version
>
> upgrade has incredibly worse performance just based on QTime numbers, even
>
> though nothing has really changed.
>
>
> If we just make the additional information available in 4.x and then update
>
> QTime to include everything in 5.0, that seems like a reasonable path. It's
>
> easier to manage expectations on a major version bump.
>
>
> Thanks,
>
> Shawn
>
>
>
>
> ---------------------------------------------------------------------
>
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
>
> ---------------------------------------------------------------------
>
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
> --
> Walter Underwood
> wunder@wunderwood.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Additional timing/debug info
Posted by Walter Underwood <wu...@wunderwood.org>.
There is some work on reporting this through the Codahale Graphics system. For us, that is way way better than a Solr-specific metrics interface.
wunder
On Jul 10, 2013, at 7:06 PM, Erick Erickson wrote:
> Yonik:
>
> Yes, but correlating these is a bit awkward. My notion is it would be
> useful to have this in a debug response and avoid having to
> reconcile things from log files.... Perhaps Shawn's idea
> would be a good thing to put in a (new?) debug section rather than
> re-purpose the QTime which we all know and love.
>
> On Wed, Jul 10, 2013 at 10:01 PM, Otis Gospodnetic
> <ot...@gmail.com> wrote:
>> This sounds attractive to me. What other times are you thinking about, Shawn?
>>
>> I think this type of info should be owned by Solr and one should not
>> rely on Jetty. Plus the plan is to ditch the servlet container
>> anyway.
>>
>> Otis
>> --
>> Solr Performance Monitoring -- http://sematext.com/spm
>>
>>
>>
>> On Wed, Jul 10, 2013 at 6:04 PM, Shawn Heisey <so...@elyograg.org> wrote:
>>> On 7/10/2013 2:46 PM, Erick Erickson wrote:
>>>>
>>>> I've been waving my hands for a while with "QTime is just the query
>>>> time, it doesn't count network latency, assembling the response blah
>>>> blah blah".
>>>>
>>>> It seems like we could at least provide the time it takes to write out
>>>> the docs that would include decompression time, disk latency, all that
>>>> stuff. Still wouldn't deal with network latency, but it'd be progress.
>>>
>>>
>>> <snip>
>>>
>>>
>>>> Does this seem do-able? What about valuable? I'm assuming that just
>>>> _adding_ a section wouldn't break back-compat. What do people think?
>>>> Should I raise a JIRA?
>>>
>>>
>>> +1 on raising a JIRA. Here's my radical notion:
>>>
>>> IMHO we should add all available timing information up and display that as
>>> QTime. Having that QTime further broken down into additional information
>>> would be very good. Any simple calculations (which shouldn't really slow
>>> down a request) should be included by default, and any calculations that do
>>> slow things down could be part of debugQuery output.
>>>
>>> My preference would be to make these changes in branch_4x, but if we do
>>> that, we'll suddenly be dealing with people who think that a minor version
>>> upgrade has incredibly worse performance just based on QTime numbers, even
>>> though nothing has really changed.
>>>
>>> If we just make the additional information available in 4.x and then update
>>> QTime to include everything in 5.0, that seems like a reasonable path. It's
>>> easier to manage expectations on a major version bump.
>>>
>>> Thanks,
>>> Shawn
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: dev-help@lucene.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
--
Walter Underwood
wunder@wunderwood.org
Re: Additional timing/debug info
Posted by Erick Erickson <er...@gmail.com>.
Yonik:
Yes, but correlating these is a bit awkward. My notion is it would be
useful to have this in a debug response and avoid having to
reconcile things from log files.... Perhaps Shawn's idea
would be a good thing to put in a (new?) debug section rather than
re-purpose the QTime which we all know and love.
On Wed, Jul 10, 2013 at 10:01 PM, Otis Gospodnetic
<ot...@gmail.com> wrote:
> This sounds attractive to me. What other times are you thinking about, Shawn?
>
> I think this type of info should be owned by Solr and one should not
> rely on Jetty. Plus the plan is to ditch the servlet container
> anyway.
>
> Otis
> --
> Solr Performance Monitoring -- http://sematext.com/spm
>
>
>
> On Wed, Jul 10, 2013 at 6:04 PM, Shawn Heisey <so...@elyograg.org> wrote:
>> On 7/10/2013 2:46 PM, Erick Erickson wrote:
>>>
>>> I've been waving my hands for a while with "QTime is just the query
>>> time, it doesn't count network latency, assembling the response blah
>>> blah blah".
>>>
>>> It seems like we could at least provide the time it takes to write out
>>> the docs that would include decompression time, disk latency, all that
>>> stuff. Still wouldn't deal with network latency, but it'd be progress.
>>
>>
>> <snip>
>>
>>
>>> Does this seem do-able? What about valuable? I'm assuming that just
>>> _adding_ a section wouldn't break back-compat. What do people think?
>>> Should I raise a JIRA?
>>
>>
>> +1 on raising a JIRA. Here's my radical notion:
>>
>> IMHO we should add all available timing information up and display that as
>> QTime. Having that QTime further broken down into additional information
>> would be very good. Any simple calculations (which shouldn't really slow
>> down a request) should be included by default, and any calculations that do
>> slow things down could be part of debugQuery output.
>>
>> My preference would be to make these changes in branch_4x, but if we do
>> that, we'll suddenly be dealing with people who think that a minor version
>> upgrade has incredibly worse performance just based on QTime numbers, even
>> though nothing has really changed.
>>
>> If we just make the additional information available in 4.x and then update
>> QTime to include everything in 5.0, that seems like a reasonable path. It's
>> easier to manage expectations on a major version bump.
>>
>> Thanks,
>> Shawn
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Additional timing/debug info
Posted by Otis Gospodnetic <ot...@gmail.com>.
This sounds attractive to me. What other times are you thinking about, Shawn?
I think this type of info should be owned by Solr and one should not
rely on Jetty. Plus the plan is to ditch the servlet container
anyway.
Otis
--
Solr Performance Monitoring -- http://sematext.com/spm
On Wed, Jul 10, 2013 at 6:04 PM, Shawn Heisey <so...@elyograg.org> wrote:
> On 7/10/2013 2:46 PM, Erick Erickson wrote:
>>
>> I've been waving my hands for a while with "QTime is just the query
>> time, it doesn't count network latency, assembling the response blah
>> blah blah".
>>
>> It seems like we could at least provide the time it takes to write out
>> the docs that would include decompression time, disk latency, all that
>> stuff. Still wouldn't deal with network latency, but it'd be progress.
>
>
> <snip>
>
>
>> Does this seem do-able? What about valuable? I'm assuming that just
>> _adding_ a section wouldn't break back-compat. What do people think?
>> Should I raise a JIRA?
>
>
> +1 on raising a JIRA. Here's my radical notion:
>
> IMHO we should add all available timing information up and display that as
> QTime. Having that QTime further broken down into additional information
> would be very good. Any simple calculations (which shouldn't really slow
> down a request) should be included by default, and any calculations that do
> slow things down could be part of debugQuery output.
>
> My preference would be to make these changes in branch_4x, but if we do
> that, we'll suddenly be dealing with people who think that a minor version
> upgrade has incredibly worse performance just based on QTime numbers, even
> though nothing has really changed.
>
> If we just make the additional information available in 4.x and then update
> QTime to include everything in 5.0, that seems like a reasonable path. It's
> easier to manage expectations on a major version bump.
>
> Thanks,
> Shawn
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Additional timing/debug info
Posted by Shawn Heisey <so...@elyograg.org>.
On 7/10/2013 2:46 PM, Erick Erickson wrote:
> I've been waving my hands for a while with "QTime is just the query
> time, it doesn't count network latency, assembling the response blah
> blah blah".
>
> It seems like we could at least provide the time it takes to write out
> the docs that would include decompression time, disk latency, all that
> stuff. Still wouldn't deal with network latency, but it'd be progress.
<snip>
> Does this seem do-able? What about valuable? I'm assuming that just
> _adding_ a section wouldn't break back-compat. What do people think?
> Should I raise a JIRA?
+1 on raising a JIRA. Here's my radical notion:
IMHO we should add all available timing information up and display that
as QTime. Having that QTime further broken down into additional
information would be very good. Any simple calculations (which
shouldn't really slow down a request) should be included by default, and
any calculations that do slow things down could be part of debugQuery
output.
My preference would be to make these changes in branch_4x, but if we do
that, we'll suddenly be dealing with people who think that a minor
version upgrade has incredibly worse performance just based on QTime
numbers, even though nothing has really changed.
If we just make the additional information available in 4.x and then
update QTime to include everything in 5.0, that seems like a reasonable
path. It's easier to manage expectations on a major version bump.
Thanks,
Shawn
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Additional timing/debug info
Posted by Otis Gospodnetic <ot...@gmail.com>.
+1 for JIRA and publishing in JMX!
Otis
--
Solr & ElasticSearch Support -- http://sematext.com/
Solr Performance Monitoring -- http://sematext.com/spm
On Wed, Jul 10, 2013 at 4:46 PM, Erick Erickson <er...@gmail.com> wrote:
> I've been waving my hands for a while with "QTime is just the query
> time, it doesn't count network latency, assembling the response blah
> blah blah".
>
> It seems like we could at least provide the time it takes to write out
> the docs that would include decompression time, disk latency, all that
> stuff. Still wouldn't deal with network latency, but it'd be progress.
>
> I'm thinking that TextResponsWriter.writeDocuments and
> BinaryResponseWriter.writeResultsBody each have a loop that could
> pretty easily be modified to collect this bit of data and it could be
> returned in another tag in the "timings" section.
>
> Does this seem do-able? What about valuable? I'm assuming that just
> _adding_ a section wouldn't break back-compat. What do people think?
> Should I raise a JIRA?
>
> Not quite sure what it would mean in a cloud setup, possibly start out
> as an element in the debug/timing/process section of the response?
> (not sure what about the binary response writer yet, haven't thought
> it through much).
>
> <lst name="assemblyTimes">
> <str name="shard1">43</str>
> <str name="shard2">54</str>
> </lst>
>
> Although the above would help pinpoint laggard shards.
>
> But _much_ lower fruit would be to just log this, which should be
> really easy unless I'm missing something.
>
> Let me know and I'll raise a JIRA if there's interest.,
> Erick
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org