You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Spyros Lambrinidis <sp...@peopleperhour.com> on 2013/05/29 23:58:52 UTC
Solr query performance tool
Hi,
Lately we are seeing increased latency times on solr and we would like to
know which queries / facet searches are the most time consuming and heavy
for our system.
Is there any tool equivalent to the mysql low log? Does solr keep the times
each query takes in some log?
Thank you for your help.
-S.
--
Spyros Lambrinidis
Head of Engineering & Commando of
PeoplePerHour.com<http://www.peopleperhour.com>
Evmolpidon 23
118 54, Gkazi
Athens, Greece
Tel: +30 210 3455480
Follow us on Facebook <http://www.facebook.com/peopleperhour>
Follow us on Twitter <http://twitter.com/#%21/peopleperhour>
Re: Solr query performance tool
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi,
The regular Solr log logs Qtime for each query.
Otis
Solr & ElasticSearch Support
http://sematext.com/
On May 29, 2013 5:59 PM, "Spyros Lambrinidis" <sp...@peopleperhour.com>
wrote:
> Hi,
>
> Lately we are seeing increased latency times on solr and we would like to
> know which queries / facet searches are the most time consuming and heavy
> for our system.
>
> Is there any tool equivalent to the mysql low log? Does solr keep the times
> each query takes in some log?
>
> Thank you for your help.
>
> -S.
>
>
> --
> Spyros Lambrinidis
> Head of Engineering & Commando of
> PeoplePerHour.com<http://www.peopleperhour.com>
> Evmolpidon 23
> 118 54, Gkazi
> Athens, Greece
> Tel: +30 210 3455480
>
> Follow us on Facebook <http://www.facebook.com/peopleperhour>
> Follow us on Twitter <http://twitter.com/#%21/peopleperhour>
>
Re: Solr query performance tool
Posted by William Bell <bi...@gmail.com>.
We should consider adding another parameter "RealTime" in the log. That
would really help all of us trying to figure out how much time a query is
taking.
On Tue, Jun 4, 2013 at 5:14 PM, Otis Gospodnetic <otis.gospodnetic@gmail.com
> wrote:
> Right. The main takeway is that QTime is not exactly what user sees.
> What users sees is always > QTime. To see what end users experience
> one needs RUM (and I don't mean the tasty kind, but
> http://en.wikipedia.org/wiki/Real_user_monitoring .
>
> Otis
> --
> Solr & ElasticSearch Support
> http://sematext.com/
>
>
>
>
>
> On Tue, Jun 4, 2013 at 11:54 AM, Walter Underwood <wu...@wunderwood.org>
> wrote:
> > On Jun 4, 2013, at 8:28 AM, Shawn Heisey wrote:
> >
> >> On 6/4/2013 8:45 AM, O'Regan, Mike wrote:
> >>> Can't a GC also kick in before the timer starts, resulting a short
> QTime but a long time from the point of view of the client? Believe I have
> seen that phenomenon in action.
> >>
> >> You are correct. I don't know why I didn't think of that.
> >>
> >> Thanks,
> >> Shawn
> >
> >
> > If I remember correctly, QTime does not include the time to fetch the
> field data after the query is evaluated and scored. It also does not
> include the time to send the data.
> >
> > wunder
> >
> >
> >
> >
>
--
Bill Bell
billnbell@gmail.com
cell 720-256-8076
Re: Solr query performance tool
Posted by Otis Gospodnetic <ot...@gmail.com>.
Right. The main takeway is that QTime is not exactly what user sees.
What users sees is always > QTime. To see what end users experience
one needs RUM (and I don't mean the tasty kind, but
http://en.wikipedia.org/wiki/Real_user_monitoring .
Otis
--
Solr & ElasticSearch Support
http://sematext.com/
On Tue, Jun 4, 2013 at 11:54 AM, Walter Underwood <wu...@wunderwood.org> wrote:
> On Jun 4, 2013, at 8:28 AM, Shawn Heisey wrote:
>
>> On 6/4/2013 8:45 AM, O'Regan, Mike wrote:
>>> Can't a GC also kick in before the timer starts, resulting a short QTime but a long time from the point of view of the client? Believe I have seen that phenomenon in action.
>>
>> You are correct. I don't know why I didn't think of that.
>>
>> Thanks,
>> Shawn
>
>
> If I remember correctly, QTime does not include the time to fetch the field data after the query is evaluated and scored. It also does not include the time to send the data.
>
> wunder
>
>
>
>
Re: Solr query performance tool
Posted by Walter Underwood <wu...@wunderwood.org>.
On Jun 4, 2013, at 8:28 AM, Shawn Heisey wrote:
> On 6/4/2013 8:45 AM, O'Regan, Mike wrote:
>> Can't a GC also kick in before the timer starts, resulting a short QTime but a long time from the point of view of the client? Believe I have seen that phenomenon in action.
>
> You are correct. I don't know why I didn't think of that.
>
> Thanks,
> Shawn
If I remember correctly, QTime does not include the time to fetch the field data after the query is evaluated and scored. It also does not include the time to send the data.
wunder
Re: Solr query performance tool
Posted by Shawn Heisey <so...@elyograg.org>.
On 6/4/2013 8:45 AM, O'Regan, Mike wrote:
> Can't a GC also kick in before the timer starts, resulting a short QTime but a long time from the point of view of the client? Believe I have seen that phenomenon in action.
You are correct. I don't know why I didn't think of that.
Thanks,
Shawn
RE: Solr query performance tool
Posted by "O'Regan, Mike" <mo...@follett.com>.
> From: Shawn Heisey [solr@elyograg.org]
>
>On 6/3/2013 3:33 PM, Greg Harris wrote:
>>
>> You have to be careful looking at the QTime's. They do not include garbage collection. I've run into issues where QTime is short (cause it was), it just happened that the query came in during a long garbage collection where everything was paused. So you can get into situations where once the 15 second GC is done everything performs as expected! I'd make sure and have an external querying tool and you can monitor GC times as well via JMX.
>
>The QTime value in the response is calculated using
>System.currentTimeMillis(), so it should include the GC time, unless the
>GC happens to hit just after the QTime is calculated but before the
>final response with all the results is sent. If you are requesting a
>lot of documents or you have very large documents where most/all of the
>fields are stored, having long GCs hit during that particular moment
>might actually be a common occurrence.
Can't a GC also kick in before the timer starts, resulting a short QTime but a long time from the point of view of the client? Believe I have seen that phenomenon in action.
Mike O'Regan
Re: Solr query performance tool
Posted by Shawn Heisey <so...@elyograg.org>.
On 6/3/2013 3:33 PM, Greg Harris wrote:
>
> You have to be careful looking at the QTime's. They do not include garbage collection. I've run into issues where QTime is short (cause it was), it just happened that the query came in during a long garbage collection where everything was paused. So you can get into situations where once the 15 second GC is done everything performs as expected! I'd make sure and have an external querying tool and you can monitor GC times as well via JMX.
The QTime value in the response is calculated using
System.currentTimeMillis(), so it should include the GC time, unless the
GC happens to hit just after the QTime is calculated but before the
final response with all the results is sent. If you are requesting a
lot of documents or you have very large documents where most/all of the
fields are stored, having long GCs hit during that particular moment
might actually be a common occurrence.
Thanks,
Shawn
RE: Solr query performance tool
Posted by Greg Harris <gr...@lucidworks.com>.
You have to be careful looking at the QTime's. They do not include garbage collection. I've run into issues where QTime is short (cause it was), it just happened that the query came in during a long garbage collection where everything was paused. So you can get into situations where once the 15 second GC is done everything performs as expected! I'd make sure and have an external querying tool and you can monitor GC times as well via JMX.
________________________________________
From: bbarani [bbarani@gmail.com]
Sent: Monday, June 03, 2013 8:58 AM
To: solr-user@lucene.apache.org
Subject: Re: Solr query performance tool
You can use this tool to analyze the logs..
https://github.com/dfdeshom/solr-loganalyzer
We use solrmeter to test the performance / Stress testing.
https://code.google.com/p/solrmeter/
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-query-performance-tool-tp4066900p4067869.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr query performance tool
Posted by bbarani <bb...@gmail.com>.
You can use this tool to analyze the logs..
https://github.com/dfdeshom/solr-loganalyzer
We use solrmeter to test the performance / Stress testing.
https://code.google.com/p/solrmeter/
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-query-performance-tool-tp4066900p4067869.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr query performance tool
Posted by Erick Erickson <er...@gmail.com>.
The qtimes are in the solr log, you'll see lines like:
params={q=*:*} hits=32 status=0 QTime=5
QTime is the time spent serving the query but does NOT include
assembling the response.
Best
Erick
On Wed, May 29, 2013 at 5:58 PM, Spyros Lambrinidis
<sp...@peopleperhour.com> wrote:
> Hi,
>
> Lately we are seeing increased latency times on solr and we would like to
> know which queries / facet searches are the most time consuming and heavy
> for our system.
>
> Is there any tool equivalent to the mysql low log? Does solr keep the times
> each query takes in some log?
>
> Thank you for your help.
>
> -S.
>
>
> --
> Spyros Lambrinidis
> Head of Engineering & Commando of
> PeoplePerHour.com<http://www.peopleperhour.com>
> Evmolpidon 23
> 118 54, Gkazi
> Athens, Greece
> Tel: +30 210 3455480
>
> Follow us on Facebook <http://www.facebook.com/peopleperhour>
> Follow us on Twitter <http://twitter.com/#%21/peopleperhour>