You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by jimtronic <ji...@gmail.com> on 2013/08/01 19:12:50 UTC
How to uncache a query to debug?
I have a query that runs slow occasionally. I'm having trouble debugging it
because once it's cached, it runs fast -- under 10 ms. But throughout the
day it occasionally takes up to 3 secs. It seems like it could be one of the
following:
1. My autoCommit (300000 and openSearcher=false) and softAutoCommit (10000)
settings
2. Something to do with distributed search -- There are three nodes, but
only 1 shard each.
3. Just a slow query that is getting blown out of cache periodically
This is in Solr 4.2.
I like that it runs fast when cached, but if it's going to be blown out
quickly, then I'd really like to just optimize the query to run fast
uncached.
*Is there any way to run a query using no caching whatsoever?*
The query changes, but has *:* for the q param and 4 fq parameters. It's
also trying to do field collapsing.
Jim
--
View this message in context: http://lucene.472066.n3.nabble.com/How-to-uncache-a-query-to-debug-tp4082010.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: How to uncache a query to debug?
Posted by jimtronic <ji...@gmail.com>.
Thanks, but that doesn't seem to do much. I've added it to all four of the
fq params and the q param, but it only makes it marginally slower -- like
50 ms instead of 2 ms. There appears to be a deeper or more widely
encompassing cache at work here.
Jim
On Thu, Aug 1, 2013 at 2:49 PM, Mikhail Khludnev [via Lucene] <
ml-node+s472066n4082044h80@n3.nabble.com> wrote:
> Hello Jim,
>
> Does q={!cache=false}lorem ipsum works for you?
>
>
> On Thu, Aug 1, 2013 at 9:12 PM, jimtronic <[hidden email]<http://user/SendEmail.jtp?type=node&node=4082044&i=0>>
> wrote:
>
> > I have a query that runs slow occasionally. I'm having trouble debugging
> it
> > because once it's cached, it runs fast -- under 10 ms. But throughout
> the
> > day it occasionally takes up to 3 secs. It seems like it could be one of
> > the
> > following:
> >
> > 1. My autoCommit (300000 and openSearcher=false) and softAutoCommit
> (10000)
> > settings
> > 2. Something to do with distributed search -- There are three nodes, but
> > only 1 shard each.
> > 3. Just a slow query that is getting blown out of cache periodically
> >
> > This is in Solr 4.2.
> >
> > I like that it runs fast when cached, but if it's going to be blown out
> > quickly, then I'd really like to just optimize the query to run fast
> > uncached.
> >
> > *Is there any way to run a query using no caching whatsoever?*
> >
> > The query changes, but has *:* for the q param and 4 fq parameters. It's
> > also trying to do field collapsing.
> >
> > Jim
> >
> >
> >
> >
> > --
> > View this message in context:
> >
> http://lucene.472066.n3.nabble.com/How-to-uncache-a-query-to-debug-tp4082010.html
> > Sent from the Solr - User mailing list archive at Nabble.com.
> >
>
>
>
> --
> Sincerely yours
> Mikhail Khludnev
> Principal Engineer,
> Grid Dynamics
>
> <http://www.griddynamics.com>
> <[hidden email] <http://user/SendEmail.jtp?type=node&node=4082044&i=1>>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://lucene.472066.n3.nabble.com/How-to-uncache-a-query-to-debug-tp4082010p4082044.html
> To unsubscribe from How to uncache a query to debug?, click here<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4082010&code=amltdHJvbmljQGdtYWlsLmNvbXw0MDgyMDEwfDEzMjQ4NDk0MTQ=>
> .
> NAML<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://lucene.472066.n3.nabble.com/How-to-uncache-a-query-to-debug-tp4082010p4082046.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: How to uncache a query to debug?
Posted by Mikhail Khludnev <mk...@griddynamics.com>.
Hello Jim,
Does q={!cache=false}lorem ipsum works for you?
On Thu, Aug 1, 2013 at 9:12 PM, jimtronic <ji...@gmail.com> wrote:
> I have a query that runs slow occasionally. I'm having trouble debugging it
> because once it's cached, it runs fast -- under 10 ms. But throughout the
> day it occasionally takes up to 3 secs. It seems like it could be one of
> the
> following:
>
> 1. My autoCommit (300000 and openSearcher=false) and softAutoCommit (10000)
> settings
> 2. Something to do with distributed search -- There are three nodes, but
> only 1 shard each.
> 3. Just a slow query that is getting blown out of cache periodically
>
> This is in Solr 4.2.
>
> I like that it runs fast when cached, but if it's going to be blown out
> quickly, then I'd really like to just optimize the query to run fast
> uncached.
>
> *Is there any way to run a query using no caching whatsoever?*
>
> The query changes, but has *:* for the q param and 4 fq parameters. It's
> also trying to do field collapsing.
>
> Jim
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/How-to-uncache-a-query-to-debug-tp4082010.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
--
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics
<http://www.griddynamics.com>
<mk...@griddynamics.com>
Re: How to uncache a query to debug?
Posted by Yonik Seeley <yo...@lucidworks.com>.
On Fri, Aug 2, 2013 at 7:34 AM, Erick Erickson <er...@gmail.com> wrote:
> My guess is you're hitting the first time something is done, like sorting
> or some such
> occasionally on these queries by chance when a new searcher is opened.
And actually, if it was something like sorting (populating a field
cache), then it may only get slow after a new searcher is open *after*
a big merge (since sorting and functions are generally per-segment).
-Yonik
http://lucidworks.com
Re: How to uncache a query to debug?
Posted by Erick Erickson <er...@gmail.com>.
You should be able to mine the logs for the query, as:
webapp=/solr path=/select params={q=manu:samsung} hits=1 status=0 QTime=3
Admittedly you have to do some reconstruction, but it's there. From there,
you should
be able to run the query against a warmed Solr (but not one that has had
that particular
query run before).
My guess is you're hitting the first time something is done, like sorting
or some such
occasionally on these queries by chance when a new searcher is opened.
Best
Erick
On Thu, Aug 1, 2013 at 4:06 PM, jimtronic <ji...@gmail.com> wrote:
> Thanks. I'd rather not turn off caching completely because it only seems to
> show up in production and I don't want to turn reboot all the solr
> processes on each node.
>
> Jim
>
> On Thu, Aug 1, 2013 at 12:30 PM, Roman Chyla [via Lucene] <
> ml-node+s472066n4082014h50@n3.nabble.com> wrote:
>
> > When you set your cache (solrconfig.xml) to size=0, you are not using a
> > cache. so you can debug more easily
> >
> > roman
> >
> >
> > On Thu, Aug 1, 2013 at 1:12 PM, jimtronic <[hidden email]<
> http://user/SendEmail.jtp?type=node&node=4082014&i=0>>
> > wrote:
> >
> > > I have a query that runs slow occasionally. I'm having trouble
> debugging
> > it
> > > because once it's cached, it runs fast -- under 10 ms. But throughout
> > the
> > > day it occasionally takes up to 3 secs. It seems like it could be one
> of
> > > the
> > > following:
> > >
> > > 1. My autoCommit (300000 and openSearcher=false) and softAutoCommit
> > (10000)
> > > settings
> > > 2. Something to do with distributed search -- There are three nodes,
> but
> > > only 1 shard each.
> > > 3. Just a slow query that is getting blown out of cache periodically
> > >
> > > This is in Solr 4.2.
> > >
> > > I like that it runs fast when cached, but if it's going to be blown out
> > > quickly, then I'd really like to just optimize the query to run fast
> > > uncached.
> > >
> > > *Is there any way to run a query using no caching whatsoever?*
> > >
> > > The query changes, but has *:* for the q param and 4 fq parameters.
> It's
> > > also trying to do field collapsing.
> > >
> > > Jim
> > >
> > >
> > >
> > >
> > > --
> > > View this message in context:
> > >
> >
> http://lucene.472066.n3.nabble.com/How-to-uncache-a-query-to-debug-tp4082010.html
> > > Sent from the Solr - User mailing list archive at Nabble.com.
> > >
> >
> >
> > ------------------------------
> > If you reply to this email, your message will be added to the discussion
> > below:
> >
> >
> http://lucene.472066.n3.nabble.com/How-to-uncache-a-query-to-debug-tp4082010p4082014.html
> > To unsubscribe from How to uncache a query to debug?, click here<
> http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4082010&code=amltdHJvbmljQGdtYWlsLmNvbXw0MDgyMDEwfDEzMjQ4NDk0MTQ=
> >
> > .
> > NAML<
> http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
> >
> >
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/How-to-uncache-a-query-to-debug-tp4082010p4082047.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: How to uncache a query to debug?
Posted by jimtronic <ji...@gmail.com>.
Thanks. I'd rather not turn off caching completely because it only seems to
show up in production and I don't want to turn reboot all the solr
processes on each node.
Jim
On Thu, Aug 1, 2013 at 12:30 PM, Roman Chyla [via Lucene] <
ml-node+s472066n4082014h50@n3.nabble.com> wrote:
> When you set your cache (solrconfig.xml) to size=0, you are not using a
> cache. so you can debug more easily
>
> roman
>
>
> On Thu, Aug 1, 2013 at 1:12 PM, jimtronic <[hidden email]<http://user/SendEmail.jtp?type=node&node=4082014&i=0>>
> wrote:
>
> > I have a query that runs slow occasionally. I'm having trouble debugging
> it
> > because once it's cached, it runs fast -- under 10 ms. But throughout
> the
> > day it occasionally takes up to 3 secs. It seems like it could be one of
> > the
> > following:
> >
> > 1. My autoCommit (300000 and openSearcher=false) and softAutoCommit
> (10000)
> > settings
> > 2. Something to do with distributed search -- There are three nodes, but
> > only 1 shard each.
> > 3. Just a slow query that is getting blown out of cache periodically
> >
> > This is in Solr 4.2.
> >
> > I like that it runs fast when cached, but if it's going to be blown out
> > quickly, then I'd really like to just optimize the query to run fast
> > uncached.
> >
> > *Is there any way to run a query using no caching whatsoever?*
> >
> > The query changes, but has *:* for the q param and 4 fq parameters. It's
> > also trying to do field collapsing.
> >
> > Jim
> >
> >
> >
> >
> > --
> > View this message in context:
> >
> http://lucene.472066.n3.nabble.com/How-to-uncache-a-query-to-debug-tp4082010.html
> > Sent from the Solr - User mailing list archive at Nabble.com.
> >
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://lucene.472066.n3.nabble.com/How-to-uncache-a-query-to-debug-tp4082010p4082014.html
> To unsubscribe from How to uncache a query to debug?, click here<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4082010&code=amltdHJvbmljQGdtYWlsLmNvbXw0MDgyMDEwfDEzMjQ4NDk0MTQ=>
> .
> NAML<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://lucene.472066.n3.nabble.com/How-to-uncache-a-query-to-debug-tp4082010p4082047.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: How to uncache a query to debug?
Posted by Roman Chyla <ro...@gmail.com>.
When you set your cache (solrconfig.xml) to size=0, you are not using a
cache. so you can debug more easily
roman
On Thu, Aug 1, 2013 at 1:12 PM, jimtronic <ji...@gmail.com> wrote:
> I have a query that runs slow occasionally. I'm having trouble debugging it
> because once it's cached, it runs fast -- under 10 ms. But throughout the
> day it occasionally takes up to 3 secs. It seems like it could be one of
> the
> following:
>
> 1. My autoCommit (300000 and openSearcher=false) and softAutoCommit (10000)
> settings
> 2. Something to do with distributed search -- There are three nodes, but
> only 1 shard each.
> 3. Just a slow query that is getting blown out of cache periodically
>
> This is in Solr 4.2.
>
> I like that it runs fast when cached, but if it's going to be blown out
> quickly, then I'd really like to just optimize the query to run fast
> uncached.
>
> *Is there any way to run a query using no caching whatsoever?*
>
> The query changes, but has *:* for the q param and 4 fq parameters. It's
> also trying to do field collapsing.
>
> Jim
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/How-to-uncache-a-query-to-debug-tp4082010.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>