debugging slowness

[Brian Whitman <br...@variogr.am>]

We've got periods where direct solr queries take 35s; on resin reboot  
they go back to taking 500ms. This is on a 16GB ram 4-core server. We  
want to debug this but don't know where to start. Is there some sample- 
type command we can run during queries to figure out where Solr/Lucene  
is spending its time (ala MySQL's slow query log)?

Re: debugging slowness

[Yonik Seeley <yo...@apache.org>]

On Dec 19, 2007 9:48 AM, Brian Whitman <br...@variogr.am> wrote:
>
> On Dec 19, 2007, at 9:34 AM, Yonik Seeley wrote:
>
> > On Dec 19, 2007 9:23 AM, Brian Whitman <br...@variogr.am>
> > wrote:
> >> It's pretty easy to parse access logs or solr query logs and find the
> > offenders (I don't have any parsing code at my fingertips though...)
> >
> > Do these periods of slowness tend to happen right after a commit?
> > Do the slow queries have anything in common?
>
> All queries become slow, so looking at the access logs don't help us
> much.

You could try thread dumps (kill -3, or use the solr admin page)
The latest devel versions of solr also break out time by component if
you give debugQuery=true

> After some amount of time (variable, but measured in days) Solr
> just becomes slow for us, every query takes 20-30s.

This is strange (if a reboot fixes it).
Anything odd in the logs?  overlapping searchers?  search for
"WARNING" in the logs.
Also check to make sure it's not a GC... that could cause everything
to start slowing down when you start running out of memory.

-Yonik

Re: debugging slowness

[Brian Whitman <br...@variogr.am>]

On Dec 19, 2007, at 9:34 AM, Yonik Seeley wrote:

> On Dec 19, 2007 9:23 AM, Brian Whitman <br...@variogr.am>  
> wrote:
>> It's pretty easy to parse access logs or solr query logs and find the
> offenders (I don't have any parsing code at my fingertips though...)
>
> Do these periods of slowness tend to happen right after a commit?
> Do the slow queries have anything in common?

All queries become slow, so looking at the access logs don't help us  
much. After some amount of time (variable, but measured in days) Solr  
just becomes slow for us, every query takes 20-30s. I was looking for  
more code-level sampling/logging (to see if I could fix it.) Or at  
least see if it's a resin thing vs. a solr thing.

The index in question has a 15m autocommit and gets about 1-3 new  
documents/second.

Re: debugging slowness

[Yonik Seeley <yo...@apache.org>]

On Dec 19, 2007 9:23 AM, Brian Whitman <br...@variogr.am> wrote:
> We've got periods where direct solr queries take 35s; on resin reboot
> they go back to taking 500ms. This is on a 16GB ram 4-core server. We
> want to debug this but don't know where to start. Is there some sample-
> type command we can run during queries to figure out where Solr/Lucene
> is spending its time (ala MySQL's slow query log)?

It's pretty easy to parse access logs or solr query logs and find the
offenders (I don't have any parsing code at my fingertips though...)

Do these periods of slowness tend to happen right after a commit?
Do the slow queries have anything in common?

-Yonik