You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by oleg_gnatovskiy <ol...@citysearch.com> on 2009/01/22 19:46:58 UTC
Random queries extremely slow
Hello. Our production servers are operating relatively smoothly most of the
time running Solr with 19 million listings. However every once in a while
the same query that used to take 100 miliseconds takes 6000. This causes out
health check to fail, and the server is taken out of service. Once the
server is put back in service, queries are back to their regular response
times. Is there anything we could do to stop this random slowness from
occurring?
--
View this message in context: http://www.nabble.com/Random-queries-extremely-slow-tp21610568p21610568.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Random queries extremely slow
Posted by oleg_gnatovskiy <ol...@citysearch.com>.
Actually my issue might merit a seperate discussion as I did tuning by
adjusting the heap to different settings to see how it affected changed. It
really had no affect, as with jdk 1.6, garbage collection is parallel which
now should no longer interfere with requests during garbage collection which
holds true based on the tests we ran.
oleg_gnatovskiy wrote:
>
> My aplogies, this is likely the same issue as "Intermittent high response
> times by hbi dev "
>
>
>
> oleg_gnatovskiy wrote:
>>
>> Hello. Our production servers are operating relatively smoothly most of
>> the time running Solr with 19 million listings. However every once in a
>> while the same query that used to take 100 miliseconds takes 6000. This
>> causes out health check to fail, and the server is taken out of service.
>> Once the server is put back in service, queries are back to their regular
>> response times. Is there anything we could do to stop this random
>> slowness from occurring?
>>
>
>
--
View this message in context: http://www.nabble.com/Random-queries-extremely-slow-tp21610568p21610972.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Random queries extremely slow
Posted by oleg_gnatovskiy <ol...@citysearch.com>.
My aplogies, this is likely the same issue as "Intermittent high response
times by hbi dev "
oleg_gnatovskiy wrote:
>
> Hello. Our production servers are operating relatively smoothly most of
> the time running Solr with 19 million listings. However every once in a
> while the same query that used to take 100 miliseconds takes 6000. This
> causes out health check to fail, and the server is taken out of service.
> Once the server is put back in service, queries are back to their regular
> response times. Is there anything we could do to stop this random slowness
> from occurring?
>
--
View this message in context: http://www.nabble.com/Random-queries-extremely-slow-tp21610568p21610660.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Random queries extremely slow
Posted by oleg_gnatovskiy <ol...@citysearch.com>.
It was actually our use of the field collapse patch. Once we disabled this
the random slow queries went away.
We also added *:* as a warmup query in order to speed up performance after
indexing.
sunnyfr wrote:
>
> Hi Oleg
>
> Did you find a way to pass over this issue ??
> thanks a lot,
>
>
> oleg_gnatovskiy wrote:
>>
>> Can you expand on this? Mirroring delay on what?
>>
>>
>>
>> zayhen wrote:
>>>
>>> Use multiple boxes, with a mirroring delaay from one to another, like a
>>> pipeline.
>>>
>>> 2009/1/22 oleg_gnatovskiy <ol...@citysearch.com>
>>>
>>>>
>>>> Well this probably isn't the cause of our random slow queries, but
>>>> might be
>>>> the cause of the slow queries after pulling a new index. Is there
>>>> anything
>>>> we could do to reduce the performance hit we take from this happening?
>>>>
>>>>
>>>>
>>>> Otis Gospodnetic wrote:
>>>> >
>>>> > Here is one example: pushing a large newly optimized index onto the
>>>> > server.
>>>> >
>>>> > Otis
>>>> > --
>>>> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>>>> >
>>>> >
>>>> >
>>>> > ----- Original Message ----
>>>> >> From: oleg_gnatovskiy <ol...@citysearch.com>
>>>> >> To: solr-user@lucene.apache.org
>>>> >> Sent: Thursday, January 22, 2009 2:22:51 PM
>>>> >> Subject: Re: Random queries extremely slow
>>>> >>
>>>> >>
>>>> >> What are some things that could happen to force files out of the
>>>> cache
>>>> on
>>>> >> a
>>>> >> Linux machine? I don't know what kinds of events to look for...
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >> yonik wrote:
>>>> >> >
>>>> >> > On Thu, Jan 22, 2009 at 1:46 PM, oleg_gnatovskiy
>>>> >> > wrote:
>>>> >> >> Hello. Our production servers are operating relatively smoothly
>>>> most
>>>> >> of
>>>> >> >> the
>>>> >> >> time running Solr with 19 million listings. However every once in
>>>> a
>>>> >> while
>>>> >> >> the same query that used to take 100 miliseconds takes 6000.
>>>> >> >
>>>> >> > Anything else happening on the system that may have forced some of
>>>> the
>>>> >> > index files out of operating system disk cache at these times?
>>>> >> >
>>>> >> > -Yonik
>>>> >> >
>>>> >> >
>>>> >>
>>>> >> --
>>>> >> View this message in context:
>>>> >>
>>>> http://www.nabble.com/Random-queries-extremely-slow-tp21610568p21611240.html
>>>> >> Sent from the Solr - User mailing list archive at Nabble.com.
>>>> >
>>>> >
>>>> >
>>>>
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/Random-queries-extremely-slow-tp21610568p21611454.html
>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>> --
>>> Alexander Ramos Jardim
>>>
>>>
>>> -----
>>> RPG da Ilha
>>>
>>
>>
>
>
--
View this message in context: http://www.nabble.com/Random-queries-extremely-slow-tp21610568p23043152.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Random queries extremely slow
Posted by sunnyfr <jo...@gmail.com>.
Hi Oleg
Did you find a way to pass over this issue ??
thanks a lot,
oleg_gnatovskiy wrote:
>
> Can you expand on this? Mirroring delay on what?
>
>
>
> zayhen wrote:
>>
>> Use multiple boxes, with a mirroring delaay from one to another, like a
>> pipeline.
>>
>> 2009/1/22 oleg_gnatovskiy <ol...@citysearch.com>
>>
>>>
>>> Well this probably isn't the cause of our random slow queries, but might
>>> be
>>> the cause of the slow queries after pulling a new index. Is there
>>> anything
>>> we could do to reduce the performance hit we take from this happening?
>>>
>>>
>>>
>>> Otis Gospodnetic wrote:
>>> >
>>> > Here is one example: pushing a large newly optimized index onto the
>>> > server.
>>> >
>>> > Otis
>>> > --
>>> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>>> >
>>> >
>>> >
>>> > ----- Original Message ----
>>> >> From: oleg_gnatovskiy <ol...@citysearch.com>
>>> >> To: solr-user@lucene.apache.org
>>> >> Sent: Thursday, January 22, 2009 2:22:51 PM
>>> >> Subject: Re: Random queries extremely slow
>>> >>
>>> >>
>>> >> What are some things that could happen to force files out of the
>>> cache
>>> on
>>> >> a
>>> >> Linux machine? I don't know what kinds of events to look for...
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> yonik wrote:
>>> >> >
>>> >> > On Thu, Jan 22, 2009 at 1:46 PM, oleg_gnatovskiy
>>> >> > wrote:
>>> >> >> Hello. Our production servers are operating relatively smoothly
>>> most
>>> >> of
>>> >> >> the
>>> >> >> time running Solr with 19 million listings. However every once in
>>> a
>>> >> while
>>> >> >> the same query that used to take 100 miliseconds takes 6000.
>>> >> >
>>> >> > Anything else happening on the system that may have forced some of
>>> the
>>> >> > index files out of operating system disk cache at these times?
>>> >> >
>>> >> > -Yonik
>>> >> >
>>> >> >
>>> >>
>>> >> --
>>> >> View this message in context:
>>> >>
>>> http://www.nabble.com/Random-queries-extremely-slow-tp21610568p21611240.html
>>> >> Sent from the Solr - User mailing list archive at Nabble.com.
>>> >
>>> >
>>> >
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Random-queries-extremely-slow-tp21610568p21611454.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>> --
>> Alexander Ramos Jardim
>>
>>
>> -----
>> RPG da Ilha
>>
>
>
--
View this message in context: http://www.nabble.com/Random-queries-extremely-slow-tp21610568p23039151.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Random queries extremely slow
Posted by oleg_gnatovskiy <ol...@citysearch.com>.
Can you expand on this? Mirroring delay on what?
zayhen wrote:
>
> Use multiple boxes, with a mirroring delaay from one to another, like a
> pipeline.
>
> 2009/1/22 oleg_gnatovskiy <ol...@citysearch.com>
>
>>
>> Well this probably isn't the cause of our random slow queries, but might
>> be
>> the cause of the slow queries after pulling a new index. Is there
>> anything
>> we could do to reduce the performance hit we take from this happening?
>>
>>
>>
>> Otis Gospodnetic wrote:
>> >
>> > Here is one example: pushing a large newly optimized index onto the
>> > server.
>> >
>> > Otis
>> > --
>> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>> >
>> >
>> >
>> > ----- Original Message ----
>> >> From: oleg_gnatovskiy <ol...@citysearch.com>
>> >> To: solr-user@lucene.apache.org
>> >> Sent: Thursday, January 22, 2009 2:22:51 PM
>> >> Subject: Re: Random queries extremely slow
>> >>
>> >>
>> >> What are some things that could happen to force files out of the cache
>> on
>> >> a
>> >> Linux machine? I don't know what kinds of events to look for...
>> >>
>> >>
>> >>
>> >>
>> >> yonik wrote:
>> >> >
>> >> > On Thu, Jan 22, 2009 at 1:46 PM, oleg_gnatovskiy
>> >> > wrote:
>> >> >> Hello. Our production servers are operating relatively smoothly
>> most
>> >> of
>> >> >> the
>> >> >> time running Solr with 19 million listings. However every once in a
>> >> while
>> >> >> the same query that used to take 100 miliseconds takes 6000.
>> >> >
>> >> > Anything else happening on the system that may have forced some of
>> the
>> >> > index files out of operating system disk cache at these times?
>> >> >
>> >> > -Yonik
>> >> >
>> >> >
>> >>
>> >> --
>> >> View this message in context:
>> >>
>> http://www.nabble.com/Random-queries-extremely-slow-tp21610568p21611240.html
>> >> Sent from the Solr - User mailing list archive at Nabble.com.
>> >
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Random-queries-extremely-slow-tp21610568p21611454.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> Alexander Ramos Jardim
>
>
> -----
> RPG da Ilha
>
--
View this message in context: http://www.nabble.com/Random-queries-extremely-slow-tp21610568p21670023.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Random queries extremely slow
Posted by Alexander Ramos Jardim <al...@gmail.com>.
Use multiple boxes, with a mirroring delaay from one to another, like a
pipeline.
2009/1/22 oleg_gnatovskiy <ol...@citysearch.com>
>
> Well this probably isn't the cause of our random slow queries, but might be
> the cause of the slow queries after pulling a new index. Is there anything
> we could do to reduce the performance hit we take from this happening?
>
>
>
> Otis Gospodnetic wrote:
> >
> > Here is one example: pushing a large newly optimized index onto the
> > server.
> >
> > Otis
> > --
> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
> >
> >
> >
> > ----- Original Message ----
> >> From: oleg_gnatovskiy <ol...@citysearch.com>
> >> To: solr-user@lucene.apache.org
> >> Sent: Thursday, January 22, 2009 2:22:51 PM
> >> Subject: Re: Random queries extremely slow
> >>
> >>
> >> What are some things that could happen to force files out of the cache
> on
> >> a
> >> Linux machine? I don't know what kinds of events to look for...
> >>
> >>
> >>
> >>
> >> yonik wrote:
> >> >
> >> > On Thu, Jan 22, 2009 at 1:46 PM, oleg_gnatovskiy
> >> > wrote:
> >> >> Hello. Our production servers are operating relatively smoothly most
> >> of
> >> >> the
> >> >> time running Solr with 19 million listings. However every once in a
> >> while
> >> >> the same query that used to take 100 miliseconds takes 6000.
> >> >
> >> > Anything else happening on the system that may have forced some of the
> >> > index files out of operating system disk cache at these times?
> >> >
> >> > -Yonik
> >> >
> >> >
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/Random-queries-extremely-slow-tp21610568p21611240.html
> >> Sent from the Solr - User mailing list archive at Nabble.com.
> >
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Random-queries-extremely-slow-tp21610568p21611454.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
Alexander Ramos Jardim
Re: Random queries extremely slow
Posted by oleg_gnatovskiy <ol...@citysearch.com>.
Well this probably isn't the cause of our random slow queries, but might be
the cause of the slow queries after pulling a new index. Is there anything
we could do to reduce the performance hit we take from this happening?
Otis Gospodnetic wrote:
>
> Here is one example: pushing a large newly optimized index onto the
> server.
>
> Otis
> --
> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>
>
>
> ----- Original Message ----
>> From: oleg_gnatovskiy <ol...@citysearch.com>
>> To: solr-user@lucene.apache.org
>> Sent: Thursday, January 22, 2009 2:22:51 PM
>> Subject: Re: Random queries extremely slow
>>
>>
>> What are some things that could happen to force files out of the cache on
>> a
>> Linux machine? I don't know what kinds of events to look for...
>>
>>
>>
>>
>> yonik wrote:
>> >
>> > On Thu, Jan 22, 2009 at 1:46 PM, oleg_gnatovskiy
>> > wrote:
>> >> Hello. Our production servers are operating relatively smoothly most
>> of
>> >> the
>> >> time running Solr with 19 million listings. However every once in a
>> while
>> >> the same query that used to take 100 miliseconds takes 6000.
>> >
>> > Anything else happening on the system that may have forced some of the
>> > index files out of operating system disk cache at these times?
>> >
>> > -Yonik
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Random-queries-extremely-slow-tp21610568p21611240.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
>
--
View this message in context: http://www.nabble.com/Random-queries-extremely-slow-tp21610568p21611454.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Random queries extremely slow
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Here is one example: pushing a large newly optimized index onto the server.
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
> From: oleg_gnatovskiy <ol...@citysearch.com>
> To: solr-user@lucene.apache.org
> Sent: Thursday, January 22, 2009 2:22:51 PM
> Subject: Re: Random queries extremely slow
>
>
> What are some things that could happen to force files out of the cache on a
> Linux machine? I don't know what kinds of events to look for...
>
>
>
>
> yonik wrote:
> >
> > On Thu, Jan 22, 2009 at 1:46 PM, oleg_gnatovskiy
> > wrote:
> >> Hello. Our production servers are operating relatively smoothly most of
> >> the
> >> time running Solr with 19 million listings. However every once in a while
> >> the same query that used to take 100 miliseconds takes 6000.
> >
> > Anything else happening on the system that may have forced some of the
> > index files out of operating system disk cache at these times?
> >
> > -Yonik
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Random-queries-extremely-slow-tp21610568p21611240.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Random queries extremely slow
Posted by Walter Underwood <wu...@netflix.com>.
The OS keeps recently accessed disk pages in memory. If another
process does a lot of disk access, like a backup, the OS might
replace the Solr index pages with that processes pages.
What kind of storage: local disk, SAN, NFS?
wunder
On 1/22/09 11:22 AM, "oleg_gnatovskiy" <ol...@citysearch.com>
wrote:
>
> What are some things that could happen to force files out of the cache on a
> Linux machine? I don't know what kinds of events to look for...
>
>
>
>
> yonik wrote:
>>
>> On Thu, Jan 22, 2009 at 1:46 PM, oleg_gnatovskiy
>> <ol...@citysearch.com> wrote:
>>> Hello. Our production servers are operating relatively smoothly most of
>>> the
>>> time running Solr with 19 million listings. However every once in a while
>>> the same query that used to take 100 miliseconds takes 6000.
>>
>> Anything else happening on the system that may have forced some of the
>> index files out of operating system disk cache at these times?
>>
>> -Yonik
>>
>>
Re: Random queries extremely slow
Posted by oleg_gnatovskiy <ol...@citysearch.com>.
What are some things that could happen to force files out of the cache on a
Linux machine? I don't know what kinds of events to look for...
yonik wrote:
>
> On Thu, Jan 22, 2009 at 1:46 PM, oleg_gnatovskiy
> <ol...@citysearch.com> wrote:
>> Hello. Our production servers are operating relatively smoothly most of
>> the
>> time running Solr with 19 million listings. However every once in a while
>> the same query that used to take 100 miliseconds takes 6000.
>
> Anything else happening on the system that may have forced some of the
> index files out of operating system disk cache at these times?
>
> -Yonik
>
>
--
View this message in context: http://www.nabble.com/Random-queries-extremely-slow-tp21610568p21611240.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Random queries extremely slow
Posted by Yonik Seeley <ys...@gmail.com>.
On Thu, Jan 22, 2009 at 1:46 PM, oleg_gnatovskiy
<ol...@citysearch.com> wrote:
> Hello. Our production servers are operating relatively smoothly most of the
> time running Solr with 19 million listings. However every once in a while
> the same query that used to take 100 miliseconds takes 6000.
Anything else happening on the system that may have forced some of the
index files out of operating system disk cache at these times?
-Yonik