You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by oleg_gnatovskiy <ol...@citysearch.com> on 2008/11/12 03:31:17 UTC
Query Performance while updating the index
Hello. We have an index with 15 million documents working on a distributed
environment, with an index distribution setup. While an index on a slave
server is being updated, query response times become extremely slow (upwards
of 5 seconds). Is there any way to decrease the hit query response times
take while an index is being pushed?
Thanks!
Oleg
--
View this message in context: http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p20452835.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query Performance while updating the index
Posted by Otis Gospodnetic <ot...@yahoo.com>.
I'll assume you are using the rsync-based replication, not SOLR-561.
In that case, bwlimit is your friend. You'll have to modify the scripts and make use of this:
--bwlimit=KBPS
This option allows you to specify a maximum transfer rate in kilobytes per second. This option is most effective
when using rsync with large files (several megabytes and up). Due to the nature of rsync transfers, blocks of data
are sent, then if rsync determines the transfer was too fast, it will wait before sending the next data block. The
result is an average transfer rate equaling the specified limit. A value of zero specifies no limit.
Actually, this is something that we could add to snappuller in Solr, no?
May be worth opening a new JIRA issue for this. Eh, attaching a modified but completely untested version of snappuller. Please reply if it works or if you have to fix it please post it to JIRA.
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
________________________________
From: oleg_gnatovskiy <ol...@citysearch.com>
To: solr-user@lucene.apache.org
Sent: Tuesday, November 11, 2008 9:31:17 PM
Subject: Query Performance while updating the index
Hello. We have an index with 15 million documents working on a distributed
environment, with an index distribution setup. While an index on a slave
server is being updated, query response times become extremely slow (upwards
of 5 seconds). Is there any way to decrease the hit query response times
take while an index is being pushed?
Thanks!
Oleg
--
View this message in context: http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p20452835.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query Performance while updating teh index
Posted by Chris Hostetter <ho...@fucit.org>.
: Just to clarify - we do not optimize on the slaves at all. We only optimize
: on the master.
that doesn't change anything about hte comments that i made before. it
*really* wouldn't make sense to optimize on a slave right before pulling a
new snapshot, but it still doesn't make any more sense to optimize on a
master right before doing some updates and then pulling a new snapshot.
my second comment also still applies: a snappull after an optimize is
always going to be involve more churn on the disk...
: > : We do optimize the index before updates but we get tehse performance
: > issues
: > : even when we pull an empty snapshot. Thus even when our update is tiny,
: > the
: > : performance issues still happen.
: >
: > FWIW: this behavior doesn't make a lot of sense -- optimizing just
: > before you are about to make updates/additions ot your data, is a complete
: > waste. the main value in optimizing your index is that you have one
: > segment, as soon as you add a docment that changes.
: >
: > the other thing to keep in mind is that an optimized index is a completley
: > new segment as a new file with a new name, so there is going to be added
: > overhead on the slave machines as the OS purges the old index files and
: > replaces them with the new optimized index files -- more overhead then if
: > you had just done your additions w/o optimizing first.
-Hoss
Re: Query Performance while updating teh index
Posted by oleg_gnatovskiy <ol...@citysearch.com>.
Just to clarify - we do not optimize on the slaves at all. We only optimize
on the master.
hossman wrote:
>
>
> : We do optimize the index before updates but we get tehse performance
> issues
> : even when we pull an empty snapshot. Thus even when our update is tiny,
> the
> : performance issues still happen.
>
> FWIW: this behavior doesn't make a lot of sense -- optimizing just
> before you are about to make updates/additions ot your data, is a complete
> waste. the main value in optimizing your index is that you have one
> segment, as soon as you add a docment that changes.
>
> the other thing to keep in mind is that an optimized index is a completley
> new segment as a new file with a new name, so there is going to be added
> overhead on the slave machines as the OS purges the old index files and
> replaces them with the new optimized index files -- more overhead then if
> you had just done your additions w/o optimizing first.
>
>
>
> -Hoss
>
>
>
--
View this message in context: http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p21678267.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query Performance while updating teh index
Posted by oleg_gnatovskiy <ol...@citysearch.com>.
Just to calrify - we do not optimize on teh slaves at all. We only optimize
on the master.
hossman wrote:
>
>
> : We do optimize the index before updates but we get tehse performance
> issues
> : even when we pull an empty snapshot. Thus even when our update is tiny,
> the
> : performance issues still happen.
>
> FWIW: this behavior doesn't make a lot of sense -- optimizing just
> before you are about to make updates/additions ot your data, is a complete
> waste. the main value in optimizing your index is that you have one
> segment, as soon as you add a docment that changes.
>
> the other thing to keep in mind is that an optimized index is a completley
> new segment as a new file with a new name, so there is going to be added
> overhead on the slave machines as the OS purges the old index files and
> replaces them with the new optimized index files -- more overhead then if
> you had just done your additions w/o optimizing first.
>
>
>
> -Hoss
>
>
>
--
View this message in context: http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p21678261.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query Performance while updating teh index
Posted by oleg_gnatovskiy <ol...@citysearch.com>.
Just to calrify - we do not optimize on the slaves at all. We only optimize
on the master.
hossman wrote:
>
>
> : We do optimize the index before updates but we get tehse performance
> issues
> : even when we pull an empty snapshot. Thus even when our update is tiny,
> the
> : performance issues still happen.
>
> FWIW: this behavior doesn't make a lot of sense -- optimizing just
> before you are about to make updates/additions ot your data, is a complete
> waste. the main value in optimizing your index is that you have one
> segment, as soon as you add a docment that changes.
>
> the other thing to keep in mind is that an optimized index is a completley
> new segment as a new file with a new name, so there is going to be added
> overhead on the slave machines as the OS purges the old index files and
> replaces them with the new optimized index files -- more overhead then if
> you had just done your additions w/o optimizing first.
>
>
>
> -Hoss
>
>
>
--
View this message in context: http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p21678265.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query Performance while updating teh index
Posted by Chris Hostetter <ho...@fucit.org>.
: We do optimize the index before updates but we get tehse performance issues
: even when we pull an empty snapshot. Thus even when our update is tiny, the
: performance issues still happen.
FWIW: this behavior doesn't make a lot of sense -- optimizing just
before you are about to make updates/additions ot your data, is a complete
waste. the main value in optimizing your index is that you have one
segment, as soon as you add a docment that changes.
the other thing to keep in mind is that an optimized index is a completley
new segment as a new file with a new name, so there is going to be added
overhead on the slave machines as the OS purges the old index files and
replaces them with the new optimized index files -- more overhead then if
you had just done your additions w/o optimizing first.
-Hoss
Re: Query Performance while updating teh index
Posted by oleg_gnatovskiy <ol...@citysearch.com>.
We've tried it. There doesn't seem to be any connection between GC and the
bad performance spikes.
Otis Gospodnetic wrote:
>
> OK. Then it's likely not this. You saw the other response about looking
> at GC to see if maybe that hits you once in a while and slows whatever
> queries are in flight? Try jconsole.
>
> Otis
> --
> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>
>
>
> ----- Original Message ----
>> From: oleg_gnatovskiy <ol...@citysearch.com>
>> To: solr-user@lucene.apache.org
>> Sent: Thursday, January 22, 2009 2:43:31 PM
>> Subject: Re: Query Performance while updating teh index
>>
>>
>> We do optimize the index before updates but we get tehse performance
>> issues
>> even when we pull an empty snapshot. Thus even when our update is tiny,
>> the
>> performance issues still happen.
>>
>>
>>
>> Otis Gospodnetic wrote:
>> >
>> > This is an old and long thread, and I no longer recall what the
>> specific
>> > suggestions were.
>> > My guess is this has to do with the OS cache of your index files. When
>> > you make the large index update, that OS cache is useless (old files
>> are
>> > gone, new ones are in) and the OS cache has get re-warmed and this
>> takes
>> > time.
>> >
>> > Are you optimizing your index before the update? Do you *really* need
>> to
>> > do that?
>> > How large is your update, what makes it big, and could you make it
>> > smaller?
>> >
>> > Otis
>> > --
>> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>> >
>> >
>> >
>> > ----- Original Message ----
>> >> From: oleg_gnatovskiy
>> >> To: solr-user@lucene.apache.org
>> >> Sent: Tuesday, January 20, 2009 6:19:46 PM
>> >> Subject: Re: Query Performance while updating teh index
>> >>
>> >>
>> >> Hello again. It seems that we are still having these problems. Queries
>> >> take
>> >> as long as 20 minutes to get back to their average response time after
>> a
>> >> large index update, so it doesn't seem like the problem is the 12
>> second
>> >> autowarm time. Are there any more suggestions for things we can try?
>> >> Taking
>> >> our servers out of teh loop for as long as 20 minutes is a bit of a
>> >> hassle,
>> >> and a risk.
>> >> --
>> >> View this message in context:
>> >>
>> http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p21573927.html
>> >> Sent from the Solr - User mailing list archive at Nabble.com.
>> >
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p21611642.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
>
--
View this message in context: http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p21611976.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query Performance while updating teh index
Posted by Otis Gospodnetic <ot...@yahoo.com>.
OK. Then it's likely not this. You saw the other response about looking at GC to see if maybe that hits you once in a while and slows whatever queries are in flight? Try jconsole.
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
> From: oleg_gnatovskiy <ol...@citysearch.com>
> To: solr-user@lucene.apache.org
> Sent: Thursday, January 22, 2009 2:43:31 PM
> Subject: Re: Query Performance while updating teh index
>
>
> We do optimize the index before updates but we get tehse performance issues
> even when we pull an empty snapshot. Thus even when our update is tiny, the
> performance issues still happen.
>
>
>
> Otis Gospodnetic wrote:
> >
> > This is an old and long thread, and I no longer recall what the specific
> > suggestions were.
> > My guess is this has to do with the OS cache of your index files. When
> > you make the large index update, that OS cache is useless (old files are
> > gone, new ones are in) and the OS cache has get re-warmed and this takes
> > time.
> >
> > Are you optimizing your index before the update? Do you *really* need to
> > do that?
> > How large is your update, what makes it big, and could you make it
> > smaller?
> >
> > Otis
> > --
> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
> >
> >
> >
> > ----- Original Message ----
> >> From: oleg_gnatovskiy
> >> To: solr-user@lucene.apache.org
> >> Sent: Tuesday, January 20, 2009 6:19:46 PM
> >> Subject: Re: Query Performance while updating teh index
> >>
> >>
> >> Hello again. It seems that we are still having these problems. Queries
> >> take
> >> as long as 20 minutes to get back to their average response time after a
> >> large index update, so it doesn't seem like the problem is the 12 second
> >> autowarm time. Are there any more suggestions for things we can try?
> >> Taking
> >> our servers out of teh loop for as long as 20 minutes is a bit of a
> >> hassle,
> >> and a risk.
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p21573927.html
> >> Sent from the Solr - User mailing list archive at Nabble.com.
> >
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p21611642.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query Performance while updating teh index
Posted by oleg_gnatovskiy <ol...@citysearch.com>.
We do optimize the index before updates but we get tehse performance issues
even when we pull an empty snapshot. Thus even when our update is tiny, the
performance issues still happen.
Otis Gospodnetic wrote:
>
> This is an old and long thread, and I no longer recall what the specific
> suggestions were.
> My guess is this has to do with the OS cache of your index files. When
> you make the large index update, that OS cache is useless (old files are
> gone, new ones are in) and the OS cache has get re-warmed and this takes
> time.
>
> Are you optimizing your index before the update? Do you *really* need to
> do that?
> How large is your update, what makes it big, and could you make it
> smaller?
>
> Otis
> --
> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>
>
>
> ----- Original Message ----
>> From: oleg_gnatovskiy <ol...@citysearch.com>
>> To: solr-user@lucene.apache.org
>> Sent: Tuesday, January 20, 2009 6:19:46 PM
>> Subject: Re: Query Performance while updating teh index
>>
>>
>> Hello again. It seems that we are still having these problems. Queries
>> take
>> as long as 20 minutes to get back to their average response time after a
>> large index update, so it doesn't seem like the problem is the 12 second
>> autowarm time. Are there any more suggestions for things we can try?
>> Taking
>> our servers out of teh loop for as long as 20 minutes is a bit of a
>> hassle,
>> and a risk.
>> --
>> View this message in context:
>> http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p21573927.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
>
--
View this message in context: http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p21611642.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query Performance while updating teh index
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Oleg,
This is more of an OS-level thing that Solr-thing, it seems from your emails. If you send answers to my questions we'll be able to help more.
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
> From: oleg_gnatovskiy <ol...@citysearch.com>
> To: solr-user@lucene.apache.org
> Sent: Wednesday, January 21, 2009 1:09:21 PM
> Subject: Re: Query Performance while updating teh index
>
>
> What exactly does Solr do when it receives a new Index? How does it keep
> serving while performing the updates? It seems that the part that causes the
> slowdown is this transition.
>
>
>
>
> Otis Gospodnetic wrote:
> >
> > This is an old and long thread, and I no longer recall what the specific
> > suggestions were.
> > My guess is this has to do with the OS cache of your index files. When
> > you make the large index update, that OS cache is useless (old files are
> > gone, new ones are in) and the OS cache has get re-warmed and this takes
> > time.
> >
> > Are you optimizing your index before the update? Do you *really* need to
> > do that?
> > How large is your update, what makes it big, and could you make it
> > smaller?
> >
> > Otis
> > --
> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
> >
> >
> >
> > ----- Original Message ----
> >> From: oleg_gnatovskiy
> >> To: solr-user@lucene.apache.org
> >> Sent: Tuesday, January 20, 2009 6:19:46 PM
> >> Subject: Re: Query Performance while updating teh index
> >>
> >>
> >> Hello again. It seems that we are still having these problems. Queries
> >> take
> >> as long as 20 minutes to get back to their average response time after a
> >> large index update, so it doesn't seem like the problem is the 12 second
> >> autowarm time. Are there any more suggestions for things we can try?
> >> Taking
> >> our servers out of teh loop for as long as 20 minutes is a bit of a
> >> hassle,
> >> and a risk.
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p21573927.html
> >> Sent from the Solr - User mailing list archive at Nabble.com.
> >
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p21588779.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query Performance while updating teh index
Posted by oleg_gnatovskiy <ol...@citysearch.com>.
What exactly does Solr do when it receives a new Index? How does it keep
serving while performing the updates? It seems that the part that causes the
slowdown is this transition.
Otis Gospodnetic wrote:
>
> This is an old and long thread, and I no longer recall what the specific
> suggestions were.
> My guess is this has to do with the OS cache of your index files. When
> you make the large index update, that OS cache is useless (old files are
> gone, new ones are in) and the OS cache has get re-warmed and this takes
> time.
>
> Are you optimizing your index before the update? Do you *really* need to
> do that?
> How large is your update, what makes it big, and could you make it
> smaller?
>
> Otis
> --
> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>
>
>
> ----- Original Message ----
>> From: oleg_gnatovskiy <ol...@citysearch.com>
>> To: solr-user@lucene.apache.org
>> Sent: Tuesday, January 20, 2009 6:19:46 PM
>> Subject: Re: Query Performance while updating teh index
>>
>>
>> Hello again. It seems that we are still having these problems. Queries
>> take
>> as long as 20 minutes to get back to their average response time after a
>> large index update, so it doesn't seem like the problem is the 12 second
>> autowarm time. Are there any more suggestions for things we can try?
>> Taking
>> our servers out of teh loop for as long as 20 minutes is a bit of a
>> hassle,
>> and a risk.
>> --
>> View this message in context:
>> http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p21573927.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
>
--
View this message in context: http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p21588779.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query Performance while updating teh index
Posted by Otis Gospodnetic <ot...@yahoo.com>.
This is an old and long thread, and I no longer recall what the specific suggestions were.
My guess is this has to do with the OS cache of your index files. When you make the large index update, that OS cache is useless (old files are gone, new ones are in) and the OS cache has get re-warmed and this takes time.
Are you optimizing your index before the update? Do you *really* need to do that?
How large is your update, what makes it big, and could you make it smaller?
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
> From: oleg_gnatovskiy <ol...@citysearch.com>
> To: solr-user@lucene.apache.org
> Sent: Tuesday, January 20, 2009 6:19:46 PM
> Subject: Re: Query Performance while updating teh index
>
>
> Hello again. It seems that we are still having these problems. Queries take
> as long as 20 minutes to get back to their average response time after a
> large index update, so it doesn't seem like the problem is the 12 second
> autowarm time. Are there any more suggestions for things we can try? Taking
> our servers out of teh loop for as long as 20 minutes is a bit of a hassle,
> and a risk.
> --
> View this message in context:
> http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p21573927.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query Performance while updating teh index
Posted by oleg_gnatovskiy <ol...@citysearch.com>.
Hello again. It seems that we are still having these problems. Queries take
as long as 20 minutes to get back to their average response time after a
large index update, so it doesn't seem like the problem is the 12 second
autowarm time. Are there any more suggestions for things we can try? Taking
our servers out of teh loop for as long as 20 minutes is a bit of a hassle,
and a risk.
--
View this message in context: http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p21573927.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query Performance while updating teh index
Posted by Yonik Seeley <ys...@gmail.com>.
Right, query cache typically has a lower hit ratio, and one check per
request - often not worth autowarming much.
The filter cache can be a different story with a higher hit ratio, and
higher number of checks per request.
-Yonik
On Fri, Dec 12, 2008 at 1:35 PM, Feak, Todd <To...@smss.sony.com> wrote:
> It's spending 4-5 seconds warming up your query cache. If 4-5 seconds is
> too much, you could reduce the number of queries to auto-warm with on
> that cache.
>
> Notice that the 4-5 seconds is spent only putting about 420 queries into
> the query cache. Your autowarm of 50000 for the query cache seems a bit
> high. If you need to reduce that autowarm time below 5 seconds, you may
> have to set that value in the hundreds, as opposed to tens of thousands.
>
> -Todd Feak
>
> -----Original Message-----
> From: oleg_gnatovskiy [mailto:oleg_gnatovskiy@citysearch.com]
> Sent: Friday, December 12, 2008 10:08 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Query Performance while updating teh index
>
>
> Here's what we have on one of the data slaves for the autowarming.
>
>
>
> --
>
> Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
>
>
> filterCache{lookups=351993,hits=347055,hitratio=0.98,inserts=8332,evicti
> ons=0,size=8245,warmupTime=215,cumulative_lookups=2837676,cumulative_hit
> s=2766551,cumulative_hitratio=0.97,cumulative_inserts=72050,cumulative_e
> victions=0}
>
> Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming result for Searcher@3f32ca2b main
>
>
> filterCache{lookups=0,hits=0,hitratio=0.00,inserts=1000,evictions=0,size
> =1000,warmupTime=317,cumulative_lookups=2837676,cumulative_hits=2766551,
> cumulative_hitratio=0.97,cumulative_inserts=72050,cumulative_evictions=0
> }
>
> Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
>
>
> queryResultCache{lookups=5309,hits=5223,hitratio=0.98,inserts=422,evicti
> ons=0,size=421,warmupTime=4628,cumulative_lookups=77802,cumulative_hits=
> 77216,cumulative_hitratio=0.99,cumulative_inserts=424,cumulative_evictio
> ns=0}
>
> --
>
> Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming result for Searcher@3f32ca2b main
>
>
> queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=421,evictions=0,
> size=421,warmupTime=5536,cumulative_lookups=77804,cumulative_hits=77218,
> cumulative_hitratio=0.99,cumulative_inserts=424,cumulative_evictions=0}
>
> Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
>
>
> documentCache{lookups=87216,hits=86686,hitratio=0.99,inserts=570,evictio
> ns=0,size=570,warmupTime=0,cumulative_lookups=1270773,cumulative_hits=12
> 68318,cumulative_hitratio=0.99,cumulative_inserts=2455,cumulative_evicti
> ons=0}
>
> Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming result for Searcher@3f32ca2b main
>
>
> documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=
> 0,warmupTime=0,cumulative_lookups=1270773,cumulative_hits=1268318,cumula
> tive_hitratio=0.99,cumulative_inserts=2455,cumulative_evictions=0}
>
> --
>
>
>
> This is our current values after I've messed with them a few times
> trying to
> get better performance.
>
>
>
> <filterCache
>
> class="solr.LRUCache"
>
> size="30000"
>
> initialSize="15000"
>
> autowarmCount="1000"/>
>
> <queryResultCache
>
> class="solr.LRUCache"
>
> size="60000"
>
> initialSize="30000"
>
> autowarmCount="50000"/>
>
> <documentCache
>
> class="solr.LRUCache"
>
> size="200000"
>
> initialSize="125000"
>
> autowarmCount="0"/>
>
>
> --
> View this message in context:
> http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452
> 835p20980669.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
>
RE: Query Performance while updating teh index
Posted by oleg_gnatovskiy <ol...@citysearch.com>.
Should this autowarm value be set based on the number of lookups? From the
info I provided that like 60k. filterCache{lookups=58522
Will 25k be enough?
Also, does that mean that we have to at least increase the size and initial
size as big as we set the autowarm?
Feak, Todd wrote:
>
> Sorry, my bad. Didn't read the entire thread.
>
> Look at your filter cache first. You are autowarming 1000, and there is
> exactly 1000 in there. Yet it looks like there may be tens of thousands
> of filter queries in your system. I would try autowarming more. Try
> 10,000 or 20,000 and see if it helps.
>
> Second look at your document cache. Document caches don't use autowarm.
> But you can add queries to your firstSeacher and newSearcher entries in
> your solrconfig to pre-populate the document cache during warming.
>
> -Todd Feak
>
>
> -----Original Message-----
> From: oleg_gnatovskiy [mailto:oleg_gnatovskiy@citysearch.com]
> Sent: Friday, December 12, 2008 11:19 AM
> To: solr-user@lucene.apache.org
> Subject: RE: Query Performance while updating teh index
>
>
> The auto warm time is not an issue. We take the server off the load
> balancer
> while it is autowarming. It seems that the slowness occurs after
> autowarm is
> done.
>
>
>
> Feak, Todd wrote:
>>
>> It's spending 4-5 seconds warming up your query cache. If 4-5 seconds
> is
>> too much, you could reduce the number of queries to auto-warm with on
>> that cache.
>>
>> Notice that the 4-5 seconds is spent only putting about 420 queries
> into
>> the query cache. Your autowarm of 50000 for the query cache seems a
> bit
>> high. If you need to reduce that autowarm time below 5 seconds, you
> may
>> have to set that value in the hundreds, as opposed to tens of
> thousands.
>>
>> -Todd Feak
>>
>> -----Original Message-----
>> From: oleg_gnatovskiy [mailto:oleg_gnatovskiy@citysearch.com]
>> Sent: Friday, December 12, 2008 10:08 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Query Performance while updating teh index
>>
>>
>> Here's what we have on one of the data slaves for the autowarming.
>>
>>
>>
>> --
>>
>> Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
>>
>> INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
>>
>>
>>
> filterCache{lookups=351993,hits=347055,hitratio=0.98,inserts=8332,evicti
>>
> ons=0,size=8245,warmupTime=215,cumulative_lookups=2837676,cumulative_hit
>>
> s=2766551,cumulative_hitratio=0.97,cumulative_inserts=72050,cumulative_e
>> victions=0}
>>
>> Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
>>
>> INFO: autowarming result for Searcher@3f32ca2b main
>>
>>
>>
> filterCache{lookups=0,hits=0,hitratio=0.00,inserts=1000,evictions=0,size
>>
> =1000,warmupTime=317,cumulative_lookups=2837676,cumulative_hits=2766551,
>>
> cumulative_hitratio=0.97,cumulative_inserts=72050,cumulative_evictions=0
>> }
>>
>> Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
>>
>> INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
>>
>>
>>
> queryResultCache{lookups=5309,hits=5223,hitratio=0.98,inserts=422,evicti
>>
> ons=0,size=421,warmupTime=4628,cumulative_lookups=77802,cumulative_hits=
>>
> 77216,cumulative_hitratio=0.99,cumulative_inserts=424,cumulative_evictio
>> ns=0}
>>
>> --
>>
>> Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
>>
>> INFO: autowarming result for Searcher@3f32ca2b main
>>
>>
>>
> queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=421,evictions=0,
>>
> size=421,warmupTime=5536,cumulative_lookups=77804,cumulative_hits=77218,
>>
> cumulative_hitratio=0.99,cumulative_inserts=424,cumulative_evictions=0}
>>
>> Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
>>
>> INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
>>
>>
>>
> documentCache{lookups=87216,hits=86686,hitratio=0.99,inserts=570,evictio
>>
> ns=0,size=570,warmupTime=0,cumulative_lookups=1270773,cumulative_hits=12
>>
> 68318,cumulative_hitratio=0.99,cumulative_inserts=2455,cumulative_evicti
>> ons=0}
>>
>> Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
>>
>> INFO: autowarming result for Searcher@3f32ca2b main
>>
>>
>>
> documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=
>>
> 0,warmupTime=0,cumulative_lookups=1270773,cumulative_hits=1268318,cumula
>> tive_hitratio=0.99,cumulative_inserts=2455,cumulative_evictions=0}
>>
>> --
>>
>>
>>
>> This is our current values after I've messed with them a few times
>> trying to
>> get better performance.
>>
>>
>>
>> <filterCache
>>
>> class="solr.LRUCache"
>>
>> size="30000"
>>
>> initialSize="15000"
>>
>> autowarmCount="1000"/>
>>
>> <queryResultCache
>>
>> class="solr.LRUCache"
>>
>> size="60000"
>>
>> initialSize="30000"
>>
>> autowarmCount="50000"/>
>>
>> <documentCache
>>
>> class="solr.LRUCache"
>>
>> size="200000"
>>
>> initialSize="125000"
>>
>> autowarmCount="0"/>
>>
>>
>> --
>> View this message in context:
>>
> http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452
>> 835p20980669.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>>
>>
>
> --
> View this message in context:
> http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452
> 835p20981862.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
>
>
--
View this message in context: http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p20982647.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Query Performance while updating teh index
Posted by "Feak, Todd" <To...@smss.sony.com>.
Sorry, my bad. Didn't read the entire thread.
Look at your filter cache first. You are autowarming 1000, and there is
exactly 1000 in there. Yet it looks like there may be tens of thousands
of filter queries in your system. I would try autowarming more. Try
10,000 or 20,000 and see if it helps.
Second look at your document cache. Document caches don't use autowarm.
But you can add queries to your firstSeacher and newSearcher entries in
your solrconfig to pre-populate the document cache during warming.
-Todd Feak
-----Original Message-----
From: oleg_gnatovskiy [mailto:oleg_gnatovskiy@citysearch.com]
Sent: Friday, December 12, 2008 11:19 AM
To: solr-user@lucene.apache.org
Subject: RE: Query Performance while updating teh index
The auto warm time is not an issue. We take the server off the load
balancer
while it is autowarming. It seems that the slowness occurs after
autowarm is
done.
Feak, Todd wrote:
>
> It's spending 4-5 seconds warming up your query cache. If 4-5 seconds
is
> too much, you could reduce the number of queries to auto-warm with on
> that cache.
>
> Notice that the 4-5 seconds is spent only putting about 420 queries
into
> the query cache. Your autowarm of 50000 for the query cache seems a
bit
> high. If you need to reduce that autowarm time below 5 seconds, you
may
> have to set that value in the hundreds, as opposed to tens of
thousands.
>
> -Todd Feak
>
> -----Original Message-----
> From: oleg_gnatovskiy [mailto:oleg_gnatovskiy@citysearch.com]
> Sent: Friday, December 12, 2008 10:08 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Query Performance while updating teh index
>
>
> Here's what we have on one of the data slaves for the autowarming.
>
>
>
> --
>
> Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
>
>
>
filterCache{lookups=351993,hits=347055,hitratio=0.98,inserts=8332,evicti
>
ons=0,size=8245,warmupTime=215,cumulative_lookups=2837676,cumulative_hit
>
s=2766551,cumulative_hitratio=0.97,cumulative_inserts=72050,cumulative_e
> victions=0}
>
> Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming result for Searcher@3f32ca2b main
>
>
>
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=1000,evictions=0,size
>
=1000,warmupTime=317,cumulative_lookups=2837676,cumulative_hits=2766551,
>
cumulative_hitratio=0.97,cumulative_inserts=72050,cumulative_evictions=0
> }
>
> Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
>
>
>
queryResultCache{lookups=5309,hits=5223,hitratio=0.98,inserts=422,evicti
>
ons=0,size=421,warmupTime=4628,cumulative_lookups=77802,cumulative_hits=
>
77216,cumulative_hitratio=0.99,cumulative_inserts=424,cumulative_evictio
> ns=0}
>
> --
>
> Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming result for Searcher@3f32ca2b main
>
>
>
queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=421,evictions=0,
>
size=421,warmupTime=5536,cumulative_lookups=77804,cumulative_hits=77218,
>
cumulative_hitratio=0.99,cumulative_inserts=424,cumulative_evictions=0}
>
> Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
>
>
>
documentCache{lookups=87216,hits=86686,hitratio=0.99,inserts=570,evictio
>
ns=0,size=570,warmupTime=0,cumulative_lookups=1270773,cumulative_hits=12
>
68318,cumulative_hitratio=0.99,cumulative_inserts=2455,cumulative_evicti
> ons=0}
>
> Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming result for Searcher@3f32ca2b main
>
>
>
documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=
>
0,warmupTime=0,cumulative_lookups=1270773,cumulative_hits=1268318,cumula
> tive_hitratio=0.99,cumulative_inserts=2455,cumulative_evictions=0}
>
> --
>
>
>
> This is our current values after I've messed with them a few times
> trying to
> get better performance.
>
>
>
> <filterCache
>
> class="solr.LRUCache"
>
> size="30000"
>
> initialSize="15000"
>
> autowarmCount="1000"/>
>
> <queryResultCache
>
> class="solr.LRUCache"
>
> size="60000"
>
> initialSize="30000"
>
> autowarmCount="50000"/>
>
> <documentCache
>
> class="solr.LRUCache"
>
> size="200000"
>
> initialSize="125000"
>
> autowarmCount="0"/>
>
>
> --
> View this message in context:
>
http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452
> 835p20980669.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
>
>
--
View this message in context:
http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452
835p20981862.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Query Performance while updating teh index
Posted by oleg_gnatovskiy <ol...@citysearch.com>.
I just verified this. The slowness occurs after auto warm is done.
Oleg
--
View this message in context: http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p20982068.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Query Performance while updating teh index
Posted by oleg_gnatovskiy <ol...@citysearch.com>.
The auto warm time is not an issue. We take the server off the load balancer
while it is autowarming. It seems that the slowness occurs after autowarm is
done.
Feak, Todd wrote:
>
> It's spending 4-5 seconds warming up your query cache. If 4-5 seconds is
> too much, you could reduce the number of queries to auto-warm with on
> that cache.
>
> Notice that the 4-5 seconds is spent only putting about 420 queries into
> the query cache. Your autowarm of 50000 for the query cache seems a bit
> high. If you need to reduce that autowarm time below 5 seconds, you may
> have to set that value in the hundreds, as opposed to tens of thousands.
>
> -Todd Feak
>
> -----Original Message-----
> From: oleg_gnatovskiy [mailto:oleg_gnatovskiy@citysearch.com]
> Sent: Friday, December 12, 2008 10:08 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Query Performance while updating teh index
>
>
> Here's what we have on one of the data slaves for the autowarming.
>
>
>
> --
>
> Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
>
>
> filterCache{lookups=351993,hits=347055,hitratio=0.98,inserts=8332,evicti
> ons=0,size=8245,warmupTime=215,cumulative_lookups=2837676,cumulative_hit
> s=2766551,cumulative_hitratio=0.97,cumulative_inserts=72050,cumulative_e
> victions=0}
>
> Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming result for Searcher@3f32ca2b main
>
>
> filterCache{lookups=0,hits=0,hitratio=0.00,inserts=1000,evictions=0,size
> =1000,warmupTime=317,cumulative_lookups=2837676,cumulative_hits=2766551,
> cumulative_hitratio=0.97,cumulative_inserts=72050,cumulative_evictions=0
> }
>
> Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
>
>
> queryResultCache{lookups=5309,hits=5223,hitratio=0.98,inserts=422,evicti
> ons=0,size=421,warmupTime=4628,cumulative_lookups=77802,cumulative_hits=
> 77216,cumulative_hitratio=0.99,cumulative_inserts=424,cumulative_evictio
> ns=0}
>
> --
>
> Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming result for Searcher@3f32ca2b main
>
>
> queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=421,evictions=0,
> size=421,warmupTime=5536,cumulative_lookups=77804,cumulative_hits=77218,
> cumulative_hitratio=0.99,cumulative_inserts=424,cumulative_evictions=0}
>
> Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
>
>
> documentCache{lookups=87216,hits=86686,hitratio=0.99,inserts=570,evictio
> ns=0,size=570,warmupTime=0,cumulative_lookups=1270773,cumulative_hits=12
> 68318,cumulative_hitratio=0.99,cumulative_inserts=2455,cumulative_evicti
> ons=0}
>
> Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming result for Searcher@3f32ca2b main
>
>
> documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=
> 0,warmupTime=0,cumulative_lookups=1270773,cumulative_hits=1268318,cumula
> tive_hitratio=0.99,cumulative_inserts=2455,cumulative_evictions=0}
>
> --
>
>
>
> This is our current values after I've messed with them a few times
> trying to
> get better performance.
>
>
>
> <filterCache
>
> class="solr.LRUCache"
>
> size="30000"
>
> initialSize="15000"
>
> autowarmCount="1000"/>
>
> <queryResultCache
>
> class="solr.LRUCache"
>
> size="60000"
>
> initialSize="30000"
>
> autowarmCount="50000"/>
>
> <documentCache
>
> class="solr.LRUCache"
>
> size="200000"
>
> initialSize="125000"
>
> autowarmCount="0"/>
>
>
> --
> View this message in context:
> http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452
> 835p20980669.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
>
>
--
View this message in context: http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p20981862.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Query Performance while updating teh index
Posted by "Feak, Todd" <To...@smss.sony.com>.
It's spending 4-5 seconds warming up your query cache. If 4-5 seconds is
too much, you could reduce the number of queries to auto-warm with on
that cache.
Notice that the 4-5 seconds is spent only putting about 420 queries into
the query cache. Your autowarm of 50000 for the query cache seems a bit
high. If you need to reduce that autowarm time below 5 seconds, you may
have to set that value in the hundreds, as opposed to tens of thousands.
-Todd Feak
-----Original Message-----
From: oleg_gnatovskiy [mailto:oleg_gnatovskiy@citysearch.com]
Sent: Friday, December 12, 2008 10:08 AM
To: solr-user@lucene.apache.org
Subject: Re: Query Performance while updating teh index
Here's what we have on one of the data slaves for the autowarming.
--
Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
filterCache{lookups=351993,hits=347055,hitratio=0.98,inserts=8332,evicti
ons=0,size=8245,warmupTime=215,cumulative_lookups=2837676,cumulative_hit
s=2766551,cumulative_hitratio=0.97,cumulative_inserts=72050,cumulative_e
victions=0}
Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@3f32ca2b main
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=1000,evictions=0,size
=1000,warmupTime=317,cumulative_lookups=2837676,cumulative_hits=2766551,
cumulative_hitratio=0.97,cumulative_inserts=72050,cumulative_evictions=0
}
Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
queryResultCache{lookups=5309,hits=5223,hitratio=0.98,inserts=422,evicti
ons=0,size=421,warmupTime=4628,cumulative_lookups=77802,cumulative_hits=
77216,cumulative_hitratio=0.99,cumulative_inserts=424,cumulative_evictio
ns=0}
--
Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@3f32ca2b main
queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=421,evictions=0,
size=421,warmupTime=5536,cumulative_lookups=77804,cumulative_hits=77218,
cumulative_hitratio=0.99,cumulative_inserts=424,cumulative_evictions=0}
Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
documentCache{lookups=87216,hits=86686,hitratio=0.99,inserts=570,evictio
ns=0,size=570,warmupTime=0,cumulative_lookups=1270773,cumulative_hits=12
68318,cumulative_hitratio=0.99,cumulative_inserts=2455,cumulative_evicti
ons=0}
Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@3f32ca2b main
documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=
0,warmupTime=0,cumulative_lookups=1270773,cumulative_hits=1268318,cumula
tive_hitratio=0.99,cumulative_inserts=2455,cumulative_evictions=0}
--
This is our current values after I've messed with them a few times
trying to
get better performance.
<filterCache
class="solr.LRUCache"
size="30000"
initialSize="15000"
autowarmCount="1000"/>
<queryResultCache
class="solr.LRUCache"
size="60000"
initialSize="30000"
autowarmCount="50000"/>
<documentCache
class="solr.LRUCache"
size="200000"
initialSize="125000"
autowarmCount="0"/>
--
View this message in context:
http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452
835p20980669.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query Performance while updating teh index
Posted by Otis Gospodnetic <ot...@yahoo.com>.
It looks like cache warming is taking about 12 seconds. It sounds like you need to see if performance is bad during warming, or right after warming (and right after the new searcher gets exposed to queries).
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
> From: oleg_gnatovskiy <ol...@citysearch.com>
> To: solr-user@lucene.apache.org
> Sent: Friday, December 12, 2008 1:07:49 PM
> Subject: Re: Query Performance while updating teh index
>
>
> Here’s what we have on one of the data slaves for the autowarming.
>
>
>
> --
>
> Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
>
>
> filterCache{lookups=351993,hits=347055,hitratio=0.98,inserts=8332,evictions=0,size=8245,warmupTime=215,cumulative_lookups=2837676,cumulative_hits=2766551,cumulative_hitratio=0.97,cumulative_inserts=72050,cumulative_evictions=0}
>
> Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming result for Searcher@3f32ca2b main
>
>
> filterCache{lookups=0,hits=0,hitratio=0.00,inserts=1000,evictions=0,size=1000,warmupTime=317,cumulative_lookups=2837676,cumulative_hits=2766551,cumulative_hitratio=0.97,cumulative_inserts=72050,cumulative_evictions=0}
>
> Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
>
>
> queryResultCache{lookups=5309,hits=5223,hitratio=0.98,inserts=422,evictions=0,size=421,warmupTime=4628,cumulative_lookups=77802,cumulative_hits=77216,cumulative_hitratio=0.99,cumulative_inserts=424,cumulative_evictions=0}
>
> --
>
> Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming result for Searcher@3f32ca2b main
>
>
> queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=421,evictions=0,size=421,warmupTime=5536,cumulative_lookups=77804,cumulative_hits=77218,cumulative_hitratio=0.99,cumulative_inserts=424,cumulative_evictions=0}
>
> Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
>
>
> documentCache{lookups=87216,hits=86686,hitratio=0.99,inserts=570,evictions=0,size=570,warmupTime=0,cumulative_lookups=1270773,cumulative_hits=1268318,cumulative_hitratio=0.99,cumulative_inserts=2455,cumulative_evictions=0}
>
> Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming result for Searcher@3f32ca2b main
>
>
> documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=1270773,cumulative_hits=1268318,cumulative_hitratio=0.99,cumulative_inserts=2455,cumulative_evictions=0}
>
> --
>
>
>
> This is our current values after I’ve messed with them a few times trying to
> get better performance.
>
>
>
>
>
> class="solr.LRUCache"
>
> size="30000"
>
> initialSize="15000"
>
> autowarmCount="1000"/>
>
>
>
> class="solr.LRUCache"
>
> size="60000"
>
> initialSize="30000"
>
> autowarmCount="50000"/>
>
>
>
> class="solr.LRUCache"
>
> size="200000"
>
> initialSize="125000"
>
> autowarmCount="0"/>
>
>
> --
> View this message in context:
> http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p20980669.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query Performance while updating teh index
Posted by oleg_gnatovskiy <ol...@citysearch.com>.
Here’s what we have on one of the data slaves for the autowarming.
--
Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
filterCache{lookups=351993,hits=347055,hitratio=0.98,inserts=8332,evictions=0,size=8245,warmupTime=215,cumulative_lookups=2837676,cumulative_hits=2766551,cumulative_hitratio=0.97,cumulative_inserts=72050,cumulative_evictions=0}
Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@3f32ca2b main
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=1000,evictions=0,size=1000,warmupTime=317,cumulative_lookups=2837676,cumulative_hits=2766551,cumulative_hitratio=0.97,cumulative_inserts=72050,cumulative_evictions=0}
Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
queryResultCache{lookups=5309,hits=5223,hitratio=0.98,inserts=422,evictions=0,size=421,warmupTime=4628,cumulative_lookups=77802,cumulative_hits=77216,cumulative_hitratio=0.99,cumulative_inserts=424,cumulative_evictions=0}
--
Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@3f32ca2b main
queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=421,evictions=0,size=421,warmupTime=5536,cumulative_lookups=77804,cumulative_hits=77218,cumulative_hitratio=0.99,cumulative_inserts=424,cumulative_evictions=0}
Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
documentCache{lookups=87216,hits=86686,hitratio=0.99,inserts=570,evictions=0,size=570,warmupTime=0,cumulative_lookups=1270773,cumulative_hits=1268318,cumulative_hitratio=0.99,cumulative_inserts=2455,cumulative_evictions=0}
Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@3f32ca2b main
documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=1270773,cumulative_hits=1268318,cumulative_hitratio=0.99,cumulative_inserts=2455,cumulative_evictions=0}
--
This is our current values after I’ve messed with them a few times trying to
get better performance.
<filterCache
class="solr.LRUCache"
size="30000"
initialSize="15000"
autowarmCount="1000"/>
<queryResultCache
class="solr.LRUCache"
size="60000"
initialSize="30000"
autowarmCount="50000"/>
<documentCache
class="solr.LRUCache"
size="200000"
initialSize="125000"
autowarmCount="0"/>
--
View this message in context: http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p20980669.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query Performance while updating teh index
Posted by oleg_gnatovskiy <ol...@citysearch.com>.
Hey Otis,
Do you think our problem is slow warm time, or too few items that are being
copied?
Oleg
--
View this message in context: http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p20980523.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query Performance while updating teh index
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Oleg,
The reliable formula is situation-specific, I think. One sure way to decrease the warm time is to minimize the number of items to copy from old caches to new caches on warmup.
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
> From: oleg_gnatovskiy <ol...@citysearch.com>
> To: solr-user@lucene.apache.org
> Sent: Thursday, December 11, 2008 8:43:26 PM
> Subject: RE: Query Performance while updating teh index
>
>
> We are still having this problem. I am wondering if it can be fixed with
> autowarm settings. Is there a reliable formula for determining the autowarm
> settings?
> --
> View this message in context:
> http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p20968516.html
> Sent from the Solr - User mailing list archive at Nabble.com.
RE: Query Performance while updating teh index
Posted by oleg_gnatovskiy <ol...@citysearch.com>.
We are still having this problem. I am wondering if it can be fixed with
autowarm settings. Is there a reliable formula for determining the autowarm
settings?
--
View this message in context: http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p20968516.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Query Performance while updating teh index
Posted by Chris Hostetter <ho...@fucit.org>.
: How about create a new core, index data, then swap the core? Old core
: is still available to handle queries till new core replaces it.
a new SolrCore shouldn't help in a situation like this ... with
snapshots and commits on a single SolrCore you at least get the benefits
of autowarming and learning from the old caches -- this would be
completley eliminated from the equation when spawning a new SolrCore, but
the same total amount of processing would happen.
-Hoss
RE: Query Performance while updating teh index
Posted by "Nguyen, Joe" <jn...@automotive.com>.
How about create a new core, index data, then swap the core? Old core
is still available to handle queries till new core replaces it.
-----Original Message-----
From: Lance Norskog [mailto:goksron@gmail.com]
Sent: Wednesday, November 12, 2008 11:16 Joe
To: solr-user@lucene.apache.org
Subject: RE: Query Performance while updating teh index
Yes, this is the cache autowarming.
We turned this off and staged separate queries that pre-warm our
standard queries. We are looking at pulling the query server out of the
load balancer during this process; it is the most effective way to give
fixed response time.
Lance
-----Original Message-----
From: oleg_gnatovskiy [mailto:oleg_gnatovskiy@citysearch.com]
Sent: Wednesday, November 12, 2008 11:07 AM
To: solr-user@lucene.apache.org
Subject: Re: Query Performance while updating teh index
The rsync seems to have nothing to do with slowness, because while the
rsync
is going on, there isn't any reload occurring, once the files are on the
system, it tries a curl request to reload the searcher, which at that
point
causes the delays. The file transfer probably has nothing to do with
this.
Does this mean that it happens during warming?
Yonik Seeley wrote:
>
> On Tue, Nov 11, 2008 at 9:31 PM, oleg_gnatovskiy
> <ol...@citysearch.com> wrote:
>> Hello. We have an index with 15 million documents working on a
>> distributed environment, with an index distribution setup. While an
>> index on a slave server is being updated, query response times become
>> extremely slow (upwards of 5 seconds). Is there any way to decrease
>> the hit query response times take while an index is being pushed?
>
> Can you tell why it's getting slow? Is this during warming, or does
> it begin during the actual transfer of the new index?
>
> One possibility is that the new index being copied forces out parts of
> the old index from the OS cache. More memory would help in that
> scenario.
>
> -Yonik
>
>
--
View this message in context:
http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452
835p
20467099.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query Performance while updating teh index
Posted by oleg_gnatovskiy <ol...@citysearch.com>.
Well we never had 1.2 deployed, so I don't know if it's a new issue or not...
Yonik Seeley wrote:
>
> Warming only uses one CPU, so it shouldn't have that much of an impact
> on a multi-CPU box.
>
> Did this issue begin with Solr 1.3? Perhaps it has something to do
> with our use of reopen() (to share parts of the index that are not in
> use). This can lead to greater lock contention while reading from the
> index.
>
> If so, we need
> 1) an option to disable using IndexReader.reopen() (I think Mark
> already has a patch for this)
> 2) NIO support to reduce/eliminate that contention on non Windows
> platforms (a work in progress - the last patch doesn't actually do it)
>
> -Yonik
>
>
> On Wed, Nov 12, 2008 at 2:16 PM, Lance Norskog <go...@gmail.com> wrote:
>> Yes, this is the cache autowarming.
>>
>> We turned this off and staged separate queries that pre-warm our standard
>> queries. We are looking at pulling the query server out of the load
>> balancer
>> during this process; it is the most effective way to give fixed response
>> time.
>>
>> Lance
>>
>> -----Original Message-----
>> From: oleg_gnatovskiy [mailto:oleg_gnatovskiy@citysearch.com]
>> Sent: Wednesday, November 12, 2008 11:07 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Query Performance while updating teh index
>>
>>
>> The rsync seems to have nothing to do with slowness, because while the
>> rsync
>> is going on, there isn't any reload occurring, once the files are on the
>> system, it tries a curl request to reload the searcher, which at that
>> point
>> causes the delays. The file transfer probably has nothing to do with
>> this.
>> Does this mean that it happens during warming?
>>
>>
>>
>> Yonik Seeley wrote:
>>>
>>> On Tue, Nov 11, 2008 at 9:31 PM, oleg_gnatovskiy
>>> <ol...@citysearch.com> wrote:
>>>> Hello. We have an index with 15 million documents working on a
>>>> distributed environment, with an index distribution setup. While an
>>>> index on a slave server is being updated, query response times become
>>>> extremely slow (upwards of 5 seconds). Is there any way to decrease
>>>> the hit query response times take while an index is being pushed?
>>>
>>> Can you tell why it's getting slow? Is this during warming, or does
>>> it begin during the actual transfer of the new index?
>>>
>>> One possibility is that the new index being copied forces out parts of
>>> the old index from the OS cache. More memory would help in that
>>> scenario.
>>>
>>> -Yonik
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p
>> 20467099.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>>
>
>
--
View this message in context: http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p20469525.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query Performance while updating teh index
Posted by Yonik Seeley <yo...@apache.org>.
Warming only uses one CPU, so it shouldn't have that much of an impact
on a multi-CPU box.
Did this issue begin with Solr 1.3? Perhaps it has something to do
with our use of reopen() (to share parts of the index that are not in
use). This can lead to greater lock contention while reading from the
index.
If so, we need
1) an option to disable using IndexReader.reopen() (I think Mark
already has a patch for this)
2) NIO support to reduce/eliminate that contention on non Windows
platforms (a work in progress - the last patch doesn't actually do it)
-Yonik
On Wed, Nov 12, 2008 at 2:16 PM, Lance Norskog <go...@gmail.com> wrote:
> Yes, this is the cache autowarming.
>
> We turned this off and staged separate queries that pre-warm our standard
> queries. We are looking at pulling the query server out of the load balancer
> during this process; it is the most effective way to give fixed response
> time.
>
> Lance
>
> -----Original Message-----
> From: oleg_gnatovskiy [mailto:oleg_gnatovskiy@citysearch.com]
> Sent: Wednesday, November 12, 2008 11:07 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Query Performance while updating teh index
>
>
> The rsync seems to have nothing to do with slowness, because while the rsync
> is going on, there isn't any reload occurring, once the files are on the
> system, it tries a curl request to reload the searcher, which at that point
> causes the delays. The file transfer probably has nothing to do with this.
> Does this mean that it happens during warming?
>
>
>
> Yonik Seeley wrote:
>>
>> On Tue, Nov 11, 2008 at 9:31 PM, oleg_gnatovskiy
>> <ol...@citysearch.com> wrote:
>>> Hello. We have an index with 15 million documents working on a
>>> distributed environment, with an index distribution setup. While an
>>> index on a slave server is being updated, query response times become
>>> extremely slow (upwards of 5 seconds). Is there any way to decrease
>>> the hit query response times take while an index is being pushed?
>>
>> Can you tell why it's getting slow? Is this during warming, or does
>> it begin during the actual transfer of the new index?
>>
>> One possibility is that the new index being copied forces out parts of
>> the old index from the OS cache. More memory would help in that
>> scenario.
>>
>> -Yonik
>>
>>
>
> --
> View this message in context:
> http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p
> 20467099.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
>
RE: Query Performance while updating teh index
Posted by "Nguyen, Joe" <jn...@automotive.com>.
Another way to handle this is not to run commit script at peak
time(still pull snapshot periodically). Keeping track of the number of
requests, resource utilization, etc.. If the number of request exceeds
the threshold, don't commit.
Also, how many segments do you see under index dir? High number of
segments (implied more files need to be opened) would impact query
response time. In that case, you could run optimize script to
consolidate into a single segment.
-----Original Message-----
From: Lance Norskog [mailto:goksron@gmail.com]
Sent: Wednesday, November 12, 2008 11:16 Joe
To: solr-user@lucene.apache.org
Subject: RE: Query Performance while updating teh index
Yes, this is the cache autowarming.
We turned this off and staged separate queries that pre-warm our
standard queries. We are looking at pulling the query server out of the
load balancer during this process; it is the most effective way to give
fixed response time.
Lance
-----Original Message-----
From: oleg_gnatovskiy [mailto:oleg_gnatovskiy@citysearch.com]
Sent: Wednesday, November 12, 2008 11:07 AM
To: solr-user@lucene.apache.org
Subject: Re: Query Performance while updating teh index
The rsync seems to have nothing to do with slowness, because while the
rsync
is going on, there isn't any reload occurring, once the files are on the
system, it tries a curl request to reload the searcher, which at that
point
causes the delays. The file transfer probably has nothing to do with
this.
Does this mean that it happens during warming?
Yonik Seeley wrote:
>
> On Tue, Nov 11, 2008 at 9:31 PM, oleg_gnatovskiy
> <ol...@citysearch.com> wrote:
>> Hello. We have an index with 15 million documents working on a
>> distributed environment, with an index distribution setup. While an
>> index on a slave server is being updated, query response times become
>> extremely slow (upwards of 5 seconds). Is there any way to decrease
>> the hit query response times take while an index is being pushed?
>
> Can you tell why it's getting slow? Is this during warming, or does
> it begin during the actual transfer of the new index?
>
> One possibility is that the new index being copied forces out parts of
> the old index from the OS cache. More memory would help in that
> scenario.
>
> -Yonik
>
>
--
View this message in context:
http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452
835p
20467099.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Query Performance while updating teh index
Posted by Lance Norskog <go...@gmail.com>.
Yes, this is the cache autowarming.
We turned this off and staged separate queries that pre-warm our standard
queries. We are looking at pulling the query server out of the load balancer
during this process; it is the most effective way to give fixed response
time.
Lance
-----Original Message-----
From: oleg_gnatovskiy [mailto:oleg_gnatovskiy@citysearch.com]
Sent: Wednesday, November 12, 2008 11:07 AM
To: solr-user@lucene.apache.org
Subject: Re: Query Performance while updating teh index
The rsync seems to have nothing to do with slowness, because while the rsync
is going on, there isn't any reload occurring, once the files are on the
system, it tries a curl request to reload the searcher, which at that point
causes the delays. The file transfer probably has nothing to do with this.
Does this mean that it happens during warming?
Yonik Seeley wrote:
>
> On Tue, Nov 11, 2008 at 9:31 PM, oleg_gnatovskiy
> <ol...@citysearch.com> wrote:
>> Hello. We have an index with 15 million documents working on a
>> distributed environment, with an index distribution setup. While an
>> index on a slave server is being updated, query response times become
>> extremely slow (upwards of 5 seconds). Is there any way to decrease
>> the hit query response times take while an index is being pushed?
>
> Can you tell why it's getting slow? Is this during warming, or does
> it begin during the actual transfer of the new index?
>
> One possibility is that the new index being copied forces out parts of
> the old index from the OS cache. More memory would help in that
> scenario.
>
> -Yonik
>
>
--
View this message in context:
http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p
20467099.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query Performance while updating teh index
Posted by Otis Gospodnetic <ot...@yahoo.com>.
And you have searcher warming set up?
Does it use sort and do your queries use sort?
What do your cache settings look like?
How big is your index, how much RAM does your machine have, how much heap does the JVM have, what does vmstat output look like during warm-up?
...
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
________________________________
From: oleg_gnatovskiy <ol...@citysearch.com>
To: solr-user@lucene.apache.org
Sent: Wednesday, November 12, 2008 2:16:44 PM
Subject: Re: Query Performance while updating teh index
Yonik Seeley wrote:
>
> On Wed, Nov 12, 2008 at 2:06 PM, oleg_gnatovskiy
> <ol...@citysearch.com> wrote:
>> The rsync seems to have nothing to do with slowness, because while the
>> rsync
>> is going on, there isn't any reload occurring, once the files are on the
>> system, it tries a curl request to reload the searcher, which at that
>> point
>> causes the delays. The file transfer probably has nothing to do with
>> this.
>> Does this mean that it happens during warming?
>
> Yes, it would seem so.
> It could either be that 1) warming the new reader slows down the
> current reader used to service queries
> or 2) the first queries to come into the new reader are slow (which
> can be solved with some static warming queries to load sort fields,
> facet caches, etc).
>
> How many CPUs does the box have that you are running on? What OS?
>
> -Yonik
>
>
>
>> Yonik Seeley wrote:
>>>
>>> On Tue, Nov 11, 2008 at 9:31 PM, oleg_gnatovskiy
>>> <ol...@citysearch.com> wrote:
>>>> Hello. We have an index with 15 million documents working on a
>>>> distributed
>>>> environment, with an index distribution setup. While an index on a
>>>> slave
>>>> server is being updated, query response times become extremely slow
>>>> (upwards
>>>> of 5 seconds). Is there any way to decrease the hit query response
>>>> times
>>>> take while an index is being pushed?
>>>
>>> Can you tell why it's getting slow? Is this during warming, or does
>>> it begin during the actual transfer of the new index?
>>>
>>> One possibility is that the new index being copied forces out parts of
>>> the old index from the OS cache. More memory would help in that
>>> scenario.
>>>
>>> -Yonik
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p20467099.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
8 CPUs and Linux OS
--
View this message in context: http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p20467281.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query Performance while updating teh index
Posted by oleg_gnatovskiy <ol...@citysearch.com>.
Yonik Seeley wrote:
>
> On Wed, Nov 12, 2008 at 2:06 PM, oleg_gnatovskiy
> <ol...@citysearch.com> wrote:
>> The rsync seems to have nothing to do with slowness, because while the
>> rsync
>> is going on, there isn't any reload occurring, once the files are on the
>> system, it tries a curl request to reload the searcher, which at that
>> point
>> causes the delays. The file transfer probably has nothing to do with
>> this.
>> Does this mean that it happens during warming?
>
> Yes, it would seem so.
> It could either be that 1) warming the new reader slows down the
> current reader used to service queries
> or 2) the first queries to come into the new reader are slow (which
> can be solved with some static warming queries to load sort fields,
> facet caches, etc).
>
> How many CPUs does the box have that you are running on? What OS?
>
> -Yonik
>
>
>
>> Yonik Seeley wrote:
>>>
>>> On Tue, Nov 11, 2008 at 9:31 PM, oleg_gnatovskiy
>>> <ol...@citysearch.com> wrote:
>>>> Hello. We have an index with 15 million documents working on a
>>>> distributed
>>>> environment, with an index distribution setup. While an index on a
>>>> slave
>>>> server is being updated, query response times become extremely slow
>>>> (upwards
>>>> of 5 seconds). Is there any way to decrease the hit query response
>>>> times
>>>> take while an index is being pushed?
>>>
>>> Can you tell why it's getting slow? Is this during warming, or does
>>> it begin during the actual transfer of the new index?
>>>
>>> One possibility is that the new index being copied forces out parts of
>>> the old index from the OS cache. More memory would help in that
>>> scenario.
>>>
>>> -Yonik
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p20467099.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
8 CPUs and Linux OS
--
View this message in context: http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p20467281.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query Performance while updating teh index
Posted by Yonik Seeley <yo...@apache.org>.
On Wed, Nov 12, 2008 at 2:06 PM, oleg_gnatovskiy
<ol...@citysearch.com> wrote:
> The rsync seems to have nothing to do with slowness, because while the rsync
> is going on, there isn't any reload occurring, once the files are on the
> system, it tries a curl request to reload the searcher, which at that point
> causes the delays. The file transfer probably has nothing to do with this.
> Does this mean that it happens during warming?
Yes, it would seem so.
It could either be that 1) warming the new reader slows down the
current reader used to service queries
or 2) the first queries to come into the new reader are slow (which
can be solved with some static warming queries to load sort fields,
facet caches, etc).
How many CPUs does the box have that you are running on? What OS?
-Yonik
> Yonik Seeley wrote:
>>
>> On Tue, Nov 11, 2008 at 9:31 PM, oleg_gnatovskiy
>> <ol...@citysearch.com> wrote:
>>> Hello. We have an index with 15 million documents working on a
>>> distributed
>>> environment, with an index distribution setup. While an index on a slave
>>> server is being updated, query response times become extremely slow
>>> (upwards
>>> of 5 seconds). Is there any way to decrease the hit query response times
>>> take while an index is being pushed?
>>
>> Can you tell why it's getting slow? Is this during warming, or does
>> it begin during the actual transfer of the new index?
>>
>> One possibility is that the new index being copied forces out parts of
>> the old index from the OS cache. More memory would help in that
>> scenario.
>>
>> -Yonik
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p20467099.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
Re: Query Performance while updating teh index
Posted by oleg_gnatovskiy <ol...@citysearch.com>.
The rsync seems to have nothing to do with slowness, because while the rsync
is going on, there isn’t any reload occurring, once the files are on the
system, it tries a curl request to reload the searcher, which at that point
causes the delays. The file transfer probably has nothing to do with this.
Does this mean that it happens during warming?
Yonik Seeley wrote:
>
> On Tue, Nov 11, 2008 at 9:31 PM, oleg_gnatovskiy
> <ol...@citysearch.com> wrote:
>> Hello. We have an index with 15 million documents working on a
>> distributed
>> environment, with an index distribution setup. While an index on a slave
>> server is being updated, query response times become extremely slow
>> (upwards
>> of 5 seconds). Is there any way to decrease the hit query response times
>> take while an index is being pushed?
>
> Can you tell why it's getting slow? Is this during warming, or does
> it begin during the actual transfer of the new index?
>
> One possibility is that the new index being copied forces out parts of
> the old index from the OS cache. More memory would help in that
> scenario.
>
> -Yonik
>
>
--
View this message in context: http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p20467099.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query Performance while updating teh index
Posted by Yonik Seeley <yo...@apache.org>.
On Tue, Nov 11, 2008 at 9:31 PM, oleg_gnatovskiy
<ol...@citysearch.com> wrote:
> Hello. We have an index with 15 million documents working on a distributed
> environment, with an index distribution setup. While an index on a slave
> server is being updated, query response times become extremely slow (upwards
> of 5 seconds). Is there any way to decrease the hit query response times
> take while an index is being pushed?
Can you tell why it's getting slow? Is this during warming, or does
it begin during the actual transfer of the new index?
One possibility is that the new index being copied forces out parts of
the old index from the OS cache. More memory would help in that
scenario.
-Yonik