You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by oleg_gnatovskiy <ol...@citysearch.com> on 2008/12/12 02:43:26 UTC
RE: Query Performance while updating teh index
We are still having this problem. I am wondering if it can be fixed with
autowarm settings. Is there a reliable formula for determining the autowarm
settings?
--
View this message in context: http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p20968516.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query Performance while updating teh index
Posted by Chris Hostetter <ho...@fucit.org>.
: Just to clarify - we do not optimize on the slaves at all. We only optimize
: on the master.
that doesn't change anything about hte comments that i made before. it
*really* wouldn't make sense to optimize on a slave right before pulling a
new snapshot, but it still doesn't make any more sense to optimize on a
master right before doing some updates and then pulling a new snapshot.
my second comment also still applies: a snappull after an optimize is
always going to be involve more churn on the disk...
: > : We do optimize the index before updates but we get tehse performance
: > issues
: > : even when we pull an empty snapshot. Thus even when our update is tiny,
: > the
: > : performance issues still happen.
: >
: > FWIW: this behavior doesn't make a lot of sense -- optimizing just
: > before you are about to make updates/additions ot your data, is a complete
: > waste. the main value in optimizing your index is that you have one
: > segment, as soon as you add a docment that changes.
: >
: > the other thing to keep in mind is that an optimized index is a completley
: > new segment as a new file with a new name, so there is going to be added
: > overhead on the slave machines as the OS purges the old index files and
: > replaces them with the new optimized index files -- more overhead then if
: > you had just done your additions w/o optimizing first.
-Hoss
Re: Query Performance while updating teh index
Posted by oleg_gnatovskiy <ol...@citysearch.com>.
Just to clarify - we do not optimize on the slaves at all. We only optimize
on the master.
hossman wrote:
>
>
> : We do optimize the index before updates but we get tehse performance
> issues
> : even when we pull an empty snapshot. Thus even when our update is tiny,
> the
> : performance issues still happen.
>
> FWIW: this behavior doesn't make a lot of sense -- optimizing just
> before you are about to make updates/additions ot your data, is a complete
> waste. the main value in optimizing your index is that you have one
> segment, as soon as you add a docment that changes.
>
> the other thing to keep in mind is that an optimized index is a completley
> new segment as a new file with a new name, so there is going to be added
> overhead on the slave machines as the OS purges the old index files and
> replaces them with the new optimized index files -- more overhead then if
> you had just done your additions w/o optimizing first.
>
>
>
> -Hoss
>
>
>
--
View this message in context: http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p21678267.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query Performance while updating teh index
Posted by oleg_gnatovskiy <ol...@citysearch.com>.
Just to calrify - we do not optimize on teh slaves at all. We only optimize
on the master.
hossman wrote:
>
>
> : We do optimize the index before updates but we get tehse performance
> issues
> : even when we pull an empty snapshot. Thus even when our update is tiny,
> the
> : performance issues still happen.
>
> FWIW: this behavior doesn't make a lot of sense -- optimizing just
> before you are about to make updates/additions ot your data, is a complete
> waste. the main value in optimizing your index is that you have one
> segment, as soon as you add a docment that changes.
>
> the other thing to keep in mind is that an optimized index is a completley
> new segment as a new file with a new name, so there is going to be added
> overhead on the slave machines as the OS purges the old index files and
> replaces them with the new optimized index files -- more overhead then if
> you had just done your additions w/o optimizing first.
>
>
>
> -Hoss
>
>
>
--
View this message in context: http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p21678261.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query Performance while updating teh index
Posted by oleg_gnatovskiy <ol...@citysearch.com>.
Just to calrify - we do not optimize on the slaves at all. We only optimize
on the master.
hossman wrote:
>
>
> : We do optimize the index before updates but we get tehse performance
> issues
> : even when we pull an empty snapshot. Thus even when our update is tiny,
> the
> : performance issues still happen.
>
> FWIW: this behavior doesn't make a lot of sense -- optimizing just
> before you are about to make updates/additions ot your data, is a complete
> waste. the main value in optimizing your index is that you have one
> segment, as soon as you add a docment that changes.
>
> the other thing to keep in mind is that an optimized index is a completley
> new segment as a new file with a new name, so there is going to be added
> overhead on the slave machines as the OS purges the old index files and
> replaces them with the new optimized index files -- more overhead then if
> you had just done your additions w/o optimizing first.
>
>
>
> -Hoss
>
>
>
--
View this message in context: http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p21678265.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query Performance while updating teh index
Posted by Chris Hostetter <ho...@fucit.org>.
: We do optimize the index before updates but we get tehse performance issues
: even when we pull an empty snapshot. Thus even when our update is tiny, the
: performance issues still happen.
FWIW: this behavior doesn't make a lot of sense -- optimizing just
before you are about to make updates/additions ot your data, is a complete
waste. the main value in optimizing your index is that you have one
segment, as soon as you add a docment that changes.
the other thing to keep in mind is that an optimized index is a completley
new segment as a new file with a new name, so there is going to be added
overhead on the slave machines as the OS purges the old index files and
replaces them with the new optimized index files -- more overhead then if
you had just done your additions w/o optimizing first.
-Hoss
Re: Query Performance while updating teh index
Posted by oleg_gnatovskiy <ol...@citysearch.com>.
We've tried it. There doesn't seem to be any connection between GC and the
bad performance spikes.
Otis Gospodnetic wrote:
>
> OK. Then it's likely not this. You saw the other response about looking
> at GC to see if maybe that hits you once in a while and slows whatever
> queries are in flight? Try jconsole.
>
> Otis
> --
> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>
>
>
> ----- Original Message ----
>> From: oleg_gnatovskiy <ol...@citysearch.com>
>> To: solr-user@lucene.apache.org
>> Sent: Thursday, January 22, 2009 2:43:31 PM
>> Subject: Re: Query Performance while updating teh index
>>
>>
>> We do optimize the index before updates but we get tehse performance
>> issues
>> even when we pull an empty snapshot. Thus even when our update is tiny,
>> the
>> performance issues still happen.
>>
>>
>>
>> Otis Gospodnetic wrote:
>> >
>> > This is an old and long thread, and I no longer recall what the
>> specific
>> > suggestions were.
>> > My guess is this has to do with the OS cache of your index files. When
>> > you make the large index update, that OS cache is useless (old files
>> are
>> > gone, new ones are in) and the OS cache has get re-warmed and this
>> takes
>> > time.
>> >
>> > Are you optimizing your index before the update? Do you *really* need
>> to
>> > do that?
>> > How large is your update, what makes it big, and could you make it
>> > smaller?
>> >
>> > Otis
>> > --
>> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>> >
>> >
>> >
>> > ----- Original Message ----
>> >> From: oleg_gnatovskiy
>> >> To: solr-user@lucene.apache.org
>> >> Sent: Tuesday, January 20, 2009 6:19:46 PM
>> >> Subject: Re: Query Performance while updating teh index
>> >>
>> >>
>> >> Hello again. It seems that we are still having these problems. Queries
>> >> take
>> >> as long as 20 minutes to get back to their average response time after
>> a
>> >> large index update, so it doesn't seem like the problem is the 12
>> second
>> >> autowarm time. Are there any more suggestions for things we can try?
>> >> Taking
>> >> our servers out of teh loop for as long as 20 minutes is a bit of a
>> >> hassle,
>> >> and a risk.
>> >> --
>> >> View this message in context:
>> >>
>> http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p21573927.html
>> >> Sent from the Solr - User mailing list archive at Nabble.com.
>> >
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p21611642.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
>
--
View this message in context: http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p21611976.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query Performance while updating teh index
Posted by Otis Gospodnetic <ot...@yahoo.com>.
OK. Then it's likely not this. You saw the other response about looking at GC to see if maybe that hits you once in a while and slows whatever queries are in flight? Try jconsole.
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
> From: oleg_gnatovskiy <ol...@citysearch.com>
> To: solr-user@lucene.apache.org
> Sent: Thursday, January 22, 2009 2:43:31 PM
> Subject: Re: Query Performance while updating teh index
>
>
> We do optimize the index before updates but we get tehse performance issues
> even when we pull an empty snapshot. Thus even when our update is tiny, the
> performance issues still happen.
>
>
>
> Otis Gospodnetic wrote:
> >
> > This is an old and long thread, and I no longer recall what the specific
> > suggestions were.
> > My guess is this has to do with the OS cache of your index files. When
> > you make the large index update, that OS cache is useless (old files are
> > gone, new ones are in) and the OS cache has get re-warmed and this takes
> > time.
> >
> > Are you optimizing your index before the update? Do you *really* need to
> > do that?
> > How large is your update, what makes it big, and could you make it
> > smaller?
> >
> > Otis
> > --
> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
> >
> >
> >
> > ----- Original Message ----
> >> From: oleg_gnatovskiy
> >> To: solr-user@lucene.apache.org
> >> Sent: Tuesday, January 20, 2009 6:19:46 PM
> >> Subject: Re: Query Performance while updating teh index
> >>
> >>
> >> Hello again. It seems that we are still having these problems. Queries
> >> take
> >> as long as 20 minutes to get back to their average response time after a
> >> large index update, so it doesn't seem like the problem is the 12 second
> >> autowarm time. Are there any more suggestions for things we can try?
> >> Taking
> >> our servers out of teh loop for as long as 20 minutes is a bit of a
> >> hassle,
> >> and a risk.
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p21573927.html
> >> Sent from the Solr - User mailing list archive at Nabble.com.
> >
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p21611642.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query Performance while updating teh index
Posted by oleg_gnatovskiy <ol...@citysearch.com>.
We do optimize the index before updates but we get tehse performance issues
even when we pull an empty snapshot. Thus even when our update is tiny, the
performance issues still happen.
Otis Gospodnetic wrote:
>
> This is an old and long thread, and I no longer recall what the specific
> suggestions were.
> My guess is this has to do with the OS cache of your index files. When
> you make the large index update, that OS cache is useless (old files are
> gone, new ones are in) and the OS cache has get re-warmed and this takes
> time.
>
> Are you optimizing your index before the update? Do you *really* need to
> do that?
> How large is your update, what makes it big, and could you make it
> smaller?
>
> Otis
> --
> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>
>
>
> ----- Original Message ----
>> From: oleg_gnatovskiy <ol...@citysearch.com>
>> To: solr-user@lucene.apache.org
>> Sent: Tuesday, January 20, 2009 6:19:46 PM
>> Subject: Re: Query Performance while updating teh index
>>
>>
>> Hello again. It seems that we are still having these problems. Queries
>> take
>> as long as 20 minutes to get back to their average response time after a
>> large index update, so it doesn't seem like the problem is the 12 second
>> autowarm time. Are there any more suggestions for things we can try?
>> Taking
>> our servers out of teh loop for as long as 20 minutes is a bit of a
>> hassle,
>> and a risk.
>> --
>> View this message in context:
>> http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p21573927.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
>
--
View this message in context: http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p21611642.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query Performance while updating teh index
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Oleg,
This is more of an OS-level thing that Solr-thing, it seems from your emails. If you send answers to my questions we'll be able to help more.
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
> From: oleg_gnatovskiy <ol...@citysearch.com>
> To: solr-user@lucene.apache.org
> Sent: Wednesday, January 21, 2009 1:09:21 PM
> Subject: Re: Query Performance while updating teh index
>
>
> What exactly does Solr do when it receives a new Index? How does it keep
> serving while performing the updates? It seems that the part that causes the
> slowdown is this transition.
>
>
>
>
> Otis Gospodnetic wrote:
> >
> > This is an old and long thread, and I no longer recall what the specific
> > suggestions were.
> > My guess is this has to do with the OS cache of your index files. When
> > you make the large index update, that OS cache is useless (old files are
> > gone, new ones are in) and the OS cache has get re-warmed and this takes
> > time.
> >
> > Are you optimizing your index before the update? Do you *really* need to
> > do that?
> > How large is your update, what makes it big, and could you make it
> > smaller?
> >
> > Otis
> > --
> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
> >
> >
> >
> > ----- Original Message ----
> >> From: oleg_gnatovskiy
> >> To: solr-user@lucene.apache.org
> >> Sent: Tuesday, January 20, 2009 6:19:46 PM
> >> Subject: Re: Query Performance while updating teh index
> >>
> >>
> >> Hello again. It seems that we are still having these problems. Queries
> >> take
> >> as long as 20 minutes to get back to their average response time after a
> >> large index update, so it doesn't seem like the problem is the 12 second
> >> autowarm time. Are there any more suggestions for things we can try?
> >> Taking
> >> our servers out of teh loop for as long as 20 minutes is a bit of a
> >> hassle,
> >> and a risk.
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p21573927.html
> >> Sent from the Solr - User mailing list archive at Nabble.com.
> >
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p21588779.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query Performance while updating teh index
Posted by oleg_gnatovskiy <ol...@citysearch.com>.
What exactly does Solr do when it receives a new Index? How does it keep
serving while performing the updates? It seems that the part that causes the
slowdown is this transition.
Otis Gospodnetic wrote:
>
> This is an old and long thread, and I no longer recall what the specific
> suggestions were.
> My guess is this has to do with the OS cache of your index files. When
> you make the large index update, that OS cache is useless (old files are
> gone, new ones are in) and the OS cache has get re-warmed and this takes
> time.
>
> Are you optimizing your index before the update? Do you *really* need to
> do that?
> How large is your update, what makes it big, and could you make it
> smaller?
>
> Otis
> --
> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>
>
>
> ----- Original Message ----
>> From: oleg_gnatovskiy <ol...@citysearch.com>
>> To: solr-user@lucene.apache.org
>> Sent: Tuesday, January 20, 2009 6:19:46 PM
>> Subject: Re: Query Performance while updating teh index
>>
>>
>> Hello again. It seems that we are still having these problems. Queries
>> take
>> as long as 20 minutes to get back to their average response time after a
>> large index update, so it doesn't seem like the problem is the 12 second
>> autowarm time. Are there any more suggestions for things we can try?
>> Taking
>> our servers out of teh loop for as long as 20 minutes is a bit of a
>> hassle,
>> and a risk.
>> --
>> View this message in context:
>> http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p21573927.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
>
--
View this message in context: http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p21588779.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query Performance while updating teh index
Posted by Otis Gospodnetic <ot...@yahoo.com>.
This is an old and long thread, and I no longer recall what the specific suggestions were.
My guess is this has to do with the OS cache of your index files. When you make the large index update, that OS cache is useless (old files are gone, new ones are in) and the OS cache has get re-warmed and this takes time.
Are you optimizing your index before the update? Do you *really* need to do that?
How large is your update, what makes it big, and could you make it smaller?
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
> From: oleg_gnatovskiy <ol...@citysearch.com>
> To: solr-user@lucene.apache.org
> Sent: Tuesday, January 20, 2009 6:19:46 PM
> Subject: Re: Query Performance while updating teh index
>
>
> Hello again. It seems that we are still having these problems. Queries take
> as long as 20 minutes to get back to their average response time after a
> large index update, so it doesn't seem like the problem is the 12 second
> autowarm time. Are there any more suggestions for things we can try? Taking
> our servers out of teh loop for as long as 20 minutes is a bit of a hassle,
> and a risk.
> --
> View this message in context:
> http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p21573927.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query Performance while updating teh index
Posted by oleg_gnatovskiy <ol...@citysearch.com>.
Hello again. It seems that we are still having these problems. Queries take
as long as 20 minutes to get back to their average response time after a
large index update, so it doesn't seem like the problem is the 12 second
autowarm time. Are there any more suggestions for things we can try? Taking
our servers out of teh loop for as long as 20 minutes is a bit of a hassle,
and a risk.
--
View this message in context: http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p21573927.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query Performance while updating teh index
Posted by Yonik Seeley <ys...@gmail.com>.
Right, query cache typically has a lower hit ratio, and one check per
request - often not worth autowarming much.
The filter cache can be a different story with a higher hit ratio, and
higher number of checks per request.
-Yonik
On Fri, Dec 12, 2008 at 1:35 PM, Feak, Todd <To...@smss.sony.com> wrote:
> It's spending 4-5 seconds warming up your query cache. If 4-5 seconds is
> too much, you could reduce the number of queries to auto-warm with on
> that cache.
>
> Notice that the 4-5 seconds is spent only putting about 420 queries into
> the query cache. Your autowarm of 50000 for the query cache seems a bit
> high. If you need to reduce that autowarm time below 5 seconds, you may
> have to set that value in the hundreds, as opposed to tens of thousands.
>
> -Todd Feak
>
> -----Original Message-----
> From: oleg_gnatovskiy [mailto:oleg_gnatovskiy@citysearch.com]
> Sent: Friday, December 12, 2008 10:08 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Query Performance while updating teh index
>
>
> Here's what we have on one of the data slaves for the autowarming.
>
>
>
> --
>
> Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
>
>
> filterCache{lookups=351993,hits=347055,hitratio=0.98,inserts=8332,evicti
> ons=0,size=8245,warmupTime=215,cumulative_lookups=2837676,cumulative_hit
> s=2766551,cumulative_hitratio=0.97,cumulative_inserts=72050,cumulative_e
> victions=0}
>
> Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming result for Searcher@3f32ca2b main
>
>
> filterCache{lookups=0,hits=0,hitratio=0.00,inserts=1000,evictions=0,size
> =1000,warmupTime=317,cumulative_lookups=2837676,cumulative_hits=2766551,
> cumulative_hitratio=0.97,cumulative_inserts=72050,cumulative_evictions=0
> }
>
> Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
>
>
> queryResultCache{lookups=5309,hits=5223,hitratio=0.98,inserts=422,evicti
> ons=0,size=421,warmupTime=4628,cumulative_lookups=77802,cumulative_hits=
> 77216,cumulative_hitratio=0.99,cumulative_inserts=424,cumulative_evictio
> ns=0}
>
> --
>
> Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming result for Searcher@3f32ca2b main
>
>
> queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=421,evictions=0,
> size=421,warmupTime=5536,cumulative_lookups=77804,cumulative_hits=77218,
> cumulative_hitratio=0.99,cumulative_inserts=424,cumulative_evictions=0}
>
> Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
>
>
> documentCache{lookups=87216,hits=86686,hitratio=0.99,inserts=570,evictio
> ns=0,size=570,warmupTime=0,cumulative_lookups=1270773,cumulative_hits=12
> 68318,cumulative_hitratio=0.99,cumulative_inserts=2455,cumulative_evicti
> ons=0}
>
> Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming result for Searcher@3f32ca2b main
>
>
> documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=
> 0,warmupTime=0,cumulative_lookups=1270773,cumulative_hits=1268318,cumula
> tive_hitratio=0.99,cumulative_inserts=2455,cumulative_evictions=0}
>
> --
>
>
>
> This is our current values after I've messed with them a few times
> trying to
> get better performance.
>
>
>
> <filterCache
>
> class="solr.LRUCache"
>
> size="30000"
>
> initialSize="15000"
>
> autowarmCount="1000"/>
>
> <queryResultCache
>
> class="solr.LRUCache"
>
> size="60000"
>
> initialSize="30000"
>
> autowarmCount="50000"/>
>
> <documentCache
>
> class="solr.LRUCache"
>
> size="200000"
>
> initialSize="125000"
>
> autowarmCount="0"/>
>
>
> --
> View this message in context:
> http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452
> 835p20980669.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
>
RE: Query Performance while updating teh index
Posted by oleg_gnatovskiy <ol...@citysearch.com>.
Should this autowarm value be set based on the number of lookups? From the
info I provided that like 60k. filterCache{lookups=58522
Will 25k be enough?
Also, does that mean that we have to at least increase the size and initial
size as big as we set the autowarm?
Feak, Todd wrote:
>
> Sorry, my bad. Didn't read the entire thread.
>
> Look at your filter cache first. You are autowarming 1000, and there is
> exactly 1000 in there. Yet it looks like there may be tens of thousands
> of filter queries in your system. I would try autowarming more. Try
> 10,000 or 20,000 and see if it helps.
>
> Second look at your document cache. Document caches don't use autowarm.
> But you can add queries to your firstSeacher and newSearcher entries in
> your solrconfig to pre-populate the document cache during warming.
>
> -Todd Feak
>
>
> -----Original Message-----
> From: oleg_gnatovskiy [mailto:oleg_gnatovskiy@citysearch.com]
> Sent: Friday, December 12, 2008 11:19 AM
> To: solr-user@lucene.apache.org
> Subject: RE: Query Performance while updating teh index
>
>
> The auto warm time is not an issue. We take the server off the load
> balancer
> while it is autowarming. It seems that the slowness occurs after
> autowarm is
> done.
>
>
>
> Feak, Todd wrote:
>>
>> It's spending 4-5 seconds warming up your query cache. If 4-5 seconds
> is
>> too much, you could reduce the number of queries to auto-warm with on
>> that cache.
>>
>> Notice that the 4-5 seconds is spent only putting about 420 queries
> into
>> the query cache. Your autowarm of 50000 for the query cache seems a
> bit
>> high. If you need to reduce that autowarm time below 5 seconds, you
> may
>> have to set that value in the hundreds, as opposed to tens of
> thousands.
>>
>> -Todd Feak
>>
>> -----Original Message-----
>> From: oleg_gnatovskiy [mailto:oleg_gnatovskiy@citysearch.com]
>> Sent: Friday, December 12, 2008 10:08 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Query Performance while updating teh index
>>
>>
>> Here's what we have on one of the data slaves for the autowarming.
>>
>>
>>
>> --
>>
>> Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
>>
>> INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
>>
>>
>>
> filterCache{lookups=351993,hits=347055,hitratio=0.98,inserts=8332,evicti
>>
> ons=0,size=8245,warmupTime=215,cumulative_lookups=2837676,cumulative_hit
>>
> s=2766551,cumulative_hitratio=0.97,cumulative_inserts=72050,cumulative_e
>> victions=0}
>>
>> Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
>>
>> INFO: autowarming result for Searcher@3f32ca2b main
>>
>>
>>
> filterCache{lookups=0,hits=0,hitratio=0.00,inserts=1000,evictions=0,size
>>
> =1000,warmupTime=317,cumulative_lookups=2837676,cumulative_hits=2766551,
>>
> cumulative_hitratio=0.97,cumulative_inserts=72050,cumulative_evictions=0
>> }
>>
>> Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
>>
>> INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
>>
>>
>>
> queryResultCache{lookups=5309,hits=5223,hitratio=0.98,inserts=422,evicti
>>
> ons=0,size=421,warmupTime=4628,cumulative_lookups=77802,cumulative_hits=
>>
> 77216,cumulative_hitratio=0.99,cumulative_inserts=424,cumulative_evictio
>> ns=0}
>>
>> --
>>
>> Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
>>
>> INFO: autowarming result for Searcher@3f32ca2b main
>>
>>
>>
> queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=421,evictions=0,
>>
> size=421,warmupTime=5536,cumulative_lookups=77804,cumulative_hits=77218,
>>
> cumulative_hitratio=0.99,cumulative_inserts=424,cumulative_evictions=0}
>>
>> Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
>>
>> INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
>>
>>
>>
> documentCache{lookups=87216,hits=86686,hitratio=0.99,inserts=570,evictio
>>
> ns=0,size=570,warmupTime=0,cumulative_lookups=1270773,cumulative_hits=12
>>
> 68318,cumulative_hitratio=0.99,cumulative_inserts=2455,cumulative_evicti
>> ons=0}
>>
>> Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
>>
>> INFO: autowarming result for Searcher@3f32ca2b main
>>
>>
>>
> documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=
>>
> 0,warmupTime=0,cumulative_lookups=1270773,cumulative_hits=1268318,cumula
>> tive_hitratio=0.99,cumulative_inserts=2455,cumulative_evictions=0}
>>
>> --
>>
>>
>>
>> This is our current values after I've messed with them a few times
>> trying to
>> get better performance.
>>
>>
>>
>> <filterCache
>>
>> class="solr.LRUCache"
>>
>> size="30000"
>>
>> initialSize="15000"
>>
>> autowarmCount="1000"/>
>>
>> <queryResultCache
>>
>> class="solr.LRUCache"
>>
>> size="60000"
>>
>> initialSize="30000"
>>
>> autowarmCount="50000"/>
>>
>> <documentCache
>>
>> class="solr.LRUCache"
>>
>> size="200000"
>>
>> initialSize="125000"
>>
>> autowarmCount="0"/>
>>
>>
>> --
>> View this message in context:
>>
> http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452
>> 835p20980669.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>>
>>
>
> --
> View this message in context:
> http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452
> 835p20981862.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
>
>
--
View this message in context: http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p20982647.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Query Performance while updating teh index
Posted by "Feak, Todd" <To...@smss.sony.com>.
Sorry, my bad. Didn't read the entire thread.
Look at your filter cache first. You are autowarming 1000, and there is
exactly 1000 in there. Yet it looks like there may be tens of thousands
of filter queries in your system. I would try autowarming more. Try
10,000 or 20,000 and see if it helps.
Second look at your document cache. Document caches don't use autowarm.
But you can add queries to your firstSeacher and newSearcher entries in
your solrconfig to pre-populate the document cache during warming.
-Todd Feak
-----Original Message-----
From: oleg_gnatovskiy [mailto:oleg_gnatovskiy@citysearch.com]
Sent: Friday, December 12, 2008 11:19 AM
To: solr-user@lucene.apache.org
Subject: RE: Query Performance while updating teh index
The auto warm time is not an issue. We take the server off the load
balancer
while it is autowarming. It seems that the slowness occurs after
autowarm is
done.
Feak, Todd wrote:
>
> It's spending 4-5 seconds warming up your query cache. If 4-5 seconds
is
> too much, you could reduce the number of queries to auto-warm with on
> that cache.
>
> Notice that the 4-5 seconds is spent only putting about 420 queries
into
> the query cache. Your autowarm of 50000 for the query cache seems a
bit
> high. If you need to reduce that autowarm time below 5 seconds, you
may
> have to set that value in the hundreds, as opposed to tens of
thousands.
>
> -Todd Feak
>
> -----Original Message-----
> From: oleg_gnatovskiy [mailto:oleg_gnatovskiy@citysearch.com]
> Sent: Friday, December 12, 2008 10:08 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Query Performance while updating teh index
>
>
> Here's what we have on one of the data slaves for the autowarming.
>
>
>
> --
>
> Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
>
>
>
filterCache{lookups=351993,hits=347055,hitratio=0.98,inserts=8332,evicti
>
ons=0,size=8245,warmupTime=215,cumulative_lookups=2837676,cumulative_hit
>
s=2766551,cumulative_hitratio=0.97,cumulative_inserts=72050,cumulative_e
> victions=0}
>
> Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming result for Searcher@3f32ca2b main
>
>
>
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=1000,evictions=0,size
>
=1000,warmupTime=317,cumulative_lookups=2837676,cumulative_hits=2766551,
>
cumulative_hitratio=0.97,cumulative_inserts=72050,cumulative_evictions=0
> }
>
> Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
>
>
>
queryResultCache{lookups=5309,hits=5223,hitratio=0.98,inserts=422,evicti
>
ons=0,size=421,warmupTime=4628,cumulative_lookups=77802,cumulative_hits=
>
77216,cumulative_hitratio=0.99,cumulative_inserts=424,cumulative_evictio
> ns=0}
>
> --
>
> Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming result for Searcher@3f32ca2b main
>
>
>
queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=421,evictions=0,
>
size=421,warmupTime=5536,cumulative_lookups=77804,cumulative_hits=77218,
>
cumulative_hitratio=0.99,cumulative_inserts=424,cumulative_evictions=0}
>
> Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
>
>
>
documentCache{lookups=87216,hits=86686,hitratio=0.99,inserts=570,evictio
>
ns=0,size=570,warmupTime=0,cumulative_lookups=1270773,cumulative_hits=12
>
68318,cumulative_hitratio=0.99,cumulative_inserts=2455,cumulative_evicti
> ons=0}
>
> Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming result for Searcher@3f32ca2b main
>
>
>
documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=
>
0,warmupTime=0,cumulative_lookups=1270773,cumulative_hits=1268318,cumula
> tive_hitratio=0.99,cumulative_inserts=2455,cumulative_evictions=0}
>
> --
>
>
>
> This is our current values after I've messed with them a few times
> trying to
> get better performance.
>
>
>
> <filterCache
>
> class="solr.LRUCache"
>
> size="30000"
>
> initialSize="15000"
>
> autowarmCount="1000"/>
>
> <queryResultCache
>
> class="solr.LRUCache"
>
> size="60000"
>
> initialSize="30000"
>
> autowarmCount="50000"/>
>
> <documentCache
>
> class="solr.LRUCache"
>
> size="200000"
>
> initialSize="125000"
>
> autowarmCount="0"/>
>
>
> --
> View this message in context:
>
http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452
> 835p20980669.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
>
>
--
View this message in context:
http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452
835p20981862.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Query Performance while updating teh index
Posted by oleg_gnatovskiy <ol...@citysearch.com>.
I just verified this. The slowness occurs after auto warm is done.
Oleg
--
View this message in context: http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p20982068.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Query Performance while updating teh index
Posted by oleg_gnatovskiy <ol...@citysearch.com>.
The auto warm time is not an issue. We take the server off the load balancer
while it is autowarming. It seems that the slowness occurs after autowarm is
done.
Feak, Todd wrote:
>
> It's spending 4-5 seconds warming up your query cache. If 4-5 seconds is
> too much, you could reduce the number of queries to auto-warm with on
> that cache.
>
> Notice that the 4-5 seconds is spent only putting about 420 queries into
> the query cache. Your autowarm of 50000 for the query cache seems a bit
> high. If you need to reduce that autowarm time below 5 seconds, you may
> have to set that value in the hundreds, as opposed to tens of thousands.
>
> -Todd Feak
>
> -----Original Message-----
> From: oleg_gnatovskiy [mailto:oleg_gnatovskiy@citysearch.com]
> Sent: Friday, December 12, 2008 10:08 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Query Performance while updating teh index
>
>
> Here's what we have on one of the data slaves for the autowarming.
>
>
>
> --
>
> Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
>
>
> filterCache{lookups=351993,hits=347055,hitratio=0.98,inserts=8332,evicti
> ons=0,size=8245,warmupTime=215,cumulative_lookups=2837676,cumulative_hit
> s=2766551,cumulative_hitratio=0.97,cumulative_inserts=72050,cumulative_e
> victions=0}
>
> Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming result for Searcher@3f32ca2b main
>
>
> filterCache{lookups=0,hits=0,hitratio=0.00,inserts=1000,evictions=0,size
> =1000,warmupTime=317,cumulative_lookups=2837676,cumulative_hits=2766551,
> cumulative_hitratio=0.97,cumulative_inserts=72050,cumulative_evictions=0
> }
>
> Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
>
>
> queryResultCache{lookups=5309,hits=5223,hitratio=0.98,inserts=422,evicti
> ons=0,size=421,warmupTime=4628,cumulative_lookups=77802,cumulative_hits=
> 77216,cumulative_hitratio=0.99,cumulative_inserts=424,cumulative_evictio
> ns=0}
>
> --
>
> Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming result for Searcher@3f32ca2b main
>
>
> queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=421,evictions=0,
> size=421,warmupTime=5536,cumulative_lookups=77804,cumulative_hits=77218,
> cumulative_hitratio=0.99,cumulative_inserts=424,cumulative_evictions=0}
>
> Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
>
>
> documentCache{lookups=87216,hits=86686,hitratio=0.99,inserts=570,evictio
> ns=0,size=570,warmupTime=0,cumulative_lookups=1270773,cumulative_hits=12
> 68318,cumulative_hitratio=0.99,cumulative_inserts=2455,cumulative_evicti
> ons=0}
>
> Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming result for Searcher@3f32ca2b main
>
>
> documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=
> 0,warmupTime=0,cumulative_lookups=1270773,cumulative_hits=1268318,cumula
> tive_hitratio=0.99,cumulative_inserts=2455,cumulative_evictions=0}
>
> --
>
>
>
> This is our current values after I've messed with them a few times
> trying to
> get better performance.
>
>
>
> <filterCache
>
> class="solr.LRUCache"
>
> size="30000"
>
> initialSize="15000"
>
> autowarmCount="1000"/>
>
> <queryResultCache
>
> class="solr.LRUCache"
>
> size="60000"
>
> initialSize="30000"
>
> autowarmCount="50000"/>
>
> <documentCache
>
> class="solr.LRUCache"
>
> size="200000"
>
> initialSize="125000"
>
> autowarmCount="0"/>
>
>
> --
> View this message in context:
> http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452
> 835p20980669.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
>
>
--
View this message in context: http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p20981862.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Query Performance while updating teh index
Posted by "Feak, Todd" <To...@smss.sony.com>.
It's spending 4-5 seconds warming up your query cache. If 4-5 seconds is
too much, you could reduce the number of queries to auto-warm with on
that cache.
Notice that the 4-5 seconds is spent only putting about 420 queries into
the query cache. Your autowarm of 50000 for the query cache seems a bit
high. If you need to reduce that autowarm time below 5 seconds, you may
have to set that value in the hundreds, as opposed to tens of thousands.
-Todd Feak
-----Original Message-----
From: oleg_gnatovskiy [mailto:oleg_gnatovskiy@citysearch.com]
Sent: Friday, December 12, 2008 10:08 AM
To: solr-user@lucene.apache.org
Subject: Re: Query Performance while updating teh index
Here's what we have on one of the data slaves for the autowarming.
--
Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
filterCache{lookups=351993,hits=347055,hitratio=0.98,inserts=8332,evicti
ons=0,size=8245,warmupTime=215,cumulative_lookups=2837676,cumulative_hit
s=2766551,cumulative_hitratio=0.97,cumulative_inserts=72050,cumulative_e
victions=0}
Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@3f32ca2b main
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=1000,evictions=0,size
=1000,warmupTime=317,cumulative_lookups=2837676,cumulative_hits=2766551,
cumulative_hitratio=0.97,cumulative_inserts=72050,cumulative_evictions=0
}
Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
queryResultCache{lookups=5309,hits=5223,hitratio=0.98,inserts=422,evicti
ons=0,size=421,warmupTime=4628,cumulative_lookups=77802,cumulative_hits=
77216,cumulative_hitratio=0.99,cumulative_inserts=424,cumulative_evictio
ns=0}
--
Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@3f32ca2b main
queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=421,evictions=0,
size=421,warmupTime=5536,cumulative_lookups=77804,cumulative_hits=77218,
cumulative_hitratio=0.99,cumulative_inserts=424,cumulative_evictions=0}
Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
documentCache{lookups=87216,hits=86686,hitratio=0.99,inserts=570,evictio
ns=0,size=570,warmupTime=0,cumulative_lookups=1270773,cumulative_hits=12
68318,cumulative_hitratio=0.99,cumulative_inserts=2455,cumulative_evicti
ons=0}
Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@3f32ca2b main
documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=
0,warmupTime=0,cumulative_lookups=1270773,cumulative_hits=1268318,cumula
tive_hitratio=0.99,cumulative_inserts=2455,cumulative_evictions=0}
--
This is our current values after I've messed with them a few times
trying to
get better performance.
<filterCache
class="solr.LRUCache"
size="30000"
initialSize="15000"
autowarmCount="1000"/>
<queryResultCache
class="solr.LRUCache"
size="60000"
initialSize="30000"
autowarmCount="50000"/>
<documentCache
class="solr.LRUCache"
size="200000"
initialSize="125000"
autowarmCount="0"/>
--
View this message in context:
http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452
835p20980669.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query Performance while updating teh index
Posted by Otis Gospodnetic <ot...@yahoo.com>.
It looks like cache warming is taking about 12 seconds. It sounds like you need to see if performance is bad during warming, or right after warming (and right after the new searcher gets exposed to queries).
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
> From: oleg_gnatovskiy <ol...@citysearch.com>
> To: solr-user@lucene.apache.org
> Sent: Friday, December 12, 2008 1:07:49 PM
> Subject: Re: Query Performance while updating teh index
>
>
> Here’s what we have on one of the data slaves for the autowarming.
>
>
>
> --
>
> Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
>
>
> filterCache{lookups=351993,hits=347055,hitratio=0.98,inserts=8332,evictions=0,size=8245,warmupTime=215,cumulative_lookups=2837676,cumulative_hits=2766551,cumulative_hitratio=0.97,cumulative_inserts=72050,cumulative_evictions=0}
>
> Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming result for Searcher@3f32ca2b main
>
>
> filterCache{lookups=0,hits=0,hitratio=0.00,inserts=1000,evictions=0,size=1000,warmupTime=317,cumulative_lookups=2837676,cumulative_hits=2766551,cumulative_hitratio=0.97,cumulative_inserts=72050,cumulative_evictions=0}
>
> Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
>
>
> queryResultCache{lookups=5309,hits=5223,hitratio=0.98,inserts=422,evictions=0,size=421,warmupTime=4628,cumulative_lookups=77802,cumulative_hits=77216,cumulative_hitratio=0.99,cumulative_inserts=424,cumulative_evictions=0}
>
> --
>
> Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming result for Searcher@3f32ca2b main
>
>
> queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=421,evictions=0,size=421,warmupTime=5536,cumulative_lookups=77804,cumulative_hits=77218,cumulative_hitratio=0.99,cumulative_inserts=424,cumulative_evictions=0}
>
> Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
>
>
> documentCache{lookups=87216,hits=86686,hitratio=0.99,inserts=570,evictions=0,size=570,warmupTime=0,cumulative_lookups=1270773,cumulative_hits=1268318,cumulative_hitratio=0.99,cumulative_inserts=2455,cumulative_evictions=0}
>
> Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
>
> INFO: autowarming result for Searcher@3f32ca2b main
>
>
> documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=1270773,cumulative_hits=1268318,cumulative_hitratio=0.99,cumulative_inserts=2455,cumulative_evictions=0}
>
> --
>
>
>
> This is our current values after I’ve messed with them a few times trying to
> get better performance.
>
>
>
>
>
> class="solr.LRUCache"
>
> size="30000"
>
> initialSize="15000"
>
> autowarmCount="1000"/>
>
>
>
> class="solr.LRUCache"
>
> size="60000"
>
> initialSize="30000"
>
> autowarmCount="50000"/>
>
>
>
> class="solr.LRUCache"
>
> size="200000"
>
> initialSize="125000"
>
> autowarmCount="0"/>
>
>
> --
> View this message in context:
> http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p20980669.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query Performance while updating teh index
Posted by oleg_gnatovskiy <ol...@citysearch.com>.
Here’s what we have on one of the data slaves for the autowarming.
--
Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
filterCache{lookups=351993,hits=347055,hitratio=0.98,inserts=8332,evictions=0,size=8245,warmupTime=215,cumulative_lookups=2837676,cumulative_hits=2766551,cumulative_hitratio=0.97,cumulative_inserts=72050,cumulative_evictions=0}
Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@3f32ca2b main
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=1000,evictions=0,size=1000,warmupTime=317,cumulative_lookups=2837676,cumulative_hits=2766551,cumulative_hitratio=0.97,cumulative_inserts=72050,cumulative_evictions=0}
Dec 12, 2008 8:46:02 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
queryResultCache{lookups=5309,hits=5223,hitratio=0.98,inserts=422,evictions=0,size=421,warmupTime=4628,cumulative_lookups=77802,cumulative_hits=77216,cumulative_hitratio=0.99,cumulative_inserts=424,cumulative_evictions=0}
--
Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@3f32ca2b main
queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=421,evictions=0,size=421,warmupTime=5536,cumulative_lookups=77804,cumulative_hits=77218,cumulative_hitratio=0.99,cumulative_inserts=424,cumulative_evictions=0}
Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming Searcher@3f32ca2b main from Searcher@443ad545 main
documentCache{lookups=87216,hits=86686,hitratio=0.99,inserts=570,evictions=0,size=570,warmupTime=0,cumulative_lookups=1270773,cumulative_hits=1268318,cumulative_hitratio=0.99,cumulative_inserts=2455,cumulative_evictions=0}
Dec 12, 2008 8:46:07 AM org.apache.solr.search.SolrIndexSearcher warm
INFO: autowarming result for Searcher@3f32ca2b main
documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=1270773,cumulative_hits=1268318,cumulative_hitratio=0.99,cumulative_inserts=2455,cumulative_evictions=0}
--
This is our current values after I’ve messed with them a few times trying to
get better performance.
<filterCache
class="solr.LRUCache"
size="30000"
initialSize="15000"
autowarmCount="1000"/>
<queryResultCache
class="solr.LRUCache"
size="60000"
initialSize="30000"
autowarmCount="50000"/>
<documentCache
class="solr.LRUCache"
size="200000"
initialSize="125000"
autowarmCount="0"/>
--
View this message in context: http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p20980669.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query Performance while updating teh index
Posted by oleg_gnatovskiy <ol...@citysearch.com>.
Hey Otis,
Do you think our problem is slow warm time, or too few items that are being
copied?
Oleg
--
View this message in context: http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p20980523.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query Performance while updating teh index
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Oleg,
The reliable formula is situation-specific, I think. One sure way to decrease the warm time is to minimize the number of items to copy from old caches to new caches on warmup.
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
> From: oleg_gnatovskiy <ol...@citysearch.com>
> To: solr-user@lucene.apache.org
> Sent: Thursday, December 11, 2008 8:43:26 PM
> Subject: RE: Query Performance while updating teh index
>
>
> We are still having this problem. I am wondering if it can be fixed with
> autowarm settings. Is there a reliable formula for determining the autowarm
> settings?
> --
> View this message in context:
> http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p20968516.html
> Sent from the Solr - User mailing list archive at Nabble.com.