You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Cloude Porteus <cl...@instructables.com> on 2009/03/25 06:06:51 UTC
Snapinstaller + Overlapping onDeckSearchers Problems
We have been running our solr slaves without autowarming our new searchers
for a long time, but that was causing us 50-75 requests in 20+ seconds
timeframe after every update on the slaves. I have turned on autowarming and
that has fixed our slow response times, but I'm running into occasional
Overlapping onDeckSearchers.
We have replication setup and are using the snapinstaller script every 10
minutes:
/home/solr/bin/snappuller -M util01 -P 18984 -D /home/solr/write/data -S
/home/solr/logs -d /home/solr/read/data -u instruct;
/home/solr/bin/snapinstaller -M util01 -S /home/solr/write/logs -d
/home/solr/read/data -u instruct
Here's what a successful update/commit log looks like:
[14:13:02.510] start
commit(optimize=false,waitFlush=false,waitSearcher=true)
[14:13:02.522] Opening Searcher@e9b4bb main
[14:13:02.524] end_commit_flush
[14:13:02.525] autowarming Searcher@e9b4bb main from Searcher@159e6e8 main
[14:13:02.525]
filterCache{lookups=1809739,hits=1766607,hitratio=0.97,inserts=43211,evictions=0,
size=43154,cumulative_lookups=1809739,cumulative_hits=1766607,cumulative_hitratio=0.97,cumulative_inserts=43211,cumulative_evictions=0}
--
[14:15:42.372] {commit=} 0 159964
[14:15:42.373] /update 0 159964
Here's what a unsuccessful update/commit log looks like, where the /update
took too long and we started another commit:
[21:03:03.829] start
commit(optimize=false,waitFlush=false,waitSearcher=true)
[21:03:03.836] Opening Searcher@b2f2d6 main
[21:03:03.836] end_commit_flush
[21:03:03.836] autowarming Searcher@b2f2d6 main from Searcher@103c520 main
[21:03:03.836]
filterCache{lookups=1062196,hits=1062160,hitratio=0.99,inserts=49144,evictions=0,size=48353,cumulative_lookups=259485564,cumulative_hits=259426904,cumulative_hitratio=0.99,cumulative_inserts=68467,cumulative_evictions=0}
--
[21:23:04.794] start
commit(optimize=false,waitFlush=false,waitSearcher=true)
[21:23:04.794] PERFORMANCE WARNING: Overlapping onDeckSearchers=2
[21:23:04.802] Opening Searcher@f11bc main
[21:23:04.802] end_commit_flush
--
[21:24:55.987] {commit=} 0 1312158
[21:24:55.987] /update 0 1312158
I don't understand why this sometimes takes two minutes between the start
commit & /update and sometimes takes 20 minutes? One of our caches has about
~40,000 items, but I can't imagine it taking 20 minutes to autowarm a
searcher.
It would be super handy if the Snapinstaller script would wait until the
previous one was done before starting a new one, but I'm not sure how to
make that happen.
Thanks for any help with this.
best,
cloude
--
VP of Product Development
Instructables.com
http://www.instructables.com/member/lebowski
Re: Snapinstaller + Overlapping onDeckSearchers Problems
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Hm, I can't quite tell from here, but that is just a warning, so it's not super problematic at this point.
Could it be that one of your other caches (query cache) is large and lots of items are copied on searcher flip?
Could it be that your JVM doesn't have large or free enough enough heap? Can you tell if lots of GCing happens during the searcher flip?
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
> From: Cloude Porteus <cl...@instructables.com>
> To: solr-user@lucene.apache.org
> Sent: Wednesday, March 25, 2009 1:06:51 AM
> Subject: Snapinstaller + Overlapping onDeckSearchers Problems
>
> We have been running our solr slaves without autowarming our new searchers
> for a long time, but that was causing us 50-75 requests in 20+ seconds
> timeframe after every update on the slaves. I have turned on autowarming and
> that has fixed our slow response times, but I'm running into occasional
> Overlapping onDeckSearchers.
>
> We have replication setup and are using the snapinstaller script every 10
> minutes:
>
> /home/solr/bin/snappuller -M util01 -P 18984 -D /home/solr/write/data -S
> /home/solr/logs -d /home/solr/read/data -u instruct;
> /home/solr/bin/snapinstaller -M util01 -S /home/solr/write/logs -d
> /home/solr/read/data -u instruct
>
> Here's what a successful update/commit log looks like:
>
> [14:13:02.510] start
> commit(optimize=false,waitFlush=false,waitSearcher=true)
> [14:13:02.522] Opening Searcher@e9b4bb main
> [14:13:02.524] end_commit_flush
> [14:13:02.525] autowarming Searcher@e9b4bb main from Searcher@159e6e8 main
> [14:13:02.525]
> filterCache{lookups=1809739,hits=1766607,hitratio=0.97,inserts=43211,evictions=0,
> size=43154,cumulative_lookups=1809739,cumulative_hits=1766607,cumulative_hitratio=0.97,cumulative_inserts=43211,cumulative_evictions=0}
> --
> [14:15:42.372] {commit=} 0 159964
> [14:15:42.373] /update 0 159964
>
> Here's what a unsuccessful update/commit log looks like, where the /update
> took too long and we started another commit:
>
> [21:03:03.829] start
> commit(optimize=false,waitFlush=false,waitSearcher=true)
> [21:03:03.836] Opening Searcher@b2f2d6 main
> [21:03:03.836] end_commit_flush
> [21:03:03.836] autowarming Searcher@b2f2d6 main from Searcher@103c520 main
> [21:03:03.836]
> filterCache{lookups=1062196,hits=1062160,hitratio=0.99,inserts=49144,evictions=0,size=48353,cumulative_lookups=259485564,cumulative_hits=259426904,cumulative_hitratio=0.99,cumulative_inserts=68467,cumulative_evictions=0}
> --
> [21:23:04.794] start
> commit(optimize=false,waitFlush=false,waitSearcher=true)
> [21:23:04.794] PERFORMANCE WARNING: Overlapping onDeckSearchers=2
> [21:23:04.802] Opening Searcher@f11bc main
> [21:23:04.802] end_commit_flush
> --
> [21:24:55.987] {commit=} 0 1312158
> [21:24:55.987] /update 0 1312158
>
>
> I don't understand why this sometimes takes two minutes between the start
> commit & /update and sometimes takes 20 minutes? One of our caches has about
> ~40,000 items, but I can't imagine it taking 20 minutes to autowarm a
> searcher.
>
> It would be super handy if the Snapinstaller script would wait until the
> previous one was done before starting a new one, but I'm not sure how to
> make that happen.
>
> Thanks for any help with this.
>
> best,
> cloude
>
> --
> VP of Product Development
> Instructables.com
>
> http://www.instructables.com/member/lebowski
Re: Snapinstaller + Overlapping onDeckSearchers Problems
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Hi,
If you want to fill up the new cache set the autowarmCount to something high (e.g. same number as the cache size), but be prepared to pay the price in warmupTime and thus hit those onDeckSearchers warming again.
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
> From: Cloude Porteus <cl...@instructables.com>
> To: solr-user@lucene.apache.org
> Sent: Wednesday, March 25, 2009 11:48:30 AM
> Subject: Re: Snapinstaller + Overlapping onDeckSearchers Problems
>
> Yes, I guess I'm running 40k queries when it starts :) I didn't know that
> each count was equal to a query. I thought it was just copying the cache
> entries from the previous searcher, but I guess that wouldn't include new
> entries. I set it to the size of our filterCache. What should I set the the
> autowarmCount to if I want to try and fill up the caches?
>
> lookups : 8720372
> hits : 8676170
> hitratio : 0.99
> inserts : 44551
> evictions : 0
> size : 44417
> cumulative_lookups : 8720372
> cumulative_hits : 8676170
> cumulative_hitratio : 0.99
> cumulative_inserts : 44551
> cumulative_evictions : 0
>
> best,
> cloude
>
> On Wed, Mar 25, 2009 at 8:38 AM, Ryan McKinley wrote:
> >>
> >> I don't understand why this sometimes takes two minutes between the start
> >> commit & /update and sometimes takes 20 minutes? One of our caches has
> about
> >> ~40,000 items, but I can't imagine it taking 20 minutes to autowarm a
> >> searcher.
> >
> >
> > What do your cache configs look like?
> >
> > How big is the autowarm count?
> >
> > If you have:
> >
> > class="solr.LRUCache"
> > size="512"
> > initialSize="512"
> > autowarmCount="32"/>
> >
> > that will run 32 queries when solr starts. Are you running 40K queries
> when it starts?
> >
> >
> > ryan
> >
> >
>
>
>
> --
> VP of Product Development
> Instructables.com
>
> http://www.instructables.com/member/lebowski
Re: Snapinstaller + Overlapping onDeckSearchers Problems
Posted by Cloude Porteus <cl...@instructables.com>.
I set the autowarm to 2000, which only takes about two minutes and resolves
my issues.
Thanks for your help!
best,
cloude
On Wed, Mar 25, 2009 at 9:34 AM, Ryan McKinley <ry...@gmail.com> wrote:
> It looks like the cache is configured big enough, but the autowarm count is
> too big to have good performance.
>
> Try something smaller and see if that fixes both problems. I imagine even
> just warming the most recent 100 queries would precache the most important
> ones, but try some higher numbers and see if the performance is acceptable.
>
> for the filterCache and queryCache, autowarm queries the new index and
> caches the results.
>
>
>
>
> On Mar 25, 2009, at 11:48 AM, Cloude Porteus wrote:
>
> Yes, I guess I'm running 40k queries when it starts :) I didn't know that
>> each count was equal to a query. I thought it was just copying the cache
>> entries from the previous searcher, but I guess that wouldn't include new
>> entries. I set it to the size of our filterCache. What should I set the
>> the
>> autowarmCount to if I want to try and fill up the caches?
>>
>> lookups : 8720372
>> hits : 8676170
>> hitratio : 0.99
>> inserts : 44551
>> evictions : 0
>> size : 44417
>> cumulative_lookups : 8720372
>> cumulative_hits : 8676170
>> cumulative_hitratio : 0.99
>> cumulative_inserts : 44551
>> cumulative_evictions : 0
>>
>> best,
>> cloude
>>
>> On Wed, Mar 25, 2009 at 8:38 AM, Ryan McKinley <ry...@gmail.com> wrote:
>>
>>>
>>>> I don't understand why this sometimes takes two minutes between the
>>>> start
>>>> commit & /update and sometimes takes 20 minutes? One of our caches has
>>>>
>>> about
>>
>>> ~40,000 items, but I can't imagine it taking 20 minutes to autowarm a
>>>> searcher.
>>>>
>>>
>>>
>>> What do your cache configs look like?
>>>
>>> How big is the autowarm count?
>>>
>>> If you have:
>>> <queryResultCache
>>> class="solr.LRUCache"
>>> size="512"
>>> initialSize="512"
>>> autowarmCount="32"/>
>>>
>>> that will run 32 queries when solr starts. Are you running 40K queries
>>>
>> when it starts?
>>
>>>
>>>
>>> ryan
>>>
>>>
>>>
>>
>>
>> --
>> VP of Product Development
>> Instructables.com
>>
>> http://www.instructables.com/member/lebowski
>>
>
>
--
VP of Product Development
Instructables.com
http://www.instructables.com/member/lebowski
Re: Snapinstaller + Overlapping onDeckSearchers Problems
Posted by Ryan McKinley <ry...@gmail.com>.
It looks like the cache is configured big enough, but the autowarm
count is too big to have good performance.
Try something smaller and see if that fixes both problems. I imagine
even just warming the most recent 100 queries would precache the most
important ones, but try some higher numbers and see if the performance
is acceptable.
for the filterCache and queryCache, autowarm queries the new index and
caches the results.
On Mar 25, 2009, at 11:48 AM, Cloude Porteus wrote:
> Yes, I guess I'm running 40k queries when it starts :) I didn't know
> that
> each count was equal to a query. I thought it was just copying the
> cache
> entries from the previous searcher, but I guess that wouldn't
> include new
> entries. I set it to the size of our filterCache. What should I set
> the the
> autowarmCount to if I want to try and fill up the caches?
>
> lookups : 8720372
> hits : 8676170
> hitratio : 0.99
> inserts : 44551
> evictions : 0
> size : 44417
> cumulative_lookups : 8720372
> cumulative_hits : 8676170
> cumulative_hitratio : 0.99
> cumulative_inserts : 44551
> cumulative_evictions : 0
>
> best,
> cloude
>
> On Wed, Mar 25, 2009 at 8:38 AM, Ryan McKinley <ry...@gmail.com>
> wrote:
>>>
>>> I don't understand why this sometimes takes two minutes between
>>> the start
>>> commit & /update and sometimes takes 20 minutes? One of our caches
>>> has
> about
>>> ~40,000 items, but I can't imagine it taking 20 minutes to
>>> autowarm a
>>> searcher.
>>
>>
>> What do your cache configs look like?
>>
>> How big is the autowarm count?
>>
>> If you have:
>> <queryResultCache
>> class="solr.LRUCache"
>> size="512"
>> initialSize="512"
>> autowarmCount="32"/>
>>
>> that will run 32 queries when solr starts. Are you running 40K
>> queries
> when it starts?
>>
>>
>> ryan
>>
>>
>
>
>
> --
> VP of Product Development
> Instructables.com
>
> http://www.instructables.com/member/lebowski
Re: Snapinstaller + Overlapping onDeckSearchers Problems
Posted by Cloude Porteus <cl...@instructables.com>.
Yes, I guess I'm running 40k queries when it starts :) I didn't know that
each count was equal to a query. I thought it was just copying the cache
entries from the previous searcher, but I guess that wouldn't include new
entries. I set it to the size of our filterCache. What should I set the the
autowarmCount to if I want to try and fill up the caches?
lookups : 8720372
hits : 8676170
hitratio : 0.99
inserts : 44551
evictions : 0
size : 44417
cumulative_lookups : 8720372
cumulative_hits : 8676170
cumulative_hitratio : 0.99
cumulative_inserts : 44551
cumulative_evictions : 0
best,
cloude
On Wed, Mar 25, 2009 at 8:38 AM, Ryan McKinley <ry...@gmail.com> wrote:
>>
>> I don't understand why this sometimes takes two minutes between the start
>> commit & /update and sometimes takes 20 minutes? One of our caches has
about
>> ~40,000 items, but I can't imagine it taking 20 minutes to autowarm a
>> searcher.
>
>
> What do your cache configs look like?
>
> How big is the autowarm count?
>
> If you have:
> <queryResultCache
> class="solr.LRUCache"
> size="512"
> initialSize="512"
> autowarmCount="32"/>
>
> that will run 32 queries when solr starts. Are you running 40K queries
when it starts?
>
>
> ryan
>
>
--
VP of Product Development
Instructables.com
http://www.instructables.com/member/lebowski
Re: Snapinstaller + Overlapping onDeckSearchers Problems
Posted by Ryan McKinley <ry...@gmail.com>.
>
> I don't understand why this sometimes takes two minutes between the
> start
> commit & /update and sometimes takes 20 minutes? One of our caches
> has about
> ~40,000 items, but I can't imagine it taking 20 minutes to autowarm a
> searcher.
What do your cache configs look like?
How big is the autowarm count?
If you have:
<queryResultCache
class="solr.LRUCache"
size="512"
initialSize="512"
autowarmCount="32"/>
that will run 32 queries when solr starts. Are you running 40K
queries when it starts?
ryan