You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Matt Kuiper <ma...@issinc.com> on 2014/04/15 19:55:03 UTC
cache warming questions
Hello,
I have a few questions regarding how Solr caches are warmed.
My understanding is that there are two ways to warm internal Solr caches (only one way for document cache and lucene FieldCache):
Auto warming - occurs when there is a current searcher handling requests and new searcher is being prepared. "When a new searcher is opened, its caches may be prepopulated or "autowarmed" with cached object from caches in the old searcher. autowarmCount is the number of cached items that will be regenerated in the new searcher." http://wiki.apache.org/solr/SolrCaching#autowarmCount
Explicit warming - where the static warming queries specified in Solrconfig.xml for newSearcher and firstSearcher listeners are executed when a new searcher is being prepared.
What does it mean that items will be regenerated or prepopulated from the current searcher's cache to the new searcher's cache? I doubt it means copy, as the index has likely changed with a commit and possibly invalidated some contents of the cache. Are the queries, or filters, that define the contents of the current caches re-executed for the new searcher's caches?
For the case where auto warming is configured, a current searcher is active, and static warming queries are defined how does auto warming and explicit warming work together? Or do they? Is only one type of warming activated to fill the caches?
Thanks,
Matt
Re: cache warming questions
Posted by Kranti Parisa <kr...@gmail.com>.
cool, thanks.
Thanks,
Kranti K. Parisa
http://www.linkedin.com/in/krantiparisa
On Thu, Apr 17, 2014 at 11:37 PM, Erick Erickson <er...@gmail.com>wrote:
> No, the 5 most recently used in a query will be used to autowarm.
>
> If you have things you _know_ are going to be popular fqs, you could
> put them in newSearcher queries.
>
> Best,
> Erick
>
> On Thu, Apr 17, 2014 at 4:51 PM, Kranti Parisa <kr...@gmail.com>
> wrote:
> > Erik,
> >
> > I have a followup question on this topic.
> >
> > If we have used 10 unique FQs and when we configure filterCache=100 &
> > autoWarm=5, then which 5 out of the 10 will be repopulated in the case of
> > new searcher?
> >
> > I don't think there is a way to set the preference or there is?
> >
> >
> > Thanks,
> > Kranti K. Parisa
> > http://www.linkedin.com/in/krantiparisa
> >
> >
> >
> > On Thu, Apr 17, 2014 at 5:25 PM, Matt Kuiper <ma...@issinc.com>
> wrote:
> >
> >> Ok, that makes sense.
> >>
> >> Thanks again,
> >> Matt
> >>
> >> Matt Kuiper - Software Engineer
> >> Intelligent Software Solutions
> >> p. 719.452.7721 | matt.kuiper@issinc.com
> >> www.issinc.com | LinkedIn: intelligent-software-solutions
> >>
> >> -----Original Message-----
> >> From: Erick Erickson [mailto:erickerickson@gmail.com]
> >> Sent: Thursday, April 17, 2014 9:26 AM
> >> To: solr-user@lucene.apache.org
> >> Subject: Re: cache warming questions
> >>
> >> Don't go overboard warming here, you often hit diminishing returns very
> >> quickly. For instance, if the size is 512 you might set your autowarm
> count
> >> to 16 and get the most bang for your buck. Beyond some (usually small)
> >> number, the additional work you put in to warming is wasted. This is
> >> especially true if your autocommit (soft, or hard with
> >> openSearcher=true) is short.
> >>
> >> So while you're correct in your sizing bit, practically it's rarely that
> >> complicated since the autowarm count is usually so much smaller than the
> >> size that there's no danger of swapping them out. YMMV of course.
> >>
> >> Best,
> >> Erick
> >>
> >> On Wed, Apr 16, 2014 at 10:33 AM, Matt Kuiper <ma...@issinc.com>
> >> wrote:
> >> > Thanks Erick, this is helpful information!
> >> >
> >> > So it sounds like, at minimum the cache size (at least for filterCache
> >> and queryResultCache) should be the sum of the autowarmCount for that
> cache
> >> and the number of queries defined for the newSearcher listener.
> Otherwise
> >> some items in the caches will be evicted right away.
> >> >
> >> > Matt
> >> >
> >> > -----Original Message-----
> >> > From: Erick Erickson [mailto:erickerickson@gmail.com]
> >> > Sent: Tuesday, April 15, 2014 5:21 PM
> >> > To: solr-user@lucene.apache.org
> >> > Subject: Re: cache warming questions
> >> >
> >> > bq: What does it mean that items will be regenerated or prepopulated
> >> from the current searcher's cache...
> >> >
> >> > You're right, the values aren't cached. They can't be since the
> internal
> >> Lucene document id is used to identify docs, and due to merging the
> >> internal ID may bear no relation to the old internal ID for a particular
> >> document.
> >> >
> >> > I find it useful to think of Solr's caches as a map where the key is
> >> the "query" and the value is some representation of the found documents.
> >> The details of the value don't matter, so I'll skip them.
> >> >
> >> > What matters is the key. Consider the filter cache. You put something
> >> like &fq=price:[0 TO 100] on a URL. Solr then uses the fq clause as the
> >> key to the filterCache.
> >> >
> >> > Here's the sneaky bit. When you specify an autowarm count of N for the
> >> filterCache, when a new searcher is opened the first N keys from the map
> >> are re-executed in the new searcher's context and the results put into
> the
> >> new searcher's filterCache.
> >> >
> >> > bq: ...how does auto warming and explicit warming work together?
> >> >
> >> > They're orthogonal. IOW, the autowarming for each cache is executed as
> >> well as the newSearcher static warming queries. Use the static queries
> to
> >> do things like fill the sort caches etc.
> >> >
> >> > Incidentally, this bears on why there's a "firstSearcher" and
> >> "newSearcher". The newSearcher queries are run in addition to the cache
> >> autowarms. firstSearcher static queries are only run when a Solr server
> is
> >> started the first time, and there are no cache entries to autowarm. So
> the
> >> firstSearcher queries might be quite a bit more complex than newSearcher
> >> queries.
> >> >
> >> > HTH,
> >> > Erick
> >> >
> >> > On Tue, Apr 15, 2014 at 1:55 PM, Matt Kuiper <ma...@issinc.com>
> >> wrote:
> >> >> Hello,
> >> >>
> >> >> I have a few questions regarding how Solr caches are warmed.
> >> >>
> >> >> My understanding is that there are two ways to warm internal Solr
> >> caches (only one way for document cache and lucene FieldCache):
> >> >>
> >> >> Auto warming - occurs when there is a current searcher handling
> >> requests and new searcher is being prepared. "When a new searcher is
> >> opened, its caches may be prepopulated or "autowarmed" with cached
> object
> >> from caches in the old searcher. autowarmCount is the number of cached
> >> items that will be regenerated in the new searcher."
> >> http://wiki.apache.org/solr/SolrCaching#autowarmCount
> >> >>
> >> >> Explicit warming - where the static warming queries specified in
> >> Solrconfig.xml for newSearcher and firstSearcher listeners are executed
> >> when a new searcher is being prepared.
> >> >>
> >> >> What does it mean that items will be regenerated or prepopulated from
> >> the current searcher's cache to the new searcher's cache? I doubt it
> means
> >> copy, as the index has likely changed with a commit and possibly
> >> invalidated some contents of the cache. Are the queries, or filters,
> that
> >> define the contents of the current caches re-executed for the new
> >> searcher's caches?
> >> >>
> >> >> For the case where auto warming is configured, a current searcher is
> >> active, and static warming queries are defined how does auto warming and
> >> explicit warming work together? Or do they? Is only one type of warming
> >> activated to fill the caches?
> >> >>
> >> >> Thanks,
> >> >> Matt
> >>
>
Re: cache warming questions
Posted by Erick Erickson <er...@gmail.com>.
No, the 5 most recently used in a query will be used to autowarm.
If you have things you _know_ are going to be popular fqs, you could
put them in newSearcher queries.
Best,
Erick
On Thu, Apr 17, 2014 at 4:51 PM, Kranti Parisa <kr...@gmail.com> wrote:
> Erik,
>
> I have a followup question on this topic.
>
> If we have used 10 unique FQs and when we configure filterCache=100 &
> autoWarm=5, then which 5 out of the 10 will be repopulated in the case of
> new searcher?
>
> I don't think there is a way to set the preference or there is?
>
>
> Thanks,
> Kranti K. Parisa
> http://www.linkedin.com/in/krantiparisa
>
>
>
> On Thu, Apr 17, 2014 at 5:25 PM, Matt Kuiper <ma...@issinc.com> wrote:
>
>> Ok, that makes sense.
>>
>> Thanks again,
>> Matt
>>
>> Matt Kuiper - Software Engineer
>> Intelligent Software Solutions
>> p. 719.452.7721 | matt.kuiper@issinc.com
>> www.issinc.com | LinkedIn: intelligent-software-solutions
>>
>> -----Original Message-----
>> From: Erick Erickson [mailto:erickerickson@gmail.com]
>> Sent: Thursday, April 17, 2014 9:26 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: cache warming questions
>>
>> Don't go overboard warming here, you often hit diminishing returns very
>> quickly. For instance, if the size is 512 you might set your autowarm count
>> to 16 and get the most bang for your buck. Beyond some (usually small)
>> number, the additional work you put in to warming is wasted. This is
>> especially true if your autocommit (soft, or hard with
>> openSearcher=true) is short.
>>
>> So while you're correct in your sizing bit, practically it's rarely that
>> complicated since the autowarm count is usually so much smaller than the
>> size that there's no danger of swapping them out. YMMV of course.
>>
>> Best,
>> Erick
>>
>> On Wed, Apr 16, 2014 at 10:33 AM, Matt Kuiper <ma...@issinc.com>
>> wrote:
>> > Thanks Erick, this is helpful information!
>> >
>> > So it sounds like, at minimum the cache size (at least for filterCache
>> and queryResultCache) should be the sum of the autowarmCount for that cache
>> and the number of queries defined for the newSearcher listener. Otherwise
>> some items in the caches will be evicted right away.
>> >
>> > Matt
>> >
>> > -----Original Message-----
>> > From: Erick Erickson [mailto:erickerickson@gmail.com]
>> > Sent: Tuesday, April 15, 2014 5:21 PM
>> > To: solr-user@lucene.apache.org
>> > Subject: Re: cache warming questions
>> >
>> > bq: What does it mean that items will be regenerated or prepopulated
>> from the current searcher's cache...
>> >
>> > You're right, the values aren't cached. They can't be since the internal
>> Lucene document id is used to identify docs, and due to merging the
>> internal ID may bear no relation to the old internal ID for a particular
>> document.
>> >
>> > I find it useful to think of Solr's caches as a map where the key is
>> the "query" and the value is some representation of the found documents.
>> The details of the value don't matter, so I'll skip them.
>> >
>> > What matters is the key. Consider the filter cache. You put something
>> like &fq=price:[0 TO 100] on a URL. Solr then uses the fq clause as the
>> key to the filterCache.
>> >
>> > Here's the sneaky bit. When you specify an autowarm count of N for the
>> filterCache, when a new searcher is opened the first N keys from the map
>> are re-executed in the new searcher's context and the results put into the
>> new searcher's filterCache.
>> >
>> > bq: ...how does auto warming and explicit warming work together?
>> >
>> > They're orthogonal. IOW, the autowarming for each cache is executed as
>> well as the newSearcher static warming queries. Use the static queries to
>> do things like fill the sort caches etc.
>> >
>> > Incidentally, this bears on why there's a "firstSearcher" and
>> "newSearcher". The newSearcher queries are run in addition to the cache
>> autowarms. firstSearcher static queries are only run when a Solr server is
>> started the first time, and there are no cache entries to autowarm. So the
>> firstSearcher queries might be quite a bit more complex than newSearcher
>> queries.
>> >
>> > HTH,
>> > Erick
>> >
>> > On Tue, Apr 15, 2014 at 1:55 PM, Matt Kuiper <ma...@issinc.com>
>> wrote:
>> >> Hello,
>> >>
>> >> I have a few questions regarding how Solr caches are warmed.
>> >>
>> >> My understanding is that there are two ways to warm internal Solr
>> caches (only one way for document cache and lucene FieldCache):
>> >>
>> >> Auto warming - occurs when there is a current searcher handling
>> requests and new searcher is being prepared. "When a new searcher is
>> opened, its caches may be prepopulated or "autowarmed" with cached object
>> from caches in the old searcher. autowarmCount is the number of cached
>> items that will be regenerated in the new searcher."
>> http://wiki.apache.org/solr/SolrCaching#autowarmCount
>> >>
>> >> Explicit warming - where the static warming queries specified in
>> Solrconfig.xml for newSearcher and firstSearcher listeners are executed
>> when a new searcher is being prepared.
>> >>
>> >> What does it mean that items will be regenerated or prepopulated from
>> the current searcher's cache to the new searcher's cache? I doubt it means
>> copy, as the index has likely changed with a commit and possibly
>> invalidated some contents of the cache. Are the queries, or filters, that
>> define the contents of the current caches re-executed for the new
>> searcher's caches?
>> >>
>> >> For the case where auto warming is configured, a current searcher is
>> active, and static warming queries are defined how does auto warming and
>> explicit warming work together? Or do they? Is only one type of warming
>> activated to fill the caches?
>> >>
>> >> Thanks,
>> >> Matt
>>
Re: cache warming questions
Posted by Kranti Parisa <kr...@gmail.com>.
Erik,
I have a followup question on this topic.
If we have used 10 unique FQs and when we configure filterCache=100 &
autoWarm=5, then which 5 out of the 10 will be repopulated in the case of
new searcher?
I don't think there is a way to set the preference or there is?
Thanks,
Kranti K. Parisa
http://www.linkedin.com/in/krantiparisa
On Thu, Apr 17, 2014 at 5:25 PM, Matt Kuiper <ma...@issinc.com> wrote:
> Ok, that makes sense.
>
> Thanks again,
> Matt
>
> Matt Kuiper - Software Engineer
> Intelligent Software Solutions
> p. 719.452.7721 | matt.kuiper@issinc.com
> www.issinc.com | LinkedIn: intelligent-software-solutions
>
> -----Original Message-----
> From: Erick Erickson [mailto:erickerickson@gmail.com]
> Sent: Thursday, April 17, 2014 9:26 AM
> To: solr-user@lucene.apache.org
> Subject: Re: cache warming questions
>
> Don't go overboard warming here, you often hit diminishing returns very
> quickly. For instance, if the size is 512 you might set your autowarm count
> to 16 and get the most bang for your buck. Beyond some (usually small)
> number, the additional work you put in to warming is wasted. This is
> especially true if your autocommit (soft, or hard with
> openSearcher=true) is short.
>
> So while you're correct in your sizing bit, practically it's rarely that
> complicated since the autowarm count is usually so much smaller than the
> size that there's no danger of swapping them out. YMMV of course.
>
> Best,
> Erick
>
> On Wed, Apr 16, 2014 at 10:33 AM, Matt Kuiper <ma...@issinc.com>
> wrote:
> > Thanks Erick, this is helpful information!
> >
> > So it sounds like, at minimum the cache size (at least for filterCache
> and queryResultCache) should be the sum of the autowarmCount for that cache
> and the number of queries defined for the newSearcher listener. Otherwise
> some items in the caches will be evicted right away.
> >
> > Matt
> >
> > -----Original Message-----
> > From: Erick Erickson [mailto:erickerickson@gmail.com]
> > Sent: Tuesday, April 15, 2014 5:21 PM
> > To: solr-user@lucene.apache.org
> > Subject: Re: cache warming questions
> >
> > bq: What does it mean that items will be regenerated or prepopulated
> from the current searcher's cache...
> >
> > You're right, the values aren't cached. They can't be since the internal
> Lucene document id is used to identify docs, and due to merging the
> internal ID may bear no relation to the old internal ID for a particular
> document.
> >
> > I find it useful to think of Solr's caches as a map where the key is
> the "query" and the value is some representation of the found documents.
> The details of the value don't matter, so I'll skip them.
> >
> > What matters is the key. Consider the filter cache. You put something
> like &fq=price:[0 TO 100] on a URL. Solr then uses the fq clause as the
> key to the filterCache.
> >
> > Here's the sneaky bit. When you specify an autowarm count of N for the
> filterCache, when a new searcher is opened the first N keys from the map
> are re-executed in the new searcher's context and the results put into the
> new searcher's filterCache.
> >
> > bq: ...how does auto warming and explicit warming work together?
> >
> > They're orthogonal. IOW, the autowarming for each cache is executed as
> well as the newSearcher static warming queries. Use the static queries to
> do things like fill the sort caches etc.
> >
> > Incidentally, this bears on why there's a "firstSearcher" and
> "newSearcher". The newSearcher queries are run in addition to the cache
> autowarms. firstSearcher static queries are only run when a Solr server is
> started the first time, and there are no cache entries to autowarm. So the
> firstSearcher queries might be quite a bit more complex than newSearcher
> queries.
> >
> > HTH,
> > Erick
> >
> > On Tue, Apr 15, 2014 at 1:55 PM, Matt Kuiper <ma...@issinc.com>
> wrote:
> >> Hello,
> >>
> >> I have a few questions regarding how Solr caches are warmed.
> >>
> >> My understanding is that there are two ways to warm internal Solr
> caches (only one way for document cache and lucene FieldCache):
> >>
> >> Auto warming - occurs when there is a current searcher handling
> requests and new searcher is being prepared. "When a new searcher is
> opened, its caches may be prepopulated or "autowarmed" with cached object
> from caches in the old searcher. autowarmCount is the number of cached
> items that will be regenerated in the new searcher."
> http://wiki.apache.org/solr/SolrCaching#autowarmCount
> >>
> >> Explicit warming - where the static warming queries specified in
> Solrconfig.xml for newSearcher and firstSearcher listeners are executed
> when a new searcher is being prepared.
> >>
> >> What does it mean that items will be regenerated or prepopulated from
> the current searcher's cache to the new searcher's cache? I doubt it means
> copy, as the index has likely changed with a commit and possibly
> invalidated some contents of the cache. Are the queries, or filters, that
> define the contents of the current caches re-executed for the new
> searcher's caches?
> >>
> >> For the case where auto warming is configured, a current searcher is
> active, and static warming queries are defined how does auto warming and
> explicit warming work together? Or do they? Is only one type of warming
> activated to fill the caches?
> >>
> >> Thanks,
> >> Matt
>
RE: cache warming questions
Posted by Matt Kuiper <ma...@issinc.com>.
Ok, that makes sense.
Thanks again,
Matt
Matt Kuiper - Software Engineer
Intelligent Software Solutions
p. 719.452.7721 | matt.kuiper@issinc.com
www.issinc.com | LinkedIn: intelligent-software-solutions
-----Original Message-----
From: Erick Erickson [mailto:erickerickson@gmail.com]
Sent: Thursday, April 17, 2014 9:26 AM
To: solr-user@lucene.apache.org
Subject: Re: cache warming questions
Don't go overboard warming here, you often hit diminishing returns very quickly. For instance, if the size is 512 you might set your autowarm count to 16 and get the most bang for your buck. Beyond some (usually small) number, the additional work you put in to warming is wasted. This is especially true if your autocommit (soft, or hard with
openSearcher=true) is short.
So while you're correct in your sizing bit, practically it's rarely that complicated since the autowarm count is usually so much smaller than the size that there's no danger of swapping them out. YMMV of course.
Best,
Erick
On Wed, Apr 16, 2014 at 10:33 AM, Matt Kuiper <ma...@issinc.com> wrote:
> Thanks Erick, this is helpful information!
>
> So it sounds like, at minimum the cache size (at least for filterCache and queryResultCache) should be the sum of the autowarmCount for that cache and the number of queries defined for the newSearcher listener. Otherwise some items in the caches will be evicted right away.
>
> Matt
>
> -----Original Message-----
> From: Erick Erickson [mailto:erickerickson@gmail.com]
> Sent: Tuesday, April 15, 2014 5:21 PM
> To: solr-user@lucene.apache.org
> Subject: Re: cache warming questions
>
> bq: What does it mean that items will be regenerated or prepopulated from the current searcher's cache...
>
> You're right, the values aren't cached. They can't be since the internal Lucene document id is used to identify docs, and due to merging the internal ID may bear no relation to the old internal ID for a particular document.
>
> I find it useful to think of Solr's caches as a map where the key is the "query" and the value is some representation of the found documents. The details of the value don't matter, so I'll skip them.
>
> What matters is the key. Consider the filter cache. You put something like &fq=price:[0 TO 100] on a URL. Solr then uses the fq clause as the key to the filterCache.
>
> Here's the sneaky bit. When you specify an autowarm count of N for the filterCache, when a new searcher is opened the first N keys from the map are re-executed in the new searcher's context and the results put into the new searcher's filterCache.
>
> bq: ...how does auto warming and explicit warming work together?
>
> They're orthogonal. IOW, the autowarming for each cache is executed as well as the newSearcher static warming queries. Use the static queries to do things like fill the sort caches etc.
>
> Incidentally, this bears on why there's a "firstSearcher" and "newSearcher". The newSearcher queries are run in addition to the cache autowarms. firstSearcher static queries are only run when a Solr server is started the first time, and there are no cache entries to autowarm. So the firstSearcher queries might be quite a bit more complex than newSearcher queries.
>
> HTH,
> Erick
>
> On Tue, Apr 15, 2014 at 1:55 PM, Matt Kuiper <ma...@issinc.com> wrote:
>> Hello,
>>
>> I have a few questions regarding how Solr caches are warmed.
>>
>> My understanding is that there are two ways to warm internal Solr caches (only one way for document cache and lucene FieldCache):
>>
>> Auto warming - occurs when there is a current searcher handling requests and new searcher is being prepared. "When a new searcher is opened, its caches may be prepopulated or "autowarmed" with cached object from caches in the old searcher. autowarmCount is the number of cached items that will be regenerated in the new searcher." http://wiki.apache.org/solr/SolrCaching#autowarmCount
>>
>> Explicit warming - where the static warming queries specified in Solrconfig.xml for newSearcher and firstSearcher listeners are executed when a new searcher is being prepared.
>>
>> What does it mean that items will be regenerated or prepopulated from the current searcher's cache to the new searcher's cache? I doubt it means copy, as the index has likely changed with a commit and possibly invalidated some contents of the cache. Are the queries, or filters, that define the contents of the current caches re-executed for the new searcher's caches?
>>
>> For the case where auto warming is configured, a current searcher is active, and static warming queries are defined how does auto warming and explicit warming work together? Or do they? Is only one type of warming activated to fill the caches?
>>
>> Thanks,
>> Matt
Re: cache warming questions
Posted by Erick Erickson <er...@gmail.com>.
Don't go overboard warming here, you often hit diminishing returns
very quickly. For instance, if the size is 512 you might set your
autowarm count to 16 and get the most bang for your buck. Beyond some
(usually small) number, the additional work you put in to warming is
wasted. This is especially true if your autocommit (soft, or hard with
openSearcher=true) is short.
So while you're correct in your sizing bit, practically it's rarely
that complicated since the autowarm count is usually so much smaller
than the size that there's no danger of swapping them out. YMMV of
course.
Best,
Erick
On Wed, Apr 16, 2014 at 10:33 AM, Matt Kuiper <ma...@issinc.com> wrote:
> Thanks Erick, this is helpful information!
>
> So it sounds like, at minimum the cache size (at least for filterCache and queryResultCache) should be the sum of the autowarmCount for that cache and the number of queries defined for the newSearcher listener. Otherwise some items in the caches will be evicted right away.
>
> Matt
>
> -----Original Message-----
> From: Erick Erickson [mailto:erickerickson@gmail.com]
> Sent: Tuesday, April 15, 2014 5:21 PM
> To: solr-user@lucene.apache.org
> Subject: Re: cache warming questions
>
> bq: What does it mean that items will be regenerated or prepopulated from the current searcher's cache...
>
> You're right, the values aren't cached. They can't be since the internal Lucene document id is used to identify docs, and due to merging the internal ID may bear no relation to the old internal ID for a particular document.
>
> I find it useful to think of Solr's caches as a map where the key is the "query" and the value is some representation of the found documents. The details of the value don't matter, so I'll skip them.
>
> What matters is the key. Consider the filter cache. You put something like &fq=price:[0 TO 100] on a URL. Solr then uses the fq clause as the key to the filterCache.
>
> Here's the sneaky bit. When you specify an autowarm count of N for the filterCache, when a new searcher is opened the first N keys from the map are re-executed in the new searcher's context and the results put into the new searcher's filterCache.
>
> bq: ...how does auto warming and explicit warming work together?
>
> They're orthogonal. IOW, the autowarming for each cache is executed as well as the newSearcher static warming queries. Use the static queries to do things like fill the sort caches etc.
>
> Incidentally, this bears on why there's a "firstSearcher" and "newSearcher". The newSearcher queries are run in addition to the cache autowarms. firstSearcher static queries are only run when a Solr server is started the first time, and there are no cache entries to autowarm. So the firstSearcher queries might be quite a bit more complex than newSearcher queries.
>
> HTH,
> Erick
>
> On Tue, Apr 15, 2014 at 1:55 PM, Matt Kuiper <ma...@issinc.com> wrote:
>> Hello,
>>
>> I have a few questions regarding how Solr caches are warmed.
>>
>> My understanding is that there are two ways to warm internal Solr caches (only one way for document cache and lucene FieldCache):
>>
>> Auto warming - occurs when there is a current searcher handling requests and new searcher is being prepared. "When a new searcher is opened, its caches may be prepopulated or "autowarmed" with cached object from caches in the old searcher. autowarmCount is the number of cached items that will be regenerated in the new searcher." http://wiki.apache.org/solr/SolrCaching#autowarmCount
>>
>> Explicit warming - where the static warming queries specified in Solrconfig.xml for newSearcher and firstSearcher listeners are executed when a new searcher is being prepared.
>>
>> What does it mean that items will be regenerated or prepopulated from the current searcher's cache to the new searcher's cache? I doubt it means copy, as the index has likely changed with a commit and possibly invalidated some contents of the cache. Are the queries, or filters, that define the contents of the current caches re-executed for the new searcher's caches?
>>
>> For the case where auto warming is configured, a current searcher is active, and static warming queries are defined how does auto warming and explicit warming work together? Or do they? Is only one type of warming activated to fill the caches?
>>
>> Thanks,
>> Matt
RE: cache warming questions
Posted by Matt Kuiper <ma...@issinc.com>.
Thanks Erick, this is helpful information!
So it sounds like, at minimum the cache size (at least for filterCache and queryResultCache) should be the sum of the autowarmCount for that cache and the number of queries defined for the newSearcher listener. Otherwise some items in the caches will be evicted right away.
Matt
-----Original Message-----
From: Erick Erickson [mailto:erickerickson@gmail.com]
Sent: Tuesday, April 15, 2014 5:21 PM
To: solr-user@lucene.apache.org
Subject: Re: cache warming questions
bq: What does it mean that items will be regenerated or prepopulated from the current searcher's cache...
You're right, the values aren't cached. They can't be since the internal Lucene document id is used to identify docs, and due to merging the internal ID may bear no relation to the old internal ID for a particular document.
I find it useful to think of Solr's caches as a map where the key is the "query" and the value is some representation of the found documents. The details of the value don't matter, so I'll skip them.
What matters is the key. Consider the filter cache. You put something like &fq=price:[0 TO 100] on a URL. Solr then uses the fq clause as the key to the filterCache.
Here's the sneaky bit. When you specify an autowarm count of N for the filterCache, when a new searcher is opened the first N keys from the map are re-executed in the new searcher's context and the results put into the new searcher's filterCache.
bq: ...how does auto warming and explicit warming work together?
They're orthogonal. IOW, the autowarming for each cache is executed as well as the newSearcher static warming queries. Use the static queries to do things like fill the sort caches etc.
Incidentally, this bears on why there's a "firstSearcher" and "newSearcher". The newSearcher queries are run in addition to the cache autowarms. firstSearcher static queries are only run when a Solr server is started the first time, and there are no cache entries to autowarm. So the firstSearcher queries might be quite a bit more complex than newSearcher queries.
HTH,
Erick
On Tue, Apr 15, 2014 at 1:55 PM, Matt Kuiper <ma...@issinc.com> wrote:
> Hello,
>
> I have a few questions regarding how Solr caches are warmed.
>
> My understanding is that there are two ways to warm internal Solr caches (only one way for document cache and lucene FieldCache):
>
> Auto warming - occurs when there is a current searcher handling requests and new searcher is being prepared. "When a new searcher is opened, its caches may be prepopulated or "autowarmed" with cached object from caches in the old searcher. autowarmCount is the number of cached items that will be regenerated in the new searcher." http://wiki.apache.org/solr/SolrCaching#autowarmCount
>
> Explicit warming - where the static warming queries specified in Solrconfig.xml for newSearcher and firstSearcher listeners are executed when a new searcher is being prepared.
>
> What does it mean that items will be regenerated or prepopulated from the current searcher's cache to the new searcher's cache? I doubt it means copy, as the index has likely changed with a commit and possibly invalidated some contents of the cache. Are the queries, or filters, that define the contents of the current caches re-executed for the new searcher's caches?
>
> For the case where auto warming is configured, a current searcher is active, and static warming queries are defined how does auto warming and explicit warming work together? Or do they? Is only one type of warming activated to fill the caches?
>
> Thanks,
> Matt
Re: cache warming questions
Posted by Erick Erickson <er...@gmail.com>.
bq: What does it mean that items will be regenerated or prepopulated
from the current searcher's cache...
You're right, the values aren't cached. They can't be since the
internal Lucene document id is used to identify docs, and due to
merging the internal ID may bear no relation to the old internal ID
for a particular document.
I find it useful to think of Solr's caches as a map where the key is
the "query" and the value is some representation of the found
documents. The details of the value don't matter, so I'll skip them.
What matters is the key. Consider the filter cache. You put something
like &fq=price:[0 TO 100] on a URL. Solr then uses the fq clause as
the key to the filterCache.
Here's the sneaky bit. When you specify an autowarm count of N for the
filterCache, when a new searcher is opened the first N keys from the
map are re-executed in the new searcher's context and the results put
into the new searcher's filterCache.
bq: ...how does auto warming and explicit warming work together?
They're orthogonal. IOW, the autowarming for each cache is executed as
well as the newSearcher static warming queries. Use the static queries
to do things like fill the sort caches etc.
Incidentally, this bears on why there's a "firstSearcher" and
"newSearcher". The newSearcher queries are run in addition to the
cache autowarms. firstSearcher static queries are only run when a Solr
server is started the first time, and there are no cache entries to
autowarm. So the firstSearcher queries might be quite a bit more
complex than newSearcher queries.
HTH,
Erick
On Tue, Apr 15, 2014 at 1:55 PM, Matt Kuiper <ma...@issinc.com> wrote:
> Hello,
>
> I have a few questions regarding how Solr caches are warmed.
>
> My understanding is that there are two ways to warm internal Solr caches (only one way for document cache and lucene FieldCache):
>
> Auto warming - occurs when there is a current searcher handling requests and new searcher is being prepared. "When a new searcher is opened, its caches may be prepopulated or "autowarmed" with cached object from caches in the old searcher. autowarmCount is the number of cached items that will be regenerated in the new searcher." http://wiki.apache.org/solr/SolrCaching#autowarmCount
>
> Explicit warming - where the static warming queries specified in Solrconfig.xml for newSearcher and firstSearcher listeners are executed when a new searcher is being prepared.
>
> What does it mean that items will be regenerated or prepopulated from the current searcher's cache to the new searcher's cache? I doubt it means copy, as the index has likely changed with a commit and possibly invalidated some contents of the cache. Are the queries, or filters, that define the contents of the current caches re-executed for the new searcher's caches?
>
> For the case where auto warming is configured, a current searcher is active, and static warming queries are defined how does auto warming and explicit warming work together? Or do they? Is only one type of warming activated to fill the caches?
>
> Thanks,
> Matt