You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by YouPeng Yang <yy...@gmail.com> on 2014/07/24 18:25:57 UTC
To warm the whole cache of Solr other than the only autowarmcount
Hi
I think it is wonderful to have caches autowarmed when commit or soft
commit happens. However ,If I want to warm the whole cache other than the
only autowarmcount,the default the auto warming operation will take long
long ~~long time.So it comes up with that maybe it good idea to just
change the reference of the caches of the newsearcher with the caches of
the oldsearcher. It will increase the time of the autowarming,also increase
the query time of NRT.
It is just not a mature idea.I am pust this idea,and hope to get more
hints or help to make more cleat about the idea.
regards
Re: To warm the whole cache of Solr other than the only autowarmcount
Posted by Umesh Prasad <um...@gmail.com>.
@Eric : As you said, each use-case is different. We actually autowarm our
caches to 80% and we have a 99% hit ratio on filter cache. For query cache,
hit ratios are like 25% but given that cache hit saves us about 10X, we
strive to increase cache hit ratio.
@Yang : You can't do a direct copy of values. Values are related to
lucene's internal document id and they can change during an index update.
The change can happen because of document being deleted, segments being
merged or new segments being created. Solr's caches refer to global doc id
which are even more prone to change (because of index merges).
On 28 July 2014 21:32, Erick Erickson <er...@gmail.com> wrote:
> bq: autowarmcount=1024...
>
> That's the point, this is quite a high number in my
> experience.
>
> I've rarely seen numbers above 128 show much of
> any improvement. I've seen a large number of
> installations use much smaller autowarm numbers,
> as in the 16-32 range and be quite content.
>
> I _really_ recommend you try to use much smaller
> numbers then _measure_ whether the first few
> queries after a commit show unacceptable
> response times before trying to make things
> "better". This really feels like premature
> optimization.
>
> Of course you know your problem space better than
> I do, it's just that I've spent too much of my
> professional life fixing the wrong "problem"; I've
> become something of a "measure first" curmudgeon.
>
> FWIW,
> Erick
>
>
> On Sun, Jul 27, 2014 at 10:48 PM, YouPeng Yang <yy...@gmail.com>
> wrote:
>
> > Hi Erick
> >
> > We do the DIH job from the DB and committed frequently.It takes a long
> time
> > to autowarm the filterCaches after commit or soft commit happened when
> > setting the autowarmcount=1024,which I do think is small enough.
> > So It comes up an idea that whether it could directly pass the
> reference
> > of the caches over to the new caches so that the autowarm processing
> will
> > take much fewer time .
> >
> >
> >
> > 2014-07-28 2:30 GMT+08:00 Erick Erickson <er...@gmail.com>:
> >
> > > Why do you think you _need_ to autowarm the entire cache? It
> > > is, after all, an LRU cache, the theory being that the most recent
> > > queries are most likely to be reused.
> > >
> > > Personally I'd run some tests on using small autowarm counts
> > > before getting at all mixed up in some complex scheme that
> > > may not be useful at all. Say an autowarm count of 16. Then
> > > measure using that, then say 32 then... Insure you have a real
> > > problem before worrying about a solution! ;)
> > >
> > > Best,
> > > Erick
> > >
> > >
> > > On Fri, Jul 25, 2014 at 6:45 AM, Shawn Heisey <so...@elyograg.org>
> wrote:
> > >
> > > > On 7/24/2014 8:45 PM, YouPeng Yang wrote:
> > > > > To Matt
> > > > >
> > > > > Thank you,your opinion is very valuable ,So I have checked the
> > source
> > > > > codes about how the cache warming up. It seems to just put items
> of
> > > the
> > > > > old caches into the new caches.
> > > > > I will pull Mark Miller into this discussion.He is the one of the
> > > > > developer of the Solr whom I had contacted with.
> > > > >
> > > > > To Mark Miller
> > > > >
> > > > > Would you please check out what we are discussing in the last
> two
> > > > > posts.I need your help.
> > > >
> > > > Matt is completely right. Any commit can drastically change the
> Lucene
> > > > document id numbers. It would be too expensive to determine which
> > > > numbers haven't changed. That means Solr must throw away all cache
> > > > information on commit.
> > > >
> > > > Two of Solr's caches support autowarming. Those caches use queries
> as
> > > > keys and results as values. Autowarming works by re-executing the
> top
> > N
> > > > queries (keys) in the old cache to obtain fresh Lucene document id
> > > > numbers (values). The cache code does take *keys* from the old cache
> > > > for the new cache, but not *values*. I'm very sure about this, as I
> > > > wrote the current (and not terribly good) LFUCache.
> > > >
> > > > Thanks,
> > > > Shawn
> > > >
> > > >
> > >
> >
>
--
---
Thanks & Regards
Umesh Prasad
Re: To warm the whole cache of Solr other than the only autowarmcount
Posted by Erick Erickson <er...@gmail.com>.
bq: autowarmcount=1024...
That's the point, this is quite a high number in my
experience.
I've rarely seen numbers above 128 show much of
any improvement. I've seen a large number of
installations use much smaller autowarm numbers,
as in the 16-32 range and be quite content.
I _really_ recommend you try to use much smaller
numbers then _measure_ whether the first few
queries after a commit show unacceptable
response times before trying to make things
"better". This really feels like premature
optimization.
Of course you know your problem space better than
I do, it's just that I've spent too much of my
professional life fixing the wrong "problem"; I've
become something of a "measure first" curmudgeon.
FWIW,
Erick
On Sun, Jul 27, 2014 at 10:48 PM, YouPeng Yang <yy...@gmail.com>
wrote:
> Hi Erick
>
> We do the DIH job from the DB and committed frequently.It takes a long time
> to autowarm the filterCaches after commit or soft commit happened when
> setting the autowarmcount=1024,which I do think is small enough.
> So It comes up an idea that whether it could directly pass the reference
> of the caches over to the new caches so that the autowarm processing will
> take much fewer time .
>
>
>
> 2014-07-28 2:30 GMT+08:00 Erick Erickson <er...@gmail.com>:
>
> > Why do you think you _need_ to autowarm the entire cache? It
> > is, after all, an LRU cache, the theory being that the most recent
> > queries are most likely to be reused.
> >
> > Personally I'd run some tests on using small autowarm counts
> > before getting at all mixed up in some complex scheme that
> > may not be useful at all. Say an autowarm count of 16. Then
> > measure using that, then say 32 then... Insure you have a real
> > problem before worrying about a solution! ;)
> >
> > Best,
> > Erick
> >
> >
> > On Fri, Jul 25, 2014 at 6:45 AM, Shawn Heisey <so...@elyograg.org> wrote:
> >
> > > On 7/24/2014 8:45 PM, YouPeng Yang wrote:
> > > > To Matt
> > > >
> > > > Thank you,your opinion is very valuable ,So I have checked the
> source
> > > > codes about how the cache warming up. It seems to just put items of
> > the
> > > > old caches into the new caches.
> > > > I will pull Mark Miller into this discussion.He is the one of the
> > > > developer of the Solr whom I had contacted with.
> > > >
> > > > To Mark Miller
> > > >
> > > > Would you please check out what we are discussing in the last two
> > > > posts.I need your help.
> > >
> > > Matt is completely right. Any commit can drastically change the Lucene
> > > document id numbers. It would be too expensive to determine which
> > > numbers haven't changed. That means Solr must throw away all cache
> > > information on commit.
> > >
> > > Two of Solr's caches support autowarming. Those caches use queries as
> > > keys and results as values. Autowarming works by re-executing the top
> N
> > > queries (keys) in the old cache to obtain fresh Lucene document id
> > > numbers (values). The cache code does take *keys* from the old cache
> > > for the new cache, but not *values*. I'm very sure about this, as I
> > > wrote the current (and not terribly good) LFUCache.
> > >
> > > Thanks,
> > > Shawn
> > >
> > >
> >
>
Re: To warm the whole cache of Solr other than the only autowarmcount
Posted by YouPeng Yang <yy...@gmail.com>.
Hi Erick
We do the DIH job from the DB and committed frequently.It takes a long time
to autowarm the filterCaches after commit or soft commit happened when
setting the autowarmcount=1024,which I do think is small enough.
So It comes up an idea that whether it could directly pass the reference
of the caches over to the new caches so that the autowarm processing will
take much fewer time .
2014-07-28 2:30 GMT+08:00 Erick Erickson <er...@gmail.com>:
> Why do you think you _need_ to autowarm the entire cache? It
> is, after all, an LRU cache, the theory being that the most recent
> queries are most likely to be reused.
>
> Personally I'd run some tests on using small autowarm counts
> before getting at all mixed up in some complex scheme that
> may not be useful at all. Say an autowarm count of 16. Then
> measure using that, then say 32 then... Insure you have a real
> problem before worrying about a solution! ;)
>
> Best,
> Erick
>
>
> On Fri, Jul 25, 2014 at 6:45 AM, Shawn Heisey <so...@elyograg.org> wrote:
>
> > On 7/24/2014 8:45 PM, YouPeng Yang wrote:
> > > To Matt
> > >
> > > Thank you,your opinion is very valuable ,So I have checked the source
> > > codes about how the cache warming up. It seems to just put items of
> the
> > > old caches into the new caches.
> > > I will pull Mark Miller into this discussion.He is the one of the
> > > developer of the Solr whom I had contacted with.
> > >
> > > To Mark Miller
> > >
> > > Would you please check out what we are discussing in the last two
> > > posts.I need your help.
> >
> > Matt is completely right. Any commit can drastically change the Lucene
> > document id numbers. It would be too expensive to determine which
> > numbers haven't changed. That means Solr must throw away all cache
> > information on commit.
> >
> > Two of Solr's caches support autowarming. Those caches use queries as
> > keys and results as values. Autowarming works by re-executing the top N
> > queries (keys) in the old cache to obtain fresh Lucene document id
> > numbers (values). The cache code does take *keys* from the old cache
> > for the new cache, but not *values*. I'm very sure about this, as I
> > wrote the current (and not terribly good) LFUCache.
> >
> > Thanks,
> > Shawn
> >
> >
>
Re: To warm the whole cache of Solr other than the only autowarmcount
Posted by Erick Erickson <er...@gmail.com>.
Why do you think you _need_ to autowarm the entire cache? It
is, after all, an LRU cache, the theory being that the most recent
queries are most likely to be reused.
Personally I'd run some tests on using small autowarm counts
before getting at all mixed up in some complex scheme that
may not be useful at all. Say an autowarm count of 16. Then
measure using that, then say 32 then... Insure you have a real
problem before worrying about a solution! ;)
Best,
Erick
On Fri, Jul 25, 2014 at 6:45 AM, Shawn Heisey <so...@elyograg.org> wrote:
> On 7/24/2014 8:45 PM, YouPeng Yang wrote:
> > To Matt
> >
> > Thank you,your opinion is very valuable ,So I have checked the source
> > codes about how the cache warming up. It seems to just put items of the
> > old caches into the new caches.
> > I will pull Mark Miller into this discussion.He is the one of the
> > developer of the Solr whom I had contacted with.
> >
> > To Mark Miller
> >
> > Would you please check out what we are discussing in the last two
> > posts.I need your help.
>
> Matt is completely right. Any commit can drastically change the Lucene
> document id numbers. It would be too expensive to determine which
> numbers haven't changed. That means Solr must throw away all cache
> information on commit.
>
> Two of Solr's caches support autowarming. Those caches use queries as
> keys and results as values. Autowarming works by re-executing the top N
> queries (keys) in the old cache to obtain fresh Lucene document id
> numbers (values). The cache code does take *keys* from the old cache
> for the new cache, but not *values*. I'm very sure about this, as I
> wrote the current (and not terribly good) LFUCache.
>
> Thanks,
> Shawn
>
>
Re: To warm the whole cache of Solr other than the only autowarmcount
Posted by Shawn Heisey <so...@elyograg.org>.
On 7/28/2014 12:06 AM, YouPeng Yang wrote:
> No affense to your work,I am still confusing about the cache warm
> processing about your explanation.So I check the warm method of
> FastLRUCache as [1].
> As far as I see,there is no values refresh during the the warm
> processing. the *regenerator.regenerateItem* just put the old value to the
> new cache.
What the cache code does is pass the key and the value from the current
cache entry to the CacheRegenerator.regenerateItem method, which is
defined elsewhere when the cache is created. I thought it was just the
key, but now that I look closer, it is both. Exactly what is done with
that information is completely up to the regenerator object. The cache
unit tests use a NoOpRegenerator, which simply populates the new cache
with the entire old entry, including the value. That is not how things
work with the actual production caches, though.
With filterCache and queryResultCache, the key contains the query, and
the regenerator is set up to execute the query against the live index
and insert the new value in the new cache. I've never looked deeply
into the regenerator code.
Thanks,
Shawn
Re: To warm the whole cache of Solr other than the only autowarmcount
Posted by YouPeng Yang <yy...@gmail.com>.
Hi Shawn
No affense to your work,I am still confusing about the cache warm
processing about your explanation.So I check the warm method of
FastLRUCache as [1].
As far as I see,there is no values refresh during the the warm
processing. the *regenerator.regenerateItem* just put the old value to the
new cache.
Did I miss anything?
[1]------------------------------------------------------------------------------
public void warm(SolrIndexSearcher searcher, SolrCache old) {
if (regenerator == null) return;
long warmingStartTime = System.nanoTime();
FastLRUCache other = (FastLRUCache) old;
// warm entries
if (isAutowarmingOn()) {
int sz = autowarm.getWarmCount(other.size());
Map items = other.cache.getLatestAccessedItems(sz);
Map.Entry[] itemsArr = new Map.Entry[items.size()];
int counter = 0;
for (Object mapEntry : items.entrySet()) {
itemsArr[counter++] = (Map.Entry) mapEntry;
}
for (int i = itemsArr.length - 1; i >= 0; i--) {
try {
boolean continueRegen = regenerator.regenerateItem(searcher,
this, old, itemsArr[i].getKey(), itemsArr[i].getValue());
if (!continueRegen) break;
}
catch (Exception e) {
SolrException.log(log, "Error during auto-warming of key:" +
itemsArr[i].getKey(), e);
}
}
}
warmupTime = TimeUnit.MILLISECONDS.convert(System.nanoTime() -
warmingStartTime, TimeUnit.NANOSECONDS);
}
2014-07-25 21:45 GMT+08:00 Shawn Heisey <so...@elyograg.org>:
> On 7/24/2014 8:45 PM, YouPeng Yang wrote:
> > To Matt
> >
> > Thank you,your opinion is very valuable ,So I have checked the source
> > codes about how the cache warming up. It seems to just put items of the
> > old caches into the new caches.
> > I will pull Mark Miller into this discussion.He is the one of the
> > developer of the Solr whom I had contacted with.
> >
> > To Mark Miller
> >
> > Would you please check out what we are discussing in the last two
> > posts.I need your help.
>
> Matt is completely right. Any commit can drastically change the Lucene
> document id numbers. It would be too expensive to determine which
> numbers haven't changed. That means Solr must throw away all cache
> information on commit.
>
> Two of Solr's caches support autowarming. Those caches use queries as
> keys and results as values. Autowarming works by re-executing the top N
> queries (keys) in the old cache to obtain fresh Lucene document id
> numbers (values). The cache code does take *keys* from the old cache
> for the new cache, but not *values*. I'm very sure about this, as I
> wrote the current (and not terribly good) LFUCache.
>
> Thanks,
> Shawn
>
>
Re: To warm the whole cache of Solr other than the only autowarmcount
Posted by Shawn Heisey <so...@elyograg.org>.
On 7/24/2014 8:45 PM, YouPeng Yang wrote:
> To Matt
>
> Thank you,your opinion is very valuable ,So I have checked the source
> codes about how the cache warming up. It seems to just put items of the
> old caches into the new caches.
> I will pull Mark Miller into this discussion.He is the one of the
> developer of the Solr whom I had contacted with.
>
> To Mark Miller
>
> Would you please check out what we are discussing in the last two
> posts.I need your help.
Matt is completely right. Any commit can drastically change the Lucene
document id numbers. It would be too expensive to determine which
numbers haven't changed. That means Solr must throw away all cache
information on commit.
Two of Solr's caches support autowarming. Those caches use queries as
keys and results as values. Autowarming works by re-executing the top N
queries (keys) in the old cache to obtain fresh Lucene document id
numbers (values). The cache code does take *keys* from the old cache
for the new cache, but not *values*. I'm very sure about this, as I
wrote the current (and not terribly good) LFUCache.
Thanks,
Shawn
Re: To warm the whole cache of Solr other than the only autowarmcount
Posted by YouPeng Yang <yy...@gmail.com>.
To Matt
Thank you,your opinion is very valuable ,So I have checked the source
codes about how the cache warming up. It seems to just put items of the
old caches into the new caches.
I will pull Mark Miller into this discussion.He is the one of the
developer of the Solr whom I had contacted with.
To Mark Miller
Would you please check out what we are discussing in the last two
posts.I need your help.
Regards.
2014-07-25 2:50 GMT+08:00 Matt Kuiper (Springblox) <
matt.kuiper@springblox.com>:
> I don't believe this would work. My understanding (please correct if I
> have this wrong) is that the underlying Lucene document ids have a
> potential to change and so when a newSearcher is created the caches must be
> regenerated and not copied.
>
> Matt
>
> -----Original Message-----
> From: YouPeng Yang [mailto:yypvsxf19870706@gmail.com]
> Sent: Thursday, July 24, 2014 10:26 AM
> To: solr-user@lucene.apache.org
> Subject: To warm the whole cache of Solr other than the only autowarmcount
>
> Hi
>
> I think it is wonderful to have caches autowarmed when commit or soft
> commit happens. However ,If I want to warm the whole cache other than the
> only autowarmcount,the default the auto warming operation will take long
> long ~~long time.So it comes up with that maybe it good idea to just
> change the reference of the caches of the newsearcher with the caches of
> the oldsearcher. It will increase the time of the autowarming,also increase
> the query time of NRT.
> It is just not a mature idea.I am pust this idea,and hope to get more
> hints or help to make more cleat about the idea.
>
>
>
> regards
>
RE: To warm the whole cache of Solr other than the only autowarmcount
Posted by "Matt Kuiper (Springblox)" <ma...@springblox.com>.
I don't believe this would work. My understanding (please correct if I have this wrong) is that the underlying Lucene document ids have a potential to change and so when a newSearcher is created the caches must be regenerated and not copied.
Matt
-----Original Message-----
From: YouPeng Yang [mailto:yypvsxf19870706@gmail.com]
Sent: Thursday, July 24, 2014 10:26 AM
To: solr-user@lucene.apache.org
Subject: To warm the whole cache of Solr other than the only autowarmcount
Hi
I think it is wonderful to have caches autowarmed when commit or soft commit happens. However ,If I want to warm the whole cache other than the only autowarmcount,the default the auto warming operation will take long long ~~long time.So it comes up with that maybe it good idea to just change the reference of the caches of the newsearcher with the caches of the oldsearcher. It will increase the time of the autowarming,also increase the query time of NRT.
It is just not a mature idea.I am pust this idea,and hope to get more hints or help to make more cleat about the idea.
regards