You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Damien Dudognon <da...@ebuzzing.com> on 2012/08/31 16:02:37 UTC
Replication lag after cache optimizations
Hi,
We get some troubles with the solr replication after cache optimizations. We use a lot the facet features.
We have increased the cache size and its initial size. We have also change the queryResultCache from LRU to FastLRU and the fieldValueCache was activated (see below the detailed configuration). This optimization step allows to divide the average time per request by 50 (from 260ms to 5ms).
However, with these modifications we noticed an important replication lag. This issue is not troublesome for smaller indexes (about 300.000 elements - 3Gb), but it becomes critical if the index size is significant (30 million elements - 70Gb). In fact, the slaves can't make up for lag and they become out-of-date (and consequently unusable).
Has anyone ever been faced this kind of problems?
Our environment :
- Solr 3.4.0
- Java 1.6.0_26
- Debian 6.0.3
Best regards,
Damien
------------------------------------------------------------------
My previous cache settings (fieldValueCache was disabled):
------------------------------------------------------------------
<filterCache class="solr.FastLRUCache" size="512" initialSize="512" autowarmCount="0" />
<queryResultCache class="solr.LRUCache" size="512" initialSize="512" autowarmCount="0" />
<documentCache class="solr.FastLRUCache" size="512" initialSize="512" autowarmCount="0" />
<!--
<fieldValueCache class="solr.FastLRUCache" size="512" autowarmCount="128" showItems="32" />
-->
------------------------------------------------------------------
The settings now used:
------------------------------------------------------------------
<filterCache class="solr.FastLRUCache" size="16384" initialSize="4096" autowarmCount="0" />
<queryResultCache class="solr.FastLRUCache" size="16384" initialSize="4096" autowarmCount="0" />
<documentCache class="solr.FastLRUCache" size="16384" initialSize="4096" autowarmCount="0" />
<fieldValueCache class="solr.FastLRUCache" size="16384" autowarmCount="1024" showItems="32" />
------------------------------------------------------------------
The replication config:
------------------------------------------------------------------
<requestHandler name="/replication" class="solr.ReplicationHandler" >
<lst name="master">
<str name="enable">${solr.enable.master:false}</str>
<str name="replicateAfter">commit</str>
<str name="replicateAfter">startup</str>
<str name="confFiles">schema.xml,stopwords.txt</str>
</lst>
<lst name="slave">
<str name="enable">${solr.enable.slave:false}</str>
<str name="masterUrl">http://solrmaster:${jetty.port:8083}/solr/en/replication</str>
<str name="pollInterval">00:00:01</str>
</lst>
</requestHandler>
Re: Replication lag after cache optimizations
Posted by Damien Dudognon <da...@ebuzzing.com>.
Thanks for all the information.
> I'm not sure how exactly you are measuring/defining "replication lag" but
> if you mean "lag in how long until the newly replicated documents are
> visible in searches"
That is exactly what I wanted to say.
I've attached the cache statistics.
If you are interested in, a few more details on our use case :
Actually, we have only few hits on Solr (about 2 req/s) but we will quickly have more than 50 req/s. The requests are mainly facet requests. The index counts about 1,5M documents and we plan a size of 15M documents in one year.
Best regards,
Damien
----------------
CACHE
----------------
name: queryResultCache
class: org.apache.solr.search.FastLRUCache
version: 1.0
description: Concurrent LRU Cache(maxSize=16384, initialSize=4096, minSize=14745, acceptableSize=15564, cleanupThread=false, autowarmCount=1024, regenerator=org.apache.solr.search.SolrIndexSearcher$3@3d762027)
stats: lookups : 4
hits : 4
hitratio : 1.00
inserts : 0
evictions : 0
size : 1024
warmupTime : 20
cumulative_lookups : 1003454
cumulative_hits : 894365
cumulative_hitratio : 0.89
cumulative_inserts : 120343
cumulative_evictions : 0
name: fieldCache
class: org.apache.solr.search.SolrFieldCacheMBean
insanity_count : 0
name: documentCache
class: org.apache.solr.search.FastLRUCache
version: 1.0
description: Concurrent LRU Cache(maxSize=16384, initialSize=4096, minSize=14745, acceptableSize=15564, cleanupThread=false, autowarmCount=1024, regenerator=null)
stats: lookups : 80
hits : 60
hitratio : 0.75
inserts : 20
evictions : 0
size : 20
warmupTime : 0
cumulative_lookups : 10844723
cumulative_hits : 8318341
cumulative_hitratio : 0.76
cumulative_inserts : 2526382
cumulative_evictions : 0
name: fieldValueCache
class: org.apache.solr.search.FastLRUCache
version: 1.0
description: Concurrent LRU Cache(maxSize=16384, initialSize=16384, minSize=14745, acceptableSize=15564, cleanupThread=false, autowarmCount=1024, regenerator=org.apache.solr.search.SolrIndexSearcher$1@38bdc9b3)
stats: lookups : 2
hits : 2
hitratio : 1.00
inserts : 0
evictions : 0
size : 1
warmupTime : 1369
cumulative_lookups : 485281
cumulative_hits : 485276
cumulative_hitratio : 0.99
cumulative_inserts : 2
cumulative_evictions : 0
item_tags : {field=tags,memSize=5804302,tindexSize=36148,time=1369,phase1=1357,nTerms=118241,bigTerms=0,termInstances=448772,uses=2}
name: filterCache
class: org.apache.solr.search.FastLRUCache
version: 1.0
description: Concurrent LRU Cache(maxSize=16384, initialSize=4096, minSize=14745, acceptableSize=15564, cleanupThread=false, autowarmCount=1024, regenerator=org.apache.solr.search.SolrIndexSearcher$2@340523df)
stats: lookups : 21
hits : 21
hitratio : 1.00
inserts : 0
evictions : 0
size : 1024
warmupTime : 1305
cumulative_lookups : 5956615
cumulative_hits : 5868136
cumulative_hitratio : 0.98
cumulative_inserts : 88479
cumulative_evictions : 0
Re: Replication lag after cache optimizations
Posted by Chris Hostetter <ho...@fucit.org>.
: However, with these modifications we noticed an important replication
I'm not sure how exactly you are measuring/defining "replication lag" but
if you mean "lag in how long until the newly replicated documents are
visible in searches" then that _may_ be fairly easy to explain...
: My previous cache settings (fieldValueCache was disabled):
FYI: for historical reasons, there is always a fieldValueCache, even if
you don't declare one....
https://wiki.apache.org/solr/SolrCaching#fieldValueCache
: <fieldValueCache class="solr.FastLRUCache" size="16384" autowarmCount="1024" showItems="32" />
you have gone from using the hardcoded deault fieldValueCache, (which had
no warming configured at all) to configuring an autowarmCount of 1024 --
you should easily be able to see in the logs that the "newSearcher" time
on your machines is much longer since this change, as it autowarms thouse
fieldValueCache entries.
This means that, compared to your preivous settings, the "first
request" that attempts to use those fieldValueCache entires should be
much faster then before, but the trade off is that you are spending
the time to generate those cache entires "up-front" before you allow any
requests to see the updated index at all.
FWIW: the entries in the fieldValueCache are keyed off of field name (they
are very bug UnInvertedField objects, and are typtically very few of them
-- this is why, IIRC, yonik recommends no wautowarming of fieldValueCache
at all) so having a size of 16384 and an autowarmCount of 1024 is probably
overkill ... i suspect if you check the actual size at runtime you'll see
that there are way fewer entires then that -- if you have anywhere close
to 16384 entires i would love to hear more about your usecase.
-Hoss
Re: Replication lag after cache optimizations
Posted by Damien DUDOGNON <da...@ebuzzing.com>.
Thanks for your answer Erick.
For the polling interval, we use 1 second for the small index and 1 minute
for the big one. I'll try to increase it up to 5 minutes and see if it
solve the problem.The issue doesn't occur with the default cache settings
(i.e. cache size=512).
Indeed, we strive for near real time searching, so I'll take a look at 4.0
with NRT support, hoping it is sufficiently stable for production
environment.
Kind regards,
Damien
2012/9/2 Erick Erickson <er...@gmail.com>
> You polling interval is much too short. 1 second
> is probably getting you into resource contention
> issues.
>
> A more reasonable interval is on the order of several
> minutes. If you really need near real time
> searching, consider 4.0 which supports NRT
>
> Best
> Erick
>
> On Fri, Aug 31, 2012 at 10:02 AM, Damien Dudognon
> <da...@ebuzzing.com> wrote:
> > Hi,
> >
> > We get some troubles with the solr replication after cache
> optimizations. We use a lot the facet features.
> >
> > We have increased the cache size and its initial size. We have also
> change the queryResultCache from LRU to FastLRU and the fieldValueCache was
> activated (see below the detailed configuration). This optimization step
> allows to divide the average time per request by 50 (from 260ms to 5ms).
> >
> > However, with these modifications we noticed an important replication
> lag. This issue is not troublesome for smaller indexes (about 300.000
> elements - 3Gb), but it becomes critical if the index size is significant
> (30 million elements - 70Gb). In fact, the slaves can't make up for lag and
> they become out-of-date (and consequently unusable).
> >
> > Has anyone ever been faced this kind of problems?
> >
> > Our environment :
> > - Solr 3.4.0
> > - Java 1.6.0_26
> > - Debian 6.0.3
> >
> > Best regards,
> > Damien
> >
> > ------------------------------------------------------------------
> > My previous cache settings (fieldValueCache was disabled):
> > ------------------------------------------------------------------
> > <filterCache class="solr.FastLRUCache" size="512" initialSize="512"
> autowarmCount="0" />
> > <queryResultCache class="solr.LRUCache" size="512" initialSize="512"
> autowarmCount="0" />
> > <documentCache class="solr.FastLRUCache" size="512" initialSize="512"
> autowarmCount="0" />
> > <!--
> > <fieldValueCache class="solr.FastLRUCache" size="512"
> autowarmCount="128" showItems="32" />
> > -->
> >
> > ------------------------------------------------------------------
> > The settings now used:
> > ------------------------------------------------------------------
> > <filterCache class="solr.FastLRUCache" size="16384" initialSize="4096"
> autowarmCount="0" />
> > <queryResultCache class="solr.FastLRUCache" size="16384"
> initialSize="4096" autowarmCount="0" />
> > <documentCache class="solr.FastLRUCache" size="16384" initialSize="4096"
> autowarmCount="0" />
> > <fieldValueCache class="solr.FastLRUCache" size="16384"
> autowarmCount="1024" showItems="32" />
> >
> > ------------------------------------------------------------------
> > The replication config:
> > ------------------------------------------------------------------
> > <requestHandler name="/replication" class="solr.ReplicationHandler" >
> > <lst name="master">
> > <str name="enable">${solr.enable.master:false}</str>
> > <str name="replicateAfter">commit</str>
> > <str name="replicateAfter">startup</str>
> > <str name="confFiles">schema.xml,stopwords.txt</str>
> > </lst>
> > <lst name="slave">
> > <str name="enable">${solr.enable.slave:false}</str>
> > <str name="masterUrl">http://solrmaster:
> ${jetty.port:8083}/solr/en/replication</str>
> > <str name="pollInterval">00:00:01</str>
> > </lst>
> > </requestHandler>
>
--
Damien DUDOGNON
R&D Engineer - PhD Student
- +33 (0)6 62 79 34 10
- damien.dudognon@ebuzzing.com
-
-
- linkedin.com/in/damiendudognon<http://www.linkedin.com/in/damiendudognon>
- www.irit.fr/~Damien.Dudognon
- 1, avenue Jean Rieux
- 31500 Toulouse
- France
- +33 (0)5 62 48 33 90
-
Re: Replication lag after cache optimizations
Posted by Erick Erickson <er...@gmail.com>.
You polling interval is much too short. 1 second
is probably getting you into resource contention
issues.
A more reasonable interval is on the order of several
minutes. If you really need near real time
searching, consider 4.0 which supports NRT
Best
Erick
On Fri, Aug 31, 2012 at 10:02 AM, Damien Dudognon
<da...@ebuzzing.com> wrote:
> Hi,
>
> We get some troubles with the solr replication after cache optimizations. We use a lot the facet features.
>
> We have increased the cache size and its initial size. We have also change the queryResultCache from LRU to FastLRU and the fieldValueCache was activated (see below the detailed configuration). This optimization step allows to divide the average time per request by 50 (from 260ms to 5ms).
>
> However, with these modifications we noticed an important replication lag. This issue is not troublesome for smaller indexes (about 300.000 elements - 3Gb), but it becomes critical if the index size is significant (30 million elements - 70Gb). In fact, the slaves can't make up for lag and they become out-of-date (and consequently unusable).
>
> Has anyone ever been faced this kind of problems?
>
> Our environment :
> - Solr 3.4.0
> - Java 1.6.0_26
> - Debian 6.0.3
>
> Best regards,
> Damien
>
> ------------------------------------------------------------------
> My previous cache settings (fieldValueCache was disabled):
> ------------------------------------------------------------------
> <filterCache class="solr.FastLRUCache" size="512" initialSize="512" autowarmCount="0" />
> <queryResultCache class="solr.LRUCache" size="512" initialSize="512" autowarmCount="0" />
> <documentCache class="solr.FastLRUCache" size="512" initialSize="512" autowarmCount="0" />
> <!--
> <fieldValueCache class="solr.FastLRUCache" size="512" autowarmCount="128" showItems="32" />
> -->
>
> ------------------------------------------------------------------
> The settings now used:
> ------------------------------------------------------------------
> <filterCache class="solr.FastLRUCache" size="16384" initialSize="4096" autowarmCount="0" />
> <queryResultCache class="solr.FastLRUCache" size="16384" initialSize="4096" autowarmCount="0" />
> <documentCache class="solr.FastLRUCache" size="16384" initialSize="4096" autowarmCount="0" />
> <fieldValueCache class="solr.FastLRUCache" size="16384" autowarmCount="1024" showItems="32" />
>
> ------------------------------------------------------------------
> The replication config:
> ------------------------------------------------------------------
> <requestHandler name="/replication" class="solr.ReplicationHandler" >
> <lst name="master">
> <str name="enable">${solr.enable.master:false}</str>
> <str name="replicateAfter">commit</str>
> <str name="replicateAfter">startup</str>
> <str name="confFiles">schema.xml,stopwords.txt</str>
> </lst>
> <lst name="slave">
> <str name="enable">${solr.enable.slave:false}</str>
> <str name="masterUrl">http://solrmaster:${jetty.port:8083}/solr/en/replication</str>
> <str name="pollInterval">00:00:01</str>
> </lst>
> </requestHandler>