Minimum time between distributions

[Kevin Lewandowski <ke...@gmail.com>]

On Discogs I'm running Solr with two slaves and one master, using the
distribution scripts. The slaves pull and install a new snapshot every
five minutes and this is working very well so far.

Are there any risks with reducing this window to every one or two
minutes? With large caches could the autowarming take longer than one
or two minutes? It isn't a business need to reduce the window but I'm
just curious about the feasibility and risks.

How often do other people run snappuller and snapinstaller?

thanks,
Kevin

Re: Minimum time between distributions

[Yonik Seeley <yo...@apache.org>]

On 11/21/06, Chris Hostetter <ho...@fucit.org> wrote:
> once upon a time Yonik and i discussed having a "maxAutoWarm" time option
> on the caches, and they would give up if that much time had elapsed

At first I thought it would be easy... just autowarm until time is up.
 But then I realized that there is a problem: one would want to warm
items from the most important to the least important if you don't know
if you are going to get to them all.  But doing them in that order
with an LRU cache means that they would end up in the wrong order (the
most recently used item would be at the tail of the list and first to
be ejected).

Even if we *did* have access to the internal list of the
LinkedHashMap, reversing the order isn't what is desired either since
generation of one cache entry can touch and thus promote another.
Perhaps the solution is to keep a separate list while warming, and
then when time runs out, clear and reinsert in the correct order.


-Yonik

Re: Minimum time between distributions

[Chris Hostetter <ho...@fucit.org>]

: Are there any risks with reducing this window to every one or two
: minutes? With large caches could the autowarming take longer than one
: or two minutes? It isn't a business need to reduce the window but I'm
: just curious about the feasibility and risks.

you can definitely run into problems if autoWarming takes longer then your
snappuller interval, thre's nothing in Solr right now that prevents
"warming" searchers from layering on top of eachother eating resources.

once upon a time Yonik and i discussed having a "maxAutoWarm" time option
on the caches, and they would give up if that much time had elapsed -- but
that approach is too simplistic to be of much use -- even if you configure
snappuller to run ever minute, there are going to be lots of times when it
doesn't acctually have anything new every minute.

: How often do other people run snappuller and snapinstaller?

most of the indexes i deal with are set to 5 minutes ... 1 that really
REALLY relies on the caches for facet counts is only 15 ... i think we
have one index snappulling/installing every minute -- but there are almost
never new indexes, that internal is just so the rare occasions when there
is a new index, the data shows up ASAP.



-Hoss