You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Dotan Cohen <do...@gmail.com> on 2013/04/03 09:48:42 UTC
maxWarmingSearchers in Solr 4.
I have been dragging the same solrconfig.xml from Solr 3.x to 4.0 to
4.1, with no customization (bad, bad me!). I'm now looking into
customizing it and I see that the Solr 4.1 solrconfig.xml is much
simpler and shorter. Is this simply because many of the examples have
been removed?
In particular, I notice that there is no mention of
maxWarmingSearchers in the Solr 4.1 solrconfig.xml. I assume that I
can simply add it in, are there any other critical config options that
are missing that I should be looking into as well? Would I be better
off using the old Solr 3.x solrconfig.xml in Solr 4.1 as it contains
so many examples?
--
Dotan Cohen
http://gibberish.co.il
http://what-is-what.com
Re: maxWarmingSearchers in Solr 4.
Posted by Dotan Cohen <do...@gmail.com>.
On Thu, Apr 4, 2013 at 10:54 PM, Shawn Heisey <so...@elyograg.org> wrote:
> You'll want to ensure that your autowarmCount value on Solr's caches is low
> enough that each commit happens quickly. If it takes 5000 milliseconds to
> warm the caches when you commit, then you want to be sure that you are
> committing less often than that, or you'll quickly reach your
> maxWarmingSearchers config value. If the commits are happening VARY
> quickly, you may need to set autowarmCount to 0, and possibly disable caches
> entirely.
>
I see. This seems to be the opposite of the approach that I was taking.
>>> I went poking in the code, and it seems that maxWarmingSearchers
>>> defaults to Integer.MAX_VALUE. I'm not sure whether this is a bad
>>> default or not. It does mean that a pathological setup without
>>> maxWarmingSearchers in the config will probably blow up with an
>>> OutOfMemory exception, but is that better or worse than commits that
>>> don't make new documents searchable? I can see arguments either way.
>>
>>
>> This is interesting, what you found is that the value in the stock
>> solrconfig.xml file differs from the Solr default value. I think that
>> this is bad practice: a single default should be decided upon and Solr
>> should use this value when nothing is specified in solrconfig.xml, and
>> that _same_value_ should be specified in the stock solrconfig.xml. Is
>> it not a reasonable assumption that this would be the case?
>
>
> That was directed more at the other committers. I would argue that either a
> low number or a relatively high number should be the default, but not
> MAX_VALUE. The example config should have a commented out section for
> maxWarmingSearchers that mentions the default. I'm having the same
> discussion about maxBooleanClauses on SOLR-4586.
>
Right.
> It's possible that this has already been discussed, and that everyone
> prefers that a badly configured setup will eventually have a spectacular
> blow up with OutOfMemory, rather than semi-silently ignoring commits. A
> searcher object contains caches and uses a lot of memory, so having lots of
> them around will eventually use up the entire heap.
>
Silently dropping data is by far the worse choice, I agree, especially
as a default setting.
--
Dotan Cohen
http://gibberish.co.il
http://what-is-what.com
Re: maxWarmingSearchers in Solr 4.
Posted by Shawn Heisey <so...@elyograg.org>.
On 4/4/2013 12:34 AM, Dotan Cohen wrote:
>> In the case of maxWarmingSearchers, I would hope that you have your
>> system set up so that you would never need more than 1 warming searcher
>> at a time. If you do a commit while a previous commit is still warming,
>> Solr will try to create a second warming searcher.
>
> How would I set the system up for that? We have very many commits
> (every few seconds) and each commit contains a few tens of documents
> (mostly smaller than 1 KiB per document). Right now we get about
> 200-300 searches per minute.
You'll want to ensure that your autowarmCount value on Solr's caches is
low enough that each commit happens quickly. If it takes 5000
milliseconds to warm the caches when you commit, then you want to be
sure that you are committing less often than that, or you'll quickly
reach your maxWarmingSearchers config value. If the commits are
happening VARY quickly, you may need to set autowarmCount to 0, and
possibly disable caches entirely.
>> I went poking in the code, and it seems that maxWarmingSearchers
>> defaults to Integer.MAX_VALUE. I'm not sure whether this is a bad
>> default or not. It does mean that a pathological setup without
>> maxWarmingSearchers in the config will probably blow up with an
>> OutOfMemory exception, but is that better or worse than commits that
>> don't make new documents searchable? I can see arguments either way.
>
> This is interesting, what you found is that the value in the stock
> solrconfig.xml file differs from the Solr default value. I think that
> this is bad practice: a single default should be decided upon and Solr
> should use this value when nothing is specified in solrconfig.xml, and
> that _same_value_ should be specified in the stock solrconfig.xml. Is
> it not a reasonable assumption that this would be the case?
That was directed more at the other committers. I would argue that
either a low number or a relatively high number should be the default,
but not MAX_VALUE. The example config should have a commented out
section for maxWarmingSearchers that mentions the default. I'm having
the same discussion about maxBooleanClauses on SOLR-4586.
It's possible that this has already been discussed, and that everyone
prefers that a badly configured setup will eventually have a spectacular
blow up with OutOfMemory, rather than semi-silently ignoring commits. A
searcher object contains caches and uses a lot of memory, so having lots
of them around will eventually use up the entire heap.
Thanks,
Shawn
Re: maxWarmingSearchers in Solr 4.
Posted by Dotan Cohen <do...@gmail.com>.
On Wed, Apr 3, 2013 at 7:55 PM, Shawn Heisey <so...@elyograg.org> wrote:
> In situations where I don't want to change the default value, I prefer
> to leave config elements out of the solrconfig. It makes the config
> smaller, and it also makes it so that I will automatically see benefits
> from the default changing in new versions.
>
Thanks. This makes sense. I take it, then, that you update (or at
least review) solrconfig for each new Solr version. As I become more
familiar with that file I will begin doing the same.
> In the case of maxWarmingSearchers, I would hope that you have your
> system set up so that you would never need more than 1 warming searcher
> at a time. If you do a commit while a previous commit is still warming,
> Solr will try to create a second warming searcher.
>
How would I set the system up for that? We have very many commits
(every few seconds) and each commit contains a few tens of documents
(mostly smaller than 1 KiB per document). Right now we get about
200-300 searches per minute.
Note that I expect both the commit rate and the search rate to
increase 2-3 times in the next month, and ideally I should be able to
scale it beyond that. I'm right now looking into sharding as a
possible solution.
> I went poking in the code, and it seems that maxWarmingSearchers
> defaults to Integer.MAX_VALUE. I'm not sure whether this is a bad
> default or not. It does mean that a pathological setup without
> maxWarmingSearchers in the config will probably blow up with an
> OutOfMemory exception, but is that better or worse than commits that
> don't make new documents searchable? I can see arguments either way.
>
This is interesting, what you found is that the value in the stock
solrconfig.xml file differs from the Solr default value. I think that
this is bad practice: a single default should be decided upon and Solr
should use this value when nothing is specified in solrconfig.xml, and
that _same_value_ should be specified in the stock solrconfig.xml. Is
it not a reasonable assumption that this would be the case?
--
Dotan Cohen
http://gibberish.co.il
http://what-is-what.com
Re: maxWarmingSearchers in Solr 4.
Posted by Shawn Heisey <so...@elyograg.org>.
On 4/3/2013 1:48 AM, Dotan Cohen wrote:
> I have been dragging the same solrconfig.xml from Solr 3.x to 4.0 to
> 4.1, with no customization (bad, bad me!). I'm now looking into
> customizing it and I see that the Solr 4.1 solrconfig.xml is much
> simpler and shorter. Is this simply because many of the examples have
> been removed?
>
> In particular, I notice that there is no mention of
> maxWarmingSearchers in the Solr 4.1 solrconfig.xml. I assume that I
> can simply add it in, are there any other critical config options that
> are missing that I should be looking into as well? Would I be better
> off using the old Solr 3.x solrconfig.xml in Solr 4.1 as it contains
> so many examples?
In situations where I don't want to change the default value, I prefer
to leave config elements out of the solrconfig. It makes the config
smaller, and it also makes it so that I will automatically see benefits
from the default changing in new versions.
In the case of maxWarmingSearchers, I would hope that you have your
system set up so that you would never need more than 1 warming searcher
at a time. If you do a commit while a previous commit is still warming,
Solr will try to create a second warming searcher.
I went poking in the code, and it seems that maxWarmingSearchers
defaults to Integer.MAX_VALUE. I'm not sure whether this is a bad
default or not. It does mean that a pathological setup without
maxWarmingSearchers in the config will probably blow up with an
OutOfMemory exception, but is that better or worse than commits that
don't make new documents searchable? I can see arguments either way.
Thanks,
Shawn