You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Kayak28 <ka...@gmail.com> on 2020/02/26 05:16:57 UTC
Rule of thumb for determining maxTime of AutoCommit
Hello, Solr Community:
Another day, I had an error "exceeded limit of maxWarmingSearchers=2."
I know this error causes when multiple commits(which opens a new searcher)
are requested too frequently.
As far as I read Solr wiki, it recommends for me to have more interval
between each commit, and make commit frequency less.
Using autoCommit, I would like to decrease the commit frequency, but I am
not sure how much I should increase the value of maxTime in autoCommit?
My current configuration is the following:
<autoCommit>
<maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
<openSearcher>false</openSearcher>
</autoCommit>
How do you determine how much you increase the value in this case?
Is there any rule of thumb advice to configure commit frequency?
Any help will be appreciated.
Sincerely,
Kaya Ota
Re: Rule of thumb for determining maxTime of AutoCommit
Posted by Dwane Hall <dw...@hotmail.com>.
Hey Kaya,
How are you adding documents to your index? Do you control this yourself or do you have multiple clients (curl, solrJ, calls directly to /update*) updating data in your index as I suspect (based on your hard and soft commit settings) that a client may be causing your soft commits making updates using the commitWithin parameter (who's default behaviour is a soft commit [openSearcher=ture]). If this is the case and you don't want to allow client commits you can prevent this behaviour by adding a processor to your updateRequestProcessorChain of your solrconfig.xml and then manage your commits by configuring the settings you've described below (https://lucene.apache.org/solr/guide/8_4/shards-and-indexing-data-in-solrcloud.html#ignoring-commits-from-client-applications-in-solrcloud).
I always go back to Erick Erikson's great Lucidworks blog post for all things indexing related so if you haven't seen it I highly recommend checking it out. There's a ton of great info in this post so it may take a few reads to completely digest it all and towards the end he provides some index and query scenarios and some sensible hard and soft commit settings to address some common use cases.
To this day his words in this post still ring true in my ear "hard commits are about durability, soft commits are about visibility" (https://lucidworks.com/post/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/)
Good luck,
Dwane
________________________________
From: Emir Arnautović <em...@sematext.com>
Sent: Thursday, 27 February 2020 9:23 PM
To: solr-user@lucene.apache.org <so...@lucene.apache.org>
Subject: Re: Rule of thumb for determining maxTime of AutoCommit
Hi Kaya,
Since you do not have soft commits, you must have explicit commits somewhere since your hard commits are configured not to open searcher.
Re warming up: yes - you are right. You need to check your queries and warmup numbers in cache configs. What you need to check is how log does warmup takes and if it takes too long reduce number of warmup queries/items. I think that there is cumulative warming time in admin console, or if you prefer some proper Solr monitoring tool, you can check out our Solr integration: https://apps.sematext.com/demo <https://apps.sematext.com/demo>
HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/
> On 27 Feb 2020, at 03:00, Kayak28 <ka...@gmail.com> wrote:
>
> Hello, Emir:
>
> Thank you for your reply.
> I do understand that the frequency of creating searcher depends on how much
> realitime-search is required.
>
> As you advise me, I have checked a soft-commit configuration.
> It is configured as:
> ${solr.autoSoftCommit.maxTime:-1}
>
> If I am correct, I have not set autoSoftCommit, which means autoSoftCommit
> does not create a new searcher.
> Next, I will take a look at my explicit commit.
>
> I am curious about what you say "warming strategy."
> Is there any good resource to learn about tuning warming up?
>
> As far as I know about warming up, there is two warming-up functions in
> Solr.
> One is static warming up, which you can configure queries in solrconfig.xml
> The other is dynamic warming up, which uses queries from old cache.
>
> How should I tune them?
> What is the first step to look at?
> (I am kinda guessing the answer can vary depends on the system, the
> service, etc... )
>
>
>
> Sincerely,
> Kaya Ota
>
>
>
> 2020年2月26日(水) 17:36 Emir Arnautović <em...@sematext.com>:
>
>> Hi Kaya,
>> The answer is simple: as much as your requirements allow delay between
>> data being indexed and changes being visible. It is sometimes seconds and
>> sometimes hours or even a day is tolerable. On each commit your caches are
>> invalidated and warmed (if it is configured like that) so in order to get
>> better use of caches, you should commit as rare as possible.
>>
>> The setting that you provided is about hard commits and those are
>> configured not to open new searcher so such commit does not cause “exceeded
>> limit” error. You either have soft auto commits configured or you do
>> explicit commits when updating documents. Check and tune those and if you
>> do explicit commits, remove those if possible. If you cannot afford less
>> frequent commits, you have to tune your warming strategy to make sure it
>> does not take as much time as period between two commits.
>>
>> HTH,
>> Emir
>> --
>> Monitoring - Log Management - Alerting - Anomaly Detection
>> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>>
>>
>>
>>> On 26 Feb 2020, at 06:16, Kayak28 <ka...@gmail.com> wrote:
>>>
>>> Hello, Solr Community:
>>>
>>> Another day, I had an error "exceeded limit of maxWarmingSearchers=2."
>>> I know this error causes when multiple commits(which opens a new
>> searcher)
>>> are requested too frequently.
>>>
>>> As far as I read Solr wiki, it recommends for me to have more interval
>>> between each commit, and make commit frequency less.
>>> Using autoCommit, I would like to decrease the commit frequency, but I
>> am
>>> not sure how much I should increase the value of maxTime in autoCommit?
>>>
>>> My current configuration is the following:
>>>
>>> <autoCommit>
>>> <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
>>> <openSearcher>false</openSearcher>
>>> </autoCommit>
>>>
>>>
>>>
>>> How do you determine how much you increase the value in this case?
>>> Is there any rule of thumb advice to configure commit frequency?
>>>
>>> Any help will be appreciated.
>>>
>>> Sincerely,
>>> Kaya Ota
>>
>>
Re: Rule of thumb for determining maxTime of AutoCommit
Posted by Emir Arnautović <em...@sematext.com>.
Hi Kaya,
Since you do not have soft commits, you must have explicit commits somewhere since your hard commits are configured not to open searcher.
Re warming up: yes - you are right. You need to check your queries and warmup numbers in cache configs. What you need to check is how log does warmup takes and if it takes too long reduce number of warmup queries/items. I think that there is cumulative warming time in admin console, or if you prefer some proper Solr monitoring tool, you can check out our Solr integration: https://apps.sematext.com/demo <https://apps.sematext.com/demo>
HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/
> On 27 Feb 2020, at 03:00, Kayak28 <ka...@gmail.com> wrote:
>
> Hello, Emir:
>
> Thank you for your reply.
> I do understand that the frequency of creating searcher depends on how much
> realitime-search is required.
>
> As you advise me, I have checked a soft-commit configuration.
> It is configured as:
> ${solr.autoSoftCommit.maxTime:-1}
>
> If I am correct, I have not set autoSoftCommit, which means autoSoftCommit
> does not create a new searcher.
> Next, I will take a look at my explicit commit.
>
> I am curious about what you say "warming strategy."
> Is there any good resource to learn about tuning warming up?
>
> As far as I know about warming up, there is two warming-up functions in
> Solr.
> One is static warming up, which you can configure queries in solrconfig.xml
> The other is dynamic warming up, which uses queries from old cache.
>
> How should I tune them?
> What is the first step to look at?
> (I am kinda guessing the answer can vary depends on the system, the
> service, etc... )
>
>
>
> Sincerely,
> Kaya Ota
>
>
>
> 2020年2月26日(水) 17:36 Emir Arnautović <em...@sematext.com>:
>
>> Hi Kaya,
>> The answer is simple: as much as your requirements allow delay between
>> data being indexed and changes being visible. It is sometimes seconds and
>> sometimes hours or even a day is tolerable. On each commit your caches are
>> invalidated and warmed (if it is configured like that) so in order to get
>> better use of caches, you should commit as rare as possible.
>>
>> The setting that you provided is about hard commits and those are
>> configured not to open new searcher so such commit does not cause “exceeded
>> limit” error. You either have soft auto commits configured or you do
>> explicit commits when updating documents. Check and tune those and if you
>> do explicit commits, remove those if possible. If you cannot afford less
>> frequent commits, you have to tune your warming strategy to make sure it
>> does not take as much time as period between two commits.
>>
>> HTH,
>> Emir
>> --
>> Monitoring - Log Management - Alerting - Anomaly Detection
>> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>>
>>
>>
>>> On 26 Feb 2020, at 06:16, Kayak28 <ka...@gmail.com> wrote:
>>>
>>> Hello, Solr Community:
>>>
>>> Another day, I had an error "exceeded limit of maxWarmingSearchers=2."
>>> I know this error causes when multiple commits(which opens a new
>> searcher)
>>> are requested too frequently.
>>>
>>> As far as I read Solr wiki, it recommends for me to have more interval
>>> between each commit, and make commit frequency less.
>>> Using autoCommit, I would like to decrease the commit frequency, but I
>> am
>>> not sure how much I should increase the value of maxTime in autoCommit?
>>>
>>> My current configuration is the following:
>>>
>>> <autoCommit>
>>> <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
>>> <openSearcher>false</openSearcher>
>>> </autoCommit>
>>>
>>>
>>>
>>> How do you determine how much you increase the value in this case?
>>> Is there any rule of thumb advice to configure commit frequency?
>>>
>>> Any help will be appreciated.
>>>
>>> Sincerely,
>>> Kaya Ota
>>
>>
Re: Rule of thumb for determining maxTime of AutoCommit
Posted by Kayak28 <ka...@gmail.com>.
Hello, Emir:
Thank you for your reply.
I do understand that the frequency of creating searcher depends on how much
realitime-search is required.
As you advise me, I have checked a soft-commit configuration.
It is configured as:
${solr.autoSoftCommit.maxTime:-1}
If I am correct, I have not set autoSoftCommit, which means autoSoftCommit
does not create a new searcher.
Next, I will take a look at my explicit commit.
I am curious about what you say "warming strategy."
Is there any good resource to learn about tuning warming up?
As far as I know about warming up, there is two warming-up functions in
Solr.
One is static warming up, which you can configure queries in solrconfig.xml
The other is dynamic warming up, which uses queries from old cache.
How should I tune them?
What is the first step to look at?
(I am kinda guessing the answer can vary depends on the system, the
service, etc... )
Sincerely,
Kaya Ota
2020年2月26日(水) 17:36 Emir Arnautović <em...@sematext.com>:
> Hi Kaya,
> The answer is simple: as much as your requirements allow delay between
> data being indexed and changes being visible. It is sometimes seconds and
> sometimes hours or even a day is tolerable. On each commit your caches are
> invalidated and warmed (if it is configured like that) so in order to get
> better use of caches, you should commit as rare as possible.
>
> The setting that you provided is about hard commits and those are
> configured not to open new searcher so such commit does not cause “exceeded
> limit” error. You either have soft auto commits configured or you do
> explicit commits when updating documents. Check and tune those and if you
> do explicit commits, remove those if possible. If you cannot afford less
> frequent commits, you have to tune your warming strategy to make sure it
> does not take as much time as period between two commits.
>
> HTH,
> Emir
> --
> Monitoring - Log Management - Alerting - Anomaly Detection
> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>
>
>
> > On 26 Feb 2020, at 06:16, Kayak28 <ka...@gmail.com> wrote:
> >
> > Hello, Solr Community:
> >
> > Another day, I had an error "exceeded limit of maxWarmingSearchers=2."
> > I know this error causes when multiple commits(which opens a new
> searcher)
> > are requested too frequently.
> >
> > As far as I read Solr wiki, it recommends for me to have more interval
> > between each commit, and make commit frequency less.
> > Using autoCommit, I would like to decrease the commit frequency, but I
> am
> > not sure how much I should increase the value of maxTime in autoCommit?
> >
> > My current configuration is the following:
> >
> > <autoCommit>
> > <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
> > <openSearcher>false</openSearcher>
> > </autoCommit>
> >
> >
> >
> > How do you determine how much you increase the value in this case?
> > Is there any rule of thumb advice to configure commit frequency?
> >
> > Any help will be appreciated.
> >
> > Sincerely,
> > Kaya Ota
>
>
Re: Rule of thumb for determining maxTime of AutoCommit
Posted by Emir Arnautović <em...@sematext.com>.
Hi Kaya,
The answer is simple: as much as your requirements allow delay between data being indexed and changes being visible. It is sometimes seconds and sometimes hours or even a day is tolerable. On each commit your caches are invalidated and warmed (if it is configured like that) so in order to get better use of caches, you should commit as rare as possible.
The setting that you provided is about hard commits and those are configured not to open new searcher so such commit does not cause “exceeded limit” error. You either have soft auto commits configured or you do explicit commits when updating documents. Check and tune those and if you do explicit commits, remove those if possible. If you cannot afford less frequent commits, you have to tune your warming strategy to make sure it does not take as much time as period between two commits.
HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/
> On 26 Feb 2020, at 06:16, Kayak28 <ka...@gmail.com> wrote:
>
> Hello, Solr Community:
>
> Another day, I had an error "exceeded limit of maxWarmingSearchers=2."
> I know this error causes when multiple commits(which opens a new searcher)
> are requested too frequently.
>
> As far as I read Solr wiki, it recommends for me to have more interval
> between each commit, and make commit frequency less.
> Using autoCommit, I would like to decrease the commit frequency, but I am
> not sure how much I should increase the value of maxTime in autoCommit?
>
> My current configuration is the following:
>
> <autoCommit>
> <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
> <openSearcher>false</openSearcher>
> </autoCommit>
>
>
>
> How do you determine how much you increase the value in this case?
> Is there any rule of thumb advice to configure commit frequency?
>
> Any help will be appreciated.
>
> Sincerely,
> Kaya Ota