You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "M, Arjun (NSN - IN/Bangalore)" <ar...@nsn.com> on 2014/05/28 05:51:01 UTC
Error enquiry- exceeded limit of maxWarmingSearchers=2
Hi,
I am getting the below error.
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later.
Can you please help?
Re: Error enquiry- exceeded limit of maxWarmingSearchers=2
Posted by Jason Hellman <jh...@innoventsolutions.com>.
I just realized I failed my own reading comprehension :)
You have maxDocs, not maxTime for hard commit. Please disregard.
On May 30, 2014, at 1:46 PM, Jason Hellman <jh...@innoventsolutions.com> wrote:
> I’m also not sure I understand the practical purpose of your hard/soft auto commit settings. You are stating the following:
>
> Every 10 seconds I want data written to disk, but not be searchable.
> Every 15 seconds I want data to be written into memory and searchable.
>
> I would consider whether your soft commit window is too long, or if you can lengthen your hard commit period. It’s typical to see hard commits occur *less* frequently than soft commits.
>
>
> On May 30, 2014, at 11:04 AM, Shawn Heisey <so...@elyograg.org> wrote:
>
>> On 5/29/2014 9:55 PM, M, Arjun (NSN - IN/Bangalore) wrote:
>>> Thanks a lot for your nice explanation.. Now I understood the difference between autoCommit and autoSoftCommit.. Now my config looks like below.
>>>
>>> <autoCommit>
>>> <maxDocs>10000</maxDocs>
>>> <openSearcher>false</openSearcher>
>>> </autoCommit>
>>>
>>> <autoSoftCommit>
>>> <maxTime>15000</maxTime>
>>> </autoSoftCommit>
>>>
>>>
>>> With this now I am getting some other error like this.
>>>
>>> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: version conflict for 1401421678039128128000000030383128128 expected=1469497192978841608 actual=1469497212082847746
>>
>> This sounds like you are including the _version_ field in your document
>> when you index. You probably shouldn't be doing that. Here's what that
>> field is for, and how it works:
>>
>> http://heliosearch.org/solr/optimistic-concurrency/
>>
>> Thanks,
>> Shawn
>>
>
Re: Error enquiry- exceeded limit of maxWarmingSearchers=2
Posted by Jason Hellman <jh...@innoventsolutions.com>.
I’m also not sure I understand the practical purpose of your hard/soft auto commit settings. You are stating the following:
Every 10 seconds I want data written to disk, but not be searchable.
Every 15 seconds I want data to be written into memory and searchable.
I would consider whether your soft commit window is too long, or if you can lengthen your hard commit period. It’s typical to see hard commits occur *less* frequently than soft commits.
On May 30, 2014, at 11:04 AM, Shawn Heisey <so...@elyograg.org> wrote:
> On 5/29/2014 9:55 PM, M, Arjun (NSN - IN/Bangalore) wrote:
>> Thanks a lot for your nice explanation.. Now I understood the difference between autoCommit and autoSoftCommit.. Now my config looks like below.
>>
>> <autoCommit>
>> <maxDocs>10000</maxDocs>
>> <openSearcher>false</openSearcher>
>> </autoCommit>
>>
>> <autoSoftCommit>
>> <maxTime>15000</maxTime>
>> </autoSoftCommit>
>>
>>
>> With this now I am getting some other error like this.
>>
>> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: version conflict for 1401421678039128128000000030383128128 expected=1469497192978841608 actual=1469497212082847746
>
> This sounds like you are including the _version_ field in your document
> when you index. You probably shouldn't be doing that. Here's what that
> field is for, and how it works:
>
> http://heliosearch.org/solr/optimistic-concurrency/
>
> Thanks,
> Shawn
>
Re: Error enquiry- exceeded limit of maxWarmingSearchers=2
Posted by Shawn Heisey <so...@elyograg.org>.
On 5/29/2014 9:55 PM, M, Arjun (NSN - IN/Bangalore) wrote:
> Thanks a lot for your nice explanation.. Now I understood the difference between autoCommit and autoSoftCommit.. Now my config looks like below.
>
> <autoCommit>
> <maxDocs>10000</maxDocs>
> <openSearcher>false</openSearcher>
> </autoCommit>
>
> <autoSoftCommit>
> <maxTime>15000</maxTime>
> </autoSoftCommit>
>
>
> With this now I am getting some other error like this.
>
> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: version conflict for 1401421678039128128000000030383128128 expected=1469497192978841608 actual=1469497212082847746
This sounds like you are including the _version_ field in your document
when you index. You probably shouldn't be doing that. Here's what that
field is for, and how it works:
http://heliosearch.org/solr/optimistic-concurrency/
Thanks,
Shawn
RE: Error enquiry- exceeded limit of maxWarmingSearchers=2
Posted by "M, Arjun (NSN - IN/Bangalore)" <ar...@nsn.com>.
Hi Shawn,
Thanks a lot for your nice explanation.. Now I understood the difference between autoCommit and autoSoftCommit.. Now my config looks like below.
<autoCommit>
<maxDocs>10000</maxDocs>
<openSearcher>false</openSearcher>
</autoCommit>
<autoSoftCommit>
<maxTime>15000</maxTime>
</autoSoftCommit>
With this now I am getting some other error like this.
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: version conflict for 1401421678039128128000000030383128128 expected=1469497192978841608 actual=1469497212082847746
What could be the reason?
Thanks & Regards,
Arjun M
-----Original Message-----
From: ext Shawn Heisey [mailto:solr@elyograg.org]
Sent: Thursday, May 29, 2014 10:14 PM
To: solr-user@lucene.apache.org
Subject: Re: Error enquiry- exceeded limit of maxWarmingSearchers=2
On 5/29/2014 7:52 AM, M, Arjun (NSN - IN/Bangalore) wrote:
> Thanks Shawn... Just one more question..
>
> Can both autoCommit and autoSoftCommit be enabled? If both are enabled, which one takes precedence?
Yes, and it's a very common configuration. If you do enable both, you
want openSearcher to be false on autoCommit, so that your hard commits
are not making documents visible. That is a job for autoSoftCommit. If
you use openSearcher=false on autoCommit, then the question of which one
takes precendence actually has no meaning, because the two kinds of
commits will be doing different things.
Read this until you completely understand it:
http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
Thanks,
Shawn
Re: Error enquiry- exceeded limit of maxWarmingSearchers=2
Posted by Shawn Heisey <so...@elyograg.org>.
On 5/29/2014 7:52 AM, M, Arjun (NSN - IN/Bangalore) wrote:
> Thanks Shawn... Just one more question..
>
> Can both autoCommit and autoSoftCommit be enabled? If both are enabled, which one takes precedence?
Yes, and it's a very common configuration. If you do enable both, you
want openSearcher to be false on autoCommit, so that your hard commits
are not making documents visible. That is a job for autoSoftCommit. If
you use openSearcher=false on autoCommit, then the question of which one
takes precendence actually has no meaning, because the two kinds of
commits will be doing different things.
Read this until you completely understand it:
http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
Thanks,
Shawn
RE: Error enquiry- exceeded limit of maxWarmingSearchers=2
Posted by "M, Arjun (NSN - IN/Bangalore)" <ar...@nsn.com>.
Thanks Shawn... Just one more question..
Can both autoCommit and autoSoftCommit be enabled? If both are enabled, which one takes precedence?
Thanks & Regards,
Arjun M
-----Original Message-----
From: ext Shawn Heisey [mailto:solr@elyograg.org]
Sent: Thursday, May 29, 2014 7:02 PM
To: solr-user@lucene.apache.org
Subject: Re: Error enquiry- exceeded limit of maxWarmingSearchers=2
On 5/29/2014 4:18 AM, M, Arjun (NSN - IN/Bangalore) wrote:
> Thanks for your valuable inputs... Find below my code and config in solrconfig.xml. Index update is successful but I am not able to see any data from solr admin console. What could be the issue? Any help here is highly appreciated.
>
> I can see the data in the solr admin gui after tomcat restart(solr is running in tomcat in my case)
>
> private void addToSolr(List<SolrInputDocument> c) throws SolrServerException, IOException {
> if (!c.isEmpty()) {
> try {
>
>
> solr.add(c);
> logger.info("Commit size after Add=" + c.size());
>
> } finally {
> //renew lock
> }
> }
> }
>
> autoCommit config in solrconfig.xml
> =====================================
>
> <autoCommit>
> <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
> <maxDocs>10000</maxDocs>
> <openSearcher>false</openSearcher>
> </autoCommit>
>
> <autoSoftCommit>
> <maxTime>${solr.autoSoftCommit.maxTime:-1}</maxTime>
> </autoSoftCommit>
The code snippet does not include a commit.
I am not really clear on what using a value of -1 would do on maxTime
here. I suspect that it efectively disables autoSoftCommit. If that's
the case, then there is nothing at all in your code or your config that
will open a new searcher -- that option is set to false in your autoCommit.
If you want Solr to automatically do commits to make documents visible,
I think you should configure a maxTime value for autoSoftCommit, and
make it as long as you can possibly stand to not have new documents
available. Then you won't have to worry about commits in your code at all.
> Few more questions..
>
> 2) If I use solrServer.add(<Doc list>,<commitwithin>), should I do solrServer.commit() also
No. The commitWithin would do a soft commit for you once that much time
has elapsed since indexing started (or the last commit with
openSearcher=true), so you would not need to do a commit().
My opinion is that you should not combine manual commits with
autoSoftCommit. Depending on exactly what your needs are, you might
want to use commitWithin, and have autoSoftCommit as a last guarantee
against errors in your indexing process.
Thanks,
Shawn
Re: Error enquiry- exceeded limit of maxWarmingSearchers=2
Posted by Shawn Heisey <so...@elyograg.org>.
On 5/29/2014 4:18 AM, M, Arjun (NSN - IN/Bangalore) wrote:
> Thanks for your valuable inputs... Find below my code and config in solrconfig.xml. Index update is successful but I am not able to see any data from solr admin console. What could be the issue? Any help here is highly appreciated.
>
> I can see the data in the solr admin gui after tomcat restart(solr is running in tomcat in my case)
>
> private void addToSolr(List<SolrInputDocument> c) throws SolrServerException, IOException {
> if (!c.isEmpty()) {
> try {
>
>
> solr.add(c);
> logger.info("Commit size after Add=" + c.size());
>
> } finally {
> //renew lock
> }
> }
> }
>
> autoCommit config in solrconfig.xml
> =====================================
>
> <autoCommit>
> <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
> <maxDocs>10000</maxDocs>
> <openSearcher>false</openSearcher>
> </autoCommit>
>
> <autoSoftCommit>
> <maxTime>${solr.autoSoftCommit.maxTime:-1}</maxTime>
> </autoSoftCommit>
The code snippet does not include a commit.
I am not really clear on what using a value of -1 would do on maxTime
here. I suspect that it efectively disables autoSoftCommit. If that's
the case, then there is nothing at all in your code or your config that
will open a new searcher -- that option is set to false in your autoCommit.
If you want Solr to automatically do commits to make documents visible,
I think you should configure a maxTime value for autoSoftCommit, and
make it as long as you can possibly stand to not have new documents
available. Then you won't have to worry about commits in your code at all.
> Few more questions..
>
> 2) If I use solrServer.add(<Doc list>,<commitwithin>), should I do solrServer.commit() also
No. The commitWithin would do a soft commit for you once that much time
has elapsed since indexing started (or the last commit with
openSearcher=true), so you would not need to do a commit().
My opinion is that you should not combine manual commits with
autoSoftCommit. Depending on exactly what your needs are, you might
want to use commitWithin, and have autoSoftCommit as a last guarantee
against errors in your indexing process.
Thanks,
Shawn
RE: Error enquiry- exceeded limit of maxWarmingSearchers=2
Posted by "M, Arjun (NSN - IN/Bangalore)" <ar...@nsn.com>.
Hi,
Thanks for your valuable inputs... Find below my code and config in solrconfig.xml. Index update is successful but I am not able to see any data from solr admin console. What could be the issue? Any help here is highly appreciated.
I can see the data in the solr admin gui after tomcat restart(solr is running in tomcat in my case)
private void addToSolr(List<SolrInputDocument> c) throws SolrServerException, IOException {
if (!c.isEmpty()) {
try {
solr.add(c);
logger.info("Commit size after Add=" + c.size());
} finally {
//renew lock
}
}
}
autoCommit config in solrconfig.xml
=====================================
<autoCommit>
<maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
<maxDocs>10000</maxDocs>
<openSearcher>false</openSearcher>
</autoCommit>
<autoSoftCommit>
<maxTime>${solr.autoSoftCommit.maxTime:-1}</maxTime>
</autoSoftCommit>
Few more questions..
2) If I use solrServer.add(<Doc list>,<commitwithin>), should I do solrServer.commit() also
Thanks & Regards,
Arjun M
-----Original Message-----
From: ext Shawn Heisey [mailto:solr@elyograg.org]
Sent: Wednesday, May 28, 2014 6:36 PM
To: solr-user@lucene.apache.org
Subject: Re: Error enquiry- exceeded limit of maxWarmingSearchers=2
On 5/28/2014 3:45 AM, M, Arjun (NSN - IN/Bangalore) wrote:
> Also is there a way to check if autowarming completed (or) how to make the next commit wait till previous commit finishes?
With Solr, probably not. There might be a statistic available from an
admin handler that I don't know about, but as far as I know, your code
must be aware of approximately how long a commit is likely to take, and
not send another commit until you can be sure that the previous commit
is done. This includes the commitWithin parameter on an update request.
Now that I've just said that, you *can* do an all documents query with
rows=0 and look for a change in numFound. An update might actually
result in no change to numFound, so you would need to build in a
time-based exit to the loop that looks for numFound changes.
In the case of commits done automatically by the configuration
(autoCommit and/or autoSoftCommit), there is definitely no way to detect
when a previous commit is done.
The general recommendation with Solr 4.x is to have autoCommit enabled
with openSearcher=false, with a relatively short maxTime -- from 5
minutes down to 15 seconds, depending on indexing rate. These commits
will not open a new searcher, and they will not make new documents visible.
For commits that affect which documents are visible, you need to
determine how long you can possibly stand to go without seeing new data
that has been indexed. Once you know that time interval, you can use it
to do a manual commit, or you can set up autoSoftCommit with that
interval. It is not at all unusual to have an autoCommit time interval
that's shorter than autoSoftCommit.
This blog post mentions SolrCloud, but it is also applicable to Solr 4.x
when NOT running in cloud mode:
http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
Thanks,
Shawn
Re: Error enquiry- exceeded limit of maxWarmingSearchers=2
Posted by Shawn Heisey <so...@elyograg.org>.
On 5/28/2014 3:45 AM, M, Arjun (NSN - IN/Bangalore) wrote:
> Also is there a way to check if autowarming completed (or) how to make the next commit wait till previous commit finishes?
With Solr, probably not. There might be a statistic available from an
admin handler that I don't know about, but as far as I know, your code
must be aware of approximately how long a commit is likely to take, and
not send another commit until you can be sure that the previous commit
is done. This includes the commitWithin parameter on an update request.
Now that I've just said that, you *can* do an all documents query with
rows=0 and look for a change in numFound. An update might actually
result in no change to numFound, so you would need to build in a
time-based exit to the loop that looks for numFound changes.
In the case of commits done automatically by the configuration
(autoCommit and/or autoSoftCommit), there is definitely no way to detect
when a previous commit is done.
The general recommendation with Solr 4.x is to have autoCommit enabled
with openSearcher=false, with a relatively short maxTime -- from 5
minutes down to 15 seconds, depending on indexing rate. These commits
will not open a new searcher, and they will not make new documents visible.
For commits that affect which documents are visible, you need to
determine how long you can possibly stand to go without seeing new data
that has been indexed. Once you know that time interval, you can use it
to do a manual commit, or you can set up autoSoftCommit with that
interval. It is not at all unusual to have an autoCommit time interval
that's shorter than autoSoftCommit.
This blog post mentions SolrCloud, but it is also applicable to Solr 4.x
when NOT running in cloud mode:
http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
Thanks,
Shawn
RE: Error enquiry- exceeded limit of maxWarmingSearchers=2
Posted by "M, Arjun (NSN - IN/Bangalore)" <ar...@nsn.com>.
Hi,
Also is there a way to check if autowarming completed (or) how to make the next commit wait till previous commit finishes?
Thanks & Regards,
Arjun M
-----Original Message-----
From: ext Shawn Heisey [mailto:solr@elyograg.org]
Sent: Wednesday, May 28, 2014 10:31 AM
To: solr-user@lucene.apache.org
Subject: Re: Error enquiry- exceeded limit of maxWarmingSearchers=2
> Hi,
>
> I am getting the below error.
>
> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
> Error opening new searcher. exceeded limit of
> maxWarmingSearchers=2, try again later.
This error is usually a symptom of a problem, not the actual problem.
Either you are running into performance issues that are making your
commits slow, or you are committing too frequently. Either way, you've
got a situation where one commit (with opensearcher=true) is not able to
finish before the next commit starts.
Solr puts a limit on the number of searcher objects that can be starting
up (warming) at the same time. You've exceeded that limit. Here's a wiki
page about slow commits:
http://wiki.apache.org/solr/SolrPerformanceProblems#Slow_commits
The rest of that wiki page discusses other things that can cause Solr
performance issues.
Thanks,
Shawn
Re: Error enquiry- exceeded limit of maxWarmingSearchers=2
Posted by Shawn Heisey <so...@elyograg.org>.
> Hi,
>
> I am getting the below error.
>
> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
> Error opening new searcher. exceeded limit of
> maxWarmingSearchers=2, try again later.
This error is usually a symptom of a problem, not the actual problem.
Either you are running into performance issues that are making your
commits slow, or you are committing too frequently. Either way, you've
got a situation where one commit (with opensearcher=true) is not able to
finish before the next commit starts.
Solr puts a limit on the number of searcher objects that can be starting
up (warming) at the same time. You've exceeded that limit. Here's a wiki
page about slow commits:
http://wiki.apache.org/solr/SolrPerformanceProblems#Slow_commits
The rest of that wiki page discusses other things that can cause Solr
performance issues.
Thanks,
Shawn