You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Adrian Liew <ad...@avanade.com> on 2015/11/16 08:40:01 UTC

Solr Cloud 5.3.0 Errors in Logs

Hi there,

Will like to get some opinions on the errors encountered below. I have currently setup a SolrCloud cluster of 3 servers (each server hosting a Solr instance and a Zookeeper instance).

I am encountering the errors below in the logs:
Monday, November 16, 2015 3:22:54 PM ERROR null SolrCore org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=6,​ try again later.
Monday, November 16, 2015 3:22:54 PM ERROR null SolrCore org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=6,​ try again later.
Monday, November 16, 2015 3:22:54 PM ERROR null SolrCore org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=6,​ try again later.
Monday, November 16, 2015 3:22:54 PM ERROR null SolrCore org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=6,​ try again later.
Monday, November 16, 2015 3:22:54 PM ERROR null SolrCmdDistributor org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://172.18.111.112:8983/solr/sitecore_master_index_shard1_replica1: Error opening new searcher. exceeded limit of maxWarmingSearchers=6,​ try again later.
Monday, November 16, 2015 3:22:54 PM ERROR null SolrCmdDistributor org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://172.18.111.112:8983/solr/sitecore_master_index_shard1_replica1: Error opening new searcher. exceeded limit of maxWarmingSearchers=6,​ try again later.
Monday, November 16, 2015 3:22:54 PM WARN null DistributedUpdateProcessor Error sending update to http://172.18.111.112:8983/solr
Monday, November 16, 2015 3:22:54 PM WARN null DistributedUpdateProcessor Error sending update to http://172.18.111.112:8983/solr
Monday, November 16, 2015 3:22:54 PM WARN null DistributedUpdateProcessor Error sending update to http://172.18.111.112:8983/solr
Monday, November 16, 2015 3:22:54 PM WARN null DistributedUpdateProcessor Error sending update to http://172.18.111.112:8983/solr

11/16/2015, 3:17:09 PM

WARN

null

DistributedUpdateProcessor

Error sending update to http://172.18.111.112:8983/solr

11/16/2015, 3:17:09 PM

WARN

null

DistributedUpdateProcessor

Error sending update to http://172.18.111.112:8983/solr

11/16/2015, 3:22:26 PM

ERROR

null

SolrCmdDistributor

org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://172.18.111.112:8983/solr/sitecore_master_index_shard1_replica1



Main errors are Timeout occurred exceptions, maxWarmingSearchers exceeded. Is anyone able to advise or have experienced something the same as the above in their SolrCloud setup?

Regards,
Adrian

RE: Solr Cloud 5.3.0 Errors in Logs

Posted by Adrian Liew <ad...@avanade.com>.
Thanks for the tip Eric. Really useful article to know.

I will keep you posted on my findings!

Regards,
Adrian

-----Original Message-----
From: Erick Erickson [mailto:erickerickson@gmail.com] 
Sent: Tuesday, November 17, 2015 8:56 AM
To: solr-user <so...@lucene.apache.org>
Subject: Re: Solr Cloud 5.3.0 Errors in Logs

Here's perhaps more than you really want to know about commits....

https://lucidworks.com/blog/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/

The short form is that most setups set autoCommit to a relatively short interval (15-60 seconds at least under heavy indexing loads, I've seen 10-15 minutes under relatively light loads) with openSearcher set to false. And I rarely set maxDocs in that configuration, it's actually not that useful IMO.

Then set up an autoSoftCommit to be as long as you can tolerate, but IMO rarely shorter than 60 seconds unless you have very aggressive near-real-time (NRT) requirements.

And if your product manager simply insists on very aggressive NRT settings, you should consider making your filterCache and queryResultCache relatively small with minimal autowarming.

Any time you exceed maxWarmingSearchers, it indicates poorly configured Solr instances and/or commits happening far too often.
Bumping that up to numbers greater than two is almost always a band-aid over that misconfiguration.

On Mon, Nov 16, 2015 at 3:45 PM, Adrian Liew <ad...@avanade.com> wrote:
> Thanks Eric.
>
> Here is my reply
>
>>> 1> that commits are happening too often, and especially if the 
>>> 1> commits
>>> are happening
>>> from a client. If commits aren't being sent by a client, then look at autoCommit and softAutoCommit in solrconfig.xml (if you can).
> Understand what you mean. Besides talking to to the folks at Sitecore on where they issue commits, is there a way I can balance these with autoCommit and softAutoCommit in solrconfig.xml? Best is, can you recommend any articles that talk about best practice configuration for a production setup.
>
>>> 2> excessive autowarm settings, again in solronfig.xml.
>>> If, as you say all of Solr is a black box, then talk to the Sitecore folks, on the surface Solr is just poorly configured.
> I will raise with Sitecore guys on this. Particularly asking them why commits are happening faster than searchers can be opened. I have seen overlappingDeckSearchers have exceeded the limit errors as well. Let you know what I get back from the guys.
>
> Regards,
> Adrian
>
> -----Original Message-----
> From: Erick Erickson [mailto:erickerickson@gmail.com]
> Sent: Tuesday, November 17, 2015 12:44 AM
> To: solr-user <so...@lucene.apache.org>
> Subject: Re: Solr Cloud 5.3.0 Errors in Logs
>
> Having 6 warming serachers is an anti-pattern. What it means is that commits are happening faster than your searcher can be opened. There is _no_ good reason that I know of for changing it from 2, having changed it in solrconfig.xml to 6 almost always indicates an improper configuration.
>
> Places to look:
> 1> that commits are happening too often, and especially if the commits
> are happening
> from a client. If commits aren't being sent by a client, then look at autoCommit and softAutoCommit in solrconfig.xml (if you can).
>
> 2> excessive autowarm settings, again in solronfig.xml.
>
> If, as you say all of Solr is a black box, then talk to the Sitecore folks, on the surface Solr is just poorly configured.
>
> Best,
> Erick
>
> On Mon, Nov 16, 2015 at 4:33 AM, Adrian Liew <ad...@avanade.com> wrote:
>> Hi Emir,
>>
>> I am working with a third party platform, Sitecore. The product is a black box that encapsulates the internal workings of solr queries and so on. If there are any questions you have with regards with the below, let me know. It will be useful for me to communicate what could cause the issues below.
>>
>> Regards,
>> Adrian
>>
>> -----Original Message-----
>> From: Emir Arnautovic [mailto:emir.arnautovic@sematext.com]
>> Sent: Monday, November 16, 2015 4:47 PM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Solr Cloud 5.3.0 Errors in Logs
>>
>> Hi Adrian,
>> Can you give us bit more details about warmup queries you use and test that you are running when error occurs.
>>
>> Thanks,
>> Emir
>>
>> On 16.11.2015 08:40, Adrian Liew wrote:
>>> Hi there,
>>>
>>> Will like to get some opinions on the errors encountered below. I have currently setup a SolrCloud cluster of 3 servers (each server hosting a Solr instance and a Zookeeper instance).
>>>
>>> I am encountering the errors below in the logs:
>>> Monday, November 16, 2015 3:22:54 PM ERROR null SolrCore org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=6, try again later.
>>> Monday, November 16, 2015 3:22:54 PM ERROR null SolrCore org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=6, try again later.
>>> Monday, November 16, 2015 3:22:54 PM ERROR null SolrCore org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=6, try again later.
>>> Monday, November 16, 2015 3:22:54 PM ERROR null SolrCore org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=6, try again later.
>>> Monday, November 16, 2015 3:22:54 PM ERROR null SolrCmdDistributor org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://172.18.111.112:8983/solr/sitecore_master_index_shard1_replica1: Error opening new searcher. exceeded limit of maxWarmingSearchers=6, try again later.
>>> Monday, November 16, 2015 3:22:54 PM ERROR null SolrCmdDistributor org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://172.18.111.112:8983/solr/sitecore_master_index_shard1_replica1: Error opening new searcher. exceeded limit of maxWarmingSearchers=6, try again later.
>>> Monday, November 16, 2015 3:22:54 PM WARN null 
>>> DistributedUpdateProcessor Error sending update to 
>>> http://172.18.111.112:8983/solr Monday, November 16, 2015 3:22:54 PM 
>>> WARN null DistributedUpdateProcessor Error sending update to 
>>> http://172.18.111.112:8983/solr Monday, November 16, 2015 3:22:54 PM 
>>> WARN null DistributedUpdateProcessor Error sending update to 
>>> http://172.18.111.112:8983/solr Monday, November 16, 2015 3:22:54 PM 
>>> WARN null DistributedUpdateProcessor Error sending update to 
>>> http://172.18.111.112:8983/solr
>>>
>>> 11/16/2015, 3:17:09 PM
>>>
>>> WARN
>>>
>>> null
>>>
>>> DistributedUpdateProcessor
>>>
>>> Error sending update to http://172.18.111.112:8983/solr
>>>
>>> 11/16/2015, 3:17:09 PM
>>>
>>> WARN
>>>
>>> null
>>>
>>> DistributedUpdateProcessor
>>>
>>> Error sending update to http://172.18.111.112:8983/solr
>>>
>>> 11/16/2015, 3:22:26 PM
>>>
>>> ERROR
>>>
>>> null
>>>
>>> SolrCmdDistributor
>>>
>>> org.apache.solr.client.solrj.SolrServerException: Timeout occured 
>>> while waiting response from server at:
>>> http://172.18.111.112:8983/solr/sitecore_master_index_shard1_replica
>>> 1
>>>
>>>
>>>
>>> Main errors are Timeout occurred exceptions, maxWarmingSearchers exceeded. Is anyone able to advise or have experienced something the same as the above in their SolrCloud setup?
>>>
>>> Regards,
>>> Adrian
>>>
>>>
>>>
>>
>> --
>> Monitoring * Alerting * Anomaly Detection * Centralized Log 
>> Management Solr & Elasticsearch Support * http://sematext.com/
>>

Re: Solr Cloud 5.3.0 Errors in Logs

Posted by Erick Erickson <er...@gmail.com>.
Here's perhaps more than you really want to know about commits....

https://lucidworks.com/blog/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/

The short form is that most setups set autoCommit to a relatively
short interval (15-60 seconds at least under heavy indexing loads,
I've seen 10-15 minutes under relatively light loads) with
openSearcher set to false. And I rarely set maxDocs in that
configuration, it's actually not that useful IMO.

Then set up an autoSoftCommit to be as long as you can tolerate, but
IMO rarely shorter than 60 seconds unless you have very aggressive
near-real-time (NRT) requirements.

And if your product manager simply insists on very aggressive NRT
settings, you should consider making your filterCache and
queryResultCache relatively small with minimal autowarming.

Any time you exceed maxWarmingSearchers, it indicates poorly
configured Solr instances and/or commits happening far too often.
Bumping that up to numbers greater than two is almost always a
band-aid over that misconfiguration.

On Mon, Nov 16, 2015 at 3:45 PM, Adrian Liew <ad...@avanade.com> wrote:
> Thanks Eric.
>
> Here is my reply
>
>>> 1> that commits are happening too often, and especially if the commits
>>> are happening
>>> from a client. If commits aren't being sent by a client, then look at autoCommit and softAutoCommit in solrconfig.xml (if you can).
> Understand what you mean. Besides talking to to the folks at Sitecore on where they issue commits, is there a way I can balance these with autoCommit and softAutoCommit in solrconfig.xml? Best is, can you recommend any articles that talk about best practice configuration for a production setup.
>
>>> 2> excessive autowarm settings, again in solronfig.xml.
>>> If, as you say all of Solr is a black box, then talk to the Sitecore folks, on the surface Solr is just poorly configured.
> I will raise with Sitecore guys on this. Particularly asking them why commits are happening faster than searchers can be opened. I have seen overlappingDeckSearchers have exceeded the limit errors as well. Let you know what I get back from the guys.
>
> Regards,
> Adrian
>
> -----Original Message-----
> From: Erick Erickson [mailto:erickerickson@gmail.com]
> Sent: Tuesday, November 17, 2015 12:44 AM
> To: solr-user <so...@lucene.apache.org>
> Subject: Re: Solr Cloud 5.3.0 Errors in Logs
>
> Having 6 warming serachers is an anti-pattern. What it means is that commits are happening faster than your searcher can be opened. There is _no_ good reason that I know of for changing it from 2, having changed it in solrconfig.xml to 6 almost always indicates an improper configuration.
>
> Places to look:
> 1> that commits are happening too often, and especially if the commits
> are happening
> from a client. If commits aren't being sent by a client, then look at autoCommit and softAutoCommit in solrconfig.xml (if you can).
>
> 2> excessive autowarm settings, again in solronfig.xml.
>
> If, as you say all of Solr is a black box, then talk to the Sitecore folks, on the surface Solr is just poorly configured.
>
> Best,
> Erick
>
> On Mon, Nov 16, 2015 at 4:33 AM, Adrian Liew <ad...@avanade.com> wrote:
>> Hi Emir,
>>
>> I am working with a third party platform, Sitecore. The product is a black box that encapsulates the internal workings of solr queries and so on. If there are any questions you have with regards with the below, let me know. It will be useful for me to communicate what could cause the issues below.
>>
>> Regards,
>> Adrian
>>
>> -----Original Message-----
>> From: Emir Arnautovic [mailto:emir.arnautovic@sematext.com]
>> Sent: Monday, November 16, 2015 4:47 PM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Solr Cloud 5.3.0 Errors in Logs
>>
>> Hi Adrian,
>> Can you give us bit more details about warmup queries you use and test that you are running when error occurs.
>>
>> Thanks,
>> Emir
>>
>> On 16.11.2015 08:40, Adrian Liew wrote:
>>> Hi there,
>>>
>>> Will like to get some opinions on the errors encountered below. I have currently setup a SolrCloud cluster of 3 servers (each server hosting a Solr instance and a Zookeeper instance).
>>>
>>> I am encountering the errors below in the logs:
>>> Monday, November 16, 2015 3:22:54 PM ERROR null SolrCore org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=6, try again later.
>>> Monday, November 16, 2015 3:22:54 PM ERROR null SolrCore org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=6, try again later.
>>> Monday, November 16, 2015 3:22:54 PM ERROR null SolrCore org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=6, try again later.
>>> Monday, November 16, 2015 3:22:54 PM ERROR null SolrCore org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=6, try again later.
>>> Monday, November 16, 2015 3:22:54 PM ERROR null SolrCmdDistributor org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://172.18.111.112:8983/solr/sitecore_master_index_shard1_replica1: Error opening new searcher. exceeded limit of maxWarmingSearchers=6, try again later.
>>> Monday, November 16, 2015 3:22:54 PM ERROR null SolrCmdDistributor org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://172.18.111.112:8983/solr/sitecore_master_index_shard1_replica1: Error opening new searcher. exceeded limit of maxWarmingSearchers=6, try again later.
>>> Monday, November 16, 2015 3:22:54 PM WARN null
>>> DistributedUpdateProcessor Error sending update to
>>> http://172.18.111.112:8983/solr Monday, November 16, 2015 3:22:54 PM
>>> WARN null DistributedUpdateProcessor Error sending update to
>>> http://172.18.111.112:8983/solr Monday, November 16, 2015 3:22:54 PM
>>> WARN null DistributedUpdateProcessor Error sending update to
>>> http://172.18.111.112:8983/solr Monday, November 16, 2015 3:22:54 PM
>>> WARN null DistributedUpdateProcessor Error sending update to
>>> http://172.18.111.112:8983/solr
>>>
>>> 11/16/2015, 3:17:09 PM
>>>
>>> WARN
>>>
>>> null
>>>
>>> DistributedUpdateProcessor
>>>
>>> Error sending update to http://172.18.111.112:8983/solr
>>>
>>> 11/16/2015, 3:17:09 PM
>>>
>>> WARN
>>>
>>> null
>>>
>>> DistributedUpdateProcessor
>>>
>>> Error sending update to http://172.18.111.112:8983/solr
>>>
>>> 11/16/2015, 3:22:26 PM
>>>
>>> ERROR
>>>
>>> null
>>>
>>> SolrCmdDistributor
>>>
>>> org.apache.solr.client.solrj.SolrServerException: Timeout occured
>>> while waiting response from server at:
>>> http://172.18.111.112:8983/solr/sitecore_master_index_shard1_replica1
>>>
>>>
>>>
>>> Main errors are Timeout occurred exceptions, maxWarmingSearchers exceeded. Is anyone able to advise or have experienced something the same as the above in their SolrCloud setup?
>>>
>>> Regards,
>>> Adrian
>>>
>>>
>>>
>>
>> --
>> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>> Solr & Elasticsearch Support * http://sematext.com/
>>

RE: Solr Cloud 5.3.0 Errors in Logs

Posted by Adrian Liew <ad...@avanade.com>.
Thanks Eric.

Here is my reply

>> 1> that commits are happening too often, and especially if the commits
>> are happening
>> from a client. If commits aren't being sent by a client, then look at autoCommit and softAutoCommit in solrconfig.xml (if you can).
Understand what you mean. Besides talking to to the folks at Sitecore on where they issue commits, is there a way I can balance these with autoCommit and softAutoCommit in solrconfig.xml? Best is, can you recommend any articles that talk about best practice configuration for a production setup.

>> 2> excessive autowarm settings, again in solronfig.xml.
>> If, as you say all of Solr is a black box, then talk to the Sitecore folks, on the surface Solr is just poorly configured.
I will raise with Sitecore guys on this. Particularly asking them why commits are happening faster than searchers can be opened. I have seen overlappingDeckSearchers have exceeded the limit errors as well. Let you know what I get back from the guys.

Regards,
Adrian

-----Original Message-----
From: Erick Erickson [mailto:erickerickson@gmail.com] 
Sent: Tuesday, November 17, 2015 12:44 AM
To: solr-user <so...@lucene.apache.org>
Subject: Re: Solr Cloud 5.3.0 Errors in Logs

Having 6 warming serachers is an anti-pattern. What it means is that commits are happening faster than your searcher can be opened. There is _no_ good reason that I know of for changing it from 2, having changed it in solrconfig.xml to 6 almost always indicates an improper configuration.

Places to look:
1> that commits are happening too often, and especially if the commits
are happening
from a client. If commits aren't being sent by a client, then look at autoCommit and softAutoCommit in solrconfig.xml (if you can).

2> excessive autowarm settings, again in solronfig.xml.

If, as you say all of Solr is a black box, then talk to the Sitecore folks, on the surface Solr is just poorly configured.

Best,
Erick

On Mon, Nov 16, 2015 at 4:33 AM, Adrian Liew <ad...@avanade.com> wrote:
> Hi Emir,
>
> I am working with a third party platform, Sitecore. The product is a black box that encapsulates the internal workings of solr queries and so on. If there are any questions you have with regards with the below, let me know. It will be useful for me to communicate what could cause the issues below.
>
> Regards,
> Adrian
>
> -----Original Message-----
> From: Emir Arnautovic [mailto:emir.arnautovic@sematext.com]
> Sent: Monday, November 16, 2015 4:47 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr Cloud 5.3.0 Errors in Logs
>
> Hi Adrian,
> Can you give us bit more details about warmup queries you use and test that you are running when error occurs.
>
> Thanks,
> Emir
>
> On 16.11.2015 08:40, Adrian Liew wrote:
>> Hi there,
>>
>> Will like to get some opinions on the errors encountered below. I have currently setup a SolrCloud cluster of 3 servers (each server hosting a Solr instance and a Zookeeper instance).
>>
>> I am encountering the errors below in the logs:
>> Monday, November 16, 2015 3:22:54 PM ERROR null SolrCore org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=6, try again later.
>> Monday, November 16, 2015 3:22:54 PM ERROR null SolrCore org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=6, try again later.
>> Monday, November 16, 2015 3:22:54 PM ERROR null SolrCore org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=6, try again later.
>> Monday, November 16, 2015 3:22:54 PM ERROR null SolrCore org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=6, try again later.
>> Monday, November 16, 2015 3:22:54 PM ERROR null SolrCmdDistributor org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://172.18.111.112:8983/solr/sitecore_master_index_shard1_replica1: Error opening new searcher. exceeded limit of maxWarmingSearchers=6, try again later.
>> Monday, November 16, 2015 3:22:54 PM ERROR null SolrCmdDistributor org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://172.18.111.112:8983/solr/sitecore_master_index_shard1_replica1: Error opening new searcher. exceeded limit of maxWarmingSearchers=6, try again later.
>> Monday, November 16, 2015 3:22:54 PM WARN null 
>> DistributedUpdateProcessor Error sending update to 
>> http://172.18.111.112:8983/solr Monday, November 16, 2015 3:22:54 PM 
>> WARN null DistributedUpdateProcessor Error sending update to 
>> http://172.18.111.112:8983/solr Monday, November 16, 2015 3:22:54 PM 
>> WARN null DistributedUpdateProcessor Error sending update to 
>> http://172.18.111.112:8983/solr Monday, November 16, 2015 3:22:54 PM 
>> WARN null DistributedUpdateProcessor Error sending update to 
>> http://172.18.111.112:8983/solr
>>
>> 11/16/2015, 3:17:09 PM
>>
>> WARN
>>
>> null
>>
>> DistributedUpdateProcessor
>>
>> Error sending update to http://172.18.111.112:8983/solr
>>
>> 11/16/2015, 3:17:09 PM
>>
>> WARN
>>
>> null
>>
>> DistributedUpdateProcessor
>>
>> Error sending update to http://172.18.111.112:8983/solr
>>
>> 11/16/2015, 3:22:26 PM
>>
>> ERROR
>>
>> null
>>
>> SolrCmdDistributor
>>
>> org.apache.solr.client.solrj.SolrServerException: Timeout occured 
>> while waiting response from server at:
>> http://172.18.111.112:8983/solr/sitecore_master_index_shard1_replica1
>>
>>
>>
>> Main errors are Timeout occurred exceptions, maxWarmingSearchers exceeded. Is anyone able to advise or have experienced something the same as the above in their SolrCloud setup?
>>
>> Regards,
>> Adrian
>>
>>
>>
>
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management 
> Solr & Elasticsearch Support * http://sematext.com/
>

Re: Solr Cloud 5.3.0 Errors in Logs

Posted by Erick Erickson <er...@gmail.com>.
Having 6 warming serachers is an anti-pattern. What it means is that commits
are happening faster than your searcher can be opened. There is _no_ good
reason that I know of for changing it from 2, having changed it in
solrconfig.xml
to 6 almost always indicates an improper configuration.

Places to look:
1> that commits are happening too often, and especially if the commits
are happening
from a client. If commits aren't being sent by a client, then look at
autoCommit and
softAutoCommit in solrconfig.xml (if you can).

2> excessive autowarm settings, again in solronfig.xml.

If, as you say all of Solr is a black box, then talk to the Sitecore
folks, on the surface
Solr is just poorly configured.

Best,
Erick

On Mon, Nov 16, 2015 at 4:33 AM, Adrian Liew <ad...@avanade.com> wrote:
> Hi Emir,
>
> I am working with a third party platform, Sitecore. The product is a black box that encapsulates the internal workings of solr queries and so on. If there are any questions you have with regards with the below, let me know. It will be useful for me to communicate what could cause the issues below.
>
> Regards,
> Adrian
>
> -----Original Message-----
> From: Emir Arnautovic [mailto:emir.arnautovic@sematext.com]
> Sent: Monday, November 16, 2015 4:47 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr Cloud 5.3.0 Errors in Logs
>
> Hi Adrian,
> Can you give us bit more details about warmup queries you use and test that you are running when error occurs.
>
> Thanks,
> Emir
>
> On 16.11.2015 08:40, Adrian Liew wrote:
>> Hi there,
>>
>> Will like to get some opinions on the errors encountered below. I have currently setup a SolrCloud cluster of 3 servers (each server hosting a Solr instance and a Zookeeper instance).
>>
>> I am encountering the errors below in the logs:
>> Monday, November 16, 2015 3:22:54 PM ERROR null SolrCore org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=6, try again later.
>> Monday, November 16, 2015 3:22:54 PM ERROR null SolrCore org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=6, try again later.
>> Monday, November 16, 2015 3:22:54 PM ERROR null SolrCore org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=6, try again later.
>> Monday, November 16, 2015 3:22:54 PM ERROR null SolrCore org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=6, try again later.
>> Monday, November 16, 2015 3:22:54 PM ERROR null SolrCmdDistributor org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://172.18.111.112:8983/solr/sitecore_master_index_shard1_replica1: Error opening new searcher. exceeded limit of maxWarmingSearchers=6, try again later.
>> Monday, November 16, 2015 3:22:54 PM ERROR null SolrCmdDistributor org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://172.18.111.112:8983/solr/sitecore_master_index_shard1_replica1: Error opening new searcher. exceeded limit of maxWarmingSearchers=6, try again later.
>> Monday, November 16, 2015 3:22:54 PM WARN null
>> DistributedUpdateProcessor Error sending update to
>> http://172.18.111.112:8983/solr Monday, November 16, 2015 3:22:54 PM
>> WARN null DistributedUpdateProcessor Error sending update to
>> http://172.18.111.112:8983/solr Monday, November 16, 2015 3:22:54 PM
>> WARN null DistributedUpdateProcessor Error sending update to
>> http://172.18.111.112:8983/solr Monday, November 16, 2015 3:22:54 PM
>> WARN null DistributedUpdateProcessor Error sending update to
>> http://172.18.111.112:8983/solr
>>
>> 11/16/2015, 3:17:09 PM
>>
>> WARN
>>
>> null
>>
>> DistributedUpdateProcessor
>>
>> Error sending update to http://172.18.111.112:8983/solr
>>
>> 11/16/2015, 3:17:09 PM
>>
>> WARN
>>
>> null
>>
>> DistributedUpdateProcessor
>>
>> Error sending update to http://172.18.111.112:8983/solr
>>
>> 11/16/2015, 3:22:26 PM
>>
>> ERROR
>>
>> null
>>
>> SolrCmdDistributor
>>
>> org.apache.solr.client.solrj.SolrServerException: Timeout occured
>> while waiting response from server at:
>> http://172.18.111.112:8983/solr/sitecore_master_index_shard1_replica1
>>
>>
>>
>> Main errors are Timeout occurred exceptions, maxWarmingSearchers exceeded. Is anyone able to advise or have experienced something the same as the above in their SolrCloud setup?
>>
>> Regards,
>> Adrian
>>
>>
>>
>
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/
>

RE: Solr Cloud 5.3.0 Errors in Logs

Posted by Adrian Liew <ad...@avanade.com>.
Hi Emir,

I am working with a third party platform, Sitecore. The product is a black box that encapsulates the internal workings of solr queries and so on. If there are any questions you have with regards with the below, let me know. It will be useful for me to communicate what could cause the issues below.

Regards,
Adrian

-----Original Message-----
From: Emir Arnautovic [mailto:emir.arnautovic@sematext.com] 
Sent: Monday, November 16, 2015 4:47 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr Cloud 5.3.0 Errors in Logs

Hi Adrian,
Can you give us bit more details about warmup queries you use and test that you are running when error occurs.

Thanks,
Emir

On 16.11.2015 08:40, Adrian Liew wrote:
> Hi there,
>
> Will like to get some opinions on the errors encountered below. I have currently setup a SolrCloud cluster of 3 servers (each server hosting a Solr instance and a Zookeeper instance).
>
> I am encountering the errors below in the logs:
> Monday, November 16, 2015 3:22:54 PM ERROR null SolrCore org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=6,​ try again later.
> Monday, November 16, 2015 3:22:54 PM ERROR null SolrCore org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=6,​ try again later.
> Monday, November 16, 2015 3:22:54 PM ERROR null SolrCore org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=6,​ try again later.
> Monday, November 16, 2015 3:22:54 PM ERROR null SolrCore org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=6,​ try again later.
> Monday, November 16, 2015 3:22:54 PM ERROR null SolrCmdDistributor org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://172.18.111.112:8983/solr/sitecore_master_index_shard1_replica1: Error opening new searcher. exceeded limit of maxWarmingSearchers=6,​ try again later.
> Monday, November 16, 2015 3:22:54 PM ERROR null SolrCmdDistributor org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://172.18.111.112:8983/solr/sitecore_master_index_shard1_replica1: Error opening new searcher. exceeded limit of maxWarmingSearchers=6,​ try again later.
> Monday, November 16, 2015 3:22:54 PM WARN null 
> DistributedUpdateProcessor Error sending update to 
> http://172.18.111.112:8983/solr Monday, November 16, 2015 3:22:54 PM 
> WARN null DistributedUpdateProcessor Error sending update to 
> http://172.18.111.112:8983/solr Monday, November 16, 2015 3:22:54 PM 
> WARN null DistributedUpdateProcessor Error sending update to 
> http://172.18.111.112:8983/solr Monday, November 16, 2015 3:22:54 PM 
> WARN null DistributedUpdateProcessor Error sending update to 
> http://172.18.111.112:8983/solr
>
> 11/16/2015, 3:17:09 PM
>
> WARN
>
> null
>
> DistributedUpdateProcessor
>
> Error sending update to http://172.18.111.112:8983/solr
>
> 11/16/2015, 3:17:09 PM
>
> WARN
>
> null
>
> DistributedUpdateProcessor
>
> Error sending update to http://172.18.111.112:8983/solr
>
> 11/16/2015, 3:22:26 PM
>
> ERROR
>
> null
>
> SolrCmdDistributor
>
> org.apache.solr.client.solrj.SolrServerException: Timeout occured 
> while waiting response from server at: 
> http://172.18.111.112:8983/solr/sitecore_master_index_shard1_replica1
>
>
>
> Main errors are Timeout occurred exceptions, maxWarmingSearchers exceeded. Is anyone able to advise or have experienced something the same as the above in their SolrCloud setup?
>
> Regards,
> Adrian
>
>
>

--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/

Re: Solr Cloud 5.3.0 Errors in Logs

Posted by Emir Arnautovic <em...@sematext.com>.
Hi Adrian,
Can you give us bit more details about warmup queries you use and test 
that you are running when error occurs.

Thanks,
Emir

On 16.11.2015 08:40, Adrian Liew wrote:
> Hi there,
>
> Will like to get some opinions on the errors encountered below. I have currently setup a SolrCloud cluster of 3 servers (each server hosting a Solr instance and a Zookeeper instance).
>
> I am encountering the errors below in the logs:
> Monday, November 16, 2015 3:22:54 PM ERROR null SolrCore org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=6,​ try again later.
> Monday, November 16, 2015 3:22:54 PM ERROR null SolrCore org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=6,​ try again later.
> Monday, November 16, 2015 3:22:54 PM ERROR null SolrCore org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=6,​ try again later.
> Monday, November 16, 2015 3:22:54 PM ERROR null SolrCore org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=6,​ try again later.
> Monday, November 16, 2015 3:22:54 PM ERROR null SolrCmdDistributor org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://172.18.111.112:8983/solr/sitecore_master_index_shard1_replica1: Error opening new searcher. exceeded limit of maxWarmingSearchers=6,​ try again later.
> Monday, November 16, 2015 3:22:54 PM ERROR null SolrCmdDistributor org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://172.18.111.112:8983/solr/sitecore_master_index_shard1_replica1: Error opening new searcher. exceeded limit of maxWarmingSearchers=6,​ try again later.
> Monday, November 16, 2015 3:22:54 PM WARN null DistributedUpdateProcessor Error sending update to http://172.18.111.112:8983/solr
> Monday, November 16, 2015 3:22:54 PM WARN null DistributedUpdateProcessor Error sending update to http://172.18.111.112:8983/solr
> Monday, November 16, 2015 3:22:54 PM WARN null DistributedUpdateProcessor Error sending update to http://172.18.111.112:8983/solr
> Monday, November 16, 2015 3:22:54 PM WARN null DistributedUpdateProcessor Error sending update to http://172.18.111.112:8983/solr
>
> 11/16/2015, 3:17:09 PM
>
> WARN
>
> null
>
> DistributedUpdateProcessor
>
> Error sending update to http://172.18.111.112:8983/solr
>
> 11/16/2015, 3:17:09 PM
>
> WARN
>
> null
>
> DistributedUpdateProcessor
>
> Error sending update to http://172.18.111.112:8983/solr
>
> 11/16/2015, 3:22:26 PM
>
> ERROR
>
> null
>
> SolrCmdDistributor
>
> org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://172.18.111.112:8983/solr/sitecore_master_index_shard1_replica1
>
>
>
> Main errors are Timeout occurred exceptions, maxWarmingSearchers exceeded. Is anyone able to advise or have experienced something the same as the above in their SolrCloud setup?
>
> Regards,
> Adrian
>
>
>

-- 
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/