You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Amit L <am...@gmail.com> on 2015/04/27 22:25:31 UTC

SolrCloud Replication Issue

Hi,

A few days ago I deployed a solr 4.9.0 cluster, which consists of 2
collections. Each collection has 1 shard with 3 replicates on 3 different
machines.

On the first day I noticed this error appear on the leader. Full Log -
http://pastebin.com/wcPMZb0s

4/23/2015, 2:34:37 PM SEVERE SolrCmdDistributor
org.apache.solr.client.solrj.SolrServerException: IOException occured when
talking to server at:
http://production-solrcloud-004:8080/solr/bookings_shard1_replica2

4/23/2015, 2:34:37 PM WARNING DistributedUpdateProcessor
Error sending update

4/23/2015, 2:34:37 PM WARNING ZkController
Leader is publishing core=bookings_shard1_replica2 state=down on behalf of
un-reachable replica
http://production-solrcloud-004:8080/solr/bookings_shard1_replica2/;
forcePublishState? false


The other 2 replicas had 0 errors.

I thought it may be a one off but the same error occured on day 2 which has
got me slighlty concerned. During these periods I didn't notice any issues
with the cluster and everything looks healthy in the cloud summary. All of
the instances are hosted on AWS.

Any idea what may be causing this issue and what I can do to mitigate?

Thanks
Amit

Re: SolrCloud Replication Issue

Posted by Erick Erickson <er...@gmail.com>.
Amit:

The fact that "all instances are using no more than 30%...." isn't
really indicative of whether or not GC pauses are a problem. If you
have a large heap allocated to Java, then the to-be-collected objects
will build up and _eventually_ you'll have a stop-the-world GC pause
even though each time you happen to look you may be < 30%. Guessing
here...

So how much memory are you allocating to the JVM? You can have GC
stats printed to the log, see:
http://lucidworks.com/blog/garbage-collection-bootcamp-1-0/

does the log of the instance that can't be reached tell you anything?

Best,
Erick

On Mon, Apr 27, 2015 at 1:55 PM, Amit L <am...@gmail.com> wrote:
> Appreciate the response, to answer your questions.
>
> * Do you see this happen often? How often?
> It has happened twice in five days. The first two days after deployment.
>
> * Are there any known network issues?
> There are no obvious network issues but as these instances reside in AWS i
> cannot rule it out network blips.
>
> * Do you have any idea about the GC on those replicas?
> I have been monitoring the memory usage and all instances are using no more
> than 30% of its JVM memory allocation.
>
>
>
>
> On 27 April 2015 at 21:36, Anshum Gupta <an...@anshumgupta.net> wrote:
>
>> Looks like LeaderInitiatedRecovery or LIR. When a leader receives a
>> document (update) but fails to successfully forward it to a replica, it
>> marks that replica as down and asks the replica to recover (hence the name,
>> Leader Initiated Recovery). It could be due to multiple reasons e.g.
>> network issue/GC. The replica generally comes back up and syncs with the
>> leader transparently. As an end-user, you don't have to really worry much
>> about this but if you want to dig deeper, here are a few questions that
>> might help us in suggesting what to do/look at.
>> * Do you see this happen often? How often?
>> * Are there any known network issues?
>> * Do you have any idea about the GC on those replicas?
>>
>>
>> On Mon, Apr 27, 2015 at 1:25 PM, Amit L <am...@gmail.com> wrote:
>>
>> > Hi,
>> >
>> > A few days ago I deployed a solr 4.9.0 cluster, which consists of 2
>> > collections. Each collection has 1 shard with 3 replicates on 3 different
>> > machines.
>> >
>> > On the first day I noticed this error appear on the leader. Full Log -
>> > http://pastebin.com/wcPMZb0s
>> >
>> > 4/23/2015, 2:34:37 PM SEVERE SolrCmdDistributor
>> > org.apache.solr.client.solrj.SolrServerException: IOException occured
>> when
>> > talking to server at:
>> > http://production-solrcloud-004:8080/solr/bookings_shard1_replica2
>> >
>> > 4/23/2015, 2:34:37 PM WARNING DistributedUpdateProcessor
>> > Error sending update
>> >
>> > 4/23/2015, 2:34:37 PM WARNING ZkController
>> > Leader is publishing core=bookings_shard1_replica2 state=down on behalf
>> of
>> > un-reachable replica
>> > http://production-solrcloud-004:8080/solr/bookings_shard1_replica2/;
>> > forcePublishState? false
>> >
>> >
>> > The other 2 replicas had 0 errors.
>> >
>> > I thought it may be a one off but the same error occured on day 2 which
>> has
>> > got me slighlty concerned. During these periods I didn't notice any
>> issues
>> > with the cluster and everything looks healthy in the cloud summary. All
>> of
>> > the instances are hosted on AWS.
>> >
>> > Any idea what may be causing this issue and what I can do to mitigate?
>> >
>> > Thanks
>> > Amit
>> >
>>
>>
>>
>> --
>> Anshum Gupta
>>

Re: SolrCloud Replication Issue

Posted by Amit L <am...@gmail.com>.
Appreciate the response, to answer your questions.

* Do you see this happen often? How often?
It has happened twice in five days. The first two days after deployment.

* Are there any known network issues?
There are no obvious network issues but as these instances reside in AWS i
cannot rule it out network blips.

* Do you have any idea about the GC on those replicas?
I have been monitoring the memory usage and all instances are using no more
than 30% of its JVM memory allocation.




On 27 April 2015 at 21:36, Anshum Gupta <an...@anshumgupta.net> wrote:

> Looks like LeaderInitiatedRecovery or LIR. When a leader receives a
> document (update) but fails to successfully forward it to a replica, it
> marks that replica as down and asks the replica to recover (hence the name,
> Leader Initiated Recovery). It could be due to multiple reasons e.g.
> network issue/GC. The replica generally comes back up and syncs with the
> leader transparently. As an end-user, you don't have to really worry much
> about this but if you want to dig deeper, here are a few questions that
> might help us in suggesting what to do/look at.
> * Do you see this happen often? How often?
> * Are there any known network issues?
> * Do you have any idea about the GC on those replicas?
>
>
> On Mon, Apr 27, 2015 at 1:25 PM, Amit L <am...@gmail.com> wrote:
>
> > Hi,
> >
> > A few days ago I deployed a solr 4.9.0 cluster, which consists of 2
> > collections. Each collection has 1 shard with 3 replicates on 3 different
> > machines.
> >
> > On the first day I noticed this error appear on the leader. Full Log -
> > http://pastebin.com/wcPMZb0s
> >
> > 4/23/2015, 2:34:37 PM SEVERE SolrCmdDistributor
> > org.apache.solr.client.solrj.SolrServerException: IOException occured
> when
> > talking to server at:
> > http://production-solrcloud-004:8080/solr/bookings_shard1_replica2
> >
> > 4/23/2015, 2:34:37 PM WARNING DistributedUpdateProcessor
> > Error sending update
> >
> > 4/23/2015, 2:34:37 PM WARNING ZkController
> > Leader is publishing core=bookings_shard1_replica2 state=down on behalf
> of
> > un-reachable replica
> > http://production-solrcloud-004:8080/solr/bookings_shard1_replica2/;
> > forcePublishState? false
> >
> >
> > The other 2 replicas had 0 errors.
> >
> > I thought it may be a one off but the same error occured on day 2 which
> has
> > got me slighlty concerned. During these periods I didn't notice any
> issues
> > with the cluster and everything looks healthy in the cloud summary. All
> of
> > the instances are hosted on AWS.
> >
> > Any idea what may be causing this issue and what I can do to mitigate?
> >
> > Thanks
> > Amit
> >
>
>
>
> --
> Anshum Gupta
>

Re: SolrCloud Replication Issue

Posted by Anshum Gupta <an...@anshumgupta.net>.
Looks like LeaderInitiatedRecovery or LIR. When a leader receives a
document (update) but fails to successfully forward it to a replica, it
marks that replica as down and asks the replica to recover (hence the name,
Leader Initiated Recovery). It could be due to multiple reasons e.g.
network issue/GC. The replica generally comes back up and syncs with the
leader transparently. As an end-user, you don't have to really worry much
about this but if you want to dig deeper, here are a few questions that
might help us in suggesting what to do/look at.
* Do you see this happen often? How often?
* Are there any known network issues?
* Do you have any idea about the GC on those replicas?


On Mon, Apr 27, 2015 at 1:25 PM, Amit L <am...@gmail.com> wrote:

> Hi,
>
> A few days ago I deployed a solr 4.9.0 cluster, which consists of 2
> collections. Each collection has 1 shard with 3 replicates on 3 different
> machines.
>
> On the first day I noticed this error appear on the leader. Full Log -
> http://pastebin.com/wcPMZb0s
>
> 4/23/2015, 2:34:37 PM SEVERE SolrCmdDistributor
> org.apache.solr.client.solrj.SolrServerException: IOException occured when
> talking to server at:
> http://production-solrcloud-004:8080/solr/bookings_shard1_replica2
>
> 4/23/2015, 2:34:37 PM WARNING DistributedUpdateProcessor
> Error sending update
>
> 4/23/2015, 2:34:37 PM WARNING ZkController
> Leader is publishing core=bookings_shard1_replica2 state=down on behalf of
> un-reachable replica
> http://production-solrcloud-004:8080/solr/bookings_shard1_replica2/;
> forcePublishState? false
>
>
> The other 2 replicas had 0 errors.
>
> I thought it may be a one off but the same error occured on day 2 which has
> got me slighlty concerned. During these periods I didn't notice any issues
> with the cluster and everything looks healthy in the cloud summary. All of
> the instances are hosted on AWS.
>
> Any idea what may be causing this issue and what I can do to mitigate?
>
> Thanks
> Amit
>



-- 
Anshum Gupta