You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "ASF subversion and git services (JIRA)" <ji...@apache.org> on 2018/04/19 08:01:00 UTC
[jira] [Commented] (SOLR-12087) Deleting replicas sometimes fails
and causes the replicas to exist in the down state
[ https://issues.apache.org/jira/browse/SOLR-12087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16443682#comment-16443682 ]
ASF subversion and git services commented on SOLR-12087:
--------------------------------------------------------
Commit ec9ccb5cd07f54eacba2cf071281cc9c37b766c1 in lucene-solr's branch refs/heads/branch_7_3 from [~caomanhdat]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ec9ccb5 ]
SOLR-12087: Deleting replicas sometimes fails and causes the replicas to exist in the down state
> Deleting replicas sometimes fails and causes the replicas to exist in the down state
> ------------------------------------------------------------------------------------
>
> Key: SOLR-12087
> URL: https://issues.apache.org/jira/browse/SOLR-12087
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
> Affects Versions: 7.2
> Reporter: Jerry Bao
> Assignee: Cao Manh Dat
> Priority: Critical
> Fix For: 7.4
>
> Attachments: SOLR-12087.patch, SOLR-12087.patch, SOLR-12087.patch, SOLR-12087.test.patch, Screen Shot 2018-03-16 at 11.50.32 AM.png
>
>
> Sometimes when deleting replicas, the replica fails to be removed from the cluster state. This occurs especially when deleting replicas en mass; the resulting cause is that the data is deleted but the replicas aren't removed from the cluster state. Attempting to delete the downed replicas causes failures because the core does not exist anymore.
> This also occurs when trying to move replicas, since that move is an add and delete.
> Some more information regarding this issue; when the MOVEREPLICA command is issued, the new replica is created successfully but the replica to be deleted fails to be removed from state.json (the core is deleted though) and we see two logs spammed.
> # The node containing the leader replica continually (read every second) attempts to initiate recovery on the replica and fails to do so because the core does not exist. As a result it continually publishes a down state for the replica to zookeeper.
> # The deleted replica node spams that it cannot locate the core because it's been deleted.
> During this period of time, we see an increase in ZK network connectivity overall, until the replica is finally deleted (spamming DELETEREPLICA on the shard until its removed from the state)
> My guess is there's two issues at hand here:
> # The leader continually attempts to recover a downed replica that is unrecoverable because the core does not exist.
> # The replica to be deleted is having trouble being deleted from state.json in ZK.
> This is mostly consistent for my use case. I'm running 7.2.1 with 66 nodes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org