[jira] [Created] (KAFKA-9961) Brokers may be left in an inconsistent state after reassignment

["Dhruvil Shah (Jira)" <ji...@apache.org>]

Dhruvil Shah created KAFKA-9961:
-----------------------------------

             Summary: Brokers may be left in an inconsistent state after reassignment
                 Key: KAFKA-9961
                 URL: https://issues.apache.org/jira/browse/KAFKA-9961
             Project: Kafka
          Issue Type: Bug
            Reporter: Dhruvil Shah


When completing a reassignment, the controller sends StopReplicaRequest to replicas that are not in the target assignment and removes them from the assignment in ZK. We do not have any retry mechanism to ensure that the broker is able to process the StopReplicaRequest successfully. Under certain circumstances, this could leave brokers in an inconsistent state, where they continue being the follower for this partition and end up with an inconsistent metadata cache.

We have seen messages like the following being spammed in the broker logs when we get into this situation:
{code:java}
While recording the replica LEO, the partition topic-1 hasn't been created.
{code}
This happens because the broker has not an updated LeaderAndIsrRequest for the new leader nor a StopReplicaRequest from the controller when the replica was removed from the assignment.

Note that we would require a restart of the affected broker to fix this situation. A controller failover would not fix it as the broker could continue being a replica for the partition until it receives a StopReplicaRequest, which would never happen in this case.

There seem to be couple of problems we should address:
 # We need a mechanism to retry replica deletions after partition reassignment is complete. The main challenge here is to be able to deal with cases where a broker has been decommissioned and may never come back up.
 # We could perhaps consider a mechanism to reconcile replica states across brokers, something similar to the solution proposed in [https://cwiki.apache.org/confluence/display/KAFKA/KIP-550%3A+Mechanism+to+Delete+Stray+Partitions+on+Broker].



--
This message was sent by Atlassian Jira
(v8.3.4#803005)