You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2016/12/11 07:04:58 UTC
[jira] [Commented] (KAFKA-4521) MirrorMaker should flush all
messages before releasing partition ownership during rebalance
[ https://issues.apache.org/jira/browse/KAFKA-4521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15739243#comment-15739243 ]
ASF GitHub Bot commented on KAFKA-4521:
---------------------------------------
GitHub user lindong28 opened a pull request:
https://github.com/apache/kafka/pull/2241
KAFKA-4521; MirrorMaker should flush all messages before releasing partition ownership during rebalance
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/lindong28/kafka KAFKA-4521
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/2241.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2241
----
commit cdbd79189f1cdb53f93d014c9b9758a5941b8168
Author: Dong Lin <do...@linkedin.com>
Date: 2016-12-11T07:03:46Z
KAFKA-4521; MirrorMaker should flush all messages before releasing partition ownership during rebalance
----
> MirrorMaker should flush all messages before releasing partition ownership during rebalance
> -------------------------------------------------------------------------------------------
>
> Key: KAFKA-4521
> URL: https://issues.apache.org/jira/browse/KAFKA-4521
> Project: Kafka
> Issue Type: Bug
> Reporter: Dong Lin
> Assignee: Dong Lin
>
> In order to ensure that messages from a given partition in the source cluster are mirrored to the same partition in the destination cluster in the *same* order, MirrorMaker needs to produce and flush all messages that its consumer has received from source cluster before giving up partition in the cluster.
> However, as of current implementation of Apache Kafka, this is not guaranteed and will cause out-of-order message delivery in the following scenario.
> - mirror maker process 1 fetches messages 1, 2, 3, from source cluster in one FetchRequest
> - mirror maker process 2 starts up and triggers rebalance.
> - `InternalRebalanceListenerForOldConsumer` is executed by zookeeper listener thread, which does producer.flush() and commit offset of this consumer. However, at this moment messages 1, 2, 3 haven't even been produced.
> - consumer of mirror maker process 1 releases ownership of this partition
> - consumer of mirror maker process 2 gets ownership of this partition.
> - mirror maker process 2 fetches messages 4, 5, 6 from source cluster.
> - messages 4, 5, 6 can be produced before messages 1, 2, 3.
> To fix this problem, the rebalance listener callback function should signal MirrorMakerThread to get all messages from consumer, produce these messages to destination cluster, flush producer, and commit offset. Rebalance listener callback function should wait for MirrorMakerThread to finish these steps before it allows ownership of this partition to be released.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)