You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Jiangjie Qin (JIRA)" <ji...@apache.org> on 2014/12/04 21:02:13 UTC
[jira] [Comment Edited] (KAFKA-1650) Mirror Maker could lose data
on unclean shutdown.
[ https://issues.apache.org/jira/browse/KAFKA-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234557#comment-14234557 ]
Jiangjie Qin edited comment on KAFKA-1650 at 12/4/14 8:02 PM:
--------------------------------------------------------------
[~junrao] Could you also help review the patch as Joel thought it is kind of a big patch of new design for mirror maker. Thanks.
Some major changes are listed below to expedite the review:
1. ZookeeperConsumerConnector auto commit is turned off.
2. A Map<SourceTopicPartition, Map<TargetTopicPartition, Offset>> is used to keep track of the offsets of messages successfully sent to target cluster. The map is updated on new producer's callback.
3. ZookeeperConsumerConnector has been modified to take a rebalance listener and take external offset map to commit offset.
4. Mirror maker wires in a consumer rebalance listener to avoid duplicates on consumer rebalance. On consumer rebalance, before releasing the partition ownership, it will first clean the messages in data channel, wait until the messages taken by producer receives callback, commit offset and continue the rebalance.
5. An offset commit thread will be responsible for committing offset periodically.
was (Author: becket_qin):
[~junrao] Could you also help review the patch as Joel thought it is kind of a big patch of new design for mirror maker.
Some major changes are listed below to expedite the review:
1. ZookeeperConsumerConnector auto commit is turned off.
2. A Map<SourceTopicPartition, Map<TargetTopicPartition, Offset>> is used to keep track of the offsets of messages successfully sent to target cluster. The map is updated on new producer's callback.
3. ZookeeperConsumerConnector has been modified to take a rebalance listener and take external offset map to commit offset.
4. Mirror maker wires in a consumer rebalance listener to avoid duplicates on consumer rebalance. On consumer rebalance, before releasing the partition ownership, it will first clean the messages in data channel, wait until the messages taken by producer receives callback, commit offset and continue the rebalance.
5. An offset commit thread will be responsible for committing offset periodically.
> Mirror Maker could lose data on unclean shutdown.
> -------------------------------------------------
>
> Key: KAFKA-1650
> URL: https://issues.apache.org/jira/browse/KAFKA-1650
> Project: Kafka
> Issue Type: Improvement
> Reporter: Jiangjie Qin
> Assignee: Jiangjie Qin
> Attachments: KAFKA-1650.patch, KAFKA-1650_2014-10-06_10:17:46.patch, KAFKA-1650_2014-11-12_09:51:30.patch, KAFKA-1650_2014-11-17_18:44:37.patch, KAFKA-1650_2014-11-20_12:00:16.patch, KAFKA-1650_2014-11-24_08:15:17.patch, KAFKA-1650_2014-12-03_15:02:31.patch, KAFKA-1650_2014-12-03_19:02:13.patch, KAFKA-1650_2014-12-04_11:59:07.patch
>
>
> Currently if mirror maker got shutdown uncleanly, the data in the data channel and buffer could potentially be lost. With the new producer's callback, this issue could be solved.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)