You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "ASF subversion and git services (JIRA)" <ji...@apache.org> on 2016/09/01 01:51:20 UTC
[jira] [Commented] (FLUME-2972) Handle offset migration in the new
Kafka Channel
[ https://issues.apache.org/jira/browse/FLUME-2972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15454011#comment-15454011 ]
ASF subversion and git services commented on FLUME-2972:
--------------------------------------------------------
Commit 330e572879c9903f5b1084d84b237685aa9970c6 in flume's branch refs/heads/trunk from [~granthenke]
[ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=330e572 ]
FLUME-2983. Handle offset migration in the new Kafka Source
Similar to FLUME-2972, offsets tracking the position in Kafka consumers
change from using zookeeper for offset storage to Kafka when moving from
0.8.x to 0.9.x.
FLUME-2821 makes the client change in the Kafka Source but does not
ensure existing offsets get migrated in order to continue consuming
where it left off. Flume should have some automated logic on startup to
check if Kafka offsets exist, if not and migration is enabled (by
default) then copy the offsets from Zookeeper and commit them to Kafka.
This change should also fix the backwards incompatibility caused by
removing the zookeeperConnect property. The bootstrap can be looked up
if zookeeperConnect is used.
Reviewers: Denes Arvay, Mike Percy
(Grant Henke via Mike Percy)
> Handle offset migration in the new Kafka Channel
> ------------------------------------------------
>
> Key: FLUME-2972
> URL: https://issues.apache.org/jira/browse/FLUME-2972
> Project: Flume
> Issue Type: Bug
> Components: Channel
> Affects Versions: v1.7.0
> Reporter: Grant Henke
> Assignee: Grant Henke
> Fix For: v1.7.0
>
> Attachments: FLUME-2972-1.patch, FLUME-2972-2.patch, FLUME-2972.patch
>
>
> Offsets tracking the position in Kafka consumers change from using zookeeper for offset storage to Kafka when moving from 0.8.x to 0.9.x.
> FLUME-2823 makes the client change in the Kafka Channel but does not ensure existing offsets get migrated in order to continue consuming where it left off.
> Flume should have some automated logic on startup to check if Kafka offsets exist, if not and migration is enabled (by default) then copy the offsets from Zookeeper and commit them to Kafka.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)