You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flume.apache.org by "Grant Henke (JIRA)" <ji...@apache.org> on 2016/08/15 21:48:22 UTC

[jira] [Created] (FLUME-2972) Handle offset migration in the new Kafka Channel

Grant Henke created FLUME-2972:
----------------------------------

             Summary: Handle offset migration in the new Kafka Channel
                 Key: FLUME-2972
                 URL: https://issues.apache.org/jira/browse/FLUME-2972
             Project: Flume
          Issue Type: Bug
          Components: Channel
    Affects Versions: v1.7.0
            Reporter: Grant Henke
            Assignee: Grant Henke
             Fix For: v1.7.0


Offsets tracking the position in Kafka consumers  change from using zookeeper for offset storage to Kafka when moving from 0.8.x to 0.9.x. 

FLUME-2823 makes the client change in the Kafka Channel but does not ensure existing offsets get migrated in order to continue consuming where it left off. 

Flume should have some automated logic on startup to check if Kafka offsets exist, if not and migration is enabled (by default) then copy the offsets from Zookeeper and commit them to Kafka. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)