[jira] [Commented] (FLINK-3294) KafkaConsumer (0.8) commit offsets using SimpleConsumer.commitOffsets()

["Robert Metzger (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-3294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15358546#comment-15358546 ] 

Robert Metzger commented on FLINK-3294:
---------------------------------------

Hi Jens,

the Kafka 0.9 consumer doesn't commit the offsets to Zookeeper. They are committed to the Kafka broker.
Only Flink's Kafka 0.8 consumer commits to ZK. That's why the commit code is different in the two implementations. The Kafka 0.9 consumer also should not have any ZK dependencies.

What you have to do is to introduce an interface "OffsetHandler" into the 0.8 fetcher that has two implementations: A Zookeeper Offset handler (the current code) and a Kafka offset handler (new code).
In the diff of the branch I posed above, you can see this structure already: https://github.com/apache/flink/compare/master...rmetzger:yangjun_fix

> KafkaConsumer (0.8) commit offsets using SimpleConsumer.commitOffsets()
> -----------------------------------------------------------------------
>
>                 Key: FLINK-3294
>                 URL: https://issues.apache.org/jira/browse/FLINK-3294
>             Project: Flink
>          Issue Type: Bug
>          Components: Kafka Connector
>            Reporter: Robert Metzger
>            Assignee: Robert Metzger
>
> Currently, the 0.8 consumer for Kafka is committing the offsets manually into Zookeeper so that users can track the lag using external tools.
> The 0.8 consumer has a pluggable design, and this component is easily pluggable.
> Since OffsetCommitRequest version=1 (supported in 0.8.2 or later), users can choose between two offset commit modes:
> a) Let the broker commit into ZK (this is  what we are doing from the consumer
> b) Let the broker commit the offset into a special topic.
> By adding a different "OffsetHandler" backend, users can commit offsets from the brokers (reducing the total number of ZK connections) or into the broker's offset topic.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)