[jira] [Commented] (KAFKA-1012) Implement an Offset Manager and hook offset requests to it

["Joe Stein (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13805368#comment-13805368 ] 

Joe Stein commented on KAFKA-1012:
----------------------------------

HI, I thought OffsetManager would abstract the storage as a trait or something like Sriram was saying.

+      try {
+        debug("Sending offset commit request.")
+        offsetCommitProducer.send(topicRegistry.flatMap(t => t._2.values.map( partitionInfo =>
+          KeyedMessage[String, String](OffsetManager.OffsetsTopicName,
+                                       OffsetManager.offsetCommitKey(config.groupId, t._1, partitionInfo.partitionId),
+                                       config.groupId,
+                                       OffsetManager.offsetCommitValue(partitionInfo.getConsumeOffset()))
+        )).toSeq: _*)

Isn't this something that has to be plugable? It seems we just bypassed the Zookeeper interface and turned around and started writing the data to the broker.  Was the thought to build out different storage options ? which I thought was Phase 2 and Phase 1 handling the abstraction? maybe is a second option people could use along with the option of sending to zookeeper?  I was thinking maybe for that we could have some REST service option too with thrift and/or avro extensions since the abstraction is plugable people can build whatever they need.




> Implement an Offset Manager and hook offset requests to it
> ----------------------------------------------------------
>
>                 Key: KAFKA-1012
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1012
>             Project: Kafka
>          Issue Type: Sub-task
>          Components: consumer
>            Reporter: Tejas Patil
>            Assignee: Tejas Patil
>            Priority: Minor
>         Attachments: KAFKA-1012.patch, KAFKA-1012-v2.patch
>
>
> After KAFKA-657, we have a protocol for consumers to commit and fetch offsets from brokers. Currently, consumers are not using this API and directly talking with Zookeeper. 
> This Jira will involve following:
> 1. Add a special topic in kafka for storing offsets
> 2. Add an OffsetManager interface which would handle storing, accessing, loading and maintaining consumer offsets
> 3. Implement offset managers for both of these 2 choices : existing ZK based storage or inbuilt storage for offsets.
> 4. Leader brokers would now maintain an additional hash table of offsets for the group-topic-partitions that they lead
> 5. Consumers should now use the OffsetCommit and OffsetFetch API



--
This message was sent by Atlassian JIRA
(v6.1#6144)