You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Tejas Patil (JIRA)" <ji...@apache.org> on 2013/08/19 22:39:48 UTC
[jira] [Updated] (KAFKA-1012) Implement an Offset Manager and hook
offset requests to it
[ https://issues.apache.org/jira/browse/KAFKA-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tejas Patil updated KAFKA-1012:
-------------------------------
Attachment: KAFKA-1012.patch
Attached KAFKA-1012-v1.patch for trunk. An overview of the implementation can be found at [0]. Would be happy to hear any suggestions / concerns wrt the design and implementation.
Things that are planned but not included in this patch
--------------------------------------------------------------
1. Use a binary format for the offsets topic messages and incorporate versioning to support changes over time if required.
2. Cleanup of offsets table when there is a change in ownership of any partition of the offsets' topic.
3. Apart from the normal use case of clearing off the unwanted offsets from offsets table, support is needed to cleanup all entries specific to a topic or group and also based on their age. (This could be even moved to KAFKA-559)
Known limitations / caveats
---------------------------------
1. Re-assign partitions: We are assuming that there is no reassignment of the offsets' topic partitions
2. Atomicity of offset commit is not currently implemented since we are using a concurrent hashmap.
PS: If you plan to run junits, use the v1 patch in KAFKA-1014 on top of this one.
[0] : https://cwiki.apache.org/confluence/display/KAFKA/Inbuilt+Consumer+Offset+Management
> Implement an Offset Manager and hook offset requests to it
> ----------------------------------------------------------
>
> Key: KAFKA-1012
> URL: https://issues.apache.org/jira/browse/KAFKA-1012
> Project: Kafka
> Issue Type: Sub-task
> Components: consumer
> Reporter: Tejas Patil
> Assignee: Tejas Patil
> Priority: Minor
> Attachments: KAFKA-1012.patch
>
>
> After KAFKA-657, we have a protocol for consumers to commit and fetch offsets from brokers. Currently, consumers are not using this API and directly talking with Zookeeper.
> This Jira will involve following:
> 1. Add a special topic in kafka for storing offsets
> 2. Add an OffsetManager interface which would handle storing, accessing, loading and maintaining consumer offsets
> 3. Implement offset managers for both of these 2 choices : existing ZK based storage or inbuilt storage for offsets.
> 4. Leader brokers would now maintain an additional hash table of offsets for the group-topic-partitions that they lead
> 5. Consumers should now use the OffsetCommit and OffsetFetch API
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira