You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Tejas Patil (JIRA)" <ji...@apache.org> on 2013/08/19 21:25:52 UTC
[jira] [Updated] (KAFKA-1000) Inbuilt consumer offset management
[ https://issues.apache.org/jira/browse/KAFKA-1000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tejas Patil updated KAFKA-1000:
-------------------------------
Description:
This Jira is for tracking the phase #2 of Offset Management [0]. After KAFKA-657 [1], we have a protocol for consumers to commit and fetch offsets from brokers. Currently, consumers are not using this API and directly talking with Zookeeper.
Joel and I have been working on this.
[0] : https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management
[1] : https://issues.apache.org/jira/browse/KAFKA-657
was:
This Jitra corresponds to the phase #2 of Offset Management [0]. After KAFKA-657 [1], we have a protocol for consumers to commit and fetch offsets from brokers. Currently, consumers are not using this API and directly talking with Zookeeper.
This Jira will involve following:
1. Consumers should now use the offset commit API
2. Add a special topic in kafka for storing offsets
3. Pluggable offset commit logic so that users can pick the existing ZK based storage or inbuilt storage for offsets.
4. Leader brokers would now maintain an additional hash table of offsets for the group-topic-partitions that they lead
5. Consumers send their offset fetch requests to brokers. Receiving broker may or may not be the leader for the requested group-topic-partitions so would have to forward the request to respective leader brokers.
Joel and I have been working on this and I would share the initial patch and a writeup covering the nitty-gritty of the implementation.
[0] : https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management
[1] : https://issues.apache.org/jira/browse/KAFKA-657
> Inbuilt consumer offset management
> ----------------------------------
>
> Key: KAFKA-1000
> URL: https://issues.apache.org/jira/browse/KAFKA-1000
> Project: Kafka
> Issue Type: New Feature
> Components: consumer
> Affects Versions: 0.8.1
> Reporter: Tejas Patil
> Assignee: Tejas Patil
> Priority: Minor
> Labels: features
>
> This Jira is for tracking the phase #2 of Offset Management [0]. After KAFKA-657 [1], we have a protocol for consumers to commit and fetch offsets from brokers. Currently, consumers are not using this API and directly talking with Zookeeper.
> Joel and I have been working on this.
> [0] : https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management
> [1] : https://issues.apache.org/jira/browse/KAFKA-657
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira