You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Jun Rao (JIRA)" <ji...@apache.org> on 2012/06/04 02:20:22 UTC
[jira] [Created] (KAFKA-362) ZookeeperConsumerConnector needs to
connect to new leader after leadership change
Jun Rao created KAFKA-362:
-----------------------------
Summary: ZookeeperConsumerConnector needs to connect to new leader after leadership change
Key: KAFKA-362
URL: https://issues.apache.org/jira/browse/KAFKA-362
Project: Kafka
Issue Type: Sub-task
Components: core
Affects Versions: 0.8
Reporter: Jun Rao
Currently, if the leader of a partition changes, the fetcher in ZookeeperConsumerConnector won't switch to the new leader automatically.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-362) ZookeeperConsumerConnector needs to
connect to new leader after leadership change
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao updated KAFKA-362:
--------------------------
Fix Version/s: 0.8
Status: Patch Available (was: Open)
> ZookeeperConsumerConnector needs to connect to new leader after leadership change
> ---------------------------------------------------------------------------------
>
> Key: KAFKA-362
> URL: https://issues.apache.org/jira/browse/KAFKA-362
> Project: Kafka
> Issue Type: Sub-task
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Jun Rao
> Fix For: 0.8
>
> Attachments: kafka-362_v1.patch
>
> Original Estimate: 252h
> Remaining Estimate: 252h
>
> Currently, if the leader of a partition changes, the fetcher in ZookeeperConsumerConnector won't switch to the new leader automatically.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-362) ZookeeperConsumerConnector needs to
connect to new leader after leadership change
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao updated KAFKA-362:
--------------------------
Attachment: kafka-362_v3.patch
Thanks for the review. Attach patch v3.
1a. fixed
1b. Didn't notice those configs already exist. Fixed.
2. Changed to not return a value in enqueue. Changed comments above processPartitionData.
3. It's needed. If we shut down the fetcher thread, we could get an Interrupted exception. In this case, we don't need to add the partitions to the list.
4. fixed.
5. It's possible to use a BlockingQueue. However, we probably still need to use a lock to synchronize the update of partitionMap and noLeaderPartitionSet (or queue). For example, if partitionMap is null, we don't want to update noLeaderPartitionSet any more. This is a bit hard to guarantee just with an atomic value since the value could change immediately after you do the check. So keeping the current implementation for now.
> ZookeeperConsumerConnector needs to connect to new leader after leadership change
> ---------------------------------------------------------------------------------
>
> Key: KAFKA-362
> URL: https://issues.apache.org/jira/browse/KAFKA-362
> Project: Kafka
> Issue Type: Sub-task
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Jun Rao
> Fix For: 0.8
>
> Attachments: kafka-362_v1.patch, kafka-362_v2.patch, kafka-362_v3.patch
>
> Original Estimate: 252h
> Remaining Estimate: 252h
>
> Currently, if the leader of a partition changes, the fetcher in ZookeeperConsumerConnector won't switch to the new leader automatically.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (KAFKA-362) ZookeeperConsumerConnector needs to
connect to new leader after leadership change
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao reassigned KAFKA-362:
-----------------------------
Assignee: Jun Rao
> ZookeeperConsumerConnector needs to connect to new leader after leadership change
> ---------------------------------------------------------------------------------
>
> Key: KAFKA-362
> URL: https://issues.apache.org/jira/browse/KAFKA-362
> Project: Kafka
> Issue Type: Sub-task
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Jun Rao
> Original Estimate: 252h
> Remaining Estimate: 252h
>
> Currently, if the leader of a partition changes, the fetcher in ZookeeperConsumerConnector won't switch to the new leader automatically.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-362) ZookeeperConsumerConnector needs to
connect to new leader after leadership change
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao updated KAFKA-362:
--------------------------
Attachment: kafka-362_v1.patch
Attach patch v1. The patch replaces Fetcher and FetcherRunnable with ConsumerFetcherManager and ConsumerFetcherThread. ConsumerFetcherManager maintains a list of partitions that have lost the leader. It keeps trying to get the new leader of the partition and add the partition to the corresponding fetcher thread.
> ZookeeperConsumerConnector needs to connect to new leader after leadership change
> ---------------------------------------------------------------------------------
>
> Key: KAFKA-362
> URL: https://issues.apache.org/jira/browse/KAFKA-362
> Project: Kafka
> Issue Type: Sub-task
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Jun Rao
> Fix For: 0.8
>
> Attachments: kafka-362_v1.patch
>
> Original Estimate: 252h
> Remaining Estimate: 252h
>
> Currently, if the leader of a partition changes, the fetcher in ZookeeperConsumerConnector won't switch to the new leader automatically.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-362) ZookeeperConsumerConnector needs to
connect to new leader after leadership change
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13288640#comment-13288640 ]
Jun Rao commented on KAFKA-362:
-------------------------------
In trunk, a partition only exists in 1 broker. So, once the partition assignment is done, the set of brokers from which a consumer fetches data doesn't change. In 0.8, each partition can have several replicas and data can only be fetched from the leader broker, which can change over time. So, a consumer needs to handle leader change properly in 0.8.
> ZookeeperConsumerConnector needs to connect to new leader after leadership change
> ---------------------------------------------------------------------------------
>
> Key: KAFKA-362
> URL: https://issues.apache.org/jira/browse/KAFKA-362
> Project: Kafka
> Issue Type: Sub-task
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Original Estimate: 252h
> Remaining Estimate: 252h
>
> Currently, if the leader of a partition changes, the fetcher in ZookeeperConsumerConnector won't switch to the new leader automatically.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-362) ZookeeperConsumerConnector needs to
connect to new leader after leadership change
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao updated KAFKA-362:
--------------------------
Attachment: kafka-362_v2.patch
Attache patch v2 with minor changes to make sure that no more fetcher threads are started after stopAllConnections() is called in ConsumerFetcherManager.
> ZookeeperConsumerConnector needs to connect to new leader after leadership change
> ---------------------------------------------------------------------------------
>
> Key: KAFKA-362
> URL: https://issues.apache.org/jira/browse/KAFKA-362
> Project: Kafka
> Issue Type: Sub-task
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Jun Rao
> Fix For: 0.8
>
> Attachments: kafka-362_v1.patch, kafka-362_v2.patch
>
> Original Estimate: 252h
> Remaining Estimate: 252h
>
> Currently, if the leader of a partition changes, the fetcher in ZookeeperConsumerConnector won't switch to the new leader automatically.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-362) ZookeeperConsumerConnector needs to
connect to new leader after leadership change
Posted by "Joel Koshy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13411127#comment-13411127 ]
Joel Koshy commented on KAFKA-362:
----------------------------------
+1 on v3
> ZookeeperConsumerConnector needs to connect to new leader after leadership change
> ---------------------------------------------------------------------------------
>
> Key: KAFKA-362
> URL: https://issues.apache.org/jira/browse/KAFKA-362
> Project: Kafka
> Issue Type: Sub-task
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Jun Rao
> Fix For: 0.8
>
> Attachments: kafka-362_v1.patch, kafka-362_v2.patch, kafka-362_v3.patch
>
> Original Estimate: 252h
> Remaining Estimate: 252h
>
> Currently, if the leader of a partition changes, the fetcher in ZookeeperConsumerConnector won't switch to the new leader automatically.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-362) ZookeeperConsumerConnector needs to
connect to new leader after leadership change
Posted by "Guozhang Wang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13288354#comment-13288354 ]
Guozhang Wang commented on KAFKA-362:
-------------------------------------
One question: in the trunk code that I saw (which I think does not have many updates including broker replication), when consumers computes partition assignment they just consider everyone. With broker replication, they will only consider leader's partitions?
> ZookeeperConsumerConnector needs to connect to new leader after leadership change
> ---------------------------------------------------------------------------------
>
> Key: KAFKA-362
> URL: https://issues.apache.org/jira/browse/KAFKA-362
> Project: Kafka
> Issue Type: Sub-task
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Original Estimate: 252h
> Remaining Estimate: 252h
>
> Currently, if the leader of a partition changes, the fetcher in ZookeeperConsumerConnector won't switch to the new leader automatically.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-362) ZookeeperConsumerConnector needs to
connect to new leader after leadership change
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao updated KAFKA-362:
--------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thanks for the view. Just committed to 0.8.
> ZookeeperConsumerConnector needs to connect to new leader after leadership change
> ---------------------------------------------------------------------------------
>
> Key: KAFKA-362
> URL: https://issues.apache.org/jira/browse/KAFKA-362
> Project: Kafka
> Issue Type: Sub-task
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Jun Rao
> Fix For: 0.8
>
> Attachments: kafka-362_v1.patch, kafka-362_v2.patch, kafka-362_v3.patch
>
> Original Estimate: 252h
> Remaining Estimate: 252h
>
> Currently, if the leader of a partition changes, the fetcher in ZookeeperConsumerConnector won't switch to the new leader automatically.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-362) ZookeeperConsumerConnector needs to
connect to new leader after leadership change
Posted by "Joel Koshy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13409931#comment-13409931 ]
Joel Koshy commented on KAFKA-362:
----------------------------------
1 - ConsumerConfig:
a - Instead of NoneReplicaFetcherId, use FetchRequest.NonFollowerId (or
make it an alias of that)
b - So fetch.wait.time.ms and fetch.min.byte deprecate the existing
min.fetch.bytes and max.fetch.wait.ms configs? - ConsoleConsumer needs
to be updated (as it still uses the old ones). Actually, why rename
the existing configs?
2 - PartitionTopicInfo: Seems that enqueue does not need to return anything
now. Actually, it's unclear: the comment above processPartitionData in
AbstractFetcherManager says that it returns the new fetch offset but the
implementation does not return anything.
3 - AbstractFetcherThread: Line 77: is the check on isRunning necessary?
4 - AbstractFetcherManager: rename numReplicaFetchers to numFetchers.
5 - ConsumerFetcherManager: just a thought (I have not looked carefully
enough for feasibility): for noLeaderPartitionSet - would it be possible
to use a SynchronousQueue[Seq[(String, Int)] instead? That way, you may be
able to get rid of the refreshLeaderBackOff, and the lock and its
associated condition (although you would need to use an atomic reference
for the partitionMap).
- completes -> completed
core/src/main/scala/kafka/consumer/ConsumerFetcherManager.scala: info("shutdown completes")
core/src/main/scala/kafka/server/ReplicaFetcherManager.scala: info("shutdown completes")
> ZookeeperConsumerConnector needs to connect to new leader after leadership change
> ---------------------------------------------------------------------------------
>
> Key: KAFKA-362
> URL: https://issues.apache.org/jira/browse/KAFKA-362
> Project: Kafka
> Issue Type: Sub-task
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Jun Rao
> Fix For: 0.8
>
> Attachments: kafka-362_v1.patch, kafka-362_v2.patch
>
> Original Estimate: 252h
> Remaining Estimate: 252h
>
> Currently, if the leader of a partition changes, the fetcher in ZookeeperConsumerConnector won't switch to the new leader automatically.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira