You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Jun Rao (JIRA)" <ji...@apache.org> on 2012/08/20 18:15:38 UTC
[jira] [Created] (KAFKA-474) support changing host/port of a broker
Jun Rao created KAFKA-474:
-----------------------------
Summary: support changing host/port of a broker
Key: KAFKA-474
URL: https://issues.apache.org/jira/browse/KAFKA-474
Project: Kafka
Issue Type: Bug
Components: core
Reporter: Jun Rao
Fix For: 0.8
Currently, the consumer client caches the host/port of a broker and never refreshes the cache. This means that if a broker changes to a different host, the consumer client won't be able to connect to the new host without a restart. One possibility is to change AbstractFetcherManager to maintain a map of <Broker, fetcher>, instead of <broker id, fetcher>.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (KAFKA-474) support changing host/port of a
broker
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao resolved KAFKA-474.
---------------------------
Resolution: Fixed
Assignee: David Arthur
Thanks for the patch. Committed to 0.8.
> support changing host/port of a broker
> --------------------------------------
>
> Key: KAFKA-474
> URL: https://issues.apache.org/jira/browse/KAFKA-474
> Project: Kafka
> Issue Type: Bug
> Components: core
> Reporter: Jun Rao
> Assignee: David Arthur
> Labels: bugs, newbie
> Fix For: 0.8
>
> Attachments: KAFKA-474.patch
>
>
> Currently, the consumer client caches the host/port of a broker and never refreshes the cache. This means that if a broker changes to a different host, the consumer client won't be able to connect to the new host without a restart. One possibility is to change AbstractFetcherManager to maintain a map of <Broker, fetcher>, instead of <broker id, fetcher>.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-474) support changing host/port of a broker
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao updated KAFKA-474:
--------------------------
Remaining Estimate: (was: 168h)
Original Estimate: (was: 168h)
> support changing host/port of a broker
> --------------------------------------
>
> Key: KAFKA-474
> URL: https://issues.apache.org/jira/browse/KAFKA-474
> Project: Kafka
> Issue Type: Bug
> Components: core
> Reporter: Jun Rao
> Labels: bugs, newbie
> Fix For: 0.8
>
>
> Currently, the consumer client caches the host/port of a broker and never refreshes the cache. This means that if a broker changes to a different host, the consumer client won't be able to connect to the new host without a restart. One possibility is to change AbstractFetcherManager to maintain a map of <Broker, fetcher>, instead of <broker id, fetcher>.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (KAFKA-474) support changing host/port of a broker
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao closed KAFKA-474.
-------------------------
> support changing host/port of a broker
> --------------------------------------
>
> Key: KAFKA-474
> URL: https://issues.apache.org/jira/browse/KAFKA-474
> Project: Kafka
> Issue Type: Bug
> Components: core
> Reporter: Jun Rao
> Assignee: David Arthur
> Labels: bugs, newbie
> Fix For: 0.8
>
> Attachments: KAFKA-474.patch
>
>
> Currently, the consumer client caches the host/port of a broker and never refreshes the cache. This means that if a broker changes to a different host, the consumer client won't be able to connect to the new host without a restart. One possibility is to change AbstractFetcherManager to maintain a map of <Broker, fetcher>, instead of <broker id, fetcher>.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-474) support changing host/port of a
broker
Posted by "David Arthur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469333#comment-13469333 ]
David Arthur commented on KAFKA-474:
------------------------------------
I took a look at AbstractFetcherManager and the broker -> thread map. I don't see where/how this map is used to actually get the fetcher thread for a particular broker. I see the fetcherSourceBroker method, but cannot find anywhere where it is invoked.
In fact, the only usage of this map I can see is when fetcher threads are created and removed.
Am I overlooking something?
> support changing host/port of a broker
> --------------------------------------
>
> Key: KAFKA-474
> URL: https://issues.apache.org/jira/browse/KAFKA-474
> Project: Kafka
> Issue Type: Bug
> Components: core
> Reporter: Jun Rao
> Labels: bugs, newbie
> Fix For: 0.8
>
>
> Currently, the consumer client caches the host/port of a broker and never refreshes the cache. This means that if a broker changes to a different host, the consumer client won't be able to connect to the new host without a restart. One possibility is to change AbstractFetcherManager to maintain a map of <Broker, fetcher>, instead of <broker id, fetcher>.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-474) support changing host/port of a broker
Posted by "David Arthur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
David Arthur updated KAFKA-474:
-------------------------------
Attachment: KAFKA-474.patch
Something as simple as this?
> support changing host/port of a broker
> --------------------------------------
>
> Key: KAFKA-474
> URL: https://issues.apache.org/jira/browse/KAFKA-474
> Project: Kafka
> Issue Type: Bug
> Components: core
> Reporter: Jun Rao
> Labels: bugs, newbie
> Fix For: 0.8
>
> Attachments: KAFKA-474.patch
>
>
> Currently, the consumer client caches the host/port of a broker and never refreshes the cache. This means that if a broker changes to a different host, the consumer client won't be able to connect to the new host without a restart. One possibility is to change AbstractFetcherManager to maintain a map of <Broker, fetcher>, instead of <broker id, fetcher>.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-474) support changing host/port of a
broker
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469522#comment-13469522 ]
Jun Rao commented on KAFKA-474:
-------------------------------
Yes, fetcherSourceBroker is actually no longer needed. It was used before when a follower switches leaders. The follower uses that method to see if it's already fetching from the new leader. We removed this when we cache the leader epoch in Partition, which avoids executing duplicated leader change requests.
The real problem here is that in addFetcher(), we use the broker id as the key. Of course, the host/port info could have changed. I think a simple solution may be just key fetcherThreadMap on Broker, instead of broker id.
> support changing host/port of a broker
> --------------------------------------
>
> Key: KAFKA-474
> URL: https://issues.apache.org/jira/browse/KAFKA-474
> Project: Kafka
> Issue Type: Bug
> Components: core
> Reporter: Jun Rao
> Labels: bugs, newbie
> Fix For: 0.8
>
>
> Currently, the consumer client caches the host/port of a broker and never refreshes the cache. This means that if a broker changes to a different host, the consumer client won't be able to connect to the new host without a restart. One possibility is to change AbstractFetcherManager to maintain a map of <Broker, fetcher>, instead of <broker id, fetcher>.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira