You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Jun Rao (JIRA)" <ji...@apache.org> on 2012/08/01 18:20:03 UTC
[jira] [Created] (KAFKA-432) allow consumer to read from followers
Jun Rao created KAFKA-432:
-----------------------------
Summary: allow consumer to read from followers
Key: KAFKA-432
URL: https://issues.apache.org/jira/browse/KAFKA-432
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.8
Reporter: Jun Rao
For debugging purpose, it would be convenient if we allow a consumer to consume from a follower replica.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-432) allow consumer to read from followers
Posted by "Yang Ye (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yang Ye updated KAFKA-432:
--------------------------
Attachment: kafka_432_v1.diff
1. removing the "ConsumerShell.scala" file
2. Use "replicaId" field in "FetchRequest" and "OffsetRequest" to indicate where the fetch is from, -1 means "OrdinaryConsumerId", -2 means "LowLevelConsumerId", instead of using "clientId" to distinguish different request
3. create a function "getLocalReplica()" in ReplicaManager, which returns the local replica even if it's not the leader. This function is used in KafkaApis, to work with requests from "LowLevelConsumerId"
4. In kafkaApis, when handling "offsetRequest" and "fetchRequest", treat differently depending on the value of the "replicaId" field. If it's from "lowLevelConsumer", use "getLocalReplica" instead of "getLocalReplicaIfLeader"
5. Move the "earliestOrLatestOffset()" function from "zookeeperConsumerConnector" to "SimpleConsumer" because it's now used by both the high level and low level consumer. (And create two overloaded functions because the client can get it from either zkConnect or from brokerList)
6. Move the "MessageFormmater" inner class out of "ConsoleConsumer", because it's now used by both the console consumer and the simple consumer shell
7. Update the logic of simpleConsumerShell so that it supports fetching from any specified replica now.
8. Removing some unused import and fixed missing arguments in some log statement
> allow consumer to read from followers
> -------------------------------------
>
> Key: KAFKA-432
> URL: https://issues.apache.org/jira/browse/KAFKA-432
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Yang Ye
> Priority: Blocker
> Labels: optimization
> Attachments: kafka_432_v1.diff
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> For debugging purpose, it would be convenient if we allow a consumer to consume from a follower replica.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-432) allow consumer to read from followers
Posted by "Yang Ye (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yang Ye updated KAFKA-432:
--------------------------
Attachment: kafka_432_v5.diff
Thanks for the review! And they're addressed in v5 patch.
> allow consumer to read from followers
> -------------------------------------
>
> Key: KAFKA-432
> URL: https://issues.apache.org/jira/browse/KAFKA-432
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Yang Ye
> Priority: Blocker
> Labels: optimization
> Attachments: kafka_432_v1.diff, kafka_432_v2.diff, kafka_432_v3.diff, kafka_432_v4.diff, kafka_432_v5.diff
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> For debugging purpose, it would be convenient if we allow a consumer to consume from a follower replica.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (KAFKA-432) allow consumer to read from followers
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede resolved KAFKA-432.
---------------------------------
Resolution: Fixed
Committed v5.
> allow consumer to read from followers
> -------------------------------------
>
> Key: KAFKA-432
> URL: https://issues.apache.org/jira/browse/KAFKA-432
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Yang Ye
> Priority: Blocker
> Labels: optimization
> Attachments: kafka_432_v1.diff, kafka_432_v2.diff, kafka_432_v3.diff, kafka_432_v4.diff, kafka_432_v5.diff
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> For debugging purpose, it would be convenient if we allow a consumer to consume from a follower replica.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-432) allow consumer to read from followers
Posted by "Yang Ye (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yang Ye updated KAFKA-432:
--------------------------
Attachment: kafka_432_v2.diff
All comments for patch v1 addressed
> allow consumer to read from followers
> -------------------------------------
>
> Key: KAFKA-432
> URL: https://issues.apache.org/jira/browse/KAFKA-432
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Yang Ye
> Priority: Blocker
> Labels: optimization
> Attachments: kafka_432_v1.diff, kafka_432_v2.diff
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> For debugging purpose, it would be convenient if we allow a consumer to consume from a follower replica.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-432) allow consumer to read from
followers
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467868#comment-13467868 ]
Jun Rao commented on KAFKA-432:
-------------------------------
Thanks for patch v1. Some comments:
1. ReplicaManager.getLocalReplica(): We shouldn't be creating a new replica in this method. Also, this function is almost the same as getReplica() except that it wants to throw an exception if the replica is not found. How about creating a new method getReplicaOrException that calls getReplica and throws an exception if replica is none?
2. object MessageFormatter: remove the extra empty lines above.
3. RequestOrResponse: Could we change LowLevelConsumerId to DebuggingConsumerId so that people is aware that it's only for debugging purpose?
4. remove unused imports
5. SimpleConsumerShell:
5.1 The comment for the --offset is inaccurate. -2 is from beginning.
5.2 The --server option should be changed to --broker-list and the format should be changed accordingly (to take in a list).
5.3 Can we add a command line option to print out the offset for each message too?
5.4 Could we add a command line option to specify the maxWaitMs for each fetch request and defaults it to 1000ms? Also, in FetchRequestBuilder, could we set the default maxWait and minBytes to that in ConsumerConfig object, instead of FetchRequest?
Could you also rebase?
> allow consumer to read from followers
> -------------------------------------
>
> Key: KAFKA-432
> URL: https://issues.apache.org/jira/browse/KAFKA-432
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Yang Ye
> Priority: Blocker
> Labels: optimization
> Attachments: kafka_432_v1.diff
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> For debugging purpose, it would be convenient if we allow a consumer to consume from a follower replica.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (KAFKA-432) allow consumer to read from followers
Posted by "Yang Ye (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yang Ye reassigned KAFKA-432:
-----------------------------
Assignee: Yang Ye
> allow consumer to read from followers
> -------------------------------------
>
> Key: KAFKA-432
> URL: https://issues.apache.org/jira/browse/KAFKA-432
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Yang Ye
> Priority: Blocker
> Labels: optimization
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> For debugging purpose, it would be convenient if we allow a consumer to consume from a follower replica.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (KAFKA-432) allow consumer to read from followers
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede closed KAFKA-432.
-------------------------------
> allow consumer to read from followers
> -------------------------------------
>
> Key: KAFKA-432
> URL: https://issues.apache.org/jira/browse/KAFKA-432
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Yang Ye
> Priority: Blocker
> Labels: optimization
> Attachments: kafka_432_v1.diff, kafka_432_v2.diff, kafka_432_v3.diff, kafka_432_v4.diff, kafka_432_v5.diff
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> For debugging purpose, it would be convenient if we allow a consumer to consume from a follower replica.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-432) allow consumer to read from
followers
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13474720#comment-13474720 ]
Neha Narkhede commented on KAFKA-432:
-------------------------------------
Thanks for the patch! Overall, looks pretty good. A few more comments -
1. FetchRequest
1.1 isFromOrdinaryConsumer and isFromLowLevelConsumer methods are unused. How about adding them when we have a use for them ?
1.2 If you still want to keep them, rename isFromLowLevelConsumer to isFromDebuggingConsumer to be consistent
2. FetchResponse
Revert this file since you changed the spacing between the ( and { at the end of a function name and that does not conform with our coding convention
3. KafkaApis
The following statements are not easy to understand and should not be at info level
info("bytes readable: " + bytesReadable)
info("data read: " + dataRead)
For example, if you read them in a busy Kafka log, would you be able to make sense of it ?
3. OffsetRequest
Rename isFromLowLevelClient to isFromDebuggingClient
4. SimpleConsumer
In earliestOrLatestOffset API, the response variable is never used
5. SimpleConsumerShell
1. You don't have a space between ) and { at the start of few blocks. We have been following the convention of including a space between those, please can you conform to convention in all your patches ?
2. There is one very-long-statement in the file. Please can you break it ?
> allow consumer to read from followers
> -------------------------------------
>
> Key: KAFKA-432
> URL: https://issues.apache.org/jira/browse/KAFKA-432
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Yang Ye
> Priority: Blocker
> Labels: optimization
> Attachments: kafka_432_v1.diff, kafka_432_v2.diff, kafka_432_v3.diff, kafka_432_v4.diff
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> For debugging purpose, it would be convenient if we allow a consumer to consume from a follower replica.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-432) allow consumer to read from
followers
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469034#comment-13469034 ]
Jun Rao commented on KAFKA-432:
-------------------------------
Thanks for patch v2. A few other minor comments:
10. FetchRequestBuilder: could we set the default maxWait and minBytes to that in ConsumerConfig object, instead of FetchRequest?
11. ReplicaManager.getLocalReplica(): Could we use getReplica?
12. Let's remove the consumer shell script in bin/
> allow consumer to read from followers
> -------------------------------------
>
> Key: KAFKA-432
> URL: https://issues.apache.org/jira/browse/KAFKA-432
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Yang Ye
> Priority: Blocker
> Labels: optimization
> Attachments: kafka_432_v1.diff, kafka_432_v2.diff
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> For debugging purpose, it would be convenient if we allow a consumer to consume from a follower replica.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-432) allow consumer to read from followers
Posted by "Yang Ye (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yang Ye updated KAFKA-432:
--------------------------
Attachment: kafka_432_v3.diff
addressed all comments for v2 patch
> allow consumer to read from followers
> -------------------------------------
>
> Key: KAFKA-432
> URL: https://issues.apache.org/jira/browse/KAFKA-432
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Yang Ye
> Priority: Blocker
> Labels: optimization
> Attachments: kafka_432_v1.diff, kafka_432_v2.diff, kafka_432_v3.diff
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> For debugging purpose, it would be convenient if we allow a consumer to consume from a follower replica.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-432) allow consumer to read from followers
Posted by "Joel Koshy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joel Koshy updated KAFKA-432:
-----------------------------
Labels: optimization (was: )
> allow consumer to read from followers
> -------------------------------------
>
> Key: KAFKA-432
> URL: https://issues.apache.org/jira/browse/KAFKA-432
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Labels: optimization
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> For debugging purpose, it would be convenient if we allow a consumer to consume from a follower replica.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-432) allow consumer to read from followers
Posted by "Yang Ye (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yang Ye updated KAFKA-432:
--------------------------
Attachment: kafka_432_v4.diff
rebase from kafka 510
> allow consumer to read from followers
> -------------------------------------
>
> Key: KAFKA-432
> URL: https://issues.apache.org/jira/browse/KAFKA-432
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Yang Ye
> Priority: Blocker
> Labels: optimization
> Attachments: kafka_432_v1.diff, kafka_432_v2.diff, kafka_432_v3.diff, kafka_432_v4.diff
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> For debugging purpose, it would be convenient if we allow a consumer to consume from a follower replica.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-432) allow consumer to read from
followers
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472126#comment-13472126 ]
Jun Rao commented on KAFKA-432:
-------------------------------
Patch no longer applies. Could you rebase?
> allow consumer to read from followers
> -------------------------------------
>
> Key: KAFKA-432
> URL: https://issues.apache.org/jira/browse/KAFKA-432
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Yang Ye
> Priority: Blocker
> Labels: optimization
> Attachments: kafka_432_v1.diff, kafka_432_v2.diff, kafka_432_v3.diff
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> For debugging purpose, it would be convenient if we allow a consumer to consume from a follower replica.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-432) allow consumer to read from
followers
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13474741#comment-13474741 ]
Neha Narkhede commented on KAFKA-432:
-------------------------------------
+1. v5 looks good.
> allow consumer to read from followers
> -------------------------------------
>
> Key: KAFKA-432
> URL: https://issues.apache.org/jira/browse/KAFKA-432
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Yang Ye
> Priority: Blocker
> Labels: optimization
> Attachments: kafka_432_v1.diff, kafka_432_v2.diff, kafka_432_v3.diff, kafka_432_v4.diff, kafka_432_v5.diff
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> For debugging purpose, it would be convenient if we allow a consumer to consume from a follower replica.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-432) allow consumer to read from followers
Posted by "Joel Koshy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joel Koshy updated KAFKA-432:
-----------------------------
Priority: Blocker (was: Major)
> allow consumer to read from followers
> -------------------------------------
>
> Key: KAFKA-432
> URL: https://issues.apache.org/jira/browse/KAFKA-432
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Priority: Blocker
> Labels: optimization
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> For debugging purpose, it would be convenient if we allow a consumer to consume from a follower replica.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira