You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Jun Rao (Created) (JIRA)" <ji...@apache.org> on 2012/01/05 19:06:39 UTC
[jira] [Created] (KAFKA-237) create/delete ZK path for a topic in
an admin tool
create/delete ZK path for a topic in an admin tool
--------------------------------------------------
Key: KAFKA-237
URL: https://issues.apache.org/jira/browse/KAFKA-237
Project: Kafka
Issue Type: Sub-task
Reporter: Jun Rao
This subtask will implement a create/delete admin tool that create/delete the ZK paths needed for a new topic. The needed ZK paths are described in the design doc in KAFKA-50 and the discussion in KAFKA-47.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-237) create/delete ZK path for a topic in
an admin tool
Posted by "Neha Narkhede (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185397#comment-13185397 ]
Neha Narkhede commented on KAFKA-237:
-------------------------------------
Thanks for uploading v2.
5. Today, we expose this sort of information in the zk consumer (ZkUtils.createEphemeralPathExpectConflict) and it has been useful in debugging. Since this API exists as part of the admin package, it does seem like useful information to have.
7. Got it. Thanks for the explanation.
> create/delete ZK path for a topic in an admin tool
> --------------------------------------------------
>
> Key: KAFKA-237
> URL: https://issues.apache.org/jira/browse/KAFKA-237
> Project: Kafka
> Issue Type: Sub-task
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Jun Rao
> Attachments: kafka-237-v2.patch, kafka-237.patch
>
>
> This subtask will implement a create/delete admin tool that create/delete the ZK paths needed for a new topic. The needed ZK paths are described in the design doc in KAFKA-50 and the discussion in KAFKA-47.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-237) create/delete ZK path for a topic in
an admin tool
Posted by "Neha Narkhede (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185158#comment-13185158 ]
Neha Narkhede commented on KAFKA-237:
-------------------------------------
Took a look at the patch. Good to see work started on KAFKA-50. I have a couple of comments -
1. It will be good to use a logger in all the tools. This is mainly to be able to cleanly log stack traces. For example, println("delection failed because of " + e) does not print out the stack trace as desired.
2. In the create topic tool, there is a getBrokerList API. This seems like a pretty useful utility API. Would it make sense to expose it in ZkUtils instead of rewriting it there ?
3. In the create topic tool, can the broker list returned by getBrokerList be a Set ? It is expected to be a Set in getManualReplicaAssignment, and also expected to be one.
4. In the create topic tool, replica-assignment-list format is very pretty to read. Can this be described in more detail ? Also, usually entries are described in csv format. Seems like it is easier to understand if the list of entries per partition are comma separated, and the replica ids within a partitions list are ":" separated.
5. In createReplicaAssignmentPathInZK API, if a topic already exists, it will be good to print its version (timestamp) as part of the exception.
6. Can getTopicPartitionsPartPath API be renamed so that it is easier to understand what it is meant for ? It seems like it returns the replication information for a particular partition id.
7. In AdminUtils.assginReplicasToBrokers(), why is secondReplicaShift required ? Can the next replica index be selected like this - (firstReplicaIndex + j + 1) % num_brokers. Or maybe I'm missing something ?
8. There are a couple of typos in the create topic admin tool
> create/delete ZK path for a topic in an admin tool
> --------------------------------------------------
>
> Key: KAFKA-237
> URL: https://issues.apache.org/jira/browse/KAFKA-237
> Project: Kafka
> Issue Type: Sub-task
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Jun Rao
> Attachments: kafka-237.patch
>
>
> This subtask will implement a create/delete admin tool that create/delete the ZK paths needed for a new topic. The needed ZK paths are described in the design doc in KAFKA-50 and the discussion in KAFKA-47.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-237) create/delete ZK path for a topic in
an admin tool
Posted by "Neha Narkhede (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185823#comment-13185823 ]
Neha Narkhede commented on KAFKA-237:
-------------------------------------
+1.
PartitionMetaData can be a case class, but I will be making that change anyways, as part of KAFKA-238
> create/delete ZK path for a topic in an admin tool
> --------------------------------------------------
>
> Key: KAFKA-237
> URL: https://issues.apache.org/jira/browse/KAFKA-237
> Project: Kafka
> Issue Type: Sub-task
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Jun Rao
> Attachments: kafka-237-v2.patch, kafka-237-v3.patch, kafka-237.patch
>
>
> This subtask will implement a create/delete admin tool that create/delete the ZK paths needed for a new topic. The needed ZK paths are described in the design doc in KAFKA-50 and the discussion in KAFKA-47.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-237) create/delete ZK path for a topic in
an admin tool
Posted by "Jun Rao (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao updated KAFKA-237:
--------------------------
Attachment: kafka-237-v3.patch
Attach v3.
5. Return version id if create topic fails.
> create/delete ZK path for a topic in an admin tool
> --------------------------------------------------
>
> Key: KAFKA-237
> URL: https://issues.apache.org/jira/browse/KAFKA-237
> Project: Kafka
> Issue Type: Sub-task
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Jun Rao
> Attachments: kafka-237-v2.patch, kafka-237-v3.patch, kafka-237.patch
>
>
> This subtask will implement a create/delete admin tool that create/delete the ZK paths needed for a new topic. The needed ZK paths are described in the design doc in KAFKA-50 and the discussion in KAFKA-47.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-237) create/delete ZK path for a topic in
an admin tool
Posted by "Jun Rao (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao updated KAFKA-237:
--------------------------
Attachment: kafka-237-v2.patch
Attach patch v2. Comments addressed:
1. Log4j may not always print to console. So, it's better if we print out to console directly. Added stack trace in the print out.
2. Moved getBrokerList to ZkUtils.
3. Sometimes we need a sorted broker list, such as doing replica assignment. So, changed code to convert list to set when needed.
4. Changed the separator format accordingly.
5. Actually, version information is mostly for internal use and probably shouldn't be exposed to users. From users' perspective, knowing that a topic exists seems enough. Not sure if returning version info is helpful to the users. Version info is potentailly useful for operators. However, such information can be got from ZK admin tools directly.
6. renamed
7. The reason is that if there are multiple partitions (first replica) assigned to the same broker, we want their second replicas to be spread around the rest of brokers. The approach you described will assign the second replica of all those partitions to the same broker.
8. fixed
> create/delete ZK path for a topic in an admin tool
> --------------------------------------------------
>
> Key: KAFKA-237
> URL: https://issues.apache.org/jira/browse/KAFKA-237
> Project: Kafka
> Issue Type: Sub-task
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Jun Rao
> Attachments: kafka-237-v2.patch, kafka-237.patch
>
>
> This subtask will implement a create/delete admin tool that create/delete the ZK paths needed for a new topic. The needed ZK paths are described in the design doc in KAFKA-50 and the discussion in KAFKA-47.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-237) create/delete ZK path for a topic in
an admin tool
Posted by "Jun Rao (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao updated KAFKA-237:
--------------------------
Component/s: core
> create/delete ZK path for a topic in an admin tool
> --------------------------------------------------
>
> Key: KAFKA-237
> URL: https://issues.apache.org/jira/browse/KAFKA-237
> Project: Kafka
> Issue Type: Sub-task
> Components: core
> Reporter: Jun Rao
>
> This subtask will implement a create/delete admin tool that create/delete the ZK paths needed for a new topic. The needed ZK paths are described in the design doc in KAFKA-50 and the discussion in KAFKA-47.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-237) create/delete ZK path for a topic in
an admin tool
Posted by "Jun Rao (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao updated KAFKA-237:
--------------------------
Resolution: Fixed
Fix Version/s: 0.8
Status: Resolved (was: Patch Available)
thanks for the review; just committed this.
> create/delete ZK path for a topic in an admin tool
> --------------------------------------------------
>
> Key: KAFKA-237
> URL: https://issues.apache.org/jira/browse/KAFKA-237
> Project: Kafka
> Issue Type: Sub-task
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Jun Rao
> Fix For: 0.8
>
> Attachments: kafka-237-v2.patch, kafka-237-v3.patch, kafka-237.patch
>
>
> This subtask will implement a create/delete admin tool that create/delete the ZK paths needed for a new topic. The needed ZK paths are described in the design doc in KAFKA-50 and the discussion in KAFKA-47.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (KAFKA-237) create/delete ZK path for a topic in
an admin tool
Posted by "Jun Rao (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao reassigned KAFKA-237:
-----------------------------
Assignee: Jun Rao
> create/delete ZK path for a topic in an admin tool
> --------------------------------------------------
>
> Key: KAFKA-237
> URL: https://issues.apache.org/jira/browse/KAFKA-237
> Project: Kafka
> Issue Type: Sub-task
> Components: core
> Reporter: Jun Rao
> Assignee: Jun Rao
>
> This subtask will implement a create/delete admin tool that create/delete the ZK paths needed for a new topic. The needed ZK paths are described in the design doc in KAFKA-50 and the discussion in KAFKA-47.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-237) create/delete ZK path for a topic in
an admin tool
Posted by "Jun Rao (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao updated KAFKA-237:
--------------------------
Affects Version/s: 0.8
Status: Patch Available (was: Open)
Patch attached. The patch contains:
1. New command line tools to create, list, and delete a topic.
2. Auto replica assignment.
3. Unit test.
There is a minor change from the doc. The ZK paths for a topic is now under:
/brokers/topics/[topic]/partitions/[partition_id], instead of /brokers/topics/[topic]/[partition_id]. This is mainly to distinguish the new ZK data structures from the old ones.
> create/delete ZK path for a topic in an admin tool
> --------------------------------------------------
>
> Key: KAFKA-237
> URL: https://issues.apache.org/jira/browse/KAFKA-237
> Project: Kafka
> Issue Type: Sub-task
> Components: core
> Affects Versions: 0.8
> Reporter: Jun Rao
> Assignee: Jun Rao
> Attachments: kafka-237.patch
>
>
> This subtask will implement a create/delete admin tool that create/delete the ZK paths needed for a new topic. The needed ZK paths are described in the design doc in KAFKA-50 and the discussion in KAFKA-47.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-237) create/delete ZK path for a topic in
an admin tool
Posted by "Jun Rao (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao updated KAFKA-237:
--------------------------
Attachment: kafka-237.patch
> create/delete ZK path for a topic in an admin tool
> --------------------------------------------------
>
> Key: KAFKA-237
> URL: https://issues.apache.org/jira/browse/KAFKA-237
> Project: Kafka
> Issue Type: Sub-task
> Components: core
> Reporter: Jun Rao
> Assignee: Jun Rao
> Attachments: kafka-237.patch
>
>
> This subtask will implement a create/delete admin tool that create/delete the ZK paths needed for a new topic. The needed ZK paths are described in the design doc in KAFKA-50 and the discussion in KAFKA-47.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira