You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Neha Narkhede (JIRA)" <ji...@apache.org> on 2013/04/04 02:09:15 UTC
[jira] [Comment Edited] (KAFKA-330) Add delete topic support
[ https://issues.apache.org/jira/browse/KAFKA-330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13621456#comment-13621456 ]
Neha Narkhede edited comment on KAFKA-330 at 4/4/13 12:08 AM:
--------------------------------------------------------------
Let's do some zookeeper math here to see how long it takes to delete one topic, 8 partitions from a 6 node kafka cluster -
# of zk ops operation during delete topic
1 val partitionAssignment = ZkUtils.getPartitionAssignmentForTopics(zkClient, topics.toSeq)
7 val brokers = ZkUtils.getAllBrokersInCluster(zkClient)
1 ZkUtils.getAllReplicasOnBroker(zkClient, topics.toSeq, brokers.map(_.id)) (This is a redundant read from zookeeper, so reuse the info read in step 1)
2 removeReplicaFromIsr -> getLeaderIsrAndEpochForPartition, conditionalUpdatePersistentPath
9 removeFromTopicsBeingDeleted -> readDataMaybeNull (1), deletePath (8)
20 zookeeper ops. With 10ms per op, (which is a what a zookeeper cluster that kafka consumers and brokers share does in best case), that is 200ms per topic
With 50 such topics, it is 10 seconds. That is the amount of time you are starving other partitions from being available!
What you can do, for simplicity purposes, is keep the existing long lock on the controller side for this patch. We can improve it later or in 0.8.1
Also, the log side of your patch does not acquire the lock. You used the delete APIs that were used by unit tests so far. So they don't deal with the issues I've mentioned above in my comments.
Regarding LogManager - Let's look at the modified version of your patch and see if that solves the problems I've outlined above wrt to interleaving other operations with delete log.
was (Author: nehanarkhede):
Let's do some zookeeper math here to see how long it takes to delete one topic, 8 partitions from a 6 node kafka cluster -
# of zk ops operation during delete topic
1 val partitionAssignment = ZkUtils.getPartitionAssignmentForTopics(zkClient, topics.toSeq)
7 val brokers = ZkUtils.getAllBrokersInCluster(zkClient)
1 ZkUtils.getAllReplicasOnBroker(zkClient, topics.toSeq, brokers.map(_.id)) (This is a redundant read from zookeeper, so reuse the info read in step 1)
2 removeReplicaFromIsr -> getLeaderIsrAndEpochForPartition, conditionalUpdatePersistentPath
9 removeFromTopicsBeingDeleted -> readDataMaybeNull (1), deletePath (8)
20 zookeeper ops. With 10ms per op, (which is a what a zookeeper cluster that kafka consumers and brokers share does in best case), that is 200ms per topic
With 50 such topics, it is 10 seconds. That is the amount of time you are starving other partitions from being available!
Also, the log side of your patch does not acquire the lock. You used the delete APIs that were used by unit tests so far. So they don't deal with the issues I've mentioned above in my comments.
Regarding LogManager - Let's look at the modified version of your patch and see if that solves the problems I've outlined above wrt to interleaving other operations with delete log.
> Add delete topic support
> -------------------------
>
> Key: KAFKA-330
> URL: https://issues.apache.org/jira/browse/KAFKA-330
> Project: Kafka
> Issue Type: Bug
> Components: controller, log, replication
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Swapnil Ghike
> Priority: Blocker
> Labels: features, kafka-0.8, p2, project
> Fix For: 0.8
>
> Attachments: kafka-330-v1.patch
>
>
> One proposal of this API is here - https://cwiki.apache.org/confluence/display/KAFKA/Kafka+replication+detailed+design+V2#KafkareplicationdetaileddesignV2-Deletetopic
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira