[jira] [Commented] (KAFKA-1787) Consumer should delete offsets and release partition ownership for deleted topics

["Jun Rao (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-1787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14230875#comment-14230875 ] 

Jun Rao commented on KAFKA-1787:
--------------------------------

Even if we trigger a rebalance in the consumer on topic deletion, it's still possible for a deleted topic to be recreated since a consumer could have issued a fetch request (with the deleted topic) after the topic is deleted on the broker, but before the rebalance is triggered on the consumer. To completely solve this problem, we need to add a create topic api and modify the TopicMetadataRequest such that it doesn't trigger auto topic creation on the broker. This is probably too big a change for 0.8.2 though.

> Consumer should delete offsets and release partition ownership for deleted topics
> ---------------------------------------------------------------------------------
>
>                 Key: KAFKA-1787
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1787
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 0.8.2
>            Reporter: Joel Koshy
>            Assignee: Onur Karaman
>            Priority: Blocker
>             Fix For: 0.8.2
>
>
> Marking as a blocker for now, but we can evaluate.
> If a topic is deleted, the consumer currently does not rebalance (handleDataDeleted is a TODO in ZooKeeperConsumerConnector)
> As a result, it will continue owning that (deleted) partition and continue to issue fetch requests. Those fetch requests will result in an UnknownTopicOrPartition error and thus will result in TopicMetadataRequests issued by the leader finder thread which can recreate the topic if auto-create is turned on.
> Furthermore if we don't delete the offsets it is possible to lose messages if the topic is recreated immediately and the previously checkpointed offset (for the old data) is small. E.g., if a consumer is at offset 10 on some partition, and if the partition is deleted and recreated and 15 messages are sent to it then the first 10 messages will be lost.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)