You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Jiangjie Qin (JIRA)" <ji...@apache.org> on 2015/12/07 19:08:11 UTC
[jira] [Commented] (KAFKA-2948) Kafka producer does not cope well
with topic deletions
[ https://issues.apache.org/jira/browse/KAFKA-2948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15045372#comment-15045372 ]
Jiangjie Qin commented on KAFKA-2948:
-------------------------------------
[~rsivaram] As you pointed out we never remove the topics set in producer metadata. I am not sure if removing the topic from the set when we see UNKNOWN_TOPIC_OR_PARTITION error code is the right way to fix this, because UNKNOWN_TOPIC_OR_PARTITION can also occur in other cases such as partition reassignment, the producer is supposed to retry in this case.
Maybe a TTL is a better solution here. e.g. If the producer hasn't sent data to a particular topic since last metadata refresh, we can remove the topic from metadata topic set on next metadata refresh.
> Kafka producer does not cope well with topic deletions
> ------------------------------------------------------
>
> Key: KAFKA-2948
> URL: https://issues.apache.org/jira/browse/KAFKA-2948
> Project: Kafka
> Issue Type: Bug
> Components: producer
> Affects Versions: 0.9.0.0
> Reporter: Rajini Sivaram
> Assignee: Rajini Sivaram
>
> Kafka producer gets metadata for topics when send is invoked and thereafter it attempts to keep the metadata up-to-date without any explicit requests from the client. This works well in static environments, but when topics are added or deleted, list of topics in Metadata grows but never shrinks. Apart from being a memory leak, this results in constant requests for metadata for deleted topics.
> We are running into this issue with the Confluent REST server where topic deletion from tests are filling up logs with warnings about unknown topics. Auto-create is turned off in our Kafka cluster.
> I am happy to provide a fix, but am not sure what the right fix is. Does it make sense to remove topics from the metadata list when UNKNOWN_TOPIC_OR_PARTITION response is received if there are no outstanding sends? It doesn't look very straightforward to do this, so any alternative suggestions are welcome.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)