[jira] [Commented] (KAFKA-9372) Add producer config to make topicExpiry configurable

["Brian Byrne (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-9372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17009933#comment-17009933 ] 

Brian Byrne commented on KAFKA-9372:
------------------------------------

Hi [~Jiao-zhang] - we actually have a KIP in progress to add just this in the form of metadata.eviction.period.ms:
  https://cwiki.apache.org/confluence/display/KAFKA/KIP-526%3A+Reduce+Producer+Metadata+Lookups+for+Large+Number+of+Topics

I'll take ownership of the ticket and update it once implemented. Thanks!

> Add producer config to make topicExpiry configurable
> ----------------------------------------------------
>
>                 Key: KAFKA-9372
>                 URL: https://issues.apache.org/jira/browse/KAFKA-9372
>             Project: Kafka
>          Issue Type: Improvement
>          Components: producer 
>    Affects Versions: 1.1.0
>            Reporter: Jiao Zhang
>            Priority: Minor
>
> Sometimes we got error "org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 1000 ms" on producer side. We did the investigation and found
>  # our producer produced messages in really low rate, the interval is more than 10 minutes
>  # by default, producer would expire topics after TOPIC_EXPIRY_MS, after topic expired if no data produce before next metadata update (automatically triggered by metadata.max.age.ms) partitions entry for the topic would disappear from the Metadata cache As a result, almost for every time's produce, producer need fetch metadata which could possibly end with timeout.
> To solve this, we propose to add a new config metadata.topic.expiry for producer to make topicExpiry configurable. Topic expiry is good only when producer is long-lived and is used for producing variable counts of topics. But in the case that producers are bounded to single or few fixed topics, there is no need to expire topics at all.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)