[GitHub] [druid] FrankChen021 opened a new issue #10008: Why metadata.max.age.ms is set to 10 seconds ?

[GitBox <gi...@apache.org>]

FrankChen021 opened a new issue #10008:
URL: https://github.com/apache/druid/issues/10008


   `metadata.max.age.ms` is a parameter provided by KafkaProducer/KafkaConsumer, which means 
   > The period of time in milliseconds after which we force a refresh of metadata even if we haven't seen any partition leadership changes to proactively discover any new brokers or partitions.
   
   It's default value is 5 minutes , and in Kafka Index Service, [its value is set to 10 seconds](https://github.com/apache/druid/blob/771870ae2d312d643e6d98f3d0af8a9618af9681/extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/KafkaConsumerConfigs.java#L37), which means every 10 seconds, the kafka consumer should issue a meta request to kafka brokers to get the latest metadata of whole cluster.
   
   I checked commit history, #7249 introduced current code, but before that, this parameter had been there. 
   
   In production it's uncommon to change this parameter.  And reducing the live time of metadata at client side is not a problem. 
   
   I'm just curious that why its value is shorten to 10 seconds ? Were there any problems related to Kafka partition leader switches during ingestion ?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org