[jira] [Created] (KAFKA-4385) producer is sending too many unnecessary meta data request if the meta data for a topic is not available

["Jun Yao (JIRA)" <ji...@apache.org>]

Jun Yao created KAFKA-4385:
------------------------------

             Summary: producer is sending too many unnecessary meta data request if the meta data for a topic is not available
                 Key: KAFKA-4385
                 URL: https://issues.apache.org/jira/browse/KAFKA-4385
             Project: Kafka
          Issue Type: Bug
            Reporter: Jun Yao


All current kafka-client producer implementation (<= 0.10.1.0),
When sending a msg to a topic, it will first check if meta data for this topic is available or not, 
when not available, it will set "metadata.requestUpdate()" and wait for meta data from brokers, 

The thing is inside "org.apache.kafka.clients.Metadata.awaitUpdate()", it's already doing a "while (this.version <= lastVersion)" loop waiting for new version response, 
So the loop inside "org.apache.kafka.clients.producer.KafkaProducer.waitOnMetadata() is not needed, 

When "auto.create.topics.enable" is false, sending msgs to a non-exist topic will trigger too many meta requests, everytime a metadata response is returned, because it does not contain the metadata for the topic, it's going to try again until TimeoutException is thrown; 
This is a waste and sometimes causes too much overhead when unexpected msgs are arrived. 





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)