[jira] [Created] (KAFKA-13901) Exactly once producer cannot start due to TimeoutException: Timeout expired after 300000ms awaiting InitProducerId

["Ben Augarten (Jira)" <ji...@apache.org>]

Ben Augarten created KAFKA-13901:
------------------------------------

             Summary: Exactly once producer cannot start due to TimeoutException: Timeout expired after 300000ms awaiting InitProducerId
                 Key: KAFKA-13901
                 URL: https://issues.apache.org/jira/browse/KAFKA-13901
             Project: Kafka
          Issue Type: Bug
    Affects Versions: 2.5.1
            Reporter: Ben Augarten
         Attachments: broker-logs-renamed-topics, client-logs-truncated

I'm currently running into this problem running org.apache.kafka:kafka and org.apache.kafka:kafka-client version 2.5.1.  

 

The symptoms of this problem are very similar to https://issues.apache.org/jira/browse/KAFKA-8803

 

I'm currently running a test that runs one embedded Kafka broker and an embedded Flink cluster (version 1.13). The Flink application uses an exactly once Kafka producer. When initializing, the FlinkKafkaProducer calls [initTransactions|https://github.com/apache/flink/blob/be969dd73b533b03acaba1d81d03b29fccc54bfa/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer.java#L1282-L1285] for each of ~50 underlying KafkaProducers. On some executions of the test (but not all of them), one of these calls to InitProducerId times out. Based on my reading of the logs and some debugging sessions, it seems that the failing producer continually tries to issue FindCoordinatorRequests, but [Sender.awaitNodeReady|https://github.com/apache/kafka/blob/2.5/clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java#L511-L526] returns null. It seems like every time [NetworkClient.leastLoadedNode|https://github.com/apache/kafka/blob/2.5/clients/src/main/java/org/apache/kafka/clients/NetworkClient.java#L662] returns null prints that the one broker "is neither ready for sending or connecting"([source|https://github.com/apache/kafka/blob/2.5/clients/src/main/java/org/apache/kafka/clients/NetworkClient.java#L692]).

I set the producer max.block.ms to be 5mins and you can see in the logs that this goes on and on for the full 5 minutes. During this time, the broker appears healthy (and even serves other requests). It seems that the client is not even attempting to reconnect to the broker during this 5 minutes, though I truthfully don't understand what could be going wrong looking through the code. Do you have any ideas? Any more information I could provide? 




--
This message was sent by Atlassian Jira
(v8.20.7#820007)