[jira] [Commented] (KAFKA-17088) REQUEST_TIMED_OUT occurs intermittently in the kafka Producer client

["Arkadiusz Dankiewicz (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-17088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17881713#comment-17881713 ] 

Arkadiusz Dankiewicz commented on KAFKA-17088:
----------------------------------------------

Hi [~janardhanag],

Regarding the intermittent timeout issue you are facing when sending messages to the Kafka broker, this may be due to certain conditions or configurations in the Kafka cluster. Producer timeouts can occur during Kafka cluster activities like leader *rebalancing* or broker {*}failures{*}. In such cases, the producer may experience delays or disconnects, causing request timeouts, as observed in the error.

To better understand and resolve this issue, it would be helpful to know more about your Kafka cluster setup. Specifically:
 # The size of the Kafka cluster (number of brokers and partitions).
 # Details on any ongoing leader rebalancing or partition changes (Kafka cluster logs, there might be corelation between cluster rebalance/failures and received producer timeouts)

Additionally, I suggest tweaking a few producer configurations to mitigate the timeouts. You might want to experiment with the following settings:
 * {*}{{request.timeout.ms}}{*}: Consider increasing this value from the default 30s to give the producer more time before timing out.
 * {*}{{delivery.timeout.ms}}{*}: This controls the total time spent retrying the message delivery, and has to be greater than *{{request.timeout.ms}}* and *{{linger.ms}}* combined
 * {*}{{linger.ms}}{*}: Setting a small delay here (e.g., {{{}5-10 ms{}}}) can help batch messages more efficiently and reduce the frequency of network calls.

These adjustments can help improve message delivery reliability and reduce the occurrence of timeouts.

Let us know your cluster details or any further observations, and we can continue troubleshooting.

> REQUEST_TIMED_OUT occurs intermittently in the kafka Producer client 
> ---------------------------------------------------------------------
>
>                 Key: KAFKA-17088
>                 URL: https://issues.apache.org/jira/browse/KAFKA-17088
>             Project: Kafka
>          Issue Type: Bug
>          Components: clients, producer 
>    Affects Versions: 3.5.1
>            Reporter: Janardhana Gopalachar
>            Priority: Major
>
> Hi 
> We observe that producer receives a request timeout ( intermittently) when trying to send message to kafka broker. Below is the properties set for kafka producer. 
> producerProps.put("acks", "all");
> producerProps.put("linger.ms", 0);
> producerProps.put("max.block.ms", 5000);
> producerProps.put("metadata.max.idle.ms", 5000); producerProps.put("delivery.timeout.ms", 10000); producerProps.put("request.timeout.ms", 1000);
> producerProps.put("key.serializer", BYTE_SERIALIZER);
> producerProps.put("value.serializer", BYTE_SERIALIZER);
>  
>  
> we receive below message intermittently. We need to know the reaon for this timeout.
> _[kafka-producer-network-thread | producer-1] o.a.k.c.u.LogContext$LocationAwareKafkaLogger:434 writeLog [Producer clientId=producer-1] Got error produce response with correlation id 231972 on topic-partition {*}health_check_topic_msg2-0{*}, retrying (2147483646 attempts left). Error: REQUEST_TIMED_OUT. Error Message: Disconnected from node 1 due to timeout_



--
This message was sent by Atlassian Jira
(v8.20.10#820010)