You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "jiafu1115 (via GitHub)" <gi...@apache.org> on 2025/11/06 14:58:46 UTC
[PR] MINOR: Add latency optimization guidance to KafkaProducer.send() [kafka]
jiafu1115 opened a new pull request, #20842:
URL: https://github.com/apache/kafka/pull/20842
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Add latency optimization guidance to KafkaProducer.send() [kafka]
Posted by "jiafu1115 (via GitHub)" <gi...@apache.org>.
jiafu1115 commented on code in PR #20842:
URL: https://github.com/apache/kafka/pull/20842#discussion_r2507706333
##########
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java:
##########
@@ -937,8 +937,15 @@ public Future<RecordMetadata> send(ProducerRecord<K, V> record) {
* once the record has been stored in the buffer of records waiting to be sent.
* This allows sending many records in parallel without blocking to wait for the response after each one.
* Can block for the following cases: 1) For the first record being sent to
- * the cluster by this client for the given topic. In this case it will block for up to {@code max.block.ms} milliseconds if
- * Kafka cluster is unreachable; 2) Allocating a buffer if buffer pool doesn't have any free buffers.
+ * the cluster by this client for the given topic. In this case it will block for up to {@code max.block.ms} milliseconds
+ * while waiting for topic's metadata if Kafka cluster is unreachable; 2) Allocating a buffer if buffer pool doesn't
+ * have any free buffers.
+ * <p>
+ * <b>Reducing first-send latency:</b> You can reduce the latency of the first send by preloading the metadata
+ * with {@link #partitionsFor(String)}. However, be aware that metadata cache will be cleared after
+ * {@code metadata.max.idle.ms} of inactivity, so subsequent sends after a long idle period will still
+ * experience delays. To address this, either increase {@code metadata.max.idle.ms} to exceed your maximum
+ * send interval, or periodically call {@link #partitionsFor(String)} to keep the metadata fresh.
Review Comment:
@chia7712 make sense.I try to remove the detailed solution for it to avoid it will be taken as best practice.
##########
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java:
##########
@@ -937,8 +937,15 @@ public Future<RecordMetadata> send(ProducerRecord<K, V> record) {
* once the record has been stored in the buffer of records waiting to be sent.
* This allows sending many records in parallel without blocking to wait for the response after each one.
* Can block for the following cases: 1) For the first record being sent to
- * the cluster by this client for the given topic. In this case it will block for up to {@code max.block.ms} milliseconds if
- * Kafka cluster is unreachable; 2) Allocating a buffer if buffer pool doesn't have any free buffers.
+ * the cluster by this client for the given topic. In this case it will block for up to {@code max.block.ms} milliseconds
+ * while waiting for topic's metadata if Kafka cluster is unreachable; 2) Allocating a buffer if buffer pool doesn't
+ * have any free buffers.
+ * <p>
+ * <b>Reducing first-send latency:</b> You can reduce the latency of the first send by preloading the metadata
+ * with {@link #partitionsFor(String)}. However, be aware that metadata cache will be cleared after
+ * {@code metadata.max.idle.ms} of inactivity, so subsequent sends after a long idle period will still
+ * experience delays. To address this, either increase {@code metadata.max.idle.ms} to exceed your maximum
+ * send interval, or periodically call {@link #partitionsFor(String)} to keep the metadata fresh.
Review Comment:
@chia7712 make sense.I try to remove the detailed solution for it to avoid it will be taken as the best practice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-14768: Add latency optimization guidance to KafkaProducer.send() [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 merged PR #20842:
URL: https://github.com/apache/kafka/pull/20842
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Add latency optimization guidance to KafkaProducer.send() [kafka]
Posted by "jiafu1115 (via GitHub)" <gi...@apache.org>.
jiafu1115 commented on PR #20842:
URL: https://github.com/apache/kafka/pull/20842#issuecomment-3500060352
open another PR to fix the flaky test:
https://github.com/apache/kafka/pull/20845
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Add latency optimization guidance to KafkaProducer.send() [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #20842:
URL: https://github.com/apache/kafka/pull/20842#discussion_r2507648523
##########
clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java:
##########
@@ -937,8 +937,15 @@ public Future<RecordMetadata> send(ProducerRecord<K, V> record) {
* once the record has been stored in the buffer of records waiting to be sent.
* This allows sending many records in parallel without blocking to wait for the response after each one.
* Can block for the following cases: 1) For the first record being sent to
- * the cluster by this client for the given topic. In this case it will block for up to {@code max.block.ms} milliseconds if
- * Kafka cluster is unreachable; 2) Allocating a buffer if buffer pool doesn't have any free buffers.
+ * the cluster by this client for the given topic. In this case it will block for up to {@code max.block.ms} milliseconds
+ * while waiting for topic's metadata if Kafka cluster is unreachable; 2) Allocating a buffer if buffer pool doesn't
+ * have any free buffers.
+ * <p>
+ * <b>Reducing first-send latency:</b> You can reduce the latency of the first send by preloading the metadata
+ * with {@link #partitionsFor(String)}. However, be aware that metadata cache will be cleared after
+ * {@code metadata.max.idle.ms} of inactivity, so subsequent sends after a long idle period will still
+ * experience delays. To address this, either increase {@code metadata.max.idle.ms} to exceed your maximum
+ * send interval, or periodically call {@link #partitionsFor(String)} to keep the metadata fresh.
Review Comment:
the suggestion "or periodically call {@link #partitionsFor(String)} to keep the metadata fresh." is not the "right" pattern to me. The normal producer users should NOT rely on the method "partitionsFor" to keep up-to-date metadata.
Perhaps, we sould highlight the "edge case" you want to remind users instead of sharing the non-normal pattern. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR: Add latency optimization guidance to KafkaProducer.send() [kafka]
Posted by "jiafu1115 (via GitHub)" <gi...@apache.org>.
jiafu1115 commented on PR #20842:
URL: https://github.com/apache/kafka/pull/20842#issuecomment-3507756076
Attach the test result:
Send the same message to the same topic
```
//No preload metadata
long startTime = System.currentTimeMillis();
Future<RecordMetadata> future = producer.send(record);
System.out.println("take " + (System.currentTimeMillis() - startTime ) + "ms" );
```
take 3053ms
```
//Preload the metadata
long startTime = System.currentTimeMillis();
producer.partitionsFor(topic);
System.out.println("take " + (System.currentTimeMillis() - startTime ) + "ms to warmup" );
startTime = System.currentTimeMillis();
Future<RecordMetadata> future = producer.send(record);
System.out.println("take " + (System.currentTimeMillis() - startTime ) + "ms to sent" );
```
take 3137ms to warmup
take 46ms to sent
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org