You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "kirktrue (via GitHub)" <gi...@apache.org> on 2025/07/30 15:39:28 UTC
[PR] KAFKA-19564: Close Consumer in ConsumerPerformance only after metrics displayed [kafka]
kirktrue opened a new pull request, #20267:
URL: https://github.com/apache/kafka/pull/20267
Ensure that metrics are retrieved and displayed (when requested) before `Consumer.close()` is called. This is important because metrics are technically supposed to be removed on `Consumer.close()`, which means retrieving them _after_ `close()` would yield an empty map.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19564: Close Consumer in ConsumerPerformance only after metrics displayed [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #20267:
URL: https://github.com/apache/kafka/pull/20267#discussion_r2318289086
##########
tools/src/main/java/org/apache/kafka/tools/ConsumerPerformance.java:
##########
@@ -66,45 +69,42 @@ public static void main(String[] args) {
if (!options.hideHeader())
printHeader(options.showDetailedStats());
- KafkaConsumer<byte[], byte[]> consumer = new KafkaConsumer<>(options.props());
- long bytesRead = 0L;
- long messagesRead = 0L;
- long lastBytesRead = 0L;
- long lastMessagesRead = 0L;
- long currentTimeMs = System.currentTimeMillis();
- long joinStartMs = currentTimeMs;
- long startMs = currentTimeMs;
- consume(consumer, options, totalMessagesRead, totalBytesRead, joinTimeMs,
- bytesRead, messagesRead, lastBytesRead, lastMessagesRead,
- joinStartMs, joinTimeMsInSingleRound);
- long endMs = System.currentTimeMillis();
-
- Map<MetricName, ? extends Metric> metrics = null;
- if (options.printMetrics())
- metrics = consumer.metrics();
- consumer.close();
-
- // print final stats
- double elapsedSec = (endMs - startMs) / 1_000.0;
- long fetchTimeInMs = (endMs - startMs) - joinTimeMs.get();
- if (!options.showDetailedStats()) {
- double totalMbRead = (totalBytesRead.get() * 1.0) / (1024 * 1024);
- System.out.printf("%s, %s, %.4f, %.4f, %d, %.4f, %d, %d, %.4f, %.4f%n",
- options.dateFormat().format(startMs),
- options.dateFormat().format(endMs),
- totalMbRead,
- totalMbRead / elapsedSec,
- totalMessagesRead.get(),
- totalMessagesRead.get() / elapsedSec,
- joinTimeMs.get(),
- fetchTimeInMs,
- totalMbRead / (fetchTimeInMs / 1000.0),
- totalMessagesRead.get() / (fetchTimeInMs / 1000.0)
- );
- }
+ try (Consumer<byte[], byte[]> consumer = consumerCreator.apply(options.props())) {
+ long bytesRead = 0L;
+ long messagesRead = 0L;
+ long lastBytesRead = 0L;
+ long lastMessagesRead = 0L;
+ long currentTimeMs = System.currentTimeMillis();
+ long joinStartMs = currentTimeMs;
+ long startMs = currentTimeMs;
+ consume(consumer, options, totalMessagesRead, totalBytesRead, joinTimeMs,
+ bytesRead, messagesRead, lastBytesRead, lastMessagesRead,
+ joinStartMs, joinTimeMsInSingleRound);
+ long endMs = System.currentTimeMillis();
+
+ // print final stats
+ double elapsedSec = (endMs - startMs) / 1_000.0;
+ long fetchTimeInMs = (endMs - startMs) - joinTimeMs.get();
+ if (!options.showDetailedStats()) {
+ double totalMbRead = (totalBytesRead.get() * 1.0) / (1024 * 1024);
+ System.out.printf("%s, %s, %.4f, %.4f, %d, %.4f, %d, %d, %.4f, %.4f%n",
+ options.dateFormat().format(startMs),
+ options.dateFormat().format(endMs),
+ totalMbRead,
+ totalMbRead / elapsedSec,
+ totalMessagesRead.get(),
+ totalMessagesRead.get() / elapsedSec,
+ joinTimeMs.get(),
+ fetchTimeInMs,
+ totalMbRead / (fetchTimeInMs / 1000.0),
+ totalMessagesRead.get() / (fetchTimeInMs / 1000.0)
+ );
+ }
- if (metrics != null)
- ToolsUtils.printMetrics(metrics);
+ if (options.printMetrics()) {
Review Comment:
oh, I just noticed that @AndrewJSchofield has already opened https://issues.apache.org/jira/browse/KAFKA-19667 for it. Sorry for the noise :(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19564: Close Consumer in ConsumerPerformance only after metrics displayed [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #20267:
URL: https://github.com/apache/kafka/pull/20267#discussion_r2318276501
##########
tools/src/main/java/org/apache/kafka/tools/ConsumerPerformance.java:
##########
@@ -66,45 +69,42 @@ public static void main(String[] args) {
if (!options.hideHeader())
printHeader(options.showDetailedStats());
- KafkaConsumer<byte[], byte[]> consumer = new KafkaConsumer<>(options.props());
- long bytesRead = 0L;
- long messagesRead = 0L;
- long lastBytesRead = 0L;
- long lastMessagesRead = 0L;
- long currentTimeMs = System.currentTimeMillis();
- long joinStartMs = currentTimeMs;
- long startMs = currentTimeMs;
- consume(consumer, options, totalMessagesRead, totalBytesRead, joinTimeMs,
- bytesRead, messagesRead, lastBytesRead, lastMessagesRead,
- joinStartMs, joinTimeMsInSingleRound);
- long endMs = System.currentTimeMillis();
-
- Map<MetricName, ? extends Metric> metrics = null;
- if (options.printMetrics())
- metrics = consumer.metrics();
- consumer.close();
-
- // print final stats
- double elapsedSec = (endMs - startMs) / 1_000.0;
- long fetchTimeInMs = (endMs - startMs) - joinTimeMs.get();
- if (!options.showDetailedStats()) {
- double totalMbRead = (totalBytesRead.get() * 1.0) / (1024 * 1024);
- System.out.printf("%s, %s, %.4f, %.4f, %d, %.4f, %d, %d, %.4f, %.4f%n",
- options.dateFormat().format(startMs),
- options.dateFormat().format(endMs),
- totalMbRead,
- totalMbRead / elapsedSec,
- totalMessagesRead.get(),
- totalMessagesRead.get() / elapsedSec,
- joinTimeMs.get(),
- fetchTimeInMs,
- totalMbRead / (fetchTimeInMs / 1000.0),
- totalMessagesRead.get() / (fetchTimeInMs / 1000.0)
- );
- }
+ try (Consumer<byte[], byte[]> consumer = consumerCreator.apply(options.props())) {
+ long bytesRead = 0L;
+ long messagesRead = 0L;
+ long lastBytesRead = 0L;
+ long lastMessagesRead = 0L;
+ long currentTimeMs = System.currentTimeMillis();
+ long joinStartMs = currentTimeMs;
+ long startMs = currentTimeMs;
+ consume(consumer, options, totalMessagesRead, totalBytesRead, joinTimeMs,
+ bytesRead, messagesRead, lastBytesRead, lastMessagesRead,
+ joinStartMs, joinTimeMsInSingleRound);
+ long endMs = System.currentTimeMillis();
+
+ // print final stats
+ double elapsedSec = (endMs - startMs) / 1_000.0;
+ long fetchTimeInMs = (endMs - startMs) - joinTimeMs.get();
+ if (!options.showDetailedStats()) {
+ double totalMbRead = (totalBytesRead.get() * 1.0) / (1024 * 1024);
+ System.out.printf("%s, %s, %.4f, %.4f, %d, %.4f, %d, %d, %.4f, %.4f%n",
+ options.dateFormat().format(startMs),
+ options.dateFormat().format(endMs),
+ totalMbRead,
+ totalMbRead / elapsedSec,
+ totalMessagesRead.get(),
+ totalMessagesRead.get() / elapsedSec,
+ joinTimeMs.get(),
+ fetchTimeInMs,
+ totalMbRead / (fetchTimeInMs / 1000.0),
+ totalMessagesRead.get() / (fetchTimeInMs / 1000.0)
+ );
+ }
- if (metrics != null)
- ToolsUtils.printMetrics(metrics);
+ if (options.printMetrics()) {
Review Comment:
Great find. It looks like `ShareConsumerPerformance` has a similar issue. @kirktrue do you have some time to fix it as well
https://github.com/apache/kafka/blob/4271fd8c8b17cc7f31be027a7c25fd31ea11a7d7/tools/src/main/java/org/apache/kafka/tools/ShareConsumerPerformance.java#L86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19564: Close Consumer in ConsumerPerformance only after metrics displayed [kafka]
Posted by "AndrewJSchofield (via GitHub)" <gi...@apache.org>.
AndrewJSchofield merged PR #20267:
URL: https://github.com/apache/kafka/pull/20267
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19564: Close Consumer in ConsumerPerformance only after metrics displayed [kafka]
Posted by "AndrewJSchofield (via GitHub)" <gi...@apache.org>.
AndrewJSchofield commented on code in PR #20267:
URL: https://github.com/apache/kafka/pull/20267#discussion_r2318287119
##########
tools/src/main/java/org/apache/kafka/tools/ConsumerPerformance.java:
##########
@@ -66,45 +69,42 @@ public static void main(String[] args) {
if (!options.hideHeader())
printHeader(options.showDetailedStats());
- KafkaConsumer<byte[], byte[]> consumer = new KafkaConsumer<>(options.props());
- long bytesRead = 0L;
- long messagesRead = 0L;
- long lastBytesRead = 0L;
- long lastMessagesRead = 0L;
- long currentTimeMs = System.currentTimeMillis();
- long joinStartMs = currentTimeMs;
- long startMs = currentTimeMs;
- consume(consumer, options, totalMessagesRead, totalBytesRead, joinTimeMs,
- bytesRead, messagesRead, lastBytesRead, lastMessagesRead,
- joinStartMs, joinTimeMsInSingleRound);
- long endMs = System.currentTimeMillis();
-
- Map<MetricName, ? extends Metric> metrics = null;
- if (options.printMetrics())
- metrics = consumer.metrics();
- consumer.close();
-
- // print final stats
- double elapsedSec = (endMs - startMs) / 1_000.0;
- long fetchTimeInMs = (endMs - startMs) - joinTimeMs.get();
- if (!options.showDetailedStats()) {
- double totalMbRead = (totalBytesRead.get() * 1.0) / (1024 * 1024);
- System.out.printf("%s, %s, %.4f, %.4f, %d, %.4f, %d, %d, %.4f, %.4f%n",
- options.dateFormat().format(startMs),
- options.dateFormat().format(endMs),
- totalMbRead,
- totalMbRead / elapsedSec,
- totalMessagesRead.get(),
- totalMessagesRead.get() / elapsedSec,
- joinTimeMs.get(),
- fetchTimeInMs,
- totalMbRead / (fetchTimeInMs / 1000.0),
- totalMessagesRead.get() / (fetchTimeInMs / 1000.0)
- );
- }
+ try (Consumer<byte[], byte[]> consumer = consumerCreator.apply(options.props())) {
+ long bytesRead = 0L;
+ long messagesRead = 0L;
+ long lastBytesRead = 0L;
+ long lastMessagesRead = 0L;
+ long currentTimeMs = System.currentTimeMillis();
+ long joinStartMs = currentTimeMs;
+ long startMs = currentTimeMs;
+ consume(consumer, options, totalMessagesRead, totalBytesRead, joinTimeMs,
+ bytesRead, messagesRead, lastBytesRead, lastMessagesRead,
+ joinStartMs, joinTimeMsInSingleRound);
+ long endMs = System.currentTimeMillis();
+
+ // print final stats
+ double elapsedSec = (endMs - startMs) / 1_000.0;
+ long fetchTimeInMs = (endMs - startMs) - joinTimeMs.get();
+ if (!options.showDetailedStats()) {
+ double totalMbRead = (totalBytesRead.get() * 1.0) / (1024 * 1024);
+ System.out.printf("%s, %s, %.4f, %.4f, %d, %.4f, %d, %d, %.4f, %.4f%n",
+ options.dateFormat().format(startMs),
+ options.dateFormat().format(endMs),
+ totalMbRead,
+ totalMbRead / elapsedSec,
+ totalMessagesRead.get(),
+ totalMessagesRead.get() / elapsedSec,
+ joinTimeMs.get(),
+ fetchTimeInMs,
+ totalMbRead / (fetchTimeInMs / 1000.0),
+ totalMessagesRead.get() / (fetchTimeInMs / 1000.0)
+ );
+ }
- if (metrics != null)
- ToolsUtils.printMetrics(metrics);
+ if (options.printMetrics()) {
Review Comment:
I'm already on it. https://issues.apache.org/jira/browse/KAFKA-19667
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org