You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2022/06/01 12:23:10 UTC
[GitHub] [kafka] ableegoldman opened a new pull request, #12235: KAFKA-13945: add bytes/records consumed and produced metrics
ableegoldman opened a new pull request, #12235:
URL: https://github.com/apache/kafka/pull/12235
Implementation of [KIP-846: Source/sink node metrics for Consumed/Produced throughput in Streams](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211886093)
Adds the following INFO node-level metrics for the total bytes/records consumed and produced:
- bytes-consumed-total
- records-consumed-total
- bytes-produced-total
- records-produced-total
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] guozhangwang commented on a diff in pull request #12235: KAFKA-13945: add bytes/records consumed and produced metrics
Posted by GitBox <gi...@apache.org>.
guozhangwang commented on code in PR #12235:
URL: https://github.com/apache/kafka/pull/12235#discussion_r887401929
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/ClientUtils.java:
##########
@@ -166,4 +169,28 @@ public static String extractThreadId(final String fullThreadName) {
final int index = fullThreadName.indexOf("StreamThread-");
return fullThreadName.substring(index);
}
+
+ public static long recordSizeInBytes(final long keyBytes,
Review Comment:
Could we use it in PartitionGroupTest#getBytesBufferedForRawRecords as well?
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/SinkNode.java:
##########
@@ -82,7 +103,11 @@ public void process(final Record<KIn, VIn> record) {
final String topic = topicExtractor.extract(key, value, contextForExtraction);
- collector.send(topic, key, value, record.headers(), timestamp, keySerializer, valSerializer, partitioner);
+ final long bytesProduced =
+ collector.send(topic, key, value, record.headers(), timestamp, keySerializer, valSerializer, partitioner);
+
+ bytesProducedSensor.record(bytesProduced, context.currentSystemTimeMs());
Review Comment:
A minor suggestion: I think we can use one sensor that contains two metrics, one as "sum" and one as "count", and then call `record` here once on that sensor which would update both metrics. Similar for consumed.
Examples can be found in those `addInvocationRateAndCountToSensor` etc functions.
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamTask.java:
##########
@@ -184,7 +184,7 @@ public StreamTask(final TaskId id,
createPartitionQueues(),
mainConsumer::currentLag,
TaskMetrics.recordLatenessSensor(threadId, taskId, streamsMetrics),
- TaskMetrics.totalBytesSensor(threadId, taskId, streamsMetrics),
+ TaskMetrics.totalInputBufferBytesSensor(threadId, taskId, streamsMetrics),
Review Comment:
+1
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/metrics/ProcessorNodeMetrics.java:
##########
@@ -136,6 +154,86 @@ public static Sensor processAtSourceSensor(final String threadId,
);
}
+ public static Sensor bytesConsumedSensor(final String threadId,
+ final String taskId,
+ final String processorNodeId,
+ final String topic,
+ final StreamsMetricsImpl streamsMetrics) {
+ final String sensorName = processorNodeId + "-" + BYTES_CONSUMED;
+ final Sensor sensor =
+ streamsMetrics.nodeLevelSensor(threadId, taskId, processorNodeId, sensorName, RecordingLevel.INFO);
+ final Map<String, String> tagMap = new HashMap<>(streamsMetrics.nodeLevelTagMap(threadId, taskId, processorNodeId));
+ tagMap.put("topic", topic);
+ addSumMetricToSensor(
+ sensor,
+ PROCESSOR_NODE_LEVEL_GROUP,
+ tagMap,
+ BYTES_CONSUMED,
+ BYTES_CONSUMED_TOTAL_DESCRIPTION
+ );
+ return sensor;
+ }
+
+ public static Sensor recordsConsumedSensor(final String threadId,
+ final String taskId,
+ final String processorNodeId,
+ final String topic,
+ final StreamsMetricsImpl streamsMetrics) {
+ final String sensorName = processorNodeId + "-" + RECORDS_CONSUMED;
+ final Sensor sensor =
+ streamsMetrics.nodeLevelSensor(threadId, taskId, processorNodeId, sensorName, RecordingLevel.INFO);
+ final Map<String, String> tagMap = new HashMap<>(streamsMetrics.nodeLevelTagMap(threadId, taskId, processorNodeId));
+ tagMap.put("topic", topic);
+ addSumMetricToSensor(
Review Comment:
Since we always record `1` for this sensor, I think we can combine this and the bytes metrics in a single sensor with `sum` and `count` (see my other comment above). Ditto for produced.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna merged pull request #12235: KAFKA-13945: add bytes/records consumed and produced metrics
Posted by GitBox <gi...@apache.org>.
cadonna merged PR #12235:
URL: https://github.com/apache/kafka/pull/12235
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on a diff in pull request #12235: KAFKA-13945: add bytes/records consumed and produced metrics
Posted by GitBox <gi...@apache.org>.
cadonna commented on code in PR #12235:
URL: https://github.com/apache/kafka/pull/12235#discussion_r887767497
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/ClientUtils.java:
##########
@@ -166,4 +169,28 @@ public static String extractThreadId(final String fullThreadName) {
final int index = fullThreadName.indexOf("StreamThread-");
return fullThreadName.substring(index);
}
+
+ public static long recordSizeInBytes(final long keyBytes,
Review Comment:
Could you add unit tests for this code?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a diff in pull request #12235: KAFKA-13945: add bytes/records consumed and produced metrics
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on code in PR #12235:
URL: https://github.com/apache/kafka/pull/12235#discussion_r896155831
##########
streams/src/test/java/org/apache/kafka/test/MockRecordCollector.java:
##########
@@ -46,13 +47,15 @@ public <K, V> void send(final String topic,
final Integer partition,
final Long timestamp,
final Serializer<K> keySerializer,
- final Serializer<V> valueSerializer) {
+ final Serializer<V> valueSerializer,
+ final String processorNodeId,
+ final InternalProcessorContext<Void, Void> context) {
collected.add(new ProducerRecord<>(topic,
- partition,
- timestamp,
- key,
- value,
- headers));
+ partition,
+ timestamp,
+ key,
+ value,
+ headers));
Review Comment:
Ah my IDE probably autoformatted it - will fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a diff in pull request #12235: KAFKA-13945: add bytes/records consumed and produced metrics
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on code in PR #12235:
URL: https://github.com/apache/kafka/pull/12235#discussion_r901061337
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/RecordCollectorImpl.java:
##########
@@ -192,6 +217,28 @@ public <K, V> void send(final String topic,
} else {
log.warn("Received offset={} in produce response for {}", metadata.offset(), tp);
}
+
+ if (!topic.endsWith("-changelog")) {
+ // we may not have created a sensor yet if the node uses dynamic topic routing
Review Comment:
No, it does refer to the `if` branch -- if `sinkNodeToProducedSensorByTopic.get` returns null, that means we haven't seen this topic yet/constructed a sensor for it during initialization.
I do see how it might be confusing if it's interpreted as referring to the line above, and I do know your preference, but I stand by this particular comment -- it follows the rule of explaining WHY we do something rather than WHAT, and I believe in this case the why is very much not obvious* and worry someone reading this code in the future might think this branch would actually indicate a bug and decide to throw an exception or something. So I'll adjust the spacing to clarify which line it's referring to but I would like to keep it.
> I believe in this case the why is very much not obvious
\* Source: I myself did not remember about dynamic topics and only realized this was possible when looking at some TTD tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on a diff in pull request #12235: KAFKA-13945: add bytes/records consumed and produced metrics
Posted by GitBox <gi...@apache.org>.
cadonna commented on code in PR #12235:
URL: https://github.com/apache/kafka/pull/12235#discussion_r887131106
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/metrics/ProcessorNodeMetricsTest.java:
##########
@@ -111,7 +111,7 @@ public void shouldGetProcessAtSourceSensor() {
expectedParentSensor
);
- verifySensor(() -> ProcessorNodeMetrics.processAtSourceSensor(THREAD_ID, TASK_ID, PROCESSOR_NODE_ID, streamsMetrics));
+ verifySensor(() -> ProcessorNodeMetrics.recordsProcessedAtSourceSensor(THREAD_ID, TASK_ID, PROCESSOR_NODE_ID, streamsMetrics));
Review Comment:
`recordsProcessedAtSourceSensor()` does not exist and causes a compilation error.
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/metrics/ProcessorNodeMetrics.java:
##########
@@ -136,6 +153,82 @@ public static Sensor processAtSourceSensor(final String threadId,
);
}
+ public static Sensor bytesConsumedSensor(final String threadId,
Review Comment:
Could you please add unit tests?
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/ProcessorNode.java:
##########
@@ -40,12 +40,12 @@
private final Processor<KIn, VIn, KOut, VOut> processor;
private final FixedKeyProcessor<KIn, VIn, VOut> fixedKeyProcessor;
private final String name;
- private final Time time;
+ protected final Time time;
public final Set<String> stateStores;
- private InternalProcessorContext<KOut, VOut> internalProcessorContext;
- private String threadId;
+ protected InternalProcessorContext<KOut, VOut> internalProcessorContext;
Review Comment:
I do not think you need to do this. You can simply use the context that is passed in into the `init()` method of `SinkNode`.
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/SinkNode.java:
##########
@@ -82,7 +107,12 @@ public void process(final Record<KIn, VIn> record) {
final String topic = topicExtractor.extract(key, value, contextForExtraction);
- collector.send(topic, key, value, record.headers(), timestamp, keySerializer, valSerializer, partitioner);
+ final int bytesProduced =
+ collector.send(topic, key, value, record.headers(), timestamp, keySerializer, valSerializer, partitioner);
+
+ // Just use the cached system time to avoid the clock lookup
Review Comment:
You can also remove the comment.
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/SinkNode.java:
##########
@@ -56,6 +61,26 @@ public void addChild(final ProcessorNode<Void, Void, ?, ?> child) {
@Override
public void init(final InternalProcessorContext<Void, Void> context) {
+ // It is important to first create the sensor before calling init on the
+ // parent object. Otherwise due to backwards compatibility an empty sensor
+ // without parent is created with the same name.
+ // Once the backwards compatibility is not needed anymore it might be possible to
+ // change this.
Review Comment:
Nice, I just realized that you copied that comment from code that I wrote! 🙂
I guess the backwards compatibility was the one with the old metrics structure that we changed in KIP-444. We removed the old structure in 3.0, so I guess that this is an instance of comments that started to lie.
Moreover, from where you copied the comment (I guess it was `SourceNode`) the sensor `processAtSourceSensor` has indeed a parent. I think the comment does not make sense in this class and we need to verify if it still makes sense in the other class. That does not need to be verifed in this PR.
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/ProcessorNode.java:
##########
@@ -40,12 +40,12 @@
private final Processor<KIn, VIn, KOut, VOut> processor;
private final FixedKeyProcessor<KIn, VIn, VOut> fixedKeyProcessor;
private final String name;
- private final Time time;
+ protected final Time time;
Review Comment:
Is this really needed? Could not find where is is used outside `ProcessorNode`.
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/SinkNode.java:
##########
@@ -56,6 +61,26 @@ public void addChild(final ProcessorNode<Void, Void, ?, ?> child) {
@Override
public void init(final InternalProcessorContext<Void, Void> context) {
+ // It is important to first create the sensor before calling init on the
+ // parent object. Otherwise due to backwards compatibility an empty sensor
+ // without parent is created with the same name.
+ // Once the backwards compatibility is not needed anymore it might be possible to
+ // change this.
Review Comment:
I actually do not understand this comment. Backwards compatibility of what? What does "empty sensor without parent" mean? `recordsProducedSensor()` and `bytesProducedSensor()` do not create sensors with parents.
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/ProcessorNode.java:
##########
@@ -40,12 +40,12 @@
private final Processor<KIn, VIn, KOut, VOut> processor;
private final FixedKeyProcessor<KIn, VIn, VOut> fixedKeyProcessor;
private final String name;
- private final Time time;
+ protected final Time time;
public final Set<String> stateStores;
- private InternalProcessorContext<KOut, VOut> internalProcessorContext;
- private String threadId;
+ protected InternalProcessorContext<KOut, VOut> internalProcessorContext;
+ protected String threadId;
Review Comment:
Also this does not seem to be strictly needed since in `SinkNode` you could use `Thread.currentThread().getName()` as in `init()` of `SourceNode`. I even think you need to use `Thread.currentThread().getName()` in `SinkNode` since you use `threadId` before the variable is set in `super.init()`.
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/RecordQueue.java:
##########
@@ -71,6 +74,20 @@ public class RecordQueue {
processorContext.taskId().toString(),
processorContext.metrics()
);
+ bytesConsumedSensor = ProcessorNodeMetrics.bytesConsumedSensor(
+ Thread.currentThread().getName(),
Review Comment:
Could you store the thread name in a local variable and use that variable in all sensor retrieval methods?
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/SinkNode.java:
##########
@@ -82,7 +107,12 @@ public void process(final Record<KIn, VIn> record) {
final String topic = topicExtractor.extract(key, value, contextForExtraction);
- collector.send(topic, key, value, record.headers(), timestamp, keySerializer, valSerializer, partitioner);
+ final int bytesProduced =
+ collector.send(topic, key, value, record.headers(), timestamp, keySerializer, valSerializer, partitioner);
+
+ // Just use the cached system time to avoid the clock lookup
Review Comment:
I think it is fine as you do it since we do it similarly in other places like `processAtSourceSensor.record(1.0d, context.currentSystemTimeMs());` in `SourceNode`.
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/RecordCollectorImpl.java:
##########
@@ -199,6 +199,7 @@ public <K, V> void send(final String topic,
log.trace("Failed record: (key {} value {} timestamp {}) topic=[{}] partition=[{}]", key, value, timestamp, topic, partition);
}
});
+ return keyBytes.length + valBytes.length;
Review Comment:
Should the metric record the actually produced bytes or the bytes that were passed to the producer for sending? Here we would record the latter. The value of the metric might be higher than the actually produced bytes to the output topic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a diff in pull request #12235: KAFKA-13945: add bytes/records consumed and produced metrics
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on code in PR #12235:
URL: https://github.com/apache/kafka/pull/12235#discussion_r888672094
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/RecordCollectorImpl.java:
##########
@@ -199,6 +201,7 @@ public <K, V> void send(final String topic,
log.trace("Failed record: (key {} value {} timestamp {}) topic=[{}] partition=[{}]", key, value, timestamp, topic, partition);
}
});
+ return recordSizeInBytes(keyBytes == null ? 0 : keyBytes.length, valBytes == null ? 0 : valBytes.length, topic, headers);
Review Comment:
Yes that's a good point -- I fixed this up by moving the sensors to the `RecordCollectorImpl` class and recording the metric in the callback of the Producer's `send` method only upon success. Thanks for clarifying
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on a diff in pull request #12235: KAFKA-13945: add bytes/records consumed and produced metrics
Posted by GitBox <gi...@apache.org>.
cadonna commented on code in PR #12235:
URL: https://github.com/apache/kafka/pull/12235#discussion_r887767497
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/ClientUtils.java:
##########
@@ -166,4 +169,28 @@ public static String extractThreadId(final String fullThreadName) {
final int index = fullThreadName.indexOf("StreamThread-");
return fullThreadName.substring(index);
}
+
+ public static long recordSizeInBytes(final long keyBytes,
Review Comment:
Could you add unit tests for this method?
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/RecordCollectorImpl.java:
##########
@@ -199,6 +201,7 @@ public <K, V> void send(final String topic,
log.trace("Failed record: (key {} value {} timestamp {}) topic=[{}] partition=[{}]", key, value, timestamp, topic, partition);
}
});
+ return recordSizeInBytes(keyBytes == null ? 0 : keyBytes.length, valBytes == null ? 0 : valBytes.length, topic, headers);
Review Comment:
Should we move the `null` checks into `recordSizeInBytes()` so that we have all `null`-checks in one place?
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/RecordQueue.java:
##########
@@ -145,8 +141,11 @@ int addRawRecords(final Iterable<ConsumerRecord<byte[], byte[]>> rawRecords) {
*
* @return StampedRecord
*/
- public StampedRecord poll() {
+ public StampedRecord poll(final long wallClockTime) {
final StampedRecord recordToReturn = headRecord;
+
+ consumedSensor.record(headRecordSizeInBytes, wallClockTime);
Review Comment:
Could you please add unit tests for this?
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/metrics/StreamsMetricsImpl.java:
##########
@@ -795,6 +795,23 @@ public static void addAvgAndSumMetricsToSensor(final Sensor sensor,
);
}
+ public static void addTotalCountAndSumMetricsToSensor(final Sensor sensor,
Review Comment:
Could you please add unit tests for this method?
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/metrics/ProcessorNodeMetricsTest.java:
##########
@@ -114,6 +116,63 @@ public void shouldGetProcessAtSourceSensor() {
verifySensor(() -> ProcessorNodeMetrics.processAtSourceSensor(THREAD_ID, TASK_ID, PROCESSOR_NODE_ID, streamsMetrics));
}
+ @Test
+ public void shouldGetRecordsAndBytesConsumedSensor() {
+ final String recordsMetricNamePrefix = "records-consumed";
+ final String bytesMetricNamePrefix = "bytes-consumed";
+ final String descriptionOfRecordsTotal = "The total number of records consumed from this topic";
+ final String descriptionOfBytesTotal = "The total number of bytes consumed from this topic";
+ when(streamsMetrics.nodeLevelSensor(THREAD_ID, TASK_ID, PROCESSOR_NODE_ID, recordsMetricNamePrefix, RecordingLevel.INFO))
+ .thenReturn(expectedSensor);
+ when(streamsMetrics.nodeLevelSensor(THREAD_ID, TASK_ID, PROCESSOR_NODE_ID, bytesMetricNamePrefix, RecordingLevel.INFO))
+ .thenReturn(expectedSensor);
+ when(streamsMetrics.nodeLevelTagMap(THREAD_ID, TASK_ID, PROCESSOR_NODE_ID)).thenReturn(tagMap);
+
+ final Map<String, String> consumedTagMap = new HashMap<>(tagMap);
+ consumedTagMap.put("topic", TOPIC_NAME);
+ StreamsMetricsImpl.addTotalCountAndSumMetricsToSensor(
Review Comment:
Apparently, there was a miss in transforming the tests in this class from using `PowerMock`/`EasyMock` to using `Mockito`. With `PowerMock`/`EasyMock` this call verifies that the method is called on the mock. With `Mockito` this call just calls the method on the actual class without any verifications. You need to do the following to get this right for your tests:
1. Add a static mock:
```
private final MockedStatic<StreamsMetricsImpl> streamsMetricsMockedStatic = mockStatic(StreamsMetricsImpl.class);
private final StreamsMetricsImpl streamsMetrics = mock(StreamsMetricsImpl.class); // This is already there
```
2. Change your test to:
```
public void shouldGetRecordsAndBytesConsumedSensor() {
final String recordsMetricNamePrefix = "records-consumed";
final String bytesMetricNamePrefix = "bytes-consumed";
final String descriptionOfRecordsTotal = "The total number of records consumed from this topic";
final String descriptionOfBytesTotal = "The total number of bytes consumed from this topic";
when(streamsMetrics.nodeLevelSensor(THREAD_ID, TASK_ID, PROCESSOR_NODE_ID, recordsMetricNamePrefix, RecordingLevel.INFO))
.thenReturn(expectedSensor);
when(streamsMetrics.nodeLevelSensor(THREAD_ID, TASK_ID, PROCESSOR_NODE_ID, bytesMetricNamePrefix, RecordingLevel.INFO))
.thenReturn(expectedSensor);
when(streamsMetrics.nodeLevelTagMap(THREAD_ID, TASK_ID, PROCESSOR_NODE_ID)).thenReturn(tagMap);
verifySensor(
() -> ProcessorNodeMetrics.consumedSensor(THREAD_ID, TASK_ID, PROCESSOR_NODE_ID, TOPIC_NAME, streamsMetrics)
);
final Map<String, String> consumedTagMap = new HashMap<>(tagMap);
consumedTagMap.put("topic", TOPIC_NAME);
streamsMetricsMockedStatic.verify(() -> addTotalCountAndSumMetricsToSensor(
expectedSensor,
PROCESSOR_NODE_LEVEL_GROUP,
consumedTagMap,
recordsMetricNamePrefix,
bytesMetricNamePrefix,
descriptionOfRecordsTotal,
descriptionOfBytesTotal
));
}
```
We can fix the other tests in a separate PR.
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/RecordCollectorImpl.java:
##########
@@ -199,6 +201,7 @@ public <K, V> void send(final String topic,
log.trace("Failed record: (key {} value {} timestamp {}) topic=[{}] partition=[{}]", key, value, timestamp, topic, partition);
}
});
+ return recordSizeInBytes(keyBytes == null ? 0 : keyBytes.length, valBytes == null ? 0 : valBytes.length, topic, headers);
Review Comment:
I think in my previous comment we misunderstood each other. I was asking what we want to record with this metric. Do we want to record the records that are passed to the producer or do we want to measure the records actually sent to the output topic? As far as I understand there is no guarantee that a record passed to a producer is indeed sent to the output topic. There could be a crash in between passing the record to the consumer and sending the record to the output topic.
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/SinkNode.java:
##########
@@ -82,7 +89,19 @@ public void process(final Record<KIn, VIn> record) {
final String topic = topicExtractor.extract(key, value, contextForExtraction);
- collector.send(topic, key, value, record.headers(), timestamp, keySerializer, valSerializer, partitioner);
+ final long bytesProduced =
+ collector.send(topic, key, value, record.headers(), timestamp, keySerializer, valSerializer, partitioner);
+
+ producedSensorByTopic.putIfAbsent(
+ topic,
+ ProcessorNodeMetrics.producedSensor(
+ Thread.currentThread().getName(),
+ context.taskId().toString(),
+ name(),
+ topic,
+ context.metrics()
+ ));
+ producedSensorByTopic.get(topic).record(bytesProduced, context.currentSystemTimeMs());
Review Comment:
Could you please add unit tests for this code?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a diff in pull request #12235: KAFKA-13945: add bytes/records consumed and produced metrics
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on code in PR #12235:
URL: https://github.com/apache/kafka/pull/12235#discussion_r901061337
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/RecordCollectorImpl.java:
##########
@@ -192,6 +217,28 @@ public <K, V> void send(final String topic,
} else {
log.warn("Received offset={} in produce response for {}", metadata.offset(), tp);
}
+
+ if (!topic.endsWith("-changelog")) {
+ // we may not have created a sensor yet if the node uses dynamic topic routing
Review Comment:
~~No, it does refer to the `if` branch -- if `sinkNodeToProducedSensorByTopic.get` returns null, that means we haven't seen this topic yet/constructed a sensor for it during initialization.~~
edit: this is wrong, see following comment
I do see how it might be confusing if it's interpreted as referring to the line above, and I do know your preference, but I stand by this particular comment -- it follows the rule of explaining WHY we do something rather than WHAT, and I believe in this case the why is very much not obvious* and worry someone reading this code in the future might think this branch would actually indicate a bug and decide to throw an exception or something. So I'll adjust the spacing to clarify which line it's referring to but I would like to keep it.
> I believe in this case the why is very much not obvious
\* Source: I myself did not remember about dynamic topics and only realized this was possible when looking at some TTD tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on a diff in pull request #12235: KAFKA-13945: add bytes/records consumed and produced metrics
Posted by GitBox <gi...@apache.org>.
cadonna commented on code in PR #12235:
URL: https://github.com/apache/kafka/pull/12235#discussion_r891064878
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/ClientUtils.java:
##########
@@ -166,4 +171,46 @@ public static String extractThreadId(final String fullThreadName) {
final int index = fullThreadName.indexOf("StreamThread-");
return fullThreadName.substring(index);
}
+
+ public static long producerRecordSizeInBytes(final ProducerRecord<byte[], byte[]> record) {
+ return recordSizeInBytes(
+ record.key().length,
+ record.value() == null ? 0 : record.value().length,
+ record.topic(),
+ record.headers()
+ );
+ }
+
+ public static long consumerRecordSizeInBytes(final ConsumerRecord<byte[], byte[]> record) {
+ return recordSizeInBytes(
+ record.serializedKeySize(),
+ record.serializedValueSize(),
+ record.topic(),
+ record.headers()
+ );
+ }
+
+ public static long recordSizeInBytes(final long keyBytes,
Review Comment:
nit: Could you make this private?
##########
streams/src/test/java/org/apache/kafka/test/MockRecordCollector.java:
##########
@@ -46,13 +47,15 @@ public <K, V> void send(final String topic,
final Integer partition,
final Long timestamp,
final Serializer<K> keySerializer,
- final Serializer<V> valueSerializer) {
+ final Serializer<V> valueSerializer,
+ final String processorNodeId,
+ final InternalProcessorContext<Void, Void> context) {
collected.add(new ProducerRecord<>(topic,
- partition,
- timestamp,
- key,
- value,
- headers));
+ partition,
+ timestamp,
+ key,
+ value,
+ headers));
Review Comment:
nit: The indentation was actually right since it is a method call and not a method declaration.
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/RecordCollectorImpl.java:
##########
@@ -192,6 +217,28 @@ public <K, V> void send(final String topic,
} else {
log.warn("Received offset={} in produce response for {}", metadata.offset(), tp);
}
+
+ if (!topic.endsWith("-changelog")) {
+ // we may not have created a sensor yet if the node uses dynamic topic routing
Review Comment:
This comment is a bit misleading here. AFAIU it refers to the `else`-branch. Please move it or remove it. I think you know my preference 🙂.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a diff in pull request #12235: KAFKA-13945: add bytes/records consumed and produced metrics
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on code in PR #12235:
URL: https://github.com/apache/kafka/pull/12235#discussion_r890725391
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/metrics/ProcessorNodeMetricsTest.java:
##########
@@ -114,6 +116,63 @@ public void shouldGetProcessAtSourceSensor() {
verifySensor(() -> ProcessorNodeMetrics.processAtSourceSensor(THREAD_ID, TASK_ID, PROCESSOR_NODE_ID, streamsMetrics));
}
+ @Test
+ public void shouldGetRecordsAndBytesConsumedSensor() {
+ final String recordsMetricNamePrefix = "records-consumed";
+ final String bytesMetricNamePrefix = "bytes-consumed";
+ final String descriptionOfRecordsTotal = "The total number of records consumed from this topic";
+ final String descriptionOfBytesTotal = "The total number of bytes consumed from this topic";
+ when(streamsMetrics.nodeLevelSensor(THREAD_ID, TASK_ID, PROCESSOR_NODE_ID, recordsMetricNamePrefix, RecordingLevel.INFO))
+ .thenReturn(expectedSensor);
+ when(streamsMetrics.nodeLevelSensor(THREAD_ID, TASK_ID, PROCESSOR_NODE_ID, bytesMetricNamePrefix, RecordingLevel.INFO))
+ .thenReturn(expectedSensor);
+ when(streamsMetrics.nodeLevelTagMap(THREAD_ID, TASK_ID, PROCESSOR_NODE_ID)).thenReturn(tagMap);
+
+ final Map<String, String> consumedTagMap = new HashMap<>(tagMap);
+ consumedTagMap.put("topic", TOPIC_NAME);
+ StreamsMetricsImpl.addTotalCountAndSumMetricsToSensor(
Review Comment:
Oh man I had no idea you had to close static mocks...not blaming you for this at all @cadonna but in the future, assume I know nothing about Mockito/PowerMock and put everything out on the table! lol 🙃
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on pull request #12235: KAFKA-13945: add bytes/records consumed and produced metrics
Posted by GitBox <gi...@apache.org>.
cadonna commented on PR #12235:
URL: https://github.com/apache/kafka/pull/12235#issuecomment-1148710618
The test failures are unrelated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a diff in pull request #12235: KAFKA-13945: add bytes/records consumed and produced metrics
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on code in PR #12235:
URL: https://github.com/apache/kafka/pull/12235#discussion_r889358327
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/metrics/StreamsMetricsImpl.java:
##########
@@ -388,6 +401,40 @@ private String nodeSensorPrefix(final String threadId, final String taskId, fina
+ SENSOR_PREFIX_DELIMITER + SENSOR_NODE_LABEL + SENSOR_PREFIX_DELIMITER + processorNodeName;
}
+ public Sensor topicLevelSensor(final String threadId,
+ final String taskId,
+ final String processorNodeName,
+ final String topicName,
Review Comment:
I did update it to include the `topic` tag already, but I haven't updated it yet to clarify that we are introducing these at a new "topic-level" metrics scope. I'll do that right now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a diff in pull request #12235: KAFKA-13945: add bytes/records consumed and produced metrics
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on code in PR #12235:
URL: https://github.com/apache/kafka/pull/12235#discussion_r888670820
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/RecordCollectorImpl.java:
##########
@@ -199,6 +201,7 @@ public <K, V> void send(final String topic,
log.trace("Failed record: (key {} value {} timestamp {}) topic=[{}] partition=[{}]", key, value, timestamp, topic, partition);
}
});
+ return recordSizeInBytes(keyBytes == null ? 0 : keyBytes.length, valBytes == null ? 0 : valBytes.length, topic, headers);
Review Comment:
I did it like this to avoid the extra/unnecessary null check for consumer records specifically, which already track the serialized size in bytes unlike the producer record. And unfortunately they don't inherit from a common interface/class -- but I added separate middle-man methods to handle them and moved the null check for the producer case there, should be addressed now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] guozhangwang commented on a diff in pull request #12235: KAFKA-13945: add bytes/records consumed and produced metrics
Posted by GitBox <gi...@apache.org>.
guozhangwang commented on code in PR #12235:
URL: https://github.com/apache/kafka/pull/12235#discussion_r889168252
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/metrics/StreamsMetricsImpl.java:
##########
@@ -388,6 +401,40 @@ private String nodeSensorPrefix(final String threadId, final String taskId, fina
+ SENSOR_PREFIX_DELIMITER + SENSOR_NODE_LABEL + SENSOR_PREFIX_DELIMITER + processorNodeName;
}
+ public Sensor topicLevelSensor(final String threadId,
+ final String taskId,
+ final String processorNodeName,
+ final String topicName,
Review Comment:
I cannot access the apache wiki atm (seems down temporarily) but just double checking is topic name part of the tag in the KIP proposal as well? If not we should update the wiki and the KIP thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a diff in pull request #12235: KAFKA-13945: add bytes/records consumed and produced metrics
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on code in PR #12235:
URL: https://github.com/apache/kafka/pull/12235#discussion_r901170933
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/RecordCollectorImpl.java:
##########
@@ -192,6 +217,28 @@ public <K, V> void send(final String topic,
} else {
log.warn("Received offset={} in produce response for {}", metadata.offset(), tp);
}
+
+ if (!topic.endsWith("-changelog")) {
+ // we may not have created a sensor yet if the node uses dynamic topic routing
Review Comment:
Oh wait sorry I'm being dumb, yes, it does refer to the `else` case -- I've moved it accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a diff in pull request #12235: KAFKA-13945: add bytes/records consumed and produced metrics
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on code in PR #12235:
URL: https://github.com/apache/kafka/pull/12235#discussion_r886760998
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/SinkNode.java:
##########
@@ -82,7 +107,12 @@ public void process(final Record<KIn, VIn> record) {
final String topic = topicExtractor.extract(key, value, contextForExtraction);
- collector.send(topic, key, value, record.headers(), timestamp, keySerializer, valSerializer, partitioner);
+ final int bytesProduced =
+ collector.send(topic, key, value, record.headers(), timestamp, keySerializer, valSerializer, partitioner);
+
+ // Just use the cached system time to avoid the clock lookup
Review Comment:
Optimization to hopefully bypass any potential concern or risk of performance regression -- it may be paranoid or an over-optimization at this point but imo it doesn't hurt to report these metrics at a slightly more course-grained time interval. Happy to go either way on this however
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a diff in pull request #12235: KAFKA-13945: add bytes/records consumed and produced metrics
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on code in PR #12235:
URL: https://github.com/apache/kafka/pull/12235#discussion_r888782190
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/metrics/ProcessorNodeMetricsTest.java:
##########
@@ -114,6 +116,63 @@ public void shouldGetProcessAtSourceSensor() {
verifySensor(() -> ProcessorNodeMetrics.processAtSourceSensor(THREAD_ID, TASK_ID, PROCESSOR_NODE_ID, streamsMetrics));
}
+ @Test
+ public void shouldGetRecordsAndBytesConsumedSensor() {
+ final String recordsMetricNamePrefix = "records-consumed";
+ final String bytesMetricNamePrefix = "bytes-consumed";
+ final String descriptionOfRecordsTotal = "The total number of records consumed from this topic";
+ final String descriptionOfBytesTotal = "The total number of bytes consumed from this topic";
+ when(streamsMetrics.nodeLevelSensor(THREAD_ID, TASK_ID, PROCESSOR_NODE_ID, recordsMetricNamePrefix, RecordingLevel.INFO))
+ .thenReturn(expectedSensor);
+ when(streamsMetrics.nodeLevelSensor(THREAD_ID, TASK_ID, PROCESSOR_NODE_ID, bytesMetricNamePrefix, RecordingLevel.INFO))
+ .thenReturn(expectedSensor);
+ when(streamsMetrics.nodeLevelTagMap(THREAD_ID, TASK_ID, PROCESSOR_NODE_ID)).thenReturn(tagMap);
+
+ final Map<String, String> consumedTagMap = new HashMap<>(tagMap);
+ consumedTagMap.put("topic", TOPIC_NAME);
+ StreamsMetricsImpl.addTotalCountAndSumMetricsToSensor(
Review Comment:
Ah, thanks for the explanation. I updated the test as suggested (now in `TopicMetricsTest`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on a diff in pull request #12235: KAFKA-13945: add bytes/records consumed and produced metrics
Posted by GitBox <gi...@apache.org>.
cadonna commented on code in PR #12235:
URL: https://github.com/apache/kafka/pull/12235#discussion_r891054931
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/metrics/ProcessorNodeMetricsTest.java:
##########
@@ -114,6 +116,63 @@ public void shouldGetProcessAtSourceSensor() {
verifySensor(() -> ProcessorNodeMetrics.processAtSourceSensor(THREAD_ID, TASK_ID, PROCESSOR_NODE_ID, streamsMetrics));
}
+ @Test
+ public void shouldGetRecordsAndBytesConsumedSensor() {
+ final String recordsMetricNamePrefix = "records-consumed";
+ final String bytesMetricNamePrefix = "bytes-consumed";
+ final String descriptionOfRecordsTotal = "The total number of records consumed from this topic";
+ final String descriptionOfBytesTotal = "The total number of bytes consumed from this topic";
+ when(streamsMetrics.nodeLevelSensor(THREAD_ID, TASK_ID, PROCESSOR_NODE_ID, recordsMetricNamePrefix, RecordingLevel.INFO))
+ .thenReturn(expectedSensor);
+ when(streamsMetrics.nodeLevelSensor(THREAD_ID, TASK_ID, PROCESSOR_NODE_ID, bytesMetricNamePrefix, RecordingLevel.INFO))
+ .thenReturn(expectedSensor);
+ when(streamsMetrics.nodeLevelTagMap(THREAD_ID, TASK_ID, PROCESSOR_NODE_ID)).thenReturn(tagMap);
+
+ final Map<String, String> consumedTagMap = new HashMap<>(tagMap);
+ consumedTagMap.put("topic", TOPIC_NAME);
+ StreamsMetricsImpl.addTotalCountAndSumMetricsToSensor(
Review Comment:
Oh no! I just prototyped that code since it didn't make sense how the code was before with Mockito. I did also not know about closing static mocks in Mockito. I am really sorry!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #12235: KAFKA-13945: add bytes/records consumed and produced metrics
Posted by GitBox <gi...@apache.org>.
vvcephei commented on PR #12235:
URL: https://github.com/apache/kafka/pull/12235#issuecomment-1146046545
@ableegoldman , you might want to take a look at the build I cancelled. Before it got stuck, there were a large number of failures: https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-12235/9/#showFailuresLink
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] vvcephei commented on pull request #12235: KAFKA-13945: add bytes/records consumed and produced metrics
Posted by GitBox <gi...@apache.org>.
vvcephei commented on PR #12235:
URL: https://github.com/apache/kafka/pull/12235#issuecomment-1146044849
The prior build was hung. I cancelled it so the latest commit could be built.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a diff in pull request #12235: KAFKA-13945: add bytes/records consumed and produced metrics
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on code in PR #12235:
URL: https://github.com/apache/kafka/pull/12235#discussion_r886742508
##########
docs/ops.html:
##########
@@ -2319,7 +2319,7 @@ <h4 class="anchor-heading"><a id="connect_monitoring" class="anchor-link"></a><a
<h4 class="anchor-heading"><a id="kafka_streams_monitoring" class="anchor-link"></a><a href="#kafka_streams_monitoring">Streams Monitoring</a></h4>
A Kafka Streams instance contains all the producer and consumer metrics as well as additional metrics specific to Streams.
- By default Kafka Streams has metrics with three recording levels: <code>info</code>, <code>debug</code>, and <code>trace</code>.
+ The metrics have three recording levels: <code>info</code>, <code>debug</code>, and <code>trace</code>.
Review Comment:
I'll follow up with a docs PR separately
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on pull request #12235: KAFKA-13945: add bytes/records consumed and produced metrics
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on PR #12235:
URL: https://github.com/apache/kafka/pull/12235#issuecomment-1145785669
@cadonna ok this is ready for another pass -- besides addressing your comments I made two changes worth noting:
1. I moved the produced/consumed metrics to a new `TopicMetrics` class as they didn't exactly fit into ProcessorNodeMetrics since I realized that due to dynamic topic routing you could have more than one topic produced to by a given sink node.
2. To address your (imo valid) concern about over-counting bytes that were sent to the producer but never made it to the topic, I moved the "-produced" sensor into `RecordCollectorImpl` and have it record inside the `send` callback after we've checked for errors
The PR has definitely ballooned in size since you last reviewed it but that's almost all due to the new test coverage for the topic-level metrics. The actual logical changes are still relatively minor, so I'm hoping you can give this a +1 by now (and again, if so, please go ahead and merge it yourself)
Thanks for the thorough reviews! :P
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a diff in pull request #12235: KAFKA-13945: add bytes/records consumed and produced metrics
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on code in PR #12235:
URL: https://github.com/apache/kafka/pull/12235#discussion_r886761698
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamTask.java:
##########
@@ -184,7 +184,7 @@ public StreamTask(final TaskId id,
createPartitionQueues(),
mainConsumer::currentLag,
TaskMetrics.recordLatenessSensor(threadId, taskId, streamsMetrics),
- TaskMetrics.totalBytesSensor(threadId, taskId, streamsMetrics),
+ TaskMetrics.totalInputBufferBytesSensor(threadId, taskId, streamsMetrics),
Review Comment:
Just clearing up an overly-generic method name (the metric name itself is not changed)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a diff in pull request #12235: KAFKA-13945: add bytes/records consumed and produced metrics
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on code in PR #12235:
URL: https://github.com/apache/kafka/pull/12235#discussion_r887355518
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/SinkNode.java:
##########
@@ -56,6 +61,26 @@ public void addChild(final ProcessorNode<Void, Void, ?, ?> child) {
@Override
public void init(final InternalProcessorContext<Void, Void> context) {
+ // It is important to first create the sensor before calling init on the
+ // parent object. Otherwise due to backwards compatibility an empty sensor
+ // without parent is created with the same name.
+ // Once the backwards compatibility is not needed anymore it might be possible to
+ // change this.
Review Comment:
Ah yeah I was wondering about it but I hoped you'd see this and remember the context. I'll remove the comment from my PR and we can follow up separately if it needs to be removed from the `SourceNode` class (where I copied it from, yes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a diff in pull request #12235: KAFKA-13945: add bytes/records consumed and produced metrics
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on code in PR #12235:
URL: https://github.com/apache/kafka/pull/12235#discussion_r887536554
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/RecordCollectorImpl.java:
##########
@@ -199,6 +199,7 @@ public <K, V> void send(final String topic,
log.trace("Failed record: (key {} value {} timestamp {}) topic=[{}] partition=[{}]", key, value, timestamp, topic, partition);
}
});
+ return keyBytes.length + valBytes.length;
Review Comment:
Yes, I forgot to fix this in my first pass. I added a utility method to `ClientUtils` to compute the full record size that we compute in various places (there may be some other locations that can replace code with this method call, but we can clean them up as we find them)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on a diff in pull request #12235: KAFKA-13945: add bytes/records consumed and produced metrics
Posted by GitBox <gi...@apache.org>.
cadonna commented on code in PR #12235:
URL: https://github.com/apache/kafka/pull/12235#discussion_r887155630
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/SinkNode.java:
##########
@@ -56,6 +61,26 @@ public void addChild(final ProcessorNode<Void, Void, ?, ?> child) {
@Override
public void init(final InternalProcessorContext<Void, Void> context) {
+ // It is important to first create the sensor before calling init on the
+ // parent object. Otherwise due to backwards compatibility an empty sensor
+ // without parent is created with the same name.
+ // Once the backwards compatibility is not needed anymore it might be possible to
+ // change this.
Review Comment:
Nice, I just realized that you copied that comment from code that I wrote! 🙂
I guess the backwards compatibility was the one with the old metrics structure that we changed in KIP-444. We removed the old structure in 3.0, so I guess that this is an instance of comments that started lying.
Moreover, from where you copied the comment (I guess it was `SourceNode`) the sensor `processAtSourceSensor` has indeed a parent. I think the comment does not make sense in this class and we need to verify if it still makes sense in the other class. That does not need to be verifed in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org