You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "alanlau28 (via GitHub)" <gi...@apache.org> on 2026/06/02 14:11:49 UTC

[PR] KAFKA-14597: record-e2e-latency-max only considers consumption latencies but not processing delays [kafka]

alanlau28 opened a new pull request, #22452:
URL: https://github.com/apache/kafka/pull/22452

   The `record-e2e-latency` metric at terminal nodes was computed as `currentSystemTimeMs() - record.timestamp()`, where `currentSystemTimeMs()` returns a wall-clock value cached once at the start of processing a record (set by `updateProcessorContext` before `currNode.process()` runs). Because this value is captured before the topology executes, processing delays are invisible to it.
   
   The fix reads the wall clock freshly at the terminal site, after `child.process()` returns. To avoid a clock read per output record the read is done lazily: `currentRecordE2ELatencyTimeMs` on `StreamTask` caches the first fresh read and is reused for every subsequent terminal recording within the same input record's processing. `updateProcessorContext` resets the field to `-1` at the start of each record and punctuation, so each input record gets its own reading taken after its processing completes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-14597: record-e2e-latency-max only considers consumption latencies but not processing delays [kafka]

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #22452:
URL: https://github.com/apache/kafka/pull/22452#discussion_r3708993547


##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamTask.java:
##########
@@ -1289,6 +1297,96 @@ void maybeRecordE2ELatency(final long recordTimestamp, final long now, final Str
         }
     }
 
+    /**
+     * Buffers a terminal node's record timestamp. The latency is recorded later by
+     * {@link #maybeFlushTerminalE2ELatency(long)}, so processing delay is captured without a
+     * wall-clock read per output record.
+     */
+    void maybeAddTerminalE2ELatency(final long recordTimestamp, final String nodeName) {
+        final Sensor e2eLatencySensor = e2eLatencySensors.get(nodeName);
+        if (e2eLatencySensor == null) {
+            throw new IllegalStateException("Requested to record e2e latency but could not find sensor for node " + nodeName);
+        } else if (e2eLatencySensor.shouldRecord() && e2eLatencySensor.hasMetrics()) {
+            terminalE2ELatencyAccumulators.get(nodeName).add(recordTimestamp);
+        }
+    }
+
+    /**
+     * Records terminal {@code record-e2e-latency} for all records buffered since the last flush,
+     * against a single already-read wall-clock time {@code endMs}. Emits the exact max and min
+     * latency and, for larger batches, fills the remaining samples with the interior-timestamp mean
+     * so the count-weighted average is exact as well. Emitting one sample per record keeps the
+     * metric's weighting unchanged; all records are attributed {@code endMs}, a conservative upper
+     * bound on their completion time.
+     */
+    @Override
+    public void maybeFlushTerminalE2ELatency(final long endMs) {
+        for (final Map.Entry<String, TerminalE2ELatencyAccumulator> entry : terminalE2ELatencyAccumulators.entrySet()) {
+            final TerminalE2ELatencyAccumulator accumulator = entry.getValue();
+            final long n = accumulator.count;
+            if (n > 0) {
+                final Sensor e2eLatencySensor = e2eLatencySensors.get(entry.getKey());
+                if (e2eLatencySensor.shouldRecord() && e2eLatencySensor.hasMetrics()) {
+                    final double maxLatency = endMs - accumulator.minTs;
+                    final double minLatency = endMs - accumulator.maxTs;
+                    if (n == 1) {
+                        e2eLatencySensor.record(maxLatency, endMs);
+                    } else {
+                        e2eLatencySensor.record(maxLatency, endMs);
+                        e2eLatencySensor.record(minLatency, endMs);
+                        if (n >= 3) {
+                            final double interiorMeanTs = accumulator.baseTs
+                                + (double) (accumulator.sumOffset
+                                            - (accumulator.minTs - accumulator.baseTs)
+                                            - (accumulator.maxTs - accumulator.baseTs)) / (n - 2);
+                            // clamp against rounding so the filler can't move the sensor's max/min
+                            final double filler = Math.min(maxLatency, Math.max(minLatency, endMs - interiorMeanTs));
+                            for (long i = 0; i < n - 2; i++) {
+                                e2eLatencySensor.record(filler, endMs);
+                            }
+                        }
+                    }
+                }
+                accumulator.reset();
+            }
+        }
+    }
+
+    /**
+     * Per-terminal-node timestamp aggregates used to reconstruct e2e latency without keeping a
+     * sample per record. Timestamps are summed as offsets from the first one so the running sum
+     * stays small and overflow-safe.
+     */
+    private static final class TerminalE2ELatencyAccumulator {
+        private long count;
+        private long baseTs;
+        private long minTs;
+        private long maxTs;
+        private long sumOffset;
+
+        void add(final long recordTimestamp) {
+            if (count == 0) {
+                baseTs = recordTimestamp;
+                minTs = recordTimestamp;
+                maxTs = recordTimestamp;
+                sumOffset = 0L;
+            } else {
+                if (recordTimestamp < minTs) {
+                    minTs = recordTimestamp;
+                }
+                if (recordTimestamp > maxTs) {
+                    maxTs = recordTimestamp;
+                }
+                sumOffset += recordTimestamp - baseTs;

Review Comment:
   This is clever. I like it!



##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamTask.java:
##########
@@ -1289,6 +1297,96 @@ void maybeRecordE2ELatency(final long recordTimestamp, final long now, final Str
         }
     }
 
+    /**
+     * Buffers a terminal node's record timestamp. The latency is recorded later by
+     * {@link #maybeFlushTerminalE2ELatency(long)}, so processing delay is captured without a
+     * wall-clock read per output record.
+     */
+    void maybeAddTerminalE2ELatency(final long recordTimestamp, final String nodeName) {
+        final Sensor e2eLatencySensor = e2eLatencySensors.get(nodeName);
+        if (e2eLatencySensor == null) {
+            throw new IllegalStateException("Requested to record e2e latency but could not find sensor for node " + nodeName);
+        } else if (e2eLatencySensor.shouldRecord() && e2eLatencySensor.hasMetrics()) {
+            terminalE2ELatencyAccumulators.get(nodeName).add(recordTimestamp);
+        }
+    }
+
+    /**
+     * Records terminal {@code record-e2e-latency} for all records buffered since the last flush,
+     * against a single already-read wall-clock time {@code endMs}. Emits the exact max and min
+     * latency and, for larger batches, fills the remaining samples with the interior-timestamp mean
+     * so the count-weighted average is exact as well. Emitting one sample per record keeps the
+     * metric's weighting unchanged; all records are attributed {@code endMs}, a conservative upper
+     * bound on their completion time.
+     */
+    @Override
+    public void maybeFlushTerminalE2ELatency(final long endMs) {
+        for (final Map.Entry<String, TerminalE2ELatencyAccumulator> entry : terminalE2ELatencyAccumulators.entrySet()) {
+            final TerminalE2ELatencyAccumulator accumulator = entry.getValue();
+            final long n = accumulator.count;
+            if (n > 0) {
+                final Sensor e2eLatencySensor = e2eLatencySensors.get(entry.getKey());
+                if (e2eLatencySensor.shouldRecord() && e2eLatencySensor.hasMetrics()) {
+                    final double maxLatency = endMs - accumulator.minTs;
+                    final double minLatency = endMs - accumulator.maxTs;
+                    if (n == 1) {
+                        e2eLatencySensor.record(maxLatency, endMs);
+                    } else {
+                        e2eLatencySensor.record(maxLatency, endMs);
+                        e2eLatencySensor.record(minLatency, endMs);
+                        if (n >= 3) {
+                            final double interiorMeanTs = accumulator.baseTs
+                                + (double) (accumulator.sumOffset
+                                            - (accumulator.minTs - accumulator.baseTs)
+                                            - (accumulator.maxTs - accumulator.baseTs)) / (n - 2);
+                            // clamp against rounding so the filler can't move the sensor's max/min
+                            final double filler = Math.min(maxLatency, Math.max(minLatency, endMs - interiorMeanTs));
+                            for (long i = 0; i < n - 2; i++) {
+                                e2eLatencySensor.record(filler, endMs);
+                            }
+                        }
+                    }
+                }
+                accumulator.reset();
+            }
+        }
+    }
+
+    /**
+     * Per-terminal-node timestamp aggregates used to reconstruct e2e latency without keeping a
+     * sample per record. Timestamps are summed as offsets from the first one so the running sum
+     * stays small and overflow-safe.
+     */
+    private static final class TerminalE2ELatencyAccumulator {
+        private long count;
+        private long baseTs;
+        private long minTs;
+        private long maxTs;
+        private long sumOffset;
+
+        void add(final long recordTimestamp) {
+            if (count == 0) {
+                baseTs = recordTimestamp;
+                minTs = recordTimestamp;
+                maxTs = recordTimestamp;
+                sumOffset = 0L;
+            } else {
+                if (recordTimestamp < minTs) {
+                    minTs = recordTimestamp;
+                }
+                if (recordTimestamp > maxTs) {
+                    maxTs = recordTimestamp;
+                }

Review Comment:
   Should we use `Math.min(...)`, and `Math.max(...)` -- does the same thing but might be easier to read?



##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamTask.java:
##########
@@ -1006,6 +1012,8 @@ record = null;
                 droppedRecordsSensor.record();
             }
         } finally {
+            // flush e2e latency for any records the punctuator forwarded to terminal nodes
+            maybeFlushTerminalE2ELatency(punctuateWallClockMs);

Review Comment:
   Claude pointed out, that this is the ts captured before the punctuator ran. -- So it seems we cannot re-use a cached value.
   
   Given that punctuators don't run on every _input record_, but time based, and usually in the seconds or even minutes, it should be ok to just pass int `time.milliseconds()` here, and pay the system-call.
   
   We also don't test that this flush is happening at all.



##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskExecutor.java:
##########
@@ -124,6 +124,8 @@ private int processTask(final Task task, final int maxNumRecords, final long beg
         } finally {
             now = time.milliseconds();
             task.recordProcessBatchTime(now - then);
+            // record terminal e2e latency for the batch against this single end-of-batch time
+            task.maybeFlushTerminalE2ELatency(now);

Review Comment:
   Same. This is not covered by any test



##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamTask.java:
##########
@@ -928,7 +932,9 @@ public void punctuate(final ProcessorNode<?, ?, ?, ?> node,
             null,
             new RecordHeaders()
         );
-        updateProcessorContext(node, time.milliseconds(), recordContext);
+        // reused as the terminal e2e flush time below, so punctuation needs no extra clock read
+        final long punctuateWallClockMs = time.milliseconds();
+        updateProcessorContext(node, punctuateWallClockMs, recordContext);

Review Comment:
   Seems you are trying to cache the ts for later usage -- I think this does not work. Cf comment blow.
   
   If we really want to save a system-call here, I think we could also pass in `now` into this call though?



##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/StreamTaskTest.java:
##########
@@ -845,8 +850,10 @@ public void process(final Record<Integer, Integer> record) {
 
 
         // e2e latency = 23
+        time.sleep(13L);

Review Comment:
   Do we need to advance time here?



##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/tasks/DefaultTaskExecutor.java:
##########
@@ -181,7 +181,10 @@ private boolean processTask(final Task task, final long now, final Time time) {
                 log.error(String.format("Failed to process stream task %s due to the following error:", task.id()), e);
                 throw new StreamsException(e, task.id());
             } finally {
-                task.recordProcessBatchTime(time.milliseconds() - now);
+                final long end = time.milliseconds();
+                task.recordProcessBatchTime(end - now);
+                // record terminal e2e latency against the end time already read for the batch metric
+                task.maybeFlushTerminalE2ELatency(end);

Review Comment:
   Claude highlighted, that this call is crucial for correctness, but it's not covered by any tests.



##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/StreamTaskTest.java:
##########
@@ -817,8 +818,11 @@ public void process(final Record<Integer, Integer> record) {
         final Metric terminalMax = getProcessorMetric("record-e2e-latency", "%s-max", task.id().toString(), terminalNodeName);
 
         // e2e latency = 10
+        time.sleep(10L);

Review Comment:
   Do we need to advance time here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-14597: record-e2e-latency-max only considers consumption latencies but not processing delays [kafka]

Posted by "lucliu1108 (via GitHub)" <gi...@apache.org>.
lucliu1108 commented on PR #22452:
URL: https://github.com/apache/kafka/pull/22452#issuecomment-4792346328

   Hi @alanlau28 , thanks for the PR!
   
   Based on Matthias' suggestion, another approach is to follow the pattern adapted by `process-latency` metrics. In `StreamThread`, after `taskManager.process()` returns:
   ```
   final long processLatency = advanceNowAndComputeLatency();
   totalProcessLatency += processLatency;
   if (processed > 0) {
     processRateSensor.record(processed, now);
     processLatencySensor.record(processLatency / (double) processed, now);
     ...
   }
   ```
   `advanceNowAndComputeLatency()` reads the wall clock once and reuses it for the whole batch's metrics — one syscall covers all records processed in that taskManager.process().
   
   For e2e latency, we could apply a similar per-task pattern. The code now reads the wallclock inside `maybeRecordE2ELatency` the first time a record hits a terminal node, then caches it for the rest of that input record's fan-out. The cost is one read per input record. 
   
   We could push the read up a level to the whole `taskManager.process()` batch. To be specific, `maybeRecordE2ELatency` stops reading the clock and just appends the record's timestamp to a per-task buffer. Then at the end of the while loop in `TaskExecutor.processTask` (the per-task numIterations) : read the wallClock once and traverse through the buffer, recording one sensor sample per buffered timestamp using that single `now`. Then clear the buffer.
   
   In this way, the cost goes from one syscall per input record to one per task call. **However, the tradeoff is it is not recording the true latency, but rather "the latency that a record would have if it finished at the end of its batch", which is always ≥ the real value. It's a conservative upper bound. Also, MIN is biased by T_batch_end − max(record.ts in batch); in workloads with uniform real latency, the new proposal artificially widens the MIN/MAX range**
   
   ---
   
   Alternatively, another approach is to read wallClock time every K records during processing. Records within the same K-window share the same cached now. Records in different K-windows get different `now`. However, how to choose this `K` for batching is another problem worth considering.
   
   Happy to hear thoughts from others on this trade-off (and better options).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-14597: record-e2e-latency-max only considers consumption latencies but not processing delays [kafka]

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax merged PR #22452:
URL: https://github.com/apache/kafka/pull/22452


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-14597: record-e2e-latency-max only considers consumption latencies but not processing delays [kafka]

Posted by "alanlau28 (via GitHub)" <gi...@apache.org>.
alanlau28 commented on PR #22452:
URL: https://github.com/apache/kafka/pull/22452#issuecomment-4926672980

   Reworked to address the per-record `System.currentTimeMillis()` overhead concern.                                                                                                                              
                                                                                                                                                 
   Terminal e2e latency is now recorded once per task batch instead of at the terminal node. It reuses the wall-clock time `TaskExecutor.processTask` already reads at batch end for `recordProcessBatchTime`, so there are **no extra clock reads** vs. the original cached-time behavior. During the batch each terminal node accumulates only `count/min/max/offset-sum` of record timestamps (constant size, independent of fan-out); at flush the sensor's avg/min/max are reconstructed exactly from those aggregates and the single end time — `max = end − minTs`, `min = end − maxTs`, and the interior samples filled so the count-weighted avg is exact. Same number of samples as recording per record, so weighting is unchanged.                                                                                                                                                               
                                                                          
   Trade-off: every record in a batch is attributed the batch-end completion time (a conservative upper bound) — the same per-batch model `process-latency` uses.                                                                                                                                   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-14597: record-e2e-latency-max only considers consumption latencies but not processing delays [kafka]

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #22452:
URL: https://github.com/apache/kafka/pull/22452#issuecomment-5246814540

   Thanks for the fix. Merged to `trunk` and cherry-picked to `4.3` and` 4.2` branches.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-14597: record-e2e-latency-max only considers consumption latencies but not processing delays [kafka]

Posted by "alanlau28 (via GitHub)" <gi...@apache.org>.
alanlau28 commented on PR #22452:
URL: https://github.com/apache/kafka/pull/22452#issuecomment-5186415230

   Thanks for the review @mjsax! Addressed the comments in <https://github.com/apache/kafka/pull/22452/commits/db966c90a58907e880e3f2b036af2b5fc24f8ff1>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-14597: record-e2e-latency-max only considers consumption latencies but not processing delays [kafka]

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #22452:
URL: https://github.com/apache/kafka/pull/22452#discussion_r3365741274


##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamTask.java:
##########
@@ -1289,6 +1293,22 @@ void maybeRecordE2ELatency(final long recordTimestamp, final long now, final Str
         }
     }
 
+    /**
+     * Records e2e latency for a terminal node using the wall-clock time at which the record was
+     * fully processed by the topology.
+     */
+    void maybeRecordE2ELatency(final long recordTimestamp, final String nodeName) {
+        final Sensor e2eLatencySensor = e2eLatencySensors.get(nodeName);
+        if (e2eLatencySensor == null) {
+            throw new IllegalStateException("Requested to record e2e latency but could not find sensor for node " + nodeName);
+        } else if (e2eLatencySensor.shouldRecord() && e2eLatencySensor.hasMetrics()) {
+            if (currentRecordE2ELatencyTimeMs < 0) {
+                currentRecordE2ELatencyTimeMs = time.milliseconds();

Review Comment:
   Same concern as on the other PR for this ticket:
   
   https://github.com/apache/kafka/pull/21618#pullrequestreview-4064367259



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org