You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "junaiddshaukat (via GitHub)" <gi...@apache.org> on 2026/08/15 14:58:55 UTC

[PR] [GSoC 2026] Kafka Streams runner: bound a source poll in time, not only in elements [beam]

junaiddshaukat opened a new pull request, #39761:
URL: https://github.com/apache/beam/pull/39761

   ## Summary
   
   Part of #18479. An unbounded source could hold the Kafka Streams thread indefinitely, and a pipeline that did so read steadily and produced no output at all.
   
   ## What happens
   
   `UnboundedReadProcessor` polls its source from a wall-clock punctuator scheduled every 50ms, and the same thread runs the rest of the topology. Each turn was bounded by a count — `--readMaxElementsPerPoll` per batch, `--readCheckpointNumBundles` batches — and a count cannot bound the time, because how long an element takes is decided by the pipeline underneath the source rather than by the source.
   
   Once a turn takes longer than the interval it is scheduled at, it is already due again when it returns and fires straight away. The source then keeps the thread and the stages below it are never scheduled. Measured with a grouping pipeline over 2000 keys:
   
   | `--readMaxElementsPerPoll` | time per poll | share of the thread | groups produced |
   | --- | --- | --- | --- |
   | 200 | 3ms | 6% | 52,000 |
   | 5000 | 57ms | 89% | none |
   
   In the second case the pipeline read 40M elements and emitted nothing, and the grouping's consumer had read 682 records out of 11.6M. It is not a pipeline falling behind — it is one that never runs the rest of itself.
   
   ## The change
   
   `--readMaxPollTimeMs` (default 10) bounds a turn in time as well as in count; whichever bound is reached first ends it. The turn's deadline is checked between batches and, since one batch can be long enough on its own, every 64 elements within a batch — often enough to bound the overshoot, not so often as to put a clock read in front of every element.
   
   Roughly, the source takes `readMaxPollTimeMs` of each 50ms interval, so the default leaves the thread four fifths of its time. Lowering it yields sooner and reads less per turn; raising it does the reverse, and above 50ms it re-creates the behaviour above.
   
   At `--readMaxElementsPerPoll=5000`, which produced no groups at all before, the pipeline now produces every window complete.
   
   ## Testing
   
   ```
   ./gradlew :runners:kafka-streams:build              # unit tests, spotless + checker + errorprone
   ./gradlew :runners:kafka-streams:validatesRunner    # 59 tests
   ```
   
   Two tests in `UnboundedReadTest`. `aPollOutOfTimeYieldsBeforeReachingItsElementBound` gives a turn no time at all, so what stops it can only be the time bound, and asserts it yields short of its element bound; `aPollWithTimeToSpareReachesItsElementBound` is the control, confirming the time bound only ever cuts a turn short. The first fails if the deadline is pushed out of reach, checked with a mutation that still compiles — a mutation that fails to compile proves nothing.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GSoC 2026] Kafka Streams runner: bound a source poll in time, not only in elements [beam]

Posted by "je-ik (via GitHub)" <gi...@apache.org>.
je-ik commented on code in PR #39761:
URL: https://github.com/apache/beam/pull/39761#discussion_r3791286447


##########
runners/kafka-streams/src/main/java/org/apache/beam/runners/kafka/streams/translation/UnboundedReadProcessor.java:
##########
@@ -79,6 +80,9 @@ class UnboundedReadProcessor<T, CheckpointT extends CheckpointMark>
   /** How often the source is polled. */
   private static final Duration POLL_INTERVAL = Duration.ofMillis(50);
 
+  /** How many elements pass between checks of the turn's deadline. */

Review Comment:
   I don't fully understand this comment, is it _INTERVAL or number of elements?



##########
runners/kafka-streams/src/main/java/org/apache/beam/runners/kafka/streams/translation/UnboundedReadProcessor.java:
##########
@@ -157,15 +164,24 @@ public void process(Record<byte[], byte[]> record) {
    * Streams thread would never get back to committing or to the rest of the topology. So at most
    * {@link #checkpointEveryNPolls} batches are taken before yielding, which is also where the
    * checkpoint mark is stored, and the next punctuation carries on from there.
+   *
+   * <p>That batch bound is a count, and a count cannot bound the time: how long an element takes is
+   * decided by the pipeline underneath it, which the source knows nothing about. Since this runs on
+   * the thread that also serves the rest of the topology, a turn that overruns the punctuation
+   * interval is already due again when it returns and fires straight away, so the source keeps the
+   * thread and the stages below it never run — a pipeline reading steadily and emitting nothing,
+   * rather than one falling behind. {@link #maxPollTimeMs} bounds the turn in time as well, and
+   * whichever bound is reached first ends it.
    */
   private void poll() {
     if (exhausted) {
       return;
     }
     ProcessorContext<byte[], KStreamsPayload<?>> ctx = checkInitialized(context);
     UnboundedReader<T> currentReader = ensureReader();
+    long deadline = System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(maxPollTimeMs);

Review Comment:
   Why do we user nanoTime() here? It is more expensive than currentTimeMillis() and tje millisecond precision should be okay, no?



##########
runners/kafka-streams/src/main/java/org/apache/beam/runners/kafka/streams/translation/UnboundedReadProcessor.java:
##########
@@ -157,15 +164,24 @@ public void process(Record<byte[], byte[]> record) {
    * Streams thread would never get back to committing or to the rest of the topology. So at most
    * {@link #checkpointEveryNPolls} batches are taken before yielding, which is also where the
    * checkpoint mark is stored, and the next punctuation carries on from there.
+   *
+   * <p>That batch bound is a count, and a count cannot bound the time: how long an element takes is
+   * decided by the pipeline underneath it, which the source knows nothing about. Since this runs on
+   * the thread that also serves the rest of the topology, a turn that overruns the punctuation
+   * interval is already due again when it returns and fires straight away, so the source keeps the

Review Comment:
   This actually sounds like a bug in Kafka Streams, no?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GSoC 2026] Kafka Streams runner: bound a source poll in time, not only in elements [beam]

Posted by "je-ik (via GitHub)" <gi...@apache.org>.
je-ik merged PR #39761:
URL: https://github.com/apache/beam/pull/39761


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GSoC 2026] Kafka Streams runner: bound a source poll in time, not only in elements [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39761:
URL: https://github.com/apache/beam/pull/39761#issuecomment-5302953336

   Assigning reviewers:
   
   R: @chamikaramj added as fallback since no labels match configuration
   
   
   Note: If you would like to opt out of this review, comment `assign to next reviewer`.
   
   Available commands:
   - `stop reviewer notifications` - opt out of the automated review tooling
   - `remind me after tests pass` - tag the comment author after tests pass
   - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
   
   The PR bot will only process comments in the main thread (not review comments).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [GSoC 2026] Kafka Streams runner: bound a source poll in time, not only in elements [beam]

Posted by "junaiddshaukat (via GitHub)" <gi...@apache.org>.
junaiddshaukat commented on code in PR #39761:
URL: https://github.com/apache/beam/pull/39761#discussion_r3791482333


##########
runners/kafka-streams/src/main/java/org/apache/beam/runners/kafka/streams/translation/UnboundedReadProcessor.java:
##########
@@ -157,15 +164,24 @@ public void process(Record<byte[], byte[]> record) {
    * Streams thread would never get back to committing or to the rest of the topology. So at most
    * {@link #checkpointEveryNPolls} batches are taken before yielding, which is also where the
    * checkpoint mark is stored, and the next punctuation carries on from there.
+   *
+   * <p>That batch bound is a count, and a count cannot bound the time: how long an element takes is
+   * decided by the pipeline underneath it, which the source knows nothing about. Since this runs on
+   * the thread that also serves the rest of the topology, a turn that overruns the punctuation
+   * interval is already due again when it returns and fires straight away, so the source keeps the
+   * thread and the stages below it never run — a pipeline reading steadily and emitting nothing,
+   * rather than one falling behind. {@link #maxPollTimeMs} bounds the turn in time as well, and
+   * whichever bound is reached first ends it.
    */
   private void poll() {
     if (exhausted) {
       return;
     }
     ProcessorContext<byte[], KStreamsPayload<?>> ctx = checkInitialized(context);
     UnboundedReader<T> currentReader = ensureReader();
+    long deadline = System.nanoTime() + TimeUnit.MILLISECONDS.toNanos(maxPollTimeMs);

Review Comment:
   All three done.
   
   nanoTime → currentTimeMillis in both places. You are right that millisecond precision is enough for a bound this size, and it is the cheaper call. The tests still catch the bound being disabled after the change, so the behaviour is unchanged.
   
   The constant is renamed ELEMENTS_BETWEEN_DEADLINE_CHECKS. Your reading was the right one — the value is a count of elements, not a time, and the old name said the opposite.
   
   On whether it is a Kafka Streams bug: I have taken the claim out of the comment, because I do not think I can support it. What I can say is what I measured — a turn of 200 elements took 3ms and held the thread 6% of the time, a turn of 5000 took 57ms and held it 89% — and that a punctuator is expected to be quick. Whether Kafka Streams ought to coalesce a punctuation it has already missed is a fair question, but the runner should not be holding the thread that long either way, so I would rather fix our side and not assert something about theirs in a code comment.
   
   I also moved the readMaxPollTimeMs row in the docs into this PR, since it documents the option this PR adds. It was in #39762 before, which would have documented an option that did not exist yet.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org