You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "MartijnVisser (via GitHub)" <gi...@apache.org> on 2026/08/28 13:09:56 UTC

[PR] [FLINK-40503][core] Document effective idleness detection timing [flink]

MartijnVisser opened a new pull request, #29039:
URL: https://github.com/apache/flink/pull/29039

   ## What is the purpose of the change
   
   Resolves FLINK-40503 by documenting when `withIdleness` actually takes effect. The `WatermarksWithIdleness` countdown is anchored at the first quiet periodic probe rather than at the last record, and the strict `>` comparison only fires at the next probe after the timeout is exceeded — so idleness is detected between idleTimeout and idleTimeout plus three watermark intervals after the last record (the worst case is attained when the timeout is an exact multiple of the auto-watermark interval). The javadoc previously implied detection at idleTimeout. The timing itself is intentionally left unchanged: silently changing detection timing in this area has a regression track record (cf. the FLIP-471 follow-ups).
   
   ## Brief change log
   
   - Clarified the `WatermarkStrategy#withIdleness` javadoc and the `WatermarksWithIdleness` class doc with the precise detection-timing contract.
   - Added an equivalent paragraph to the "Dealing With Idle Sources" section of the event-time documentation (English page; updating the Chinese translation is a follow-up).
   - Added `WatermarksWithIdlenessTimeoutTest`, pinning the documented timing deterministically with a manual clock (timeout 10ms, probes every 5ms: not idle at the probes 11ms and 16ms after the last event, idle at the probe 21ms after it).
   
   ## Verifying this change
   
   This change added tests and can be verified as follows:
   
   - New deterministic `WatermarksWithIdlenessTimeoutTest` pins the documented detection timing.
   - Existing `WatermarksWithIdlenessTest` and `WatermarkStrategyTest` pass unchanged; no existing tests were modified.
   - Otherwise documentation-only; no behavior change.
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): no
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no (javadoc-only changes on `WatermarkStrategy`/`WatermarksWithIdleness`)
     - The serializers: no
     - The runtime per-record code paths (performance sensitive): no
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
     - The S3 file system connector: no
   
   ## Documentation
   
     - Does this pull request introduce a new feature? no
     - If yes, how is the feature documented? not applicable (clarifies existing documentation)
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes (please specify the tool below)
   
   Generated-by: Claude Code (Fable 5)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-40503][core] Document effective idleness detection timing [flink]

Posted by "SEPURI-SAI-KRISHNA (via GitHub)" <gi...@apache.org>.
SEPURI-SAI-KRISHNA commented on PR #29039:
URL: https://github.com/apache/flink/pull/29039#issuecomment-5662202012

   Small one on the lower bound.
   
   The countdown cannot start until a periodic probe sees silence, so detection is never within one interval of the timeout either. Driving `WatermarksWithIdleness` with a manual clock and recording when `markIdle()` fires, across timeout/interval ratios from 0.2 to 12 and every probe phase:
   
   ```
   earliest markIdle: idleTimeout + 1.00 x interval (timeout=999)
   latest   markIdle: idleTimeout + 3.00 x interval (timeout=1000)
   ```
   
   Your upper bound matches exactly. "no earlier than idleTimeout" is correct but a full interval loose, "no earlier than idleTimeout plus one watermark interval" held everywhere I measured.
   
   <details>
   <summary>Harness</summary>
   
   `javac -cp flink-core/target/classes IdlenessTiming.java && java -cp .:flink-core/target/classes IdlenessTiming`
   
   ```java
   import org.apache.flink.api.common.eventtime.*;
   import org.apache.flink.util.clock.RelativeClock;
   import java.time.Duration;
   
   /** Measures when WatermarksWithIdleness actually calls markIdle() after the last event. */
   public class IdlenessTiming {
   
       static long detect(long timeout, long interval, long firstProbeOffset) {
           long[] now = {0};
           RelativeClock clock = new RelativeClock() {
               public long relativeTimeNanos() { return now[0]; }
               public long relativeTimeMillis() { return now[0] / 1_000_000L; }
           };
           boolean[] idle = {false};
           WatermarkOutput out = new WatermarkOutput() {
               public void emitWatermark(Watermark w) {}
               public void markIdle() { idle[0] = true; }
               public void markActive() { idle[0] = false; }
           };
           WatermarksWithIdleness<Long> g = new WatermarksWithIdleness<>(
                   new AscendingTimestampsWatermarks<>(), Duration.ofNanos(timeout), clock);
   
           long ts = 0;
           for (int i = 0; i < 3; i++) {            // events flowing, probes in between
               g.onEvent(1L, ++ts, out);
               now[0] += interval;
               g.onPeriodicEmit(out);
           }
           g.onEvent(1L, ++ts, out);                // last event, at t = 0
           long base = now[0];
           for (long t = base + firstProbeOffset; ; t += interval) {
               now[0] = t;
               g.onPeriodicEmit(out);
               if (idle[0]) return t - base;        // delay after the last event
           }
       }
   
       public static void main(String[] args) {
           final long I = 1000;                     // watermark interval
           double min = Double.MAX_VALUE, max = -1;
           long tMin = 0, tMax = 0;
           for (long timeout = 200; timeout <= 12000; timeout++) {
               for (long phase = 1; phase <= I; phase++) {
                   double over = (detect(timeout, I, phase) - timeout) / (double) I;
                   if (over < min) { min = over; tMin = timeout; }
                   if (over > max) { max = over; tMax = timeout; }
               }
           }
           System.out.printf("earliest markIdle: idleTimeout + %.2f x interval (timeout=%d)%n", min, tMin);
           System.out.printf("latest   markIdle: idleTimeout + %.2f x interval (timeout=%d)%n", max, tMax);
       }
   }
   ```
   
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-40503][core] Document effective idleness detection timing [flink]

Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #29039:
URL: https://github.com/apache/flink/pull/29039#issuecomment-5453034420

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "ca757a08127ac69d25f7ae5da346671659fafc43",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ca757a08127ac69d25f7ae5da346671659fafc43",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * ca757a08127ac69d25f7ae5da346671659fafc43 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-40503][core] Document effective idleness detection timing [flink]

Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #29039:
URL: https://github.com/apache/flink/pull/29039#issuecomment-5464949803

   LGTM, also kudos for nice testing


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org