You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "wilmerdooley (via GitHub)" <gi...@apache.org> on 2026/06/19 03:29:16 UTC

[PR] [FLINK-39898][runtime] Fire processing-time timers between unaligned checkpoints when mini-batch is enabled [flink]

wilmerdooley opened a new pull request, #28484:
URL: https://github.com/apache/flink/pull/28484

   ## What is the purpose of the change
   
   When both `table.exec.mini-batch.enabled` and `execution.checkpointing.unaligned.enabled` are on, ProcessingTimeService timer callbacks are not fired between checkpoints; they only run during checkpoint barrier handling. This is a regression from 1.20 introduced by the urgent-mail system (FLINK-35796): unaligned checkpoint barriers are submitted as urgent mails, and `TaskMailboxImpl` then skipped non-urgent mails (the timer callbacks) while an urgent mail was present, starving them until the next checkpoint.
   
   This change lets non-urgent mails make progress when the batch would otherwise be empty, without changing urgent-mail prioritization.
   
   ## Brief change log
   
     - `TaskMailboxImpl.moveUrgentMailsToBatchIfNeeded`: compute `shouldOnlyMoveUrgentMails = onlyMoveUrgentMails && !isBatchEmpty` and gate the non-urgent put-back/break on it, so when the batch is empty a non-urgent mail is moved into the batch instead of being returned to the queue and skipped. Urgent mails still take FIFO priority via `batch.addFirst`.
   
   ## Verifying this change
   
   This change added a unit test: `TaskMailboxImplTest.testNonUrgentMailQueuedBehindUrgentMailIsTakenFromBatch` asserts that a non-urgent mail queued behind an urgent mail is still taken from the batch (not starved) while the urgent mail is taken first. The test fails on the pre-fix code, where the non-urgent mail was returned to the queue and skipped.
   
   ## Does this pull request potentially affect one of the following parts
   
     - Dependencies (does it add or upgrade a dependency): no
     - Public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
     - The serializers: no
     - The runtime per-record code paths (performance sensitive): yes (the task mailbox take/createBatch path runs once per mailbox-loop iteration; this change adds two local boolean computations and changes only the empty-batch branch, with no new lock or allocation, so no measurable per-record overhead is expected; it does change how often non-urgent mails are drained on the task thread)
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: yes (it interacts with unaligned-checkpoint barrier delivery and processing-time timer firing between checkpoints; the change is confined to the mailbox take path and does not touch JobManager, Kubernetes, Yarn, ZooKeeper, snapshot format, or recovery state)
     - The S3 file system connector: no
   
   ## Documentation
   
     - Does this pull request introduce a new feature? no
     - If yes, how is the feature documented? not applicable
   
   This PR was written with the assistance of generative AI tooling.
   - [X] Was generative AI tooling used to co-author this PR?
   
   Generated-by: Claude Code
   
   JIRA: https://issues.apache.org/jira/browse/FLINK-39898
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39898][runtime] Fire processing-time timers between unaligned checkpoints when mini-batch is enabled [flink]

Posted by "weiqingy (via GitHub)" <gi...@apache.org>.
weiqingy commented on code in PR #28484:
URL: https://github.com/apache/flink/pull/28484#discussion_r3447997281


##########
flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/mailbox/TaskMailboxImpl.java:
##########
@@ -247,6 +248,7 @@ private void moveUrgentMailsToBatchIfNeeded(boolean onlyMoveUrgentMails) {
             }
         }
 
+        final boolean shouldOnlyMoveUrgentMails = onlyMoveUrgentMails && !isBatchEmpty;

Review Comment:
   Perfect — sitting right above the assignment, the reasoning is impossible to miss now. That was my only open item, and the fix itself stays nicely contained to the one boolean. LGTM. I'll leave the final call to the maintainers.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39898][runtime] Fire processing-time timers between unaligned checkpoints when mini-batch is enabled [flink]

Posted by "wilmerdooley (via GitHub)" <gi...@apache.org>.
wilmerdooley commented on code in PR #28484:
URL: https://github.com/apache/flink/pull/28484#discussion_r3447706592


##########
flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/mailbox/TaskMailboxImpl.java:
##########
@@ -247,6 +248,7 @@ private void moveUrgentMailsToBatchIfNeeded(boolean onlyMoveUrgentMails) {
             }
         }
 
+        final boolean shouldOnlyMoveUrgentMails = onlyMoveUrgentMails && !isBatchEmpty;

Review Comment:
   Good call, thanks. I added your suggested wording above the line so the reasoning sits right where the fix lives.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39898][runtime] Fire processing-time timers between unaligned checkpoints when mini-batch is enabled [flink]

Posted by "weiqingy (via GitHub)" <gi...@apache.org>.
weiqingy commented on code in PR #28484:
URL: https://github.com/apache/flink/pull/28484#discussion_r3447997281


##########
flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/mailbox/TaskMailboxImpl.java:
##########
@@ -247,6 +248,7 @@ private void moveUrgentMailsToBatchIfNeeded(boolean onlyMoveUrgentMails) {
             }
         }
 
+        final boolean shouldOnlyMoveUrgentMails = onlyMoveUrgentMails && !isBatchEmpty;

Review Comment:
   Perfect — sitting right above the assignment, the reasoning is impossible to miss now. That was my only open item, and the fix itself stays nicely contained to the one boolean. LGTM. I'll leave the final call to the maintainers.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39898][runtime] Fire processing-time timers between unaligned checkpoints when mini-batch is enabled [flink]

Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #28484:
URL: https://github.com/apache/flink/pull/28484#issuecomment-4748289605

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "adfdf13ffae54899d3a117992e8b4adb387c0a1e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "adfdf13ffae54899d3a117992e8b4adb387c0a1e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * adfdf13ffae54899d3a117992e8b4adb387c0a1e UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39898][runtime] Fire processing-time timers between unaligned checkpoints when mini-batch is enabled [flink]

Posted by "wilmerdooley (via GitHub)" <gi...@apache.org>.
wilmerdooley commented on code in PR #28484:
URL: https://github.com/apache/flink/pull/28484#discussion_r3447706592


##########
flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/mailbox/TaskMailboxImpl.java:
##########
@@ -247,6 +248,7 @@ private void moveUrgentMailsToBatchIfNeeded(boolean onlyMoveUrgentMails) {
             }
         }
 
+        final boolean shouldOnlyMoveUrgentMails = onlyMoveUrgentMails && !isBatchEmpty;

Review Comment:
   I have added the suggested comment explaining the logic behind the shouldOnlyMoveUrgentMails initialization.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-39898][runtime] Fire processing-time timers between unaligned checkpoints when mini-batch is enabled [flink]

Posted by "weiqingy (via GitHub)" <gi...@apache.org>.
weiqingy commented on code in PR #28484:
URL: https://github.com/apache/flink/pull/28484#discussion_r3447506481


##########
flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/mailbox/TaskMailboxImpl.java:
##########
@@ -247,6 +248,7 @@ private void moveUrgentMailsToBatchIfNeeded(boolean onlyMoveUrgentMails) {
             }
         }
 
+        final boolean shouldOnlyMoveUrgentMails = onlyMoveUrgentMails && !isBatchEmpty;

Review Comment:
   This boolean is where the whole fix lives, and the *why* is the non-obvious part: when the batch is empty on entry, you deliberately let non-urgent mails move in too, because otherwise a non-urgent timer mail queued behind an urgent barrier gets put back and stranded until the next batch — which is exactly the between-unaligned-checkpoint timer starvation this PR fixes. A reader six months out sees `onlyMoveUrgentMails && !isBatchEmpty` with no way to recover that reasoning, and the surrounding early-returns are all commented, so this one stands out by its silence. Would a short why-comment here be worth it? Something like this, if it helps:
   
   ```java
   // When the batch is empty, also pull non-urgent mails in (not just urgent ones): otherwise a
   // non-urgent mail queued behind an urgent one is put back and stranded until the next batch,
   // starving processing-time timers between unaligned checkpoints.
   final boolean shouldOnlyMoveUrgentMails = onlyMoveUrgentMails && !isBatchEmpty;
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org