You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Rui Fan (Jira)" <ji...@apache.org> on 2026/08/31 11:34:00 UTC
[jira] [Updated] (FLINK-40518) CDR is wrongly kept enabled on the aligned path when unaligned checkpoint is disabled

     [ https://issues.apache.org/jira/browse/FLINK-40518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rui Fan updated FLINK-40518:
----------------------------
    Description: 
isCheckpointingDuringRecoveryEnabled checks only UNALIGNED_RECOVER_OUTPUT_ON_DOWNSTREAM and CHECKPOINTING_DURING_RECOVERY_ENABLED, not unaligned.enabled [1]. So a job restoring from an unaligned checkpoint but running with UC disabled still takes the CDR recovery branch [2] and goes RUNNING before the drain finishes.

But InputProcessorUtil threads recoveryCheckpointTrigger only into alternating() (UC on); the aligned() handler used when UC is off gets NO_OP [3]. The aligned path never considered a barrier delivered mid-recovery: the barrier is stashed, so it can neither snapshot-during-recovery nor decline TASK_NOT_READY. A checkpoint triggered in the recovery window hangs and expires (tolerable-failed-checkpoints default 0), leading to a failover loop.

Fix: the aligned path can't do CDR anyway, so don't enable CDR there — add "&& isUnalignedCheckpointEnabled(config)" inside isCheckpointingDuringRecoveryEnabled. It is the single central gate for all CDR call sites, so UC-off falls every path back to the tested non-CDR recovery. No functional loss.

[1] [https://github.com/apache/flink/blob/76e774e04ff84ea305714153c5f4790fff98437c/flink-core/src/main/java/org/apache/flink/configuration/CheckpointingOptions.java#L844-L849] 
[2] [https://github.com/apache/flink/blob/76e774e04ff84ea305714153c5f4790fff98437c/flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java#L943-L945] 
[3] [https://github.com/apache/flink/blob/76e774e04ff84ea305714153c5f4790fff98437c/flink-runtime/src/main/java/org/apache/flink/streaming/runtime/io/checkpointing/InputProcessorUtil.java#L186-L199] 

  was:
`isCheckpointingDuringRecoveryEnabled` checks only `UNALIGNED_RECOVER_OUTPUT_ON_DOWNSTREAM` and `CHECKPOINTING_DURING_RECOVERY_ENABLED`, **not** `unaligned.enabled` ([CheckpointingOptions.java#L844-L849](https://github.com/apache/flink/blob/76e774e04ff84ea305714153c5f4790fff98437c/flink-core/src/main/java/org/apache/flink/configuration/CheckpointingOptions.java#L844-L849)). So a job restoring from an unaligned checkpoint but running with UC disabled still takes the CDR recovery branch ([StreamTask.java#L943-L945](https://github.com/apache/flink/blob/76e774e04ff84ea305714153c5f4790fff98437c/flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java#L943-L945)) and goes RUNNING before the drain finishes.

But `InputProcessorUtil` threads `recoveryCheckpointTrigger` only into `alternating()` (UC on); the `aligned()` handler used when UC is off gets `NO_OP` ([InputProcessorUtil.java#L186-L199](https://github.com/apache/flink/blob/76e774e04ff84ea305714153c5f4790fff98437c/flink-runtime/src/main/java/org/apache/flink/streaming/runtime/io/checkpointing/InputProcessorUtil.java#L186-L199)). The aligned path never considered a barrier delivered mid-recovery: the barrier is stashed, so it can neither snapshot-during-recovery nor decline `TASK_NOT_READY`. A checkpoint triggered in the recovery window hangs and expires (`tolerable-failed-checkpoints` default 0) → failover loop.

**Fix**: the aligned path can't do CDR anyway, so don't enable CDR there — add `&& isUnalignedCheckpointEnabled(config)` inside `isCheckpointingDuringRecoveryEnabled`. It is the single central gate for all CDR call sites, so UC-off falls every path back to the tested non-CDR recovery. No functional loss.


> CDR is wrongly kept enabled on the aligned path when unaligned checkpoint is disabled
> -------------------------------------------------------------------------------------
>
>                 Key: FLINK-40518
>                 URL: https://issues.apache.org/jira/browse/FLINK-40518
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Runtime / Checkpointing
>            Reporter: Rui Fan
>            Assignee: Rui Fan
>            Priority: Major
>
> isCheckpointingDuringRecoveryEnabled checks only UNALIGNED_RECOVER_OUTPUT_ON_DOWNSTREAM and CHECKPOINTING_DURING_RECOVERY_ENABLED, not unaligned.enabled [1]. So a job restoring from an unaligned checkpoint but running with UC disabled still takes the CDR recovery branch [2] and goes RUNNING before the drain finishes.
> But InputProcessorUtil threads recoveryCheckpointTrigger only into alternating() (UC on); the aligned() handler used when UC is off gets NO_OP [3]. The aligned path never considered a barrier delivered mid-recovery: the barrier is stashed, so it can neither snapshot-during-recovery nor decline TASK_NOT_READY. A checkpoint triggered in the recovery window hangs and expires (tolerable-failed-checkpoints default 0), leading to a failover loop.
> Fix: the aligned path can't do CDR anyway, so don't enable CDR there — add "&& isUnalignedCheckpointEnabled(config)" inside isCheckpointingDuringRecoveryEnabled. It is the single central gate for all CDR call sites, so UC-off falls every path back to the tested non-CDR recovery. No functional loss.
> [1] [https://github.com/apache/flink/blob/76e774e04ff84ea305714153c5f4790fff98437c/flink-core/src/main/java/org/apache/flink/configuration/CheckpointingOptions.java#L844-L849] 
> [2] [https://github.com/apache/flink/blob/76e774e04ff84ea305714153c5f4790fff98437c/flink-runtime/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java#L943-L945] 
> [3] [https://github.com/apache/flink/blob/76e774e04ff84ea305714153c5f4790fff98437c/flink-runtime/src/main/java/org/apache/flink/streaming/runtime/io/checkpointing/InputProcessorUtil.java#L186-L199] 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)