[jira] [Created] (FLINK-39680) Enable unaligned checkpoint for pointwise connection by FORCE_UNALIGNED

["Gen Luo (Jira)" <ji...@apache.org>]

Gen Luo created FLINK-39680:
-------------------------------

             Summary: Enable unaligned checkpoint for pointwise connection by FORCE_UNALIGNED
                 Key: FLINK-39680
                 URL: https://issues.apache.org/jira/browse/FLINK-39680
             Project: Flink
          Issue Type: Improvement
            Reporter: Gen Luo


Unaligned checkpoints on pointwise edges are currently prohibited by FLINK-21936 and FLINK-21945 to ensure data consistency during parallelism changes. This restriction is necessary when a keyed stream or keyed source feeds into a pointwise edge, as channel state is redistributed upon recovery with changed parallelism. This breaks the guarantee that records of the same key reach the same downstream task, potentially compromising logic that relies on key ordering.

However, there are valid scenarios where key ordering is irrelevant, but unaligned checkpoints are essential. For instance, in bounded streaming jobs processing datasets like images, each record may take seconds or minutes to process. This can block checkpoint barriers, making aligned checkpoints infeasible within expected timeouts. 

Since Flink cannot automatically determine whether a job requires strict key ordering, we should preserve the current default behavior. Instead, I propose allowing users to explicitly opt-in to unaligned checkpoints for such cases. The existing configuration `execution.checkpointing.unaligned.forced`—originally introduced to support iterative jobs—can be leveraged for this purpose.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)