[jira] [Created] (FLINK-40269) Unaligned checkpoint restore may fail after rescale when a two-input task reads from the same upstream

["Rui Fan (Jira)" <ji...@apache.org>]

Rui Fan created FLINK-40269:
-------------------------------

             Summary: Unaligned checkpoint restore may fail after rescale when a two-input task reads from the same upstream
                 Key: FLINK-40269
                 URL: https://issues.apache.org/jira/browse/FLINK-40269
             Project: Flink
          Issue Type: Bug
          Components: Runtime / Checkpointing
            Reporter: Rui Fan


We found a restore failure in Flink 2.3.0 when restoring from an unaligned checkpoint after rescaling the job.
This issue likely exists since the early unaligned-checkpoint-rescaling support around Flink 1.13/1.14.
It can happen when one two-input task has both inputs derived from the same upstream stream:
 
{code:java}
source 
|-- rebalance() --> CoMap input 1 
|-- keyBy(...) --> CoMap input 2{code}
 
 
During rescale restore, Flink can confuse the two input edges and route channel state with the wrong mapping. The job then fails during restore with an error like:
 
{code:java}
text Cannot select SubtaskConnectionDescriptor{inputSubtaskIndex=1, outputSubtaskIndex=0}; 
known channels are [SubtaskConnectionDescriptor{inputSubtaskIndex=0, outputSubtaskIndex=0}]{code}
 
 
Reproducer branch:
https://github.com/1996fanrui/flink/tree/repro/unaligned-channel-remap
 
Commit `2711b807112e50b4bdabbd1ca181f800e8968a15` adds a minimal ITCase that reproduces the failure 100% before the runtime fix: UnalignedCheckpointRescaleSameUpstreamITCase.java
 
The proposed fix is in commit `9b8560e69818e25ac20e19875aca55bdea208fcc`.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)