[jira] [Created] (FLINK-21817) FLIP-27 Source might break subtask and split assignment upon rescale

["Kezhu Wang (Jira)" <ji...@apache.org>]

Kezhu Wang created FLINK-21817:
----------------------------------

             Summary: FLIP-27 Source might break subtask and split assignment upon rescale
                 Key: FLINK-21817
                 URL: https://issues.apache.org/jira/browse/FLINK-21817
             Project: Flink
          Issue Type: Improvement
          Components: Connectors / Common
    Affects Versions: 1.13.0
            Reporter: Kezhu Wang


On restoring, splits are add back directly to {{SourceReader}} in {{SourceOperator}}. In no rescaled restoring, bindings between splits and subtasks are preserved due to no repartition in {{RoundRobinOperatorStateRepartitioner}}. But in rescaled restoring, these operator states will be redistributed cross existing subtasks. This might break possible assignment from {{SourceEnumerator}}.

Given {{KafkaSource}} as an example, the partition to subtask assignment is decided by {{KafkaSourceEnumerator.getSplitOwner}}. The mappings will break after rescaling.

I pushed [a test case|https://github.com/kezhuw/flink/commit/9dc13cd9d7355d468a6ac8aae1b14b3a267581b6#diff-ad6e86c3757199ac3247687a71f9c34ee67b9ac743ae88a9f660950f27bec6eeR238] using {{KafkaSource}} for evaluation.

I think it requires api addition to solve in generic and configurable way.

Is it a valid issue ? I am not that sure.

cc  [~jqin] [~sewen]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)