[jira] [Commented] (FLINK-37417) Implement better exception message to explain why KinesisSource doesn't support partial recovery

["Lorenzo Nicora (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-37417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18019372#comment-18019372 ] 

Lorenzo Nicora commented on FLINK-37417:
----------------------------------------

[~hong] this should have component = `Connectors/AWS`

> Implement better exception message to explain why KinesisSource doesn't support partial recovery
> ------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-37417
>                 URL: https://issues.apache.org/jira/browse/FLINK-37417
>             Project: Flink
>          Issue Type: Improvement
>            Reporter: Hong Liang Teoh
>            Priority: Major
>
> The KinesisSource doesn't currently support partial recovery because we want to prevent duplicate records being read into KinesisSource when restart + partial failure happens.
>  
> Partial recovery is when Flink restarts a subset of subtasks to minimize downtime during a job failure. Flink determines which subtasks need to be restarted by checking all connected subtasks to the failed subtasks. 
> However, this algorithm doesn't work for a KDS source, because the KinesisSource:
>  # Has parent-child shard ordering within the KDS stream.
>  # Does not ensure that all child shards are assigned to the same subtask (to prevent skew)
> These two mean that when we restart "selected" subtasks, we cannot ensure that the child shards have not been assigned to other subtasks "not connected" to the selected subtask.
>  
> This is very much an edge case, where users will have to be doing the following:
>  # NOT have a keyBy immediately after the KDS source.
>  # Partial failure happens on an operator BEFORE the first keyBy in the job graph.
>  
> This JIRA is to enrich the Exception message thrown
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)