[jira] [Commented] (FLINK-19011) Parallelize the restore operation in OperatorStateBackend

["Stephan Ewen (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-19011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17181898#comment-17181898 ] 

Stephan Ewen commented on FLINK-19011:
--------------------------------------

Union state has caused us much trouble through the years.
I am leaning towards deprecating and dropping the feature.

  - The new sources should not need it any more, the new sinks neither.
  - The feature is not really scalable, it has an inherent limit, worse than broadcast, because it works linearly until a failure and then becomes quadratic (like a broadcast). So it is a non-scalability you discover only when it is too late.

What do you think?

> Parallelize the restore operation in OperatorStateBackend 
> ----------------------------------------------------------
>
>                 Key: FLINK-19011
>                 URL: https://issues.apache.org/jira/browse/FLINK-19011
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / Checkpointing
>    Affects Versions: 1.11.1
>            Reporter: Jiayi Liao
>            Priority: Major
>
> To restore the states, union state needs to read state handles produced by all operators. And currently during the restore operation, Flink iterates the state handles one by one, which could last tens of minutes if the magnitude of state handles exceeds ten thousand. 
> To accelerate the process, I propose to parallelize the random reads on HDFS and deserialization. We can create a runnable for each state handle and let it return the metadata and deserialized data, which can be aggregated in main thread.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)