You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by "Jiayi Liao (Jira)" <ji...@apache.org> on 2020/08/21 03:56:00 UTC

[jira] [Updated] (FLINK-19011) Parallelize the restore operation in OperatorStateBackend

     [ https://issues.apache.org/jira/browse/FLINK-19011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jiayi Liao updated FLINK-19011:
-------------------------------
    Summary: Parallelize the restore operation in OperatorStateBackend   (was: Parallelize the restoreOperation in OperatorStateBackend )

> Parallelize the restore operation in OperatorStateBackend 
> ----------------------------------------------------------
>
>                 Key: FLINK-19011
>                 URL: https://issues.apache.org/jira/browse/FLINK-19011
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / Checkpointing
>    Affects Versions: 1.11.1
>            Reporter: Jiayi Liao
>            Priority: Major
>
> To restore the states, union state needs to read state handles produced by all operators. And currently during the restore operation, Flink iterates the state handles one by one, which could last tens of minutes if the magnitude of state handles exceeds ten thousand. 
> To accelerate the process, I propose to parallelize the random reads on HDFS and deserialization. We can create a runnable for each state handle and let it return the metadata and deserialized data, which can be aggregated in main thread.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)