You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by "Jiayi Liao (Jira)" <ji...@apache.org> on 2020/08/21 03:55:00 UTC

[jira] [Created] (FLINK-19011) Parallelize the restoreOperation in OperatorStateBackend

Jiayi Liao created FLINK-19011:
----------------------------------

             Summary: Parallelize the restoreOperation in OperatorStateBackend 
                 Key: FLINK-19011
                 URL: https://issues.apache.org/jira/browse/FLINK-19011
             Project: Flink
          Issue Type: Improvement
    Affects Versions: 1.11.1
            Reporter: Jiayi Liao


To restore the states, union state needs to read state handles produced by all operators. And currently during the restore operation, Flink iterates the state handles one by one, which could last tens of minutes if the magnitude of state handles exceeds ten thousand. 

To accelerate the process, I propose to parallelize the random reads on HDFS and deserialization. We can create a runnable for each state handle and let it return the metadata and deserialized data, which can be aggregated in main thread.





--
This message was sent by Atlassian Jira
(v8.3.4#803005)