You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "ming li (Jira)" <ji...@apache.org> on 2020/12/28 09:16:00 UTC
[jira] [Commented] (FLINK-8790) Improve performance for recovery
from incremental checkpoint
[ https://issues.apache.org/jira/browse/FLINK-8790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17255484#comment-17255484 ]
ming li commented on FLINK-8790:
--------------------------------
Hi, [~sihuazhou]. At present, we use the incremental state of rocksdb to restore when scale up, and found that it is very slow to traverse the state of the corresponding keygroup. Can we now use rocksdb.deleteRange for state clipping?
> Improve performance for recovery from incremental checkpoint
> ------------------------------------------------------------
>
> Key: FLINK-8790
> URL: https://issues.apache.org/jira/browse/FLINK-8790
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
> Affects Versions: 1.5.0
> Reporter: Sihua Zhou
> Assignee: Sihua Zhou
> Priority: Major
> Fix For: 1.6.0
>
>
> When there are multi state handle to be restored, we can improve the performance as follow:
> 1. Choose the best state handle to init the target db
> 2. Use the other state handles to create temp db, and clip the db according to the target key group range (via rocksdb.deleteRange()), this can help use get rid of the `key group check` in
> `data insertion loop` and also help us get rid of traversing the useless record.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)