[jira] [Updated] (FLINK-12699) Reduce CPU consumption when snapshot/restore the spilled key-group

["Flink Jira Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-12699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Flink Jira Bot updated FLINK-12699:
-----------------------------------
    Labels: stale-assigned  (was: )

I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 14, so it has been labeled "stale-assigned".
If you are still working on the issue, please remove the label and add a comment updating the community on your progress.  If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience.
If you are no longer working on the issue, please unassign yourself so someone else may work on it. If the "warning_label" label is not removed in 7 days, the issue will be automatically unassigned.


> Reduce CPU consumption when snapshot/restore the spilled key-group
> ------------------------------------------------------------------
>
>                 Key: FLINK-12699
>                 URL: https://issues.apache.org/jira/browse/FLINK-12699
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Runtime / State Backends
>            Reporter: Yu Li
>            Assignee: PengFei Li
>            Priority: Major
>              Labels: stale-assigned
>
> We need to prevent the unnecessary de/serialization when snapshotting/restoring the spilled state key-group. To achieve this, we need to:
> 1. Add meta information for {{HeapKeyedStatebackend}} checkpoint on DFS, separating the on-heap and on-disk part
> 2. Write the off-heap bytes directly to DFS when checkpointing and mark it as on-disk
> 3. Directly write the bytes onto disk when restoring the data back from DFS, if it's marked as on-disk
> Notice that we cannot directly use file copy since we use mmap meanwhile support copy-on-write.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)