You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flink.apache.org by "Weijie Guo (Jira)" <ji...@apache.org> on 2022/12/08 03:48:00 UTC

[jira] [Created] (FLINK-30332) HsFileDataIndex should supports spill to file

Weijie Guo created FLINK-30332:
----------------------------------

             Summary: HsFileDataIndex should supports spill to file
                 Key: FLINK-30332
                 URL: https://issues.apache.org/jira/browse/FLINK-30332
             Project: Flink
          Issue Type: Sub-task
            Reporter: Weijie Guo
            Assignee: Weijie Guo


`HsFileDataIndex` is maintains in a list of treeMap(i.e. in memory), there is no problem in our 10T TPC-DS test. However, with the increase of job's data volume, the index will continue to grow, and there is a risk of OOM. We should support index data spilling and loading to avoid this problem.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)