You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "Bo Cui (Jira)" <ji...@apache.org> on 2022/02/21 06:44:00 UTC

[jira] [Created] (HUDI-3460) Flink TM Memory Optimization

Bo Cui created HUDI-3460:
----------------------------

             Summary: Flink TM Memory Optimization
                 Key: HUDI-3460
                 URL: https://issues.apache.org/jira/browse/HUDI-3460
             Project: Apache Hudi
          Issue Type: Improvement
          Components: flink
            Reporter: Bo Cui


HoodieTableSource uses the MapReduce Memory Config,

[https://github.com/apache/hudi/blob/359fbfde798b50edc06ee1d0520efcd971a289bc/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeRecordReaderUtils.java#L81]

but the config value may be the  `MaximumHeapSizeMB * 0.8`, excessive memory usage affects flink TM stability (https://github.com/apache/hadoop/blob/007c2011efe552d54956dce35e8a3ee095cdef2c/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java#L1966)

so HoodieTableSource should uses FlinkOptions's own config and 64MB(Flink rocksdb default is 64MB).

 

and write.merge.max_memory,compaction.max_memory should also be changed to 64MB



--
This message was sent by Atlassian Jira
(v8.20.1#820001)