You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Bo Cui (Jira)" <ji...@apache.org> on 2022/02/21 08:25:00 UTC
[jira] [Updated] (HUDI-3460) Flink TM Memory Optimization
[ https://issues.apache.org/jira/browse/HUDI-3460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bo Cui updated HUDI-3460:
-------------------------
Description:
HoodieTableSource uses the MapReduce Memory Config,
[https://github.com/apache/hudi/blob/359fbfde798b50edc06ee1d0520efcd971a289bc/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeRecordReaderUtils.java#L81]
but the config value may be the `parseMaximumHeapSizeMB* 0.8`, excessive memory usage affects flink TM stability ([https://github.com/apache/hadoop/blob/007c2011efe552d54956dce35e8a3ee095cdef2c/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java#L1966])
so HoodieTableSource should uses FlinkOptions's own config and 64MB(Flink rocksdb default is 64MB).
and write.merge.max_memory,compaction.max_memory should also be changed to 64MB
was:
HoodieTableSource uses the MapReduce Memory Config,
[https://github.com/apache/hudi/blob/359fbfde798b50edc06ee1d0520efcd971a289bc/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeRecordReaderUtils.java#L81]
but the config value may be the `MaximumHeapSizeMB * 0.8`, excessive memory usage affects flink TM stability (https://github.com/apache/hadoop/blob/007c2011efe552d54956dce35e8a3ee095cdef2c/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java#L1966)
so HoodieTableSource should uses FlinkOptions's own config and 64MB(Flink rocksdb default is 64MB).
and write.merge.max_memory,compaction.max_memory should also be changed to 64MB
> Flink TM Memory Optimization
> ----------------------------
>
> Key: HUDI-3460
> URL: https://issues.apache.org/jira/browse/HUDI-3460
> Project: Apache Hudi
> Issue Type: Improvement
> Components: flink
> Reporter: Bo Cui
> Priority: Major
>
> HoodieTableSource uses the MapReduce Memory Config,
> [https://github.com/apache/hudi/blob/359fbfde798b50edc06ee1d0520efcd971a289bc/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieRealtimeRecordReaderUtils.java#L81]
> but the config value may be the `parseMaximumHeapSizeMB* 0.8`, excessive memory usage affects flink TM stability ([https://github.com/apache/hadoop/blob/007c2011efe552d54956dce35e8a3ee095cdef2c/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobConf.java#L1966])
> so HoodieTableSource should uses FlinkOptions's own config and 64MB(Flink rocksdb default is 64MB).
>
> and write.merge.max_memory,compaction.max_memory should also be changed to 64MB
--
This message was sent by Atlassian Jira
(v8.20.1#820001)