[GitHub] [hudi] swuferhong commented on a change in pull request #3169: [HUDI-2085] Support specify compaction paralleism and compaction targ…

[GitBox <gi...@apache.org>]

swuferhong commented on a change in pull request #3169:
URL: https://github.com/apache/hudi/pull/3169#discussion_r660232507



##########
File path: hudi-flink/src/main/java/org/apache/hudi/sink/compact/FlinkCompactionConfig.java
##########
@@ -83,6 +83,12 @@
   @Parameter(names = {"--compaction-max-memory"}, description = "Max memory in MB for compaction spillable map, default 100MB.", required = false)
   public Integer compactionMaxMemory = 100;
 
+  @Parameter(names = {"--compaction-target-io"}, description = "Target IO per compaction (both read and write) for batching compaction, default 512000M.", required = false)

Review comment:
       > Should FlinkOptions#COMPACTION_TARGET_IO default value change to 512000 to align with HoodieCompactionConfig#DEFAULT_TARGET_IO_PER_COMPACTION_IN_MB and here?
   
   After we used a large amount of data to test writing, we found that the online compaction setting FlinkOptions#COMPACTION_TARGET_IO  cannot be setted too large, which would seriously affect the running of the writing tasks. In contrast, the offline compaction target io default value can be set larger while we have enough memory.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org