[GitHub] [flink-table-store] JingsongLi commented on a diff in pull request #131: [FLINK-27696] Add bin-pack strategy to split the whole bucket data files into several small splits for append-only table.

[GitBox <gi...@apache.org>]

JingsongLi commented on code in PR #131:
URL: https://github.com/apache/flink-table-store/pull/131#discussion_r878813019


##########
flink-table-store-core/src/main/java/org/apache/flink/table/store/file/FileStoreOptions.java:
##########
@@ -84,6 +84,13 @@ public class FileStoreOptions implements Serializable {
                             "To avoid frequent manifest merges, this parameter specifies the minimum number "
                                     + "of ManifestFileMeta to merge.");
 
+    public static final ConfigOption<MemorySize> TARGET_TASK_SPLIT_SIZE =
+            ConfigOptions.key("target-task-split-size")

Review Comment:
   target-source-split-size?



##########
flink-table-store-core/src/main/java/org/apache/flink/table/store/file/FileStoreOptions.java:
##########
@@ -84,6 +84,13 @@ public class FileStoreOptions implements Serializable {
                             "To avoid frequent manifest merges, this parameter specifies the minimum number "
                                     + "of ManifestFileMeta to merge.");
 
+    public static final ConfigOption<MemorySize> TARGET_TASK_SPLIT_SIZE =
+            ConfigOptions.key("target-task-split-size")
+                    .memoryType()
+                    .defaultValue(MemorySize.ofMebiBytes(256))

Review Comment:
   maybe 128 MB？



##########
flink-table-store-core/src/main/java/org/apache/flink/table/store/file/FileStoreOptions.java:
##########
@@ -84,6 +84,13 @@ public class FileStoreOptions implements Serializable {
                             "To avoid frequent manifest merges, this parameter specifies the minimum number "
                                     + "of ManifestFileMeta to merge.");
 
+    public static final ConfigOption<MemorySize> TARGET_TASK_SPLIT_SIZE =
+            ConfigOptions.key("target-task-split-size")
+                    .memoryType()
+                    .defaultValue(MemorySize.ofMebiBytes(256))
+                    .withDescription(
+                            "Target size of a split task when scanning a bucket in the append-only table.");

Review Comment:
   We can use this for changelog table too.
   We can use `IntervalPartition` for files. The files can be split into multiple sets without intersections, i.e. multiple splits.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org