[GitHub] [iceberg] sunchao commented on a change in pull request #2276: Spark: Add option to combine tasks by partition

[GitBox <gi...@apache.org>]

sunchao commented on a change in pull request #2276:
URL: https://github.com/apache/iceberg/pull/2276#discussion_r586931684



##########
File path: core/src/main/java/org/apache/iceberg/TableProperties.java
##########
@@ -78,6 +78,9 @@ private TableProperties() {
   public static final String SPLIT_OPEN_FILE_COST = "read.split.open-file-cost";
   public static final long SPLIT_OPEN_FILE_COST_DEFAULT = 4 * 1024 * 1024; // 4MB
 
+  public static final String SPLIT_BY_PARTITION = "read.split.by-partition";

Review comment:
       It may not always be desired even if the table is partitioned. For instance, if the partitions of a table only contain one 4MB file each, sometimes it may be more beneficial to combine them into larger splits, especially when you know that the table is not used in a join, or that it is not joined against another table that has the same partitioning (and thus cannot leverage bucket join).




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org