[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #3292: Spark: Compact Medium Size Files (#460)

[GitBox <gi...@apache.org>]

RussellSpitzer commented on a change in pull request #3292:
URL: https://github.com/apache/iceberg/pull/3292#discussion_r785002484



##########
File path: core/src/main/java/org/apache/iceberg/BaseFileScanTask.java
##########
@@ -224,5 +217,51 @@ public Expression residual() {
     public Iterable<FileScanTask> split(long splitSize) {
       throw new UnsupportedOperationException("Cannot split a task which is already split");
     }
+
+    public boolean isAdjacent(SplitScanTask other) {
+      return (other != null) &&
+          (this.file().equals(other.file())) &&
+          (this.offset + this.len == other.offset);
+    }
+  }
+
+  static FileScanTask[] combineSimilarTasks(List<FileScanTask> tasks) {
+    if (tasks.isEmpty()) {
+      return new FileScanTask[0];
+    }
+
+    List<FileScanTask> combinedScans = Lists.newArrayList();
+    SplitScanTask lastSplit = null;
+
+    for (FileScanTask fileScanTask : tasks) {
+      if (!(fileScanTask instanceof SplitScanTask)) {
+        // We do not know how to combine anything but SplitScanTasks
+        combinedScans.add(fileScanTask);
+      } else {
+        SplitScanTask split = (SplitScanTask) fileScanTask;
+        if (lastSplit != null) {
+          if (lastSplit.isAdjacent(split)) {
+            // We can merge with the last split
+            lastSplit = new SplitScanTask(
+                lastSplit.offset,
+                lastSplit.len + split.len,
+                lastSplit.fileScanTask);
+          } else {
+            // We cannot merge with the last split, add it to the set of tasks
+            combinedScans.add(lastSplit);
+            lastSplit = split;
+          }
+        } else {
+          // We haven't seen another split yet, save this split in-case we can combine it later
+          lastSplit = split;
+        }
+      }
+    }
+
+    if (lastSplit != null) {
+      combinedScans.add(lastSplit);
+    }
+
+    return combinedScans.stream().toArray(FileScanTask[]::new);

Review comment:
       Sounds good to me




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org