[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #204: Combine tasks to scan up to target split size using parquet row group information

[GitBox <gi...@apache.org>]

rdblue commented on a change in pull request #204: Combine tasks to scan up to target split size using parquet row group information
URL: https://github.com/apache/incubator-iceberg/pull/204#discussion_r290070370
 
 

 ##########
 File path: core/src/main/java/org/apache/iceberg/BaseFileScanTask.java
 ##########
 @@ -91,28 +93,66 @@ public String toString() {
         .toString();
   }
 
+
+  /**
+   * This iterator returns {@link FileScanTask} using guidance provided by split offsets.
+   */
   @VisibleForTesting
-  static final class OffsetsBasedSplitScanTaskIterator implements Iterator<FileScanTask> {
-    private final List<Long> splitOffsets;
+  static final class OffsetsAwareTargetSplitSizeScanTaskIterator implements Iterator<FileScanTask> {
+    private final List<Long> offsets;
+    private final List<Long> splitSizes;
     private final FileScanTask parentScanTask;
-    private int idx = 0;
-
-    OffsetsBasedSplitScanTaskIterator(List<Long> splitOffsets, FileScanTask fileScanTask) {
-      this.splitOffsets = splitOffsets;
-      this.parentScanTask = fileScanTask;
+    private final long targetSplitSize;
+    private int offsetIdx = 0;
+    private int sizeIdx = 0;
+
+    OffsetsAwareTargetSplitSizeScanTaskIterator(
+        List<Long> offsetList,
+        FileScanTask parentScanTask,
+        long targetSplitSize
+    ) {
+      this.offsets = ImmutableList.copyOf(offsetList);
+      this.parentScanTask = parentScanTask;
+      this.targetSplitSize = targetSplitSize;
+      this.splitSizes = new ArrayList<>(offsetList.size());
+      int idx = 0;
+      while (idx < offsets.size()) {
 
 Review comment:
   Could you use a for loop instead of a while?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org