You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "pvary (via GitHub)" <gi...@apache.org> on 2024/11/11 10:54:57 UTC

[PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

pvary opened a new pull request, #11513:
URL: https://github.com/apache/iceberg/pull/11513

   As requested in #11497 extracted the Spark and Core related changes.
   The change refactors the Spark RewriteFileGroup related code to core (`RewriteFileGroupPlanner`), so Flink could later reuse it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor FileRewriter interface to separate planning and execution [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#issuecomment-2582253213

   not stale


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor FileRewriter interface to separate planning and execution [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1876309503


##########
core/src/main/java/org/apache/iceberg/actions/FileRewritePlan.java:
##########
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.util.Map;
+import java.util.stream.Stream;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.ContentScanTask;
+import org.apache.iceberg.StructLike;
+
+/**
+ * Result of the file rewrite planning.
+ *
+ * <p>Contains the planned groups, calculated values required by the {@link FileRewriteExecutor}s
+ * and statistics.
+ *
+ * @param <I> the Java type of the plan info
+ * @param <T> the Java type of the tasks to read content files
+ * @param <F> the Java type of the content files
+ * @param <G> the Java type of the planned groups
+ */
+public abstract class FileRewritePlan<
+    I,
+    T extends ContentScanTask<F>,
+    F extends ContentFile<F>,
+    G extends FileRewriteGroup<I, T, F>> {
+  private final Stream<G> groups;
+  private final int totalGroupCount;
+  private final Map<StructLike, Integer> groupsInPartition;
+  private final long writeMaxFileSize;
+
+  protected FileRewritePlan(
+      Stream<G> groups,
+      int totalGroupCount,
+      Map<StructLike, Integer> groupsInPartition,
+      long writeMaxFileSize) {
+    this.groups = groups;
+    this.totalGroupCount = totalGroupCount;
+    this.groupsInPartition = groupsInPartition;
+    this.writeMaxFileSize = writeMaxFileSize;
+  }
+
+  /** The stream of the generated {@link FileRewriteGroup}s. */
+  public Stream<G> groups() {
+    return groups;
+  }
+
+  /** The number of the generated groups in the given partition. */
+  public int groupsInPartition(StructLike partition) {
+    return groupsInPartition.get(partition);
+  }
+
+  /** The total number of the groups generated by this plan. */
+  public int totalGroupCount() {
+    return totalGroupCount;
+  }
+
+  /** Calculated maximum file size for the target files */

Review Comment:
   Not sure what "target" means here. Maximum output file size?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor FileRewriter interface to separate planning and execution [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#issuecomment-2800203357

   This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1844627456


##########
core/src/main/java/org/apache/iceberg/actions/RewriteFileGroupPlanner.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Stream;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.RewriteJobOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.StructLikeMap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Checks the files in the table, and using the {@link FileRewriter} plans the groups for

Review Comment:
   I've been thinking a lot about the refactor. Trying to figure out what we should do.
   
   we could possibly do
   
   Rewriter[T]
     has a Planner[T] // Responsible for generating tasks (move actual filtering into rewriter)
   Maybe we should also have
   ```
   Planner[T]
       DeletePlanner[DeleteScanTask]
       FilePlanner[FileScanTask]
   ```    
       
   So then you would have
   ```
   SizeBasedDataRewriter <FileScanTask, DataFile>
      private planner = new FilePlanner<FileScanTask>
   ```
   
   Would that end up being more complicated? I feel like we are getting some interdependencies here I'm not comfortable with extending. @aokolnychyi Probably has some smart things to say here as well



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1867335924


##########
core/src/main/java/org/apache/iceberg/actions/FileRewritePlan.java:
##########
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.util.Map;
+import java.util.stream.Stream;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.ContentScanTask;
+import org.apache.iceberg.StructLike;
+
+/**
+ * Result of the file rewrite planning.
+ *
+ * <p>Contains the planned groups, calculated values required by the {@link FileRewriteExecutor}s
+ * and statistics.
+ *
+ * @param <I> the Java type of the plan info
+ * @param <T> the Java type of the tasks to read content files
+ * @param <F> the Java type of the content files
+ * @param <G> the Java type of the planned groups
+ */
+public abstract class FileRewritePlan<
+    I,
+    T extends ContentScanTask<F>,
+    F extends ContentFile<F>,
+    G extends FileRewriteGroup<I, T, F>> {
+  private final Stream<G> groups;

Review Comment:
   Currently the groups are calculated in advance to allow sorting, so instead of a stream we could provide a `Collection` or `Set`.
   Kept it as `Stream` as the previous iteration of the code used stream, but I would prefer `Collection`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor FileRewriter interface to separate planning and execution [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1941147406


##########
core/src/main/java/org/apache/iceberg/actions/RewriteFileGroup.java:
##########
@@ -31,26 +31,26 @@
 import org.apache.iceberg.util.DataFileSet;
 
 /**
- * Container class representing a set of files to be rewritten by a RewriteAction and the new files
- * which have been written by the action.
+ * Container class representing a set of data files to be rewritten by a RewriteAction and the new
+ * files which have been written by the action.
  */
-public class RewriteFileGroup {
-  private final FileGroupInfo info;
-  private final List<FileScanTask> fileScanTasks;
-
+public class RewriteFileGroup extends FileRewriteGroup<FileGroupInfo, FileScanTask, DataFile> {

Review Comment:
   Maybe `RewriteDataFileGroup` could be better?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1852559207


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/SparkRewriteExecutor.java:
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.actions;
+
+import java.util.Map;
+import java.util.Set;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.ContentScanTask;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.actions.FileRewriteExecutor;
+import org.apache.iceberg.actions.FileRewriteGroup;
+import org.apache.iceberg.actions.FileRewritePlan;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+
+/**
+ * Common parent for data and positional delete rewrite executors.
+ *
+ * @param <I> the Java type of the plan info
+ * @param <T> the Java type of the tasks to read content files
+ * @param <F> the Java type of the content files
+ * @param <G> the Java type of the planned groups
+ */
+abstract class SparkRewriteExecutor<
+        I,
+        T extends ContentScanTask<F>,
+        F extends ContentFile<F>,
+        G extends FileRewriteGroup<I, T, F>>
+    implements FileRewriteExecutor<I, T, F, G> {
+  private final Table table;
+  private long writeMaxFileSize;
+  private int outputSpecId;
+
+  SparkRewriteExecutor(Table table) {
+    this.table = table;
+  }
+
+  Table table() {
+    return table;
+  }
+
+  long writeMaxFileSize() {
+    return writeMaxFileSize;
+  }
+
+  int outputSpecId() {
+    return outputSpecId;
+  }
+
+  PartitionSpec outputSpec() {
+    return table.specs().get(outputSpecId);
+  }
+
+  @Override
+  public void initPlan(FileRewritePlan<I, T, F, G> plan) {

Review Comment:
   If we are fairly confident, that only `writeMaxFileSize` and `outputSpecId` comes from the plan, then we can add `setWriteMaxFileSize` and `setOutputSpecId` to the `FileRewriteExecutor` API instead of the `initPlan(FileRewritePlan)`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor FileRewriter interface to separate planning and execution [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1876292543


##########
core/src/main/java/org/apache/iceberg/actions/FileRewriteGroup.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.util.Comparator;
+import java.util.List;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.ContentScanTask;
+import org.apache.iceberg.RewriteJobOrder;
+
+/**
+ * Container class representing a set of files to be rewritten by a {@link FileRewriteExecutor}.

Review Comment:
   This doesn't seem to be the case since it also has 
   ```java
     /** Expected split size for the output files. */
     public long splitSize() {
       return splitSize;
     }
   
     /** Expected number of the output files. */
     public int expectedOutputFiles() {
       return expectedOutputFiles;
     }
     ```
     
    The first one "splitSize" feels like it's global to a lot of FileRewriteGroups and not a particular property of 1 RewriteGroup, so may that belongs in Plan
    
    "estimatedOutputFiles" could live here but maybe just makes sense as 
    estimatedOutputFiles(splitSize), where split size is passed in from elsewhere?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor FileRewriter interface to separate planning and execution [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1876417391


##########
core/src/main/java/org/apache/iceberg/actions/FileRewriteExecutor.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.util.Map;
+import java.util.Set;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.ContentScanTask;
+
+/**
+ * A class for rewriting content file groups ({@link FileRewriteGroup}).
+ *
+ * @param <I> the Java type of the plan info
+ * @param <T> the Java type of the tasks to read content files
+ * @param <F> the Java type of the content files
+ * @param <G> the Java type of the planned groups
+ * @param <P> the Java type of the plan to execute
+ */
+public interface FileRewriteExecutor<
+    I,
+    T extends ContentScanTask<F>,
+    F extends ContentFile<F>,
+    G extends FileRewriteGroup<I, T, F>,
+    P extends FileRewritePlan<I, T, F, G>> {
+
+  /** Returns a description for this rewriter. */
+  default String description() {
+    return getClass().getName();
+  }
+
+  /**
+   * Returns a set of supported options for this rewriter. Only options specified in this list will
+   * be accepted at runtime. Any other options will be rejected.
+   */
+  Set<String> validOptions();
+
+  /**
+   * Initializes this rewriter using provided options.
+   *
+   * @param options options to initialize this rewriter
+   */
+  void init(Map<String, String> options);

Review Comment:
   Will add javadoc.
   - `init` is called when the executor is initialized
   - `initPlan` is called when the plan has been generated and the parameters needed for the executor is calculated (`writeMaxFileSize`, `outputSpecId`)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor FileRewriter interface to separate planning and execution [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1876636215


##########
core/src/main/java/org/apache/iceberg/actions/FileRewritePlan.java:
##########
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.util.Map;
+import java.util.stream.Stream;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.ContentScanTask;
+import org.apache.iceberg.StructLike;
+
+/**
+ * Result of the file rewrite planning.

Review Comment:
   Refactored the javadoc



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#issuecomment-2476466989

   @RussellSpitzer: This is ready to another round if you have time.
   Thanks,
   Peter


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1844346161


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java:
##########
@@ -341,20 +274,19 @@ private Builder doExecute(
   }
 
   private Builder doExecuteWithPartialProgress(
-      RewriteExecutionContext ctx,
-      Stream<RewriteFileGroup> groupStream,
-      RewriteDataFilesCommitManager commitManager) {
+      RewritePlan planResult, RewriteDataFilesCommitManager commitManager) {

Review Comment:
   Nit: why not just variable name plan (same comment for other relevant parts of the file) ?
   
   Planner => Plan (no need for result imo)



##########
core/src/main/java/org/apache/iceberg/actions/RewriteFileGroupPlanner.java:
##########
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Stream;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.RewriteJobOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.StructLikeMap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Checks the files in the {@link Table}. The {@link RewriteFileGroup}s are grouped by partitions

Review Comment:
   Nit: "Checks" may be a bit not specific.  How about:
   
   ```Groups specified files in the Table by RewriteFileGroups.  These will be grouped by partitions.```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor FileRewriter interface to separate planning and execution [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1876590583


##########
core/src/main/java/org/apache/iceberg/actions/FileRewriteExecutor.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.util.Map;
+import java.util.Set;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.ContentScanTask;
+
+/**
+ * A class for rewriting content file groups ({@link FileRewriteGroup}).

Review Comment:
   Refactored the javadoc



##########
core/src/main/java/org/apache/iceberg/actions/FileRewriteExecutor.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.util.Map;
+import java.util.Set;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.ContentScanTask;
+
+/**
+ * A class for rewriting content file groups ({@link FileRewriteGroup}).
+ *
+ * @param <I> the Java type of the plan info
+ * @param <T> the Java type of the tasks to read content files
+ * @param <F> the Java type of the content files
+ * @param <G> the Java type of the planned groups
+ * @param <P> the Java type of the plan to execute
+ */
+public interface FileRewriteExecutor<
+    I,
+    T extends ContentScanTask<F>,
+    F extends ContentFile<F>,
+    G extends FileRewriteGroup<I, T, F>,
+    P extends FileRewritePlan<I, T, F, G>> {
+
+  /** Returns a description for this rewriter. */
+  default String description() {
+    return getClass().getName();
+  }
+
+  /**
+   * Returns a set of supported options for this rewriter. Only options specified in this list will
+   * be accepted at runtime. Any other options will be rejected.
+   */
+  Set<String> validOptions();
+
+  /**
+   * Initializes this rewriter using provided options.
+   *
+   * @param options options to initialize this rewriter
+   */
+  void init(Map<String, String> options);

Review Comment:
   Refactored the javadoc



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor FileRewriter interface to separate planning and execution [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1876644495


##########
core/src/main/java/org/apache/iceberg/actions/FileRewritePlanner.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.util.Map;
+import java.util.Set;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.ContentScanTask;
+
+/**
+ * A class for planning content file rewrites.
+ *
+ * <p>The entire rewrite operation is broken down into pieces based on partitioning, and size-based
+ * groups within a partition. These subunits of the rewrite are referred to as file groups. A file

Review Comment:
   Refactored the javadoc



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor FileRewriter interface to separate planning and execution [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1876318833


##########
core/src/main/java/org/apache/iceberg/actions/FileRewritePlanner.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.util.Map;
+import java.util.Set;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.ContentScanTask;
+
+/**
+ * A class for planning content file rewrites.
+ *
+ * <p>The entire rewrite operation is broken down into pieces based on partitioning, and size-based
+ * groups within a partition. These subunits of the rewrite are referred to as file groups. A file

Review Comment:
   Some of these details belong in the details of FileGroup. This doc needs to talk about the actual planning



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1841611540


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java:
##########
@@ -276,29 +215,25 @@ RewriteDataFilesCommitManager commitManager(long startingSnapshotId) {
   }
 
   private Builder doExecute(
-      RewriteExecutionContext ctx,
-      Stream<RewriteFileGroup> groupStream,
-      RewriteDataFilesCommitManager commitManager) {
+      RewritePlanResult planResult, RewriteDataFilesCommitManager commitManager) {
     ExecutorService rewriteService = rewriteService();
 
     ConcurrentLinkedQueue<RewriteFileGroup> rewrittenGroups = Queues.newConcurrentLinkedQueue();
 
     Tasks.Builder<RewriteFileGroup> rewriteTaskBuilder =
-        Tasks.foreach(groupStream)
+        Tasks.foreach(planResult.groups())
             .executeWith(rewriteService)
             .stopOnFailure()
             .noRetry()
             .onFailure(
-                (fileGroup, exception) -> {
-                  LOG.warn(
-                      "Failure during rewrite process for group {}", fileGroup.info(), exception);
-                });
+                (fileGroup, exception) ->
+                    LOG.warn(

Review Comment:
   My mistake. I changed the line in my previous version, and fixed an IntelliJ warning too, then reverted my change but not the removal of the curly brackets



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1867620928


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/SparkShufflingDataRewritePlanner.java:
##########
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.actions;
+
+import java.util.Map;
+import java.util.Set;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.actions.RewriteFileGroupPlanner;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+import org.apache.iceberg.util.PropertyUtil;
+
+/**
+ * Extends the {@link RewriteFileGroupPlanner} with the possibility to set the expected compression
+ * factor.
+ */
+public class SparkShufflingDataRewritePlanner extends RewriteFileGroupPlanner {

Review Comment:
   The only reason for this class is that `compression-factor` generates different target groups.
   If we agree that this configuration should be available in `RewriteFileGroupPlanner`, or every Spark Planner, then we can get rid of this extra class and some related nastiness in `RewriteDataFilesSparkAction` (shufflingPlanner).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor FileRewriter interface to separate planning and execution [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#issuecomment-2581488381

   This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor FileRewriter interface to separate planning and execution [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1876409895


##########
core/src/main/java/org/apache/iceberg/actions/FileRewriteGroup.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.util.Comparator;
+import java.util.List;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.ContentScanTask;
+import org.apache.iceberg.RewriteJobOrder;
+
+/**
+ * Container class representing a set of files to be rewritten by a {@link FileRewriteExecutor}.
+ *
+ * @param <I> the Java type of the plan info
+ * @param <T> the Java type of the tasks to read content files
+ * @param <F> the Java type of the content files
+ */
+public abstract class FileRewriteGroup<I, T extends ContentScanTask<F>, F extends ContentFile<F>> {
+  private final I info;
+  private final List<T> fileScanTasks;
+  private final long splitSize;
+  private final int expectedOutputFiles;
+
+  FileRewriteGroup(I info, List<T> fileScanTasks, long splitSize, int expectedOutputFiles) {
+    this.info = info;
+    this.fileScanTasks = fileScanTasks;
+    this.splitSize = splitSize;
+    this.expectedOutputFiles = expectedOutputFiles;
+  }
+
+  /** Identifiers and partition information about the group. */

Review Comment:
   This is `RewriteDataFiles.FileGroupInfo` and `RewritePositionDeleteFiles.FileGroupInfo`
   These are inherited from the old code



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor FileRewriter interface to separate planning and execution [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1876279906


##########
core/src/main/java/org/apache/iceberg/actions/FileRewriteGroup.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.util.Comparator;
+import java.util.List;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.ContentScanTask;
+import org.apache.iceberg.RewriteJobOrder;
+
+/**
+ * Container class representing a set of files to be rewritten by a {@link FileRewriteExecutor}.
+ *
+ * @param <I> the Java type of the plan info
+ * @param <T> the Java type of the tasks to read content files
+ * @param <F> the Java type of the content files
+ */
+public abstract class FileRewriteGroup<I, T extends ContentScanTask<F>, F extends ContentFile<F>> {
+  private final I info;
+  private final List<T> fileScanTasks;
+  private final long splitSize;
+  private final int expectedOutputFiles;
+
+  FileRewriteGroup(I info, List<T> fileScanTasks, long splitSize, int expectedOutputFiles) {
+    this.info = info;
+    this.fileScanTasks = fileScanTasks;
+    this.splitSize = splitSize;
+    this.expectedOutputFiles = expectedOutputFiles;
+  }
+
+  /** Identifiers and partition information about the group. */

Review Comment:
   Why isn't this just a structlike? What other class types might we be passing through here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor FileRewriter interface to separate planning and execution [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#issuecomment-2705224536

   This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#issuecomment-2483756188

   @pvary and I were talking this over a bit, I think we really want to get a stronger division between "Planning" and "Excecution" since the two are very intertwined right now.
   
   Ideally we end up in a situation where core has base classes that are only responsible for planning out exactly which files should be rewritten and in what logical groups they should be rewritten in, while another class is responsible for the physical implementation on how that actually occurs. Currently I think our structure is: Action contains Rewriter which extends a Planning Class with an Implementation. Because of this the Action and rewriter talk back and forth causing planning to occur in both the action and with parts of code in the rewriter.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1841924134


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java:
##########
@@ -168,89 +160,36 @@ public RewriteDataFiles.Result execute() {
 
     validateAndInitOptions();
 
-    StructLikeMap<List<List<FileScanTask>>> fileGroupsByPartition =
-        planFileGroups(startingSnapshotId);
-    RewriteExecutionContext ctx = new RewriteExecutionContext(fileGroupsByPartition);
+    RewritePlanResult result = plan(startingSnapshotId);

Review Comment:
   Renamed.
   Thanks for the suggestion!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1867596164


##########
core/src/main/java/org/apache/iceberg/actions/RewritePositionDeletePlan.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.util.Map;
+import java.util.stream.Stream;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.PositionDeletesScanTask;
+import org.apache.iceberg.StructLike;
+
+/** Result of the positional delete file rewrite planning. */
+public class RewritePositionDeletePlan

Review Comment:
   We could use `FileRewritePlan<
           RewritePositionDeleteFiles.FileGroupInfo,
           PositionDeletesScanTask,
           DeleteFile,
           RewritePositionDeletesGroup>` everywhere instead of creating a new class, but after I had to create the `RewriteFilePlan` to handle `outputSpecId`, this seems more elegant



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor FileRewriter interface to separate planning and execution [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1876273357


##########
core/src/main/java/org/apache/iceberg/actions/FileRewriteExecutor.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.util.Map;
+import java.util.Set;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.ContentScanTask;
+
+/**
+ * A class for rewriting content file groups ({@link FileRewriteGroup}).
+ *
+ * @param <I> the Java type of the plan info
+ * @param <T> the Java type of the tasks to read content files
+ * @param <F> the Java type of the content files
+ * @param <G> the Java type of the planned groups
+ * @param <P> the Java type of the plan to execute
+ */
+public interface FileRewriteExecutor<
+    I,
+    T extends ContentScanTask<F>,
+    F extends ContentFile<F>,
+    G extends FileRewriteGroup<I, T, F>,
+    P extends FileRewritePlan<I, T, F, G>> {
+
+  /** Returns a description for this rewriter. */
+  default String description() {
+    return getClass().getName();
+  }
+
+  /**
+   * Returns a set of supported options for this rewriter. Only options specified in this list will
+   * be accepted at runtime. Any other options will be rejected.
+   */
+  Set<String> validOptions();
+
+  /**
+   * Initializes this rewriter using provided options.
+   *
+   * @param options options to initialize this rewriter
+   */
+  void init(Map<String, String> options);

Review Comment:
   it is not clear from the definition here why initPlan and init are both required. It is also not clear when they are called or if they are both called etc ...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor FileRewriter interface to separate planning and execution [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1876643808


##########
core/src/main/java/org/apache/iceberg/actions/FileRewritePlanner.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.util.Map;
+import java.util.Set;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.ContentScanTask;
+
+/**
+ * A class for planning content file rewrites.
+ *
+ * <p>The entire rewrite operation is broken down into pieces based on partitioning, and size-based

Review Comment:
   Copied from [FileRewriter](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/actions/FileRewriter.java#L30), but agree with your assessment, so rewritten the javadoc.
   
   The partitioning is currently necessary (see `FileRewritePlan#groupsInPartition`). Removed the size-based part.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1836493600


##########
core/src/main/java/org/apache/iceberg/actions/SizeBasedFileRewriter.java:
##########
@@ -191,7 +191,7 @@ protected long inputSize(List<T> group) {
    * of output files. The final split size is adjusted to be at least as big as the target file size
    * but less than the max write file size.
    */
-  protected long splitSize(long inputSize) {
+  public long splitSize(long inputSize) {

Review Comment:
   This is somewhat unrelated change - Flink will need to extract the split size since in the Flink compaction implementation the planner which needs its own rewriter, and the executor (where the rewriter is not used) runs on a different instance.
   We could put into the `FileGroupInfo`, but as a first run I opted for the minimal change, especially on public APIs.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1841892746


##########
core/src/main/java/org/apache/iceberg/actions/RewriteFileGroupPlanner.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Stream;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.RewriteJobOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.StructLikeMap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Checks the files in the table, and using the {@link FileRewriter} plans the groups for
+ * compaction.
+ */
+public class RewriteFileGroupPlanner {
+  private static final Logger LOG = LoggerFactory.getLogger(RewriteFileGroupPlanner.class);
+
+  private final FileRewriter<FileScanTask, DataFile> rewriter;
+  private final RewriteJobOrder rewriteJobOrder;
+
+  public RewriteFileGroupPlanner(
+      FileRewriter<FileScanTask, DataFile> rewriter, RewriteJobOrder rewriteJobOrder) {
+    this.rewriter = rewriter;
+    this.rewriteJobOrder = rewriteJobOrder;
+  }
+
+  public RewritePlanResult plan(Table table, Expression filter, long startingSnapshotId) {
+    StructLikeMap<List<List<FileScanTask>>> plan =
+        planFileGroups(table, filter, startingSnapshotId);
+    RewriteExecutionContext ctx = new RewriteExecutionContext();
+    Stream<RewriteFileGroup> groups =
+        plan.entrySet().stream()
+            .filter(e -> !e.getValue().isEmpty())
+            .flatMap(
+                e -> {
+                  StructLike partition = e.getKey();
+                  List<List<FileScanTask>> scanGroups = e.getValue();
+                  return scanGroups.stream().map(tasks -> newRewriteGroup(ctx, partition, tasks));
+                })
+            .sorted(RewriteFileGroup.comparator(rewriteJobOrder));
+    Map<StructLike, Integer> groupsInPartition = plan.transformValues(List::size);
+    int totalGroupCount = groupsInPartition.values().stream().reduce(Integer::sum).orElse(0);
+    return new RewritePlanResult(groups, totalGroupCount, groupsInPartition);
+  }
+
+  private StructLikeMap<List<List<FileScanTask>>> planFileGroups(
+      Table table, Expression filter, long startingSnapshotId) {
+    CloseableIterable<FileScanTask> fileScanTasks =
+        table
+            .newScan()
+            .useSnapshot(startingSnapshotId)
+            .filter(filter)
+            .ignoreResiduals()
+            .planFiles();
+
+    try {
+      Types.StructType partitionType = table.spec().partitionType();
+      StructLikeMap<List<FileScanTask>> filesByPartition =
+          groupByPartition(table, partitionType, fileScanTasks);
+      return filesByPartition.transformValues(
+          tasks -> ImmutableList.copyOf(rewriter.planFileGroups(tasks)));
+    } finally {
+      try {
+        fileScanTasks.close();
+      } catch (IOException io) {
+        LOG.error("Cannot properly close file iterable while planning for rewrite", io);
+      }
+    }
+  }
+
+  private StructLikeMap<List<FileScanTask>> groupByPartition(
+      Table table, Types.StructType partitionType, Iterable<FileScanTask> tasks) {
+    StructLikeMap<List<FileScanTask>> filesByPartition = StructLikeMap.create(partitionType);
+    StructLike emptyStruct = GenericRecord.create(partitionType);
+
+    for (FileScanTask task : tasks) {
+      // If a task uses an incompatible partition spec the data inside could contain values
+      // which belong to multiple partitions in the current spec. Treating all such files as
+      // un-partitioned and grouping them together helps to minimize new files made.
+      StructLike taskPartition =
+          task.file().specId() == table.spec().specId() ? task.file().partition() : emptyStruct;
+
+      filesByPartition.computeIfAbsent(taskPartition, unused -> Lists.newArrayList()).add(task);
+    }
+
+    return filesByPartition;
+  }
+
+  private RewriteFileGroup newRewriteGroup(
+      RewriteExecutionContext ctx, StructLike partition, List<FileScanTask> tasks) {
+    RewriteDataFiles.FileGroupInfo info =
+        ImmutableRewriteDataFiles.FileGroupInfo.builder()
+            .globalIndex(ctx.currentGlobalIndex())
+            .partitionIndex(ctx.currentPartitionIndex(partition))
+            .partition(partition)
+            .build();
+    return new RewriteFileGroup(info, Lists.newArrayList(tasks));
+  }
+
+  public static class RewritePlanResult {

Review Comment:
   Added the javadoc.
   
   I think shipping around a stream is a non-trivial task, so I would keep it not Serializable for now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1842135605


##########
core/src/main/java/org/apache/iceberg/actions/RewriteFileGroupPlanner.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Stream;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.RewriteJobOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.StructLikeMap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Checks the files in the table, and using the {@link FileRewriter} plans the groups for
+ * compaction.
+ */
+public class RewriteFileGroupPlanner {
+  private static final Logger LOG = LoggerFactory.getLogger(RewriteFileGroupPlanner.class);
+
+  private final FileRewriter<FileScanTask, DataFile> rewriter;
+  private final RewriteJobOrder rewriteJobOrder;
+
+  public RewriteFileGroupPlanner(
+      FileRewriter<FileScanTask, DataFile> rewriter, RewriteJobOrder rewriteJobOrder) {
+    this.rewriter = rewriter;
+    this.rewriteJobOrder = rewriteJobOrder;
+  }
+
+  public RewritePlanResult plan(Table table, Expression filter, long startingSnapshotId) {
+    StructLikeMap<List<List<FileScanTask>>> plan =
+        planFileGroups(table, filter, startingSnapshotId);
+    RewriteExecutionContext ctx = new RewriteExecutionContext();
+    Stream<RewriteFileGroup> groups =
+        plan.entrySet().stream()
+            .filter(e -> !e.getValue().isEmpty())
+            .flatMap(
+                e -> {
+                  StructLike partition = e.getKey();
+                  List<List<FileScanTask>> scanGroups = e.getValue();
+                  return scanGroups.stream().map(tasks -> newRewriteGroup(ctx, partition, tasks));
+                })
+            .sorted(RewriteFileGroup.comparator(rewriteJobOrder));
+    Map<StructLike, Integer> groupsInPartition = plan.transformValues(List::size);
+    int totalGroupCount = groupsInPartition.values().stream().reduce(Integer::sum).orElse(0);
+    return new RewritePlanResult(groups, totalGroupCount, groupsInPartition);
+  }
+
+  private StructLikeMap<List<List<FileScanTask>>> planFileGroups(
+      Table table, Expression filter, long startingSnapshotId) {
+    CloseableIterable<FileScanTask> fileScanTasks =
+        table
+            .newScan()
+            .useSnapshot(startingSnapshotId)
+            .filter(filter)
+            .ignoreResiduals()
+            .planFiles();
+
+    try {
+      Types.StructType partitionType = table.spec().partitionType();
+      StructLikeMap<List<FileScanTask>> filesByPartition =
+          groupByPartition(table, partitionType, fileScanTasks);
+      return filesByPartition.transformValues(
+          tasks -> ImmutableList.copyOf(rewriter.planFileGroups(tasks)));
+    } finally {
+      try {
+        fileScanTasks.close();
+      } catch (IOException io) {
+        LOG.error("Cannot properly close file iterable while planning for rewrite", io);
+      }
+    }
+  }
+
+  private StructLikeMap<List<FileScanTask>> groupByPartition(
+      Table table, Types.StructType partitionType, Iterable<FileScanTask> tasks) {
+    StructLikeMap<List<FileScanTask>> filesByPartition = StructLikeMap.create(partitionType);
+    StructLike emptyStruct = GenericRecord.create(partitionType);
+
+    for (FileScanTask task : tasks) {
+      // If a task uses an incompatible partition spec the data inside could contain values
+      // which belong to multiple partitions in the current spec. Treating all such files as
+      // un-partitioned and grouping them together helps to minimize new files made.
+      StructLike taskPartition =
+          task.file().specId() == table.spec().specId() ? task.file().partition() : emptyStruct;

Review Comment:
   Got this exception:
   ```
   TestRewriteDataFilesAction > testBinPackRewriterWithSpecificOutputSpec() > formatVersion = 3 FAILED
       java.lang.UnsupportedOperationException: Can't retrieve values from an empty struct
           at org.apache.iceberg.EmptyStructLike.get(EmptyStructLike.java:40)
           at org.apache.iceberg.types.JavaHashes$StructLikeHash.hash(JavaHashes.java:96)
           at org.apache.iceberg.types.JavaHashes$StructLikeHash.hash(JavaHashes.java:75)
           at org.apache.iceberg.util.StructLikeWrapper.hashCode(StructLikeWrapper.java:96)
           at java.base/java.util.HashMap.hash(HashMap.java:338)
   ```
   
   So reverted the `EmptyStructLike` changes



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1841265128


##########
core/src/main/java/org/apache/iceberg/actions/RewriteFileGroupPlanner.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Stream;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.RewriteJobOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.StructLikeMap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Checks the files in the table, and using the {@link FileRewriter} plans the groups for
+ * compaction.
+ */
+public class RewriteFileGroupPlanner {
+  private static final Logger LOG = LoggerFactory.getLogger(RewriteFileGroupPlanner.class);
+
+  private final FileRewriter<FileScanTask, DataFile> rewriter;
+  private final RewriteJobOrder rewriteJobOrder;
+
+  public RewriteFileGroupPlanner(
+      FileRewriter<FileScanTask, DataFile> rewriter, RewriteJobOrder rewriteJobOrder) {
+    this.rewriter = rewriter;
+    this.rewriteJobOrder = rewriteJobOrder;
+  }
+
+  public RewritePlanResult plan(Table table, Expression filter, long startingSnapshotId) {
+    StructLikeMap<List<List<FileScanTask>>> plan =
+        planFileGroups(table, filter, startingSnapshotId);
+    RewriteExecutionContext ctx = new RewriteExecutionContext();
+    Stream<RewriteFileGroup> groups =
+        plan.entrySet().stream()
+            .filter(e -> !e.getValue().isEmpty())
+            .flatMap(
+                e -> {
+                  StructLike partition = e.getKey();
+                  List<List<FileScanTask>> scanGroups = e.getValue();
+                  return scanGroups.stream().map(tasks -> newRewriteGroup(ctx, partition, tasks));
+                })
+            .sorted(RewriteFileGroup.comparator(rewriteJobOrder));
+    Map<StructLike, Integer> groupsInPartition = plan.transformValues(List::size);
+    int totalGroupCount = groupsInPartition.values().stream().reduce(Integer::sum).orElse(0);
+    return new RewritePlanResult(groups, totalGroupCount, groupsInPartition);
+  }
+
+  private StructLikeMap<List<List<FileScanTask>>> planFileGroups(
+      Table table, Expression filter, long startingSnapshotId) {
+    CloseableIterable<FileScanTask> fileScanTasks =
+        table
+            .newScan()
+            .useSnapshot(startingSnapshotId)
+            .filter(filter)
+            .ignoreResiduals()
+            .planFiles();
+
+    try {
+      Types.StructType partitionType = table.spec().partitionType();
+      StructLikeMap<List<FileScanTask>> filesByPartition =
+          groupByPartition(table, partitionType, fileScanTasks);
+      return filesByPartition.transformValues(
+          tasks -> ImmutableList.copyOf(rewriter.planFileGroups(tasks)));
+    } finally {
+      try {
+        fileScanTasks.close();
+      } catch (IOException io) {
+        LOG.error("Cannot properly close file iterable while planning for rewrite", io);
+      }
+    }
+  }
+
+  private StructLikeMap<List<FileScanTask>> groupByPartition(
+      Table table, Types.StructType partitionType, Iterable<FileScanTask> tasks) {
+    StructLikeMap<List<FileScanTask>> filesByPartition = StructLikeMap.create(partitionType);
+    StructLike emptyStruct = GenericRecord.create(partitionType);
+
+    for (FileScanTask task : tasks) {
+      // If a task uses an incompatible partition spec the data inside could contain values
+      // which belong to multiple partitions in the current spec. Treating all such files as
+      // un-partitioned and grouping them together helps to minimize new files made.
+      StructLike taskPartition =
+          task.file().specId() == table.spec().specId() ? task.file().partition() : emptyStruct;

Review Comment:
   Minor fix; we can drop this now and use EmptyStructLike.get()



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1841614061


##########
core/src/main/java/org/apache/iceberg/actions/RewriteFileGroupPlanner.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Stream;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.RewriteJobOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.StructLikeMap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Checks the files in the table, and using the {@link FileRewriter} plans the groups for
+ * compaction.
+ */
+public class RewriteFileGroupPlanner {
+  private static final Logger LOG = LoggerFactory.getLogger(RewriteFileGroupPlanner.class);
+
+  private final FileRewriter<FileScanTask, DataFile> rewriter;
+  private final RewriteJobOrder rewriteJobOrder;
+
+  public RewriteFileGroupPlanner(
+      FileRewriter<FileScanTask, DataFile> rewriter, RewriteJobOrder rewriteJobOrder) {
+    this.rewriter = rewriter;
+    this.rewriteJobOrder = rewriteJobOrder;
+  }
+
+  public RewritePlanResult plan(Table table, Expression filter, long startingSnapshotId) {
+    StructLikeMap<List<List<FileScanTask>>> plan =
+        planFileGroups(table, filter, startingSnapshotId);
+    RewriteExecutionContext ctx = new RewriteExecutionContext();
+    Stream<RewriteFileGroup> groups =
+        plan.entrySet().stream()
+            .filter(e -> !e.getValue().isEmpty())
+            .flatMap(
+                e -> {
+                  StructLike partition = e.getKey();
+                  List<List<FileScanTask>> scanGroups = e.getValue();
+                  return scanGroups.stream().map(tasks -> newRewriteGroup(ctx, partition, tasks));
+                })
+            .sorted(RewriteFileGroup.comparator(rewriteJobOrder));
+    Map<StructLike, Integer> groupsInPartition = plan.transformValues(List::size);
+    int totalGroupCount = groupsInPartition.values().stream().reduce(Integer::sum).orElse(0);
+    return new RewritePlanResult(groups, totalGroupCount, groupsInPartition);
+  }
+
+  private StructLikeMap<List<List<FileScanTask>>> planFileGroups(
+      Table table, Expression filter, long startingSnapshotId) {
+    CloseableIterable<FileScanTask> fileScanTasks =
+        table
+            .newScan()
+            .useSnapshot(startingSnapshotId)
+            .filter(filter)
+            .ignoreResiduals()
+            .planFiles();
+
+    try {
+      Types.StructType partitionType = table.spec().partitionType();
+      StructLikeMap<List<FileScanTask>> filesByPartition =
+          groupByPartition(table, partitionType, fileScanTasks);
+      return filesByPartition.transformValues(
+          tasks -> ImmutableList.copyOf(rewriter.planFileGroups(tasks)));
+    } finally {
+      try {
+        fileScanTasks.close();
+      } catch (IOException io) {
+        LOG.error("Cannot properly close file iterable while planning for rewrite", io);
+      }
+    }
+  }
+
+  private StructLikeMap<List<FileScanTask>> groupByPartition(
+      Table table, Types.StructType partitionType, Iterable<FileScanTask> tasks) {
+    StructLikeMap<List<FileScanTask>> filesByPartition = StructLikeMap.create(partitionType);
+    StructLike emptyStruct = GenericRecord.create(partitionType);
+
+    for (FileScanTask task : tasks) {
+      // If a task uses an incompatible partition spec the data inside could contain values
+      // which belong to multiple partitions in the current spec. Treating all such files as
+      // un-partitioned and grouping them together helps to minimize new files made.
+      StructLike taskPartition =
+          task.file().specId() == table.spec().specId() ? task.file().partition() : emptyStruct;
+
+      filesByPartition.computeIfAbsent(taskPartition, unused -> Lists.newArrayList()).add(task);
+    }
+
+    return filesByPartition;
+  }
+
+  private RewriteFileGroup newRewriteGroup(
+      RewriteExecutionContext ctx, StructLike partition, List<FileScanTask> tasks) {
+    RewriteDataFiles.FileGroupInfo info =
+        ImmutableRewriteDataFiles.FileGroupInfo.builder()
+            .globalIndex(ctx.currentGlobalIndex())
+            .partitionIndex(ctx.currentPartitionIndex(partition))
+            .partition(partition)
+            .build();
+    return new RewriteFileGroup(info, Lists.newArrayList(tasks));
+  }
+
+  public static class RewritePlanResult {

Review Comment:
   I will add the javadoc.
   Since we make this public, I would remove the Serializable. We can always add back if needed, but I restrict ourselves on a public API only when it is strictly necessary.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor FileRewriter interface to separate planning and execution [iceberg]

Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1934818356


##########
core/src/main/java/org/apache/iceberg/actions/RewriteFilePlan.java:
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.util.Map;
+import java.util.stream.Stream;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.StructLike;
+
+/** Result of the data file rewrite planning. */
+public class RewriteFilePlan

Review Comment:
   similarly, I found it confusing to have `RewriteFilePlan` and `FileRewritePlan`. We need better naming to distinguish the two classes (not just order swapping).
   
   Maybe `FileRewritePlan` can be called `RewriteFilePlanBase`?



##########
core/src/main/java/org/apache/iceberg/actions/RewritePositionDeletesGroupPlanner.java:
##########
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Stream;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.MetadataTableType;
+import org.apache.iceberg.MetadataTableUtils;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.PositionDeletesScanTask;
+import org.apache.iceberg.PositionDeletesTable;
+import org.apache.iceberg.RewriteJobOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.actions.RewritePositionDeleteFiles.FileGroupInfo;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+import org.apache.iceberg.util.PropertyUtil;
+import org.apache.iceberg.util.StructLikeMap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Groups specified files in the {@link Table} by {@link RewriteFileGroup}s. These will be grouped
+ * by partitions. Extends the {@link SizeBasedFileRewritePlanner} with {@link
+ * RewritePositionDeleteFiles#REWRITE_JOB_ORDER} handling.
+ */
+public class RewritePositionDeletesGroupPlanner

Review Comment:
   `RewritePositionDeletesPlanner` is probably more accurate?



##########
core/src/main/java/org/apache/iceberg/actions/RewriteFileGroup.java:
##########
@@ -31,26 +31,26 @@
 import org.apache.iceberg.util.DataFileSet;
 
 /**
- * Container class representing a set of files to be rewritten by a RewriteAction and the new files
- * which have been written by the action.
+ * Container class representing a set of data files to be rewritten by a RewriteAction and the new
+ * files which have been written by the action.
  */
-public class RewriteFileGroup {
-  private final FileGroupInfo info;
-  private final List<FileScanTask> fileScanTasks;
-
+public class RewriteFileGroup extends FileRewriteGroup<FileGroupInfo, FileScanTask, DataFile> {

Review Comment:
   It is kind of confusing to have both `RewriteFileGroup` and `FileWriteGroup` and have former extending from later. From names, they are identical. 



##########
core/src/main/java/org/apache/iceberg/actions/FileRewriteExecutor.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.util.Map;
+import java.util.Set;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.ContentScanTask;
+
+/**
+ * A class for rewriting content file groups ({@link FileRewriteGroup}).
+ *
+ * @param <I> the Java type of the plan info
+ * @param <T> the Java type of the tasks to read content files
+ * @param <F> the Java type of the content files
+ * @param <G> the Java type of the planned groups
+ * @param <P> the Java type of the plan to execute
+ */
+public interface FileRewriteExecutor<
+    I,
+    T extends ContentScanTask<F>,
+    F extends ContentFile<F>,
+    G extends FileRewriteGroup<I, T, F>,
+    P extends FileRewritePlan<I, T, F, G>> {
+
+  /** Returns a description for this rewriter. */
+  default String description() {
+    return getClass().getName();
+  }
+
+  /**
+   * Returns a set of supported options for this rewriter. Only options specified in this list will
+   * be accepted at runtime. Any other options will be rejected.
+   */
+  Set<String> validOptions();
+
+  /**
+   * Initializes this rewriter using provided options.
+   *
+   * @param options options to initialize this rewriter
+   */
+  void init(Map<String, String> options);

Review Comment:
   I also found it confusing. Can the `init` and `initPlan` be combined into one method?
   
   Alternatively, can the `plan` be passed in via the `rewrite` method?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1842585965


##########
core/src/main/java/org/apache/iceberg/actions/RewriteFileGroupPlanner.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Stream;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.RewriteJobOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.StructLikeMap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Checks the files in the table, and using the {@link FileRewriter} plans the groups for
+ * compaction.
+ */
+public class RewriteFileGroupPlanner {
+  private static final Logger LOG = LoggerFactory.getLogger(RewriteFileGroupPlanner.class);
+
+  private final FileRewriter<FileScanTask, DataFile> rewriter;
+  private final RewriteJobOrder rewriteJobOrder;
+
+  public RewriteFileGroupPlanner(
+      FileRewriter<FileScanTask, DataFile> rewriter, RewriteJobOrder rewriteJobOrder) {
+    this.rewriter = rewriter;
+    this.rewriteJobOrder = rewriteJobOrder;
+  }
+
+  public RewritePlanResult plan(Table table, Expression filter, long startingSnapshotId) {
+    StructLikeMap<List<List<FileScanTask>>> plan =
+        planFileGroups(table, filter, startingSnapshotId);
+    RewriteExecutionContext ctx = new RewriteExecutionContext();
+    Stream<RewriteFileGroup> groups =
+        plan.entrySet().stream()
+            .filter(e -> !e.getValue().isEmpty())
+            .flatMap(
+                e -> {
+                  StructLike partition = e.getKey();
+                  List<List<FileScanTask>> scanGroups = e.getValue();
+                  return scanGroups.stream().map(tasks -> newRewriteGroup(ctx, partition, tasks));
+                })
+            .sorted(RewriteFileGroup.comparator(rewriteJobOrder));
+    Map<StructLike, Integer> groupsInPartition = plan.transformValues(List::size);
+    int totalGroupCount = groupsInPartition.values().stream().reduce(Integer::sum).orElse(0);
+    return new RewritePlanResult(groups, totalGroupCount, groupsInPartition);
+  }
+
+  private StructLikeMap<List<List<FileScanTask>>> planFileGroups(
+      Table table, Expression filter, long startingSnapshotId) {
+    CloseableIterable<FileScanTask> fileScanTasks =
+        table
+            .newScan()
+            .useSnapshot(startingSnapshotId)
+            .filter(filter)
+            .ignoreResiduals()
+            .planFiles();
+
+    try {
+      Types.StructType partitionType = table.spec().partitionType();
+      StructLikeMap<List<FileScanTask>> filesByPartition =
+          groupByPartition(table, partitionType, fileScanTasks);
+      return filesByPartition.transformValues(
+          tasks -> ImmutableList.copyOf(rewriter.planFileGroups(tasks)));
+    } finally {
+      try {
+        fileScanTasks.close();
+      } catch (IOException io) {
+        LOG.error("Cannot properly close file iterable while planning for rewrite", io);
+      }
+    }
+  }
+
+  private StructLikeMap<List<FileScanTask>> groupByPartition(
+      Table table, Types.StructType partitionType, Iterable<FileScanTask> tasks) {
+    StructLikeMap<List<FileScanTask>> filesByPartition = StructLikeMap.create(partitionType);
+    StructLike emptyStruct = GenericRecord.create(partitionType);
+
+    for (FileScanTask task : tasks) {
+      // If a task uses an incompatible partition spec the data inside could contain values
+      // which belong to multiple partitions in the current spec. Treating all such files as
+      // un-partitioned and grouping them together helps to minimize new files made.
+      StructLike taskPartition =
+          task.file().specId() == table.spec().specId() ? task.file().partition() : emptyStruct;

Review Comment:
   Blargh



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor FileRewriter interface to separate planning and execution [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #11513: Core, Spark: Refactor FileRewriter interface to separate planning and execution
URL: https://github.com/apache/iceberg/pull/11513


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor FileRewriter interface to separate planning and execution [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#issuecomment-2706765443

   Not stale


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1841270029


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java:
##########
@@ -168,89 +160,36 @@ public RewriteDataFiles.Result execute() {
 
     validateAndInitOptions();
 
-    StructLikeMap<List<List<FileScanTask>>> fileGroupsByPartition =
-        planFileGroups(startingSnapshotId);
-    RewriteExecutionContext ctx = new RewriteExecutionContext(fileGroupsByPartition);
+    RewritePlanResult result = plan(startingSnapshotId);

Review Comment:
   minor nit here because we have a "result" variable which is not actually the "result" of this function but is actually just an intermediary input. We could probably change the class name to just RewritePlan and the variable to just plan



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1841925053


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java:
##########
@@ -276,29 +215,25 @@ RewriteDataFilesCommitManager commitManager(long startingSnapshotId) {
   }
 
   private Builder doExecute(
-      RewriteExecutionContext ctx,
-      Stream<RewriteFileGroup> groupStream,
-      RewriteDataFilesCommitManager commitManager) {
+      RewritePlanResult planResult, RewriteDataFilesCommitManager commitManager) {
     ExecutorService rewriteService = rewriteService();
 
     ConcurrentLinkedQueue<RewriteFileGroup> rewrittenGroups = Queues.newConcurrentLinkedQueue();
 
     Tasks.Builder<RewriteFileGroup> rewriteTaskBuilder =
-        Tasks.foreach(groupStream)
+        Tasks.foreach(planResult.groups())
             .executeWith(rewriteService)
             .stopOnFailure()
             .noRetry()
             .onFailure(
-                (fileGroup, exception) -> {
-                  LOG.warn(
-                      "Failure during rewrite process for group {}", fileGroup.info(), exception);
-                });
+                (fileGroup, exception) ->
+                    LOG.warn(

Review Comment:
   reverted



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#issuecomment-2475910745

   > For easier review it would be great if you could highlight the changes made.
   > 
   > I see the note on RewritePlanResult but i'm not sure where this class came from. It is unclear what has been extracted and renamed and what is new code.
   
   Let me amend this now, and thanks for taking a look anyway!
   
   So there were no significant new code in the refactor.
   Created a single new class (`RewriteFileGroupPlanner`) for the functionality, and created a new class (`RewritePlan`) for storing result. The responsibility of the new class is to group the rewrite tasks together and return the calculated plan.
   
   Originally the `RewriteDataFilesSparkAction.toGroupStream` did the grouping. It used several other methods (planFileGroups, groupByPartition, fileGroupsByPartition, newRewriteGroup) and the RewriteExecutionContext inner class to do the grouping.
   
   The methods and the class got moved to the `RewriteFileGroupPlanner`. The result got its own class to be able to return not only the stream, but the total counts as well which are needed by the current Spark implementation.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor FileRewriter interface to separate planning and execution [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1876827034


##########
core/src/main/java/org/apache/iceberg/actions/FileRewriteGroup.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.util.Comparator;
+import java.util.List;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.ContentScanTask;
+import org.apache.iceberg.RewriteJobOrder;
+
+/**
+ * Container class representing a set of files to be rewritten by a {@link FileRewriteExecutor}.

Review Comment:
   I think basically anything having to do with "output" should be part of "Plan" or "Execute" depending. Like in this case we basically are just using these properties because they make it easy to control spark's behavior



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1865929273


##########
core/src/main/java/org/apache/iceberg/actions/RewriteFilePlan.java:
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.util.Map;
+import java.util.stream.Stream;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.StructLike;
+
+/** Result of the data file rewrite planning. */
+public class RewriteFilePlan
+    extends FileRewritePlan<
+        RewriteDataFiles.FileGroupInfo, FileScanTask, DataFile, RewriteFileGroup> {
+  private final int outputSpecId;

Review Comment:
   This information is not relevant for the position delete compaction, and it is not available during the planning. This is the reason why we have 2 different plan objects.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1867596164


##########
core/src/main/java/org/apache/iceberg/actions/RewritePositionDeletePlan.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.util.Map;
+import java.util.stream.Stream;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.PositionDeletesScanTask;
+import org.apache.iceberg.StructLike;
+
+/** Result of the positional delete file rewrite planning. */
+public class RewritePositionDeletePlan

Review Comment:
   We could use `FileRewritePlan<RewritePositionDeleteFiles.FileGroupInfo,
           PositionDeletesScanTask,
           DeleteFile,
           RewritePositionDeletesGroup>` everywhere instead of creating a new class, but after I had to create the `RewriteFilePlan` to handle `outputSpecId`, this seems more elegant



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1841227042


##########
core/src/main/java/org/apache/iceberg/actions/RewriteFileGroupPlanner.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Stream;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.RewriteJobOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.StructLikeMap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Checks the files in the table, and using the {@link FileRewriter} plans the groups for
+ * compaction.
+ */
+public class RewriteFileGroupPlanner {
+  private static final Logger LOG = LoggerFactory.getLogger(RewriteFileGroupPlanner.class);
+
+  private final FileRewriter<FileScanTask, DataFile> rewriter;
+  private final RewriteJobOrder rewriteJobOrder;
+
+  public RewriteFileGroupPlanner(
+      FileRewriter<FileScanTask, DataFile> rewriter, RewriteJobOrder rewriteJobOrder) {
+    this.rewriter = rewriter;
+    this.rewriteJobOrder = rewriteJobOrder;
+  }
+
+  public RewritePlanResult plan(Table table, Expression filter, long startingSnapshotId) {
+    StructLikeMap<List<List<FileScanTask>>> plan =
+        planFileGroups(table, filter, startingSnapshotId);
+    RewriteExecutionContext ctx = new RewriteExecutionContext();
+    Stream<RewriteFileGroup> groups =
+        plan.entrySet().stream()
+            .filter(e -> !e.getValue().isEmpty())
+            .flatMap(
+                e -> {
+                  StructLike partition = e.getKey();
+                  List<List<FileScanTask>> scanGroups = e.getValue();
+                  return scanGroups.stream().map(tasks -> newRewriteGroup(ctx, partition, tasks));
+                })
+            .sorted(RewriteFileGroup.comparator(rewriteJobOrder));
+    Map<StructLike, Integer> groupsInPartition = plan.transformValues(List::size);
+    int totalGroupCount = groupsInPartition.values().stream().reduce(Integer::sum).orElse(0);
+    return new RewritePlanResult(groups, totalGroupCount, groupsInPartition);
+  }
+
+  private StructLikeMap<List<List<FileScanTask>>> planFileGroups(
+      Table table, Expression filter, long startingSnapshotId) {
+    CloseableIterable<FileScanTask> fileScanTasks =
+        table
+            .newScan()
+            .useSnapshot(startingSnapshotId)
+            .filter(filter)
+            .ignoreResiduals()
+            .planFiles();
+
+    try {
+      Types.StructType partitionType = table.spec().partitionType();
+      StructLikeMap<List<FileScanTask>> filesByPartition =
+          groupByPartition(table, partitionType, fileScanTasks);
+      return filesByPartition.transformValues(
+          tasks -> ImmutableList.copyOf(rewriter.planFileGroups(tasks)));
+    } finally {
+      try {
+        fileScanTasks.close();
+      } catch (IOException io) {
+        LOG.error("Cannot properly close file iterable while planning for rewrite", io);
+      }
+    }
+  }
+
+  private StructLikeMap<List<FileScanTask>> groupByPartition(
+      Table table, Types.StructType partitionType, Iterable<FileScanTask> tasks) {
+    StructLikeMap<List<FileScanTask>> filesByPartition = StructLikeMap.create(partitionType);
+    StructLike emptyStruct = GenericRecord.create(partitionType);
+
+    for (FileScanTask task : tasks) {
+      // If a task uses an incompatible partition spec the data inside could contain values
+      // which belong to multiple partitions in the current spec. Treating all such files as
+      // un-partitioned and grouping them together helps to minimize new files made.
+      StructLike taskPartition =
+          task.file().specId() == table.spec().specId() ? task.file().partition() : emptyStruct;
+
+      filesByPartition.computeIfAbsent(taskPartition, unused -> Lists.newArrayList()).add(task);
+    }
+
+    return filesByPartition;
+  }
+
+  private RewriteFileGroup newRewriteGroup(
+      RewriteExecutionContext ctx, StructLike partition, List<FileScanTask> tasks) {
+    RewriteDataFiles.FileGroupInfo info =
+        ImmutableRewriteDataFiles.FileGroupInfo.builder()
+            .globalIndex(ctx.currentGlobalIndex())
+            .partitionIndex(ctx.currentPartitionIndex(partition))
+            .partition(partition)
+            .build();
+    return new RewriteFileGroup(info, Lists.newArrayList(tasks));
+  }
+
+  public static class RewritePlanResult {

Review Comment:
   I would just hesitate to remove something that was there before. I think originally we wanted it because we were considering shipping this class around but I don't see any evidence that it is currently serialized.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1841867737


##########
core/src/main/java/org/apache/iceberg/actions/RewriteFileGroupPlanner.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Stream;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.RewriteJobOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.StructLikeMap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Checks the files in the table, and using the {@link FileRewriter} plans the groups for
+ * compaction.
+ */
+public class RewriteFileGroupPlanner {
+  private static final Logger LOG = LoggerFactory.getLogger(RewriteFileGroupPlanner.class);
+
+  private final FileRewriter<FileScanTask, DataFile> rewriter;
+  private final RewriteJobOrder rewriteJobOrder;
+
+  public RewriteFileGroupPlanner(
+      FileRewriter<FileScanTask, DataFile> rewriter, RewriteJobOrder rewriteJobOrder) {
+    this.rewriter = rewriter;
+    this.rewriteJobOrder = rewriteJobOrder;
+  }
+
+  public RewritePlanResult plan(Table table, Expression filter, long startingSnapshotId) {
+    StructLikeMap<List<List<FileScanTask>>> plan =
+        planFileGroups(table, filter, startingSnapshotId);
+    RewriteExecutionContext ctx = new RewriteExecutionContext();
+    Stream<RewriteFileGroup> groups =
+        plan.entrySet().stream()
+            .filter(e -> !e.getValue().isEmpty())
+            .flatMap(
+                e -> {
+                  StructLike partition = e.getKey();
+                  List<List<FileScanTask>> scanGroups = e.getValue();
+                  return scanGroups.stream().map(tasks -> newRewriteGroup(ctx, partition, tasks));
+                })
+            .sorted(RewriteFileGroup.comparator(rewriteJobOrder));
+    Map<StructLike, Integer> groupsInPartition = plan.transformValues(List::size);
+    int totalGroupCount = groupsInPartition.values().stream().reduce(Integer::sum).orElse(0);
+    return new RewritePlanResult(groups, totalGroupCount, groupsInPartition);
+  }
+
+  private StructLikeMap<List<List<FileScanTask>>> planFileGroups(
+      Table table, Expression filter, long startingSnapshotId) {
+    CloseableIterable<FileScanTask> fileScanTasks =
+        table
+            .newScan()
+            .useSnapshot(startingSnapshotId)
+            .filter(filter)
+            .ignoreResiduals()
+            .planFiles();
+
+    try {
+      Types.StructType partitionType = table.spec().partitionType();
+      StructLikeMap<List<FileScanTask>> filesByPartition =
+          groupByPartition(table, partitionType, fileScanTasks);
+      return filesByPartition.transformValues(
+          tasks -> ImmutableList.copyOf(rewriter.planFileGroups(tasks)));
+    } finally {
+      try {
+        fileScanTasks.close();
+      } catch (IOException io) {
+        LOG.error("Cannot properly close file iterable while planning for rewrite", io);
+      }
+    }
+  }
+
+  private StructLikeMap<List<FileScanTask>> groupByPartition(
+      Table table, Types.StructType partitionType, Iterable<FileScanTask> tasks) {
+    StructLikeMap<List<FileScanTask>> filesByPartition = StructLikeMap.create(partitionType);
+    StructLike emptyStruct = GenericRecord.create(partitionType);
+
+    for (FileScanTask task : tasks) {
+      // If a task uses an incompatible partition spec the data inside could contain values
+      // which belong to multiple partitions in the current spec. Treating all such files as
+      // un-partitioned and grouping them together helps to minimize new files made.
+      StructLike taskPartition =
+          task.file().specId() == table.spec().specId() ? task.file().partition() : emptyStruct;

Review Comment:
   Do we want to make `EmptyStructLike`, and `EmptyStructLike.get` public?
   Currently it is package private.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor FileRewriter interface to separate planning and execution [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#issuecomment-2516532807

   @jackye1995: Would you happen to be interested in a standardized interface for compaction planning?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1852375036


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/SparkRewriteExecutor.java:
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.spark.actions;
+
+import java.util.Map;
+import java.util.Set;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.ContentScanTask;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.actions.FileRewriteExecutor;
+import org.apache.iceberg.actions.FileRewriteGroup;
+import org.apache.iceberg.actions.FileRewritePlan;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+
+/**
+ * Common parent for data and positional delete rewrite executors.
+ *
+ * @param <I> the Java type of the plan info
+ * @param <T> the Java type of the tasks to read content files
+ * @param <F> the Java type of the content files
+ * @param <G> the Java type of the planned groups
+ */
+abstract class SparkRewriteExecutor<
+        I,
+        T extends ContentScanTask<F>,
+        F extends ContentFile<F>,
+        G extends FileRewriteGroup<I, T, F>>
+    implements FileRewriteExecutor<I, T, F, G> {
+  private final Table table;
+  private long writeMaxFileSize;
+  private int outputSpecId;
+
+  SparkRewriteExecutor(Table table) {
+    this.table = table;
+  }
+
+  Table table() {
+    return table;
+  }
+
+  long writeMaxFileSize() {
+    return writeMaxFileSize;
+  }
+
+  int outputSpecId() {
+    return outputSpecId;
+  }
+
+  PartitionSpec outputSpec() {
+    return table.specs().get(outputSpecId);
+  }
+
+  @Override
+  public void initPlan(FileRewritePlan<I, T, F, G> plan) {

Review Comment:
   Write now this method is just used to set config from the plan in the rewrite executor, i'm wondering if we should just be setting those parameters directly on the rewriter rather than inheriting them from the plan?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor FileRewriter interface to separate planning and execution [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1876278981


##########
core/src/main/java/org/apache/iceberg/actions/FileRewriteGroup.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.util.Comparator;
+import java.util.List;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.ContentScanTask;
+import org.apache.iceberg.RewriteJobOrder;
+
+/**
+ * Container class representing a set of files to be rewritten by a {@link FileRewriteExecutor}.
+ *
+ * @param <I> the Java type of the plan info

Review Comment:
   This is very ambiguous to me, and it's not clear what I should be. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor FileRewriter interface to separate planning and execution [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1876410481


##########
core/src/main/java/org/apache/iceberg/actions/FileRewriteGroup.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.util.Comparator;
+import java.util.List;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.ContentScanTask;
+import org.apache.iceberg.RewriteJobOrder;
+
+/**
+ * Container class representing a set of files to be rewritten by a {@link FileRewriteExecutor}.
+ *
+ * @param <I> the Java type of the plan info

Review Comment:
   These are the `RewriteDataFiles.FileGroupInfo` and `RewritePositionDeleteFiles.FileGroupInfo`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor FileRewriter interface to separate planning and execution [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1876315283


##########
core/src/main/java/org/apache/iceberg/actions/FileRewritePlanner.java:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.util.Map;
+import java.util.Set;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.ContentScanTask;
+
+/**
+ * A class for planning content file rewrites.
+ *
+ * <p>The entire rewrite operation is broken down into pieces based on partitioning, and size-based

Review Comment:
   "partitioning and size-based" are implementation details right? Not necessarily always true?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor FileRewriter interface to separate planning and execution [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1876637865


##########
core/src/main/java/org/apache/iceberg/actions/FileRewritePlan.java:
##########
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.util.Map;
+import java.util.stream.Stream;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.ContentScanTask;
+import org.apache.iceberg.StructLike;
+
+/**
+ * Result of the file rewrite planning.
+ *
+ * <p>Contains the planned groups, calculated values required by the {@link FileRewriteExecutor}s
+ * and statistics.
+ *
+ * @param <I> the Java type of the plan info
+ * @param <T> the Java type of the tasks to read content files
+ * @param <F> the Java type of the content files
+ * @param <G> the Java type of the planned groups
+ */
+public abstract class FileRewritePlan<
+    I,
+    T extends ContentScanTask<F>,
+    F extends ContentFile<F>,
+    G extends FileRewriteGroup<I, T, F>> {
+  private final Stream<G> groups;
+  private final int totalGroupCount;
+  private final Map<StructLike, Integer> groupsInPartition;
+  private final long writeMaxFileSize;
+
+  protected FileRewritePlan(
+      Stream<G> groups,
+      int totalGroupCount,
+      Map<StructLike, Integer> groupsInPartition,
+      long writeMaxFileSize) {
+    this.groups = groups;
+    this.totalGroupCount = totalGroupCount;
+    this.groupsInPartition = groupsInPartition;
+    this.writeMaxFileSize = writeMaxFileSize;
+  }
+
+  /** The stream of the generated {@link FileRewriteGroup}s. */
+  public Stream<G> groups() {
+    return groups;
+  }
+
+  /** The number of the generated groups in the given partition. */
+  public int groupsInPartition(StructLike partition) {
+    return groupsInPartition.get(partition);
+  }
+
+  /** The total number of the groups generated by this plan. */
+  public int totalGroupCount() {
+    return totalGroupCount;
+  }
+
+  /** Calculated maximum file size for the target files */

Review Comment:
   Updated the javadoc.
   Also see: https://github.com/apache/iceberg/pull/11513#discussion_r1876635103



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor FileRewriter interface to separate planning and execution [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1876635103


##########
core/src/main/java/org/apache/iceberg/actions/FileRewriteGroup.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.util.Comparator;
+import java.util.List;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.ContentScanTask;
+import org.apache.iceberg.RewriteJobOrder;
+
+/**
+ * Container class representing a set of files to be rewritten by a {@link FileRewriteExecutor}.

Review Comment:
   The current code makes sure that the exact same compaction result is generated than with the previous algorithm.
   
   In the old code we had:
   - Plan level
       - writeMaxFileSize - which governs the target file size when rewriting (SparkWriteOptions.TARGET_FILE_SIZE_BYTES)
   - Group level
       - splitSize -  which governs the input split size when rewriting (SparkReadOptions.SPLIT_SIZE, splitSize)]
       - expectedOutputFiles - which governs the shuffle partition numbers for shuffling rewriters
   
   In the new code I have mirrored this.
   
   I think the new code would be much cleaner if we put all of this information into the group level (we could get rid of the `initPlan` step)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor FileRewriter interface to separate planning and execution [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#issuecomment-2516530755

   @aokolnychyi: @RussellSpitzer is out until Thursday. Could you please review it in the meantime?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor FileRewriter interface to separate planning and execution [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1876302993


##########
core/src/main/java/org/apache/iceberg/actions/FileRewritePlan.java:
##########
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.util.Map;
+import java.util.stream.Stream;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.ContentScanTask;
+import org.apache.iceberg.StructLike;
+
+/**
+ * Result of the file rewrite planning.

Review Comment:
   Needs more details. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1846675235


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java:
##########
@@ -341,20 +274,19 @@ private Builder doExecute(
   }
 
   private Builder doExecuteWithPartialProgress(
-      RewriteExecutionContext ctx,
-      Stream<RewriteFileGroup> groupStream,
-      RewriteDataFilesCommitManager commitManager) {
+      RewritePlan planResult, RewriteDataFilesCommitManager commitManager) {

Review Comment:
   Done



##########
core/src/main/java/org/apache/iceberg/actions/RewriteFileGroupPlanner.java:
##########
@@ -0,0 +1,181 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Stream;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.RewriteJobOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.StructLikeMap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Checks the files in the {@link Table}. The {@link RewriteFileGroup}s are grouped by partitions

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1836488118


##########
core/src/main/java/org/apache/iceberg/actions/RewriteFileGroupPlanner.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Stream;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.RewriteJobOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.StructLikeMap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Checks the files in the table, and using the {@link FileRewriter} plans the groups for
+ * compaction.
+ */
+public class RewriteFileGroupPlanner {
+  private static final Logger LOG = LoggerFactory.getLogger(RewriteFileGroupPlanner.class);
+
+  private final FileRewriter<FileScanTask, DataFile> rewriter;
+  private final RewriteJobOrder rewriteJobOrder;
+
+  public RewriteFileGroupPlanner(
+      FileRewriter<FileScanTask, DataFile> rewriter, RewriteJobOrder rewriteJobOrder) {
+    this.rewriter = rewriter;
+    this.rewriteJobOrder = rewriteJobOrder;
+  }
+
+  public RewritePlanResult plan(Table table, Expression filter, long startingSnapshotId) {
+    StructLikeMap<List<List<FileScanTask>>> plan =
+        planFileGroups(table, filter, startingSnapshotId);
+    RewriteExecutionContext ctx = new RewriteExecutionContext();
+    Stream<RewriteFileGroup> groups =
+        plan.entrySet().stream()
+            .filter(e -> !e.getValue().isEmpty())
+            .flatMap(
+                e -> {
+                  StructLike partition = e.getKey();
+                  List<List<FileScanTask>> scanGroups = e.getValue();
+                  return scanGroups.stream().map(tasks -> newRewriteGroup(ctx, partition, tasks));
+                })
+            .sorted(RewriteFileGroup.comparator(rewriteJobOrder));
+    Map<StructLike, Integer> groupsInPartition = plan.transformValues(List::size);
+    int totalGroupCount = groupsInPartition.values().stream().reduce(Integer::sum).orElse(0);
+    return new RewritePlanResult(groups, totalGroupCount, groupsInPartition);
+  }
+
+  private StructLikeMap<List<List<FileScanTask>>> planFileGroups(
+      Table table, Expression filter, long startingSnapshotId) {
+    CloseableIterable<FileScanTask> fileScanTasks =
+        table
+            .newScan()
+            .useSnapshot(startingSnapshotId)
+            .filter(filter)
+            .ignoreResiduals()
+            .planFiles();
+
+    try {
+      Types.StructType partitionType = table.spec().partitionType();
+      StructLikeMap<List<FileScanTask>> filesByPartition =
+          groupByPartition(table, partitionType, fileScanTasks);
+      return filesByPartition.transformValues(
+          tasks -> ImmutableList.copyOf(rewriter.planFileGroups(tasks)));
+    } finally {
+      try {
+        fileScanTasks.close();
+      } catch (IOException io) {
+        LOG.error("Cannot properly close file iterable while planning for rewrite", io);
+      }
+    }
+  }
+
+  private StructLikeMap<List<FileScanTask>> groupByPartition(
+      Table table, Types.StructType partitionType, Iterable<FileScanTask> tasks) {
+    StructLikeMap<List<FileScanTask>> filesByPartition = StructLikeMap.create(partitionType);
+    StructLike emptyStruct = GenericRecord.create(partitionType);
+
+    for (FileScanTask task : tasks) {
+      // If a task uses an incompatible partition spec the data inside could contain values
+      // which belong to multiple partitions in the current spec. Treating all such files as
+      // un-partitioned and grouping them together helps to minimize new files made.
+      StructLike taskPartition =
+          task.file().specId() == table.spec().specId() ? task.file().partition() : emptyStruct;
+
+      filesByPartition.computeIfAbsent(taskPartition, unused -> Lists.newArrayList()).add(task);
+    }
+
+    return filesByPartition;
+  }
+
+  private RewriteFileGroup newRewriteGroup(
+      RewriteExecutionContext ctx, StructLike partition, List<FileScanTask> tasks) {
+    RewriteDataFiles.FileGroupInfo info =
+        ImmutableRewriteDataFiles.FileGroupInfo.builder()
+            .globalIndex(ctx.currentGlobalIndex())
+            .partitionIndex(ctx.currentPartitionIndex(partition))
+            .partition(partition)
+            .build();
+    return new RewriteFileGroup(info, Lists.newArrayList(tasks));
+  }
+
+  public static class RewritePlanResult {

Review Comment:
   Removed the serializable as it was not required by the tests.
   @RussellSpitzer, @szehon-ho: Could you please check if this is ok?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1841247322


##########
core/src/main/java/org/apache/iceberg/actions/RewriteFileGroupPlanner.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Stream;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.RewriteJobOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.StructLikeMap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Checks the files in the table, and using the {@link FileRewriter} plans the groups for
+ * compaction.
+ */
+public class RewriteFileGroupPlanner {
+  private static final Logger LOG = LoggerFactory.getLogger(RewriteFileGroupPlanner.class);
+
+  private final FileRewriter<FileScanTask, DataFile> rewriter;
+  private final RewriteJobOrder rewriteJobOrder;
+
+  public RewriteFileGroupPlanner(
+      FileRewriter<FileScanTask, DataFile> rewriter, RewriteJobOrder rewriteJobOrder) {
+    this.rewriter = rewriter;
+    this.rewriteJobOrder = rewriteJobOrder;
+  }
+
+  public RewritePlanResult plan(Table table, Expression filter, long startingSnapshotId) {
+    StructLikeMap<List<List<FileScanTask>>> plan =
+        planFileGroups(table, filter, startingSnapshotId);
+    RewriteExecutionContext ctx = new RewriteExecutionContext();
+    Stream<RewriteFileGroup> groups =
+        plan.entrySet().stream()
+            .filter(e -> !e.getValue().isEmpty())
+            .flatMap(
+                e -> {
+                  StructLike partition = e.getKey();
+                  List<List<FileScanTask>> scanGroups = e.getValue();
+                  return scanGroups.stream().map(tasks -> newRewriteGroup(ctx, partition, tasks));
+                })
+            .sorted(RewriteFileGroup.comparator(rewriteJobOrder));
+    Map<StructLike, Integer> groupsInPartition = plan.transformValues(List::size);
+    int totalGroupCount = groupsInPartition.values().stream().reduce(Integer::sum).orElse(0);
+    return new RewritePlanResult(groups, totalGroupCount, groupsInPartition);
+  }
+
+  private StructLikeMap<List<List<FileScanTask>>> planFileGroups(
+      Table table, Expression filter, long startingSnapshotId) {
+    CloseableIterable<FileScanTask> fileScanTasks =
+        table
+            .newScan()
+            .useSnapshot(startingSnapshotId)
+            .filter(filter)
+            .ignoreResiduals()
+            .planFiles();
+
+    try {
+      Types.StructType partitionType = table.spec().partitionType();
+      StructLikeMap<List<FileScanTask>> filesByPartition =
+          groupByPartition(table, partitionType, fileScanTasks);
+      return filesByPartition.transformValues(
+          tasks -> ImmutableList.copyOf(rewriter.planFileGroups(tasks)));
+    } finally {
+      try {
+        fileScanTasks.close();
+      } catch (IOException io) {
+        LOG.error("Cannot properly close file iterable while planning for rewrite", io);
+      }
+    }
+  }
+
+  private StructLikeMap<List<FileScanTask>> groupByPartition(
+      Table table, Types.StructType partitionType, Iterable<FileScanTask> tasks) {
+    StructLikeMap<List<FileScanTask>> filesByPartition = StructLikeMap.create(partitionType);
+    StructLike emptyStruct = GenericRecord.create(partitionType);
+
+    for (FileScanTask task : tasks) {
+      // If a task uses an incompatible partition spec the data inside could contain values
+      // which belong to multiple partitions in the current spec. Treating all such files as
+      // un-partitioned and grouping them together helps to minimize new files made.
+      StructLike taskPartition =
+          task.file().specId() == table.spec().specId() ? task.file().partition() : emptyStruct;
+
+      filesByPartition.computeIfAbsent(taskPartition, unused -> Lists.newArrayList()).add(task);
+    }
+
+    return filesByPartition;
+  }
+
+  private RewriteFileGroup newRewriteGroup(
+      RewriteExecutionContext ctx, StructLike partition, List<FileScanTask> tasks) {
+    RewriteDataFiles.FileGroupInfo info =
+        ImmutableRewriteDataFiles.FileGroupInfo.builder()
+            .globalIndex(ctx.currentGlobalIndex())
+            .partitionIndex(ctx.currentPartitionIndex(partition))
+            .partition(partition)
+            .build();
+    return new RewriteFileGroup(info, Lists.newArrayList(tasks));
+  }
+
+  public static class RewritePlanResult {

Review Comment:
   Now that these are Public we need java docs describing their purpose



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1841263893


##########
core/src/main/java/org/apache/iceberg/actions/RewriteFileGroupPlanner.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Stream;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.RewriteJobOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.StructLikeMap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Checks the files in the table, and using the {@link FileRewriter} plans the groups for

Review Comment:
   I'm a little confused about the inheritance structure here. If we want to make this common, maybe we should just have this belong to the FileRewriter as a base class? 
   
   If I can think through this correctly in this draft we have
   
   RewriteFileGroupPlanner Responsible for 
   1. Scanning Table
   2. Grouping Tasks by Partition
   3. Further Grouping and Filtering in FileRewriter
   
   FileRewriter Responsible for
   1. Filtering Tasks based on Rewriter Config
   2. Grouping tasks within partitions
   
   Feels like all of that could just be in the FileRewriter without having a separate GroupPlanner? Do we have an argument against that? I think we could even just have some new default methods on the FileRewriter interface for planning?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1841263893


##########
core/src/main/java/org/apache/iceberg/actions/RewriteFileGroupPlanner.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Stream;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.RewriteJobOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.StructLikeMap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Checks the files in the table, and using the {@link FileRewriter} plans the groups for

Review Comment:
   I'm a little confused about the inheritance structure here. If we want to make this common, maybe we should just have this belong to the FileRewriter as a base class? 
   
   If I can think through this correctly in this draft we have
   
   RewriteFileGroupPlanner Responsible for 
   1. Scanning Table
   2. Grouping Tasks by Partition
   3. Further Grouping and Filtering in FileRewriter
   
   FileRewriter Responsible for
   1. Filtering Tasks based on Rewriter Config
   2. Grouping tasks within partitions
   
   Feels like all of that could just be in the FileRewriter without having a separate GroupPlanner? Do we have an argument against that? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1841605909


##########
core/src/main/java/org/apache/iceberg/actions/RewriteFileGroupPlanner.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Stream;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.RewriteJobOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.StructLikeMap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Checks the files in the table, and using the {@link FileRewriter} plans the groups for

Review Comment:
   Let me check that. I was hesitant to change existing classes, but if we have a consensus, I'm happy to do that.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1841840398


##########
core/src/main/java/org/apache/iceberg/actions/RewriteFileGroupPlanner.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Stream;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.RewriteJobOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.StructLikeMap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Checks the files in the table, and using the {@link FileRewriter} plans the groups for

Review Comment:
   The `FileRewriter` is a generic class with `<T extends ContentScanTask<F>, F extends ContentFile<F>>`.
   The `RewriteFileGroup` which is returned by the planning is specific for `FileScanTask`.
   
   I'm not sure that we use the `FileRewriter` with any other parametrization ATM, but to put everything into a single class we need to do one of the things below:
   1. Changing `FileRewriter<T extends FileScanTask>` instead of the current `FileRewriter<T extends ContentScanTask<F>, F extends ContentFile<F>>`, or 
   2. Changing the `public List<FileScanTask> RewriteFileGroup.fileScans()` to return `public <T extends ContentScanTask<F>, F extends ContentFile<F>>  List<T> fileScans`
   
   Both of the would be a breaking change in the public API so we should be careful about them.
   
   We still can "duplicate" and create a new FileRewriter, and deprecate the old one - but I would only do this, if we think that the original API needs a serious refactor anyways.
   
   WDYT?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1841923395


##########
core/src/main/java/org/apache/iceberg/actions/RewriteFileGroupPlanner.java:
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Stream;
+import org.apache.iceberg.DataFile;
+import org.apache.iceberg.FileScanTask;
+import org.apache.iceberg.RewriteJobOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.StructLikeMap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Checks the files in the table, and using the {@link FileRewriter} plans the groups for
+ * compaction.
+ */
+public class RewriteFileGroupPlanner {
+  private static final Logger LOG = LoggerFactory.getLogger(RewriteFileGroupPlanner.class);
+
+  private final FileRewriter<FileScanTask, DataFile> rewriter;
+  private final RewriteJobOrder rewriteJobOrder;
+
+  public RewriteFileGroupPlanner(
+      FileRewriter<FileScanTask, DataFile> rewriter, RewriteJobOrder rewriteJobOrder) {
+    this.rewriter = rewriter;
+    this.rewriteJobOrder = rewriteJobOrder;
+  }
+
+  public RewritePlanResult plan(Table table, Expression filter, long startingSnapshotId) {
+    StructLikeMap<List<List<FileScanTask>>> plan =
+        planFileGroups(table, filter, startingSnapshotId);
+    RewriteExecutionContext ctx = new RewriteExecutionContext();
+    Stream<RewriteFileGroup> groups =
+        plan.entrySet().stream()
+            .filter(e -> !e.getValue().isEmpty())
+            .flatMap(
+                e -> {
+                  StructLike partition = e.getKey();
+                  List<List<FileScanTask>> scanGroups = e.getValue();
+                  return scanGroups.stream().map(tasks -> newRewriteGroup(ctx, partition, tasks));
+                })
+            .sorted(RewriteFileGroup.comparator(rewriteJobOrder));
+    Map<StructLike, Integer> groupsInPartition = plan.transformValues(List::size);
+    int totalGroupCount = groupsInPartition.values().stream().reduce(Integer::sum).orElse(0);
+    return new RewritePlanResult(groups, totalGroupCount, groupsInPartition);
+  }
+
+  private StructLikeMap<List<List<FileScanTask>>> planFileGroups(
+      Table table, Expression filter, long startingSnapshotId) {
+    CloseableIterable<FileScanTask> fileScanTasks =
+        table
+            .newScan()
+            .useSnapshot(startingSnapshotId)
+            .filter(filter)
+            .ignoreResiduals()
+            .planFiles();
+
+    try {
+      Types.StructType partitionType = table.spec().partitionType();
+      StructLikeMap<List<FileScanTask>> filesByPartition =
+          groupByPartition(table, partitionType, fileScanTasks);
+      return filesByPartition.transformValues(
+          tasks -> ImmutableList.copyOf(rewriter.planFileGroups(tasks)));
+    } finally {
+      try {
+        fileScanTasks.close();
+      } catch (IOException io) {
+        LOG.error("Cannot properly close file iterable while planning for rewrite", io);
+      }
+    }
+  }
+
+  private StructLikeMap<List<FileScanTask>> groupByPartition(
+      Table table, Types.StructType partitionType, Iterable<FileScanTask> tasks) {
+    StructLikeMap<List<FileScanTask>> filesByPartition = StructLikeMap.create(partitionType);
+    StructLike emptyStruct = GenericRecord.create(partitionType);
+
+    for (FileScanTask task : tasks) {
+      // If a task uses an incompatible partition spec the data inside could contain values
+      // which belong to multiple partitions in the current spec. Treating all such files as
+      // un-partitioned and grouping them together helps to minimize new files made.
+      StructLike taskPartition =
+          task.file().specId() == table.spec().specId() ? task.file().partition() : emptyStruct;

Review Comment:
   Made `EmptyStructLike` public for now, but if you think this change doesn't worth it to make it public, I can revert this part.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor FileRewriter interface to separate planning and execution [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#issuecomment-2781750135

   This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor FileRewriter interface to separate planning and execution [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1941169223


##########
core/src/main/java/org/apache/iceberg/actions/FileRewriteExecutor.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.util.Map;
+import java.util.Set;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.ContentScanTask;
+
+/**
+ * A class for rewriting content file groups ({@link FileRewriteGroup}).
+ *
+ * @param <I> the Java type of the plan info
+ * @param <T> the Java type of the tasks to read content files
+ * @param <F> the Java type of the content files
+ * @param <G> the Java type of the planned groups
+ * @param <P> the Java type of the plan to execute
+ */
+public interface FileRewriteExecutor<
+    I,
+    T extends ContentScanTask<F>,
+    F extends ContentFile<F>,
+    G extends FileRewriteGroup<I, T, F>,
+    P extends FileRewritePlan<I, T, F, G>> {
+
+  /** Returns a description for this rewriter. */
+  default String description() {
+    return getClass().getName();
+  }
+
+  /**
+   * Returns a set of supported options for this rewriter. Only options specified in this list will
+   * be accepted at runtime. Any other options will be rejected.
+   */
+  Set<String> validOptions();
+
+  /**
+   * Initializes this rewriter using provided options.
+   *
+   * @param options options to initialize this rewriter
+   */
+  void init(Map<String, String> options);

Review Comment:
   Without modifying the current Spark behavior we can not merge the `init` and the `initPlan`.
   The `init` is called when the executor is created and based only on the user provided parameters, the `initPlan` is used to set parameters calculated during the planning phase.
   
   I have posted an email to the dev list: https://lists.apache.org/thread/6lj2jn3dbvqjscc96w0mc32bhxq0qfqv
   There I have asked a question about the "Data organization". If we decide on the 2nd answer "group should contain every information" then we don't need the `initPlan` anymore.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor FileRewriter interface to separate planning and execution [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#issuecomment-2633937405

   @stevenzwu: Thanks for the review! Could you please take a look at the relevant dev list email thread: https://lists.apache.org/thread/6lj2jn3dbvqjscc96w0mc32bhxq0qfqv? There are some basic questions which we have to answer. If we decide on those, I can simplify the API and we can move forward with the PR.
   
   Thanks!
   Peter


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#issuecomment-2474919672

   For easier review it would be great if you could highlight the changes made. 
   
   I see the note on RewritePlanResult but i'm not sure where this class came from. It is unclear what has been extracted and renamed and what is new code.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1841271760


##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteDataFilesSparkAction.java:
##########
@@ -276,29 +215,25 @@ RewriteDataFilesCommitManager commitManager(long startingSnapshotId) {
   }
 
   private Builder doExecute(
-      RewriteExecutionContext ctx,
-      Stream<RewriteFileGroup> groupStream,
-      RewriteDataFilesCommitManager commitManager) {
+      RewritePlanResult planResult, RewriteDataFilesCommitManager commitManager) {
     ExecutorService rewriteService = rewriteService();
 
     ConcurrentLinkedQueue<RewriteFileGroup> rewrittenGroups = Queues.newConcurrentLinkedQueue();
 
     Tasks.Builder<RewriteFileGroup> rewriteTaskBuilder =
-        Tasks.foreach(groupStream)
+        Tasks.foreach(planResult.groups())
             .executeWith(rewriteService)
             .stopOnFailure()
             .noRetry()
             .onFailure(
-                (fileGroup, exception) -> {
-                  LOG.warn(
-                      "Failure during rewrite process for group {}", fileGroup.info(), exception);
-                });
+                (fileGroup, exception) ->
+                    LOG.warn(

Review Comment:
   Not sure why this was changed? Did Spotless get more strict?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor FileRewriter interface to separate planning and execution [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1877424024


##########
core/src/main/java/org/apache/iceberg/actions/FileRewriteGroup.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.util.Comparator;
+import java.util.List;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.ContentScanTask;
+import org.apache.iceberg.RewriteJobOrder;
+
+/**
+ * Container class representing a set of files to be rewritten by a {@link FileRewriteExecutor}.

Review Comment:
   Let me apply this to the current situation, so we can check if I understand correctly:
   - We should keep `splitSize` in the group, as this drives the reading
   - We should move `expectedOutputFiles` to the plan, as it is driving the output
   
   Is this the change that you are suggesting?
   
   I see 3 possible separation of concerns:
   1. Plan is the 'result' - everything below that is only organized based on the multiplicity of the data. So if some value applies to every group, then that value belongs to the 'global' plan variables. If a value is different for every group, then that value belongs to the group (current code)
   2. Plan is the `write` config, group is the `read` config. If I understand correctly, this is what you are suggesting. IMHO this is a bit awkward, as currently the groups are part of the plan. Maybe we could have a `readConfig`, `writeConfig` map in the plan instead of adding extra attributes to the plan and to the groups. This comes with a cost of extra parsing for the configs, but allows us more flexibility (fewer classes)
   3. The group should contain every information which is required for a single job. So the job (executor) only receives a single group and every other bit of information is global. The drawback is that some information is duplicated, but cleaner on the executor side.
   
   Your thoughts?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#issuecomment-2491342980

   This is just a WIP for the change, which uses git file renames. I hope it is easier to review this way.
   
   After the initial reviews, I need to do these steps
   - Instead of the renames, deprecate the old API, and create a new one with these classes
   - Testing needs to be reviewed (kept the current ones, so I know that I don’t break known features)
   - CI and gradle changes can be reverted (these are needed now because of the renames)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1867601536


##########
core/src/main/java/org/apache/iceberg/actions/RewritePositionDeletesGroupPlanner.java:
##########
@@ -0,0 +1,238 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.stream.Stream;
+import org.apache.iceberg.DeleteFile;
+import org.apache.iceberg.MetadataTableType;
+import org.apache.iceberg.MetadataTableUtils;
+import org.apache.iceberg.Partitioning;
+import org.apache.iceberg.PositionDeletesScanTask;
+import org.apache.iceberg.PositionDeletesTable;
+import org.apache.iceberg.RewriteJobOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.TableProperties;
+import org.apache.iceberg.actions.RewritePositionDeleteFiles.FileGroupInfo;
+import org.apache.iceberg.expressions.Expression;
+import org.apache.iceberg.expressions.Expressions;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Maps;
+import org.apache.iceberg.types.Types;
+import org.apache.iceberg.util.PartitionUtil;
+import org.apache.iceberg.util.PropertyUtil;
+import org.apache.iceberg.util.StructLikeMap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Groups specified files in the {@link Table} by {@link RewriteFileGroup}s. These will be grouped
+ * by partitions. Extends the {@link SizeBasedFileRewritePlanner} with {@link
+ * RewritePositionDeleteFiles#REWRITE_JOB_ORDER} handling.
+ */
+public class RewritePositionDeletesGroupPlanner
+    extends SizeBasedFileRewritePlanner<
+        FileGroupInfo, PositionDeletesScanTask, DeleteFile, RewritePositionDeletesGroup> {
+  private static final Logger LOG =
+      LoggerFactory.getLogger(RewritePositionDeletesGroupPlanner.class);
+
+  private final Expression filter;
+  private RewriteJobOrder rewriteJobOrder;
+
+  public RewritePositionDeletesGroupPlanner(Table table) {
+    this(table, Expressions.alwaysTrue());
+  }
+
+  /**
+   * Creates the planner for the given table.
+   *
+   * @param table to plan for
+   * @param filter used to remove files from the plan
+   */
+  public RewritePositionDeletesGroupPlanner(Table table, Expression filter) {
+    super(table);
+    this.filter = filter;
+  }
+
+  @Override
+  public Set<String> validOptions() {
+    return ImmutableSet.<String>builder()
+        .addAll(super.validOptions())
+        .add(RewritePositionDeleteFiles.REWRITE_JOB_ORDER)

Review Comment:
   It is strange for me that currently we have `RewritePositionDeleteFiles.REWRITE_JOB_ORDER` and `RewriteDataFiles.REWRITE_JOB_ORDER`. If we merge these then we could move the `REWRITE_JOB_ORDER` stuff to the `SizeBasedFileRewritePlanner` class, which might be a bit nicer.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor FileRewriter interface to separate planning and execution [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1876268701


##########
core/src/main/java/org/apache/iceberg/actions/FileRewriteExecutor.java:
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.util.Map;
+import java.util.Set;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.ContentScanTask;
+
+/**
+ * A class for rewriting content file groups ({@link FileRewriteGroup}).

Review Comment:
   This is really just a rephrasing of the class name, we need to be much more clear about what this is for.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor FileRewriter interface to separate planning and execution [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1876523608


##########
core/src/main/java/org/apache/iceberg/actions/FileRewriteGroup.java:
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.actions;
+
+import java.util.Comparator;
+import java.util.List;
+import org.apache.iceberg.ContentFile;
+import org.apache.iceberg.ContentScanTask;
+import org.apache.iceberg.RewriteJobOrder;
+
+/**
+ * Container class representing a set of files to be rewritten by a {@link FileRewriteExecutor}.

Review Comment:
   > The first one "splitSize" feels like it's global to a lot of FileRewriteGroups and not a particular property of 1 RewriteGroup, so may that belongs in Plan
   
   By my understanding **currently** this could be different for different groups in the same plan. The `splitSize(long inputSize)` and the `numOutputFiles(long inputSize)` is dependent on the input size, and big part of that is dependent on the configuration of the planner (see: https://github.com/apache/iceberg/pull/9069)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core, Spark: Refactor RewriteFileGroup planner to core [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #11513:
URL: https://github.com/apache/iceberg/pull/11513#discussion_r1865930536


##########
core/src/main/java/org/apache/iceberg/actions/RewriteFileGroup.java:
##########
@@ -31,26 +31,26 @@
 import org.apache.iceberg.util.DataFileSet;
 
 /**
- * Container class representing a set of files to be rewritten by a RewriteAction and the new files
- * which have been written by the action.
+ * Container class representing a set of data files to be rewritten by a RewriteAction and the new
+ * files which have been written by the action.
  */
-public class RewriteFileGroup {
-  private final FileGroupInfo info;
-  private final List<FileScanTask> fileScanTasks;
-
+public class RewriteFileGroup extends FileRewriteGroup<FileGroupInfo, FileScanTask, DataFile> {
   private DataFileSet addedFiles = DataFileSet.create();
 
+  /**
+   * @deprecated since 1.8.0, will be removed in 1.9.0.
+   */
+  @Deprecated

Review Comment:
   Deprecated every object from the old API, but kept them for compatibility reasons.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org