You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "pvary (via GitHub)" <gi...@apache.org> on 2026/02/16 12:41:01 UTC

[PR] Data: Moving GenericFileWriterFactory to the new FormatModel API [iceberg]

pvary opened a new pull request, #15334:
URL: https://github.com/apache/iceberg/pull/15334

   Part of: https://github.com/apache/iceberg/pull/12298
   Implementation of the new API: https://github.com/apache/iceberg/pull/12774
   
   Migrating of the GenericFileWriterFactory


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Moving GenericFileWriterFactory to the new FormatModel API [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #15334:
URL: https://github.com/apache/iceberg/pull/15334#discussion_r2813155613


##########
data/src/main/java/org/apache/iceberg/data/GenericFileWriterFactory.java:
##########
@@ -107,62 +130,163 @@ public class GenericFileWriterFactory extends BaseFileWriterFactory<Record> {
     super(
         table,
         dataFileFormat,
+        Record.class,
         dataSchema,
         dataSortOrder,
         deleteFileFormat,
         equalityFieldIds,
         equalityDeleteRowSchema,
         equalityDeleteSortOrder,
-        positionDeleteRowSchema);
+        ImmutableMap.of(),
+        dataSchema,
+        equalityDeleteRowSchema);
+    this.table = table;
+    this.format = dataFileFormat;
+    this.positionDeleteRowSchema = positionDeleteRowSchema;
   }
 
   static Builder builderFor(Table table) {
     return new Builder(table);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.10.0, will be removed in 1.11.0. It won't be called starting 1.10.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureDataWrite(Avro.DataWriteBuilder builder) {
-    builder.createWriterFunc(DataWriter::create);
+    throwUnsupportedOperationException();
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.10.0, will be removed in 1.11.0. It won't be called starting 1.10.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureEqualityDelete(Avro.DeleteWriteBuilder builder) {
-    builder.createWriterFunc(DataWriter::create);
+    throwUnsupportedOperationException();
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.10.0, will be removed in 1.11.0. It won't be called starting 1.10.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configurePositionDelete(Avro.DeleteWriteBuilder builder) {
-    builder.createWriterFunc(DataWriter::create);
+    throwUnsupportedOperationException();
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.10.0, will be removed in 1.11.0. It won't be called starting 1.10.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureDataWrite(Parquet.DataWriteBuilder builder) {
-    builder.createWriterFunc(GenericParquetWriter::create);
+    throwUnsupportedOperationException();
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.10.0, will be removed in 1.11.0. It won't be called starting 1.10.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureEqualityDelete(Parquet.DeleteWriteBuilder builder) {
-    builder.createWriterFunc(GenericParquetWriter::create);
+    throwUnsupportedOperationException();
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.10.0, will be removed in 1.11.0. It won't be called starting 1.10.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configurePositionDelete(Parquet.DeleteWriteBuilder builder) {
-    builder.createWriterFunc(GenericParquetWriter::create);
+    throwUnsupportedOperationException();
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.10.0, will be removed in 1.11.0. It won't be called starting 1.10.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureDataWrite(ORC.DataWriteBuilder builder) {
-    builder.createWriterFunc(GenericOrcWriter::buildWriter);
+    throwUnsupportedOperationException();
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.10.0, will be removed in 1.11.0. It won't be called starting 1.10.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureEqualityDelete(ORC.DeleteWriteBuilder builder) {
-    builder.createWriterFunc(GenericOrcWriter::buildWriter);
+    throwUnsupportedOperationException();
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.10.0, will be removed in 1.11.0. It won't be called starting 1.10.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configurePositionDelete(ORC.DeleteWriteBuilder builder) {
-    builder.createWriterFunc(GenericOrcWriter::buildWriter);
+    throwUnsupportedOperationException();
+  }
+
+  private void throwUnsupportedOperationException() {
+    throw new UnsupportedOperationException(
+        "Method is deprecated and should not be called. "
+            + "Configuration is already done by the registry.");
+  }
+
+  @Override
+  public PositionDeleteWriter<Record> newPositionDeleteWriter(
+      EncryptedOutputFile file, PartitionSpec spec, StructLike partition) {
+    if (positionDeleteRowSchema == null) {
+      return super.newPositionDeleteWriter(file, spec, partition);
+    } else {
+      LOG.info(

Review Comment:
   WARN maybe?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Moving GenericFileWriterFactory to the new FormatModel API [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15334:
URL: https://github.com/apache/iceberg/pull/15334#discussion_r2813882936


##########
data/src/main/java/org/apache/iceberg/data/GenericFileWriterFactory.java:
##########
@@ -107,62 +130,163 @@ public class GenericFileWriterFactory extends BaseFileWriterFactory<Record> {
     super(
         table,
         dataFileFormat,
+        Record.class,
         dataSchema,
         dataSortOrder,
         deleteFileFormat,
         equalityFieldIds,
         equalityDeleteRowSchema,
         equalityDeleteSortOrder,
-        positionDeleteRowSchema);
+        ImmutableMap.of(),
+        dataSchema,
+        equalityDeleteRowSchema);
+    this.table = table;
+    this.format = dataFileFormat;
+    this.positionDeleteRowSchema = positionDeleteRowSchema;
   }
 
   static Builder builderFor(Table table) {
     return new Builder(table);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.10.0, will be removed in 1.11.0. It won't be called starting 1.10.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureDataWrite(Avro.DataWriteBuilder builder) {
-    builder.createWriterFunc(DataWriter::create);
+    throwUnsupportedOperationException();
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.10.0, will be removed in 1.11.0. It won't be called starting 1.10.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureEqualityDelete(Avro.DeleteWriteBuilder builder) {
-    builder.createWriterFunc(DataWriter::create);
+    throwUnsupportedOperationException();
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.10.0, will be removed in 1.11.0. It won't be called starting 1.10.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configurePositionDelete(Avro.DeleteWriteBuilder builder) {
-    builder.createWriterFunc(DataWriter::create);
+    throwUnsupportedOperationException();
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.10.0, will be removed in 1.11.0. It won't be called starting 1.10.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureDataWrite(Parquet.DataWriteBuilder builder) {
-    builder.createWriterFunc(GenericParquetWriter::create);
+    throwUnsupportedOperationException();
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.10.0, will be removed in 1.11.0. It won't be called starting 1.10.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureEqualityDelete(Parquet.DeleteWriteBuilder builder) {
-    builder.createWriterFunc(GenericParquetWriter::create);
+    throwUnsupportedOperationException();
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.10.0, will be removed in 1.11.0. It won't be called starting 1.10.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configurePositionDelete(Parquet.DeleteWriteBuilder builder) {
-    builder.createWriterFunc(GenericParquetWriter::create);
+    throwUnsupportedOperationException();
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.10.0, will be removed in 1.11.0. It won't be called starting 1.10.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureDataWrite(ORC.DataWriteBuilder builder) {
-    builder.createWriterFunc(GenericOrcWriter::buildWriter);
+    throwUnsupportedOperationException();
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.10.0, will be removed in 1.11.0. It won't be called starting 1.10.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureEqualityDelete(ORC.DeleteWriteBuilder builder) {
-    builder.createWriterFunc(GenericOrcWriter::buildWriter);
+    throwUnsupportedOperationException();
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.10.0, will be removed in 1.11.0. It won't be called starting 1.10.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configurePositionDelete(ORC.DeleteWriteBuilder builder) {
-    builder.createWriterFunc(GenericOrcWriter::buildWriter);
+    throwUnsupportedOperationException();
+  }
+
+  private void throwUnsupportedOperationException() {
+    throw new UnsupportedOperationException(

Review Comment:
   These methods were originally defined in the abstract `BaseFileWriterFactory` so that subclasses, such as `GenericFileWriterFactory` could implement them. BaseFileWriterFactory also relied on them to configure writers. The new RegistryBasedFileWriterFactory neither needs nor uses these methods, so any external calls to them would now represent a potential source of errors.
   
   I considered three options:
   1. Remove the methods – This causes compile‑time failures when subclasses use `@Override`.
   2. Keep the methods but throw an exception – This fails fast but breaks callers that still rely on the old configuration path.
   3. Keep the methods as no‑ops – This avoids failures but silently ignores calls, which can mask bugs.
   
   Both options (2) and (3) behave unpredictably if someone overrides `newDataWriter`, `newEqualityDeleteWriter`, or `newPositionDeleteWriter `and continues to rely on the old `configure...` methods: option (2) throws, option (3) silently discards the call.
   After thinking through the trade‑offs, the best approach seems to be keeping the methods as-is and marking them as deprecated, leaving their behavior unchanged. This provides a clear migration signal without introducing new failure modes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Moving GenericFileWriterFactory to the new FormatModel API [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15334:
URL: https://github.com/apache/iceberg/pull/15334#discussion_r2813797973


##########
data/src/main/java/org/apache/iceberg/data/BaseFileWriterFactory.java:
##########
@@ -75,13 +81,6 @@ protected BaseFileWriterFactory(
     this.positionDeleteRowSchema = null;
   }
 
-  /**
-   * @deprecated This constructor is deprecated as of version 1.11.0 and will be removed in 1.12.0.
-   *     Position deletes that include row data are no longer supported. Use {@link

Review Comment:
   I removed the deprecation comment here because this method was introduced in the current release cycle, before we decided to deprecate the entire class. Now that the class itself is deprecated, and the replacement class doesn’t provide a way to set the position‑delete row‑data schema, the constuctor is effectively deprecated as well. However, since there’s no new equivalent method, we don’t really have a suitable place to add a dedicated deprecation message.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Moving GenericFileWriterFactory to the new FormatModel API [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #15334:
URL: https://github.com/apache/iceberg/pull/15334#discussion_r2813147084


##########
data/src/main/java/org/apache/iceberg/data/GenericFileWriterFactory.java:
##########
@@ -107,62 +130,163 @@ public class GenericFileWriterFactory extends BaseFileWriterFactory<Record> {
     super(
         table,
         dataFileFormat,
+        Record.class,
         dataSchema,
         dataSortOrder,
         deleteFileFormat,
         equalityFieldIds,
         equalityDeleteRowSchema,
         equalityDeleteSortOrder,
-        positionDeleteRowSchema);
+        ImmutableMap.of(),
+        dataSchema,
+        equalityDeleteRowSchema);
+    this.table = table;
+    this.format = dataFileFormat;
+    this.positionDeleteRowSchema = positionDeleteRowSchema;
   }
 
   static Builder builderFor(Table table) {
     return new Builder(table);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.10.0, will be removed in 1.11.0. It won't be called starting 1.10.0 as the

Review Comment:
   I think this needs to be updated to start the deprecation with 1.11.0



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Moving GenericFileWriterFactory to the new FormatModel API [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #15334:
URL: https://github.com/apache/iceberg/pull/15334#discussion_r2815183374


##########
data/src/main/java/org/apache/iceberg/data/GenericFileWriterFactory.java:
##########
@@ -107,64 +130,159 @@ public class GenericFileWriterFactory extends BaseFileWriterFactory<Record> {
     super(
         table,
         dataFileFormat,
+        Record.class,
         dataSchema,
         dataSortOrder,
         deleteFileFormat,
         equalityFieldIds,
         equalityDeleteRowSchema,
         equalityDeleteSortOrder,
-        positionDeleteRowSchema);
+        ImmutableMap.of(),
+        dataSchema,
+        equalityDeleteRowSchema);
+    this.table = table;
+    this.format = dataFileFormat;
+    this.positionDeleteRowSchema = positionDeleteRowSchema;
   }
 
   static Builder builderFor(Table table) {
     return new Builder(table);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting 1.11.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureDataWrite(Avro.DataWriteBuilder builder) {
     builder.createWriterFunc(DataWriter::create);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting 1.11.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureEqualityDelete(Avro.DeleteWriteBuilder builder) {
     builder.createWriterFunc(DataWriter::create);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting 1.11.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configurePositionDelete(Avro.DeleteWriteBuilder builder) {
     builder.createWriterFunc(DataWriter::create);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting 1.11.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureDataWrite(Parquet.DataWriteBuilder builder) {
     builder.createWriterFunc(GenericParquetWriter::create);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting 1.11.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureEqualityDelete(Parquet.DeleteWriteBuilder builder) {
     builder.createWriterFunc(GenericParquetWriter::create);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting 1.11.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configurePositionDelete(Parquet.DeleteWriteBuilder builder) {
     builder.createWriterFunc(GenericParquetWriter::create);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting 1.11.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureDataWrite(ORC.DataWriteBuilder builder) {
     builder.createWriterFunc(GenericOrcWriter::buildWriter);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting 1.11.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureEqualityDelete(ORC.DeleteWriteBuilder builder) {
     builder.createWriterFunc(GenericOrcWriter::buildWriter);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting 1.11.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configurePositionDelete(ORC.DeleteWriteBuilder builder) {
     builder.createWriterFunc(GenericOrcWriter::buildWriter);
   }
 
+  @Override
+  public PositionDeleteWriter<Record> newPositionDeleteWriter(
+      EncryptedOutputFile file, PartitionSpec spec, StructLike partition) {
+    if (positionDeleteRowSchema == null) {
+      return super.newPositionDeleteWriter(file, spec, partition);
+    } else {
+      LOG.warn(
+          "Deprecated feature used. Position delete row schema is used to create the position delete writer.");
+      MetricsConfig metricsConfig =

Review Comment:
   should this part of the code actually mimic https://github.com/apache/iceberg/blob/03f2af8ab465d4a5238753be206bcf366130f9a0/data/src/main/java/org/apache/iceberg/data/BaseFileWriterFactory.java#L293 to maintain existing behavior? Because right now I think it's a copy of https://github.com/apache/iceberg/blob/03f2af8ab465d4a5238753be206bcf366130f9a0/data/src/main/java/org/apache/iceberg/data/GenericAppenderFactory.java#L288



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Moving GenericFileWriterFactory to the new FormatModel API [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #15334:
URL: https://github.com/apache/iceberg/pull/15334#issuecomment-3914271484

   Merged to main.
   Thanks @nastra and @kevinjqliu for the reviews!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Moving GenericFileWriterFactory to the new FormatModel API [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary merged PR #15334:
URL: https://github.com/apache/iceberg/pull/15334


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Moving GenericFileWriterFactory to the new FormatModel API [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #15334:
URL: https://github.com/apache/iceberg/pull/15334#discussion_r2815169968


##########
data/src/main/java/org/apache/iceberg/data/RegistryBasedFileWriterFactory.java:
##########
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.io.UncheckedIOException;
+import java.util.Map;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.formats.FileWriterBuilder;
+import org.apache.iceberg.formats.FormatModelRegistry;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileWriterFactory;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+
+/**
+ * A base writer factory to be extended by query engine integrations.
+ *
+ * @param <T> row type
+ */
+public abstract class RegistryBasedFileWriterFactory<T, S>
+    implements FileWriterFactory<T>, Serializable {
+  private final Table table;
+  private final FileFormat dataFileFormat;
+  private final Class<T> inputType;
+  private final Schema dataSchema;
+  private final SortOrder dataSortOrder;
+  private final FileFormat deleteFileFormat;
+  private final int[] equalityFieldIds;
+  private final Schema equalityDeleteRowSchema;
+  private final SortOrder equalityDeleteSortOrder;
+  private final Map<String, String> writerProperties;
+  private final S inputSchema;
+  private final S equalityDeleteInputSchema;
+
+  protected RegistryBasedFileWriterFactory(
+      Table table,
+      FileFormat dataFileFormat,
+      Class<T> inputType,
+      Schema dataSchema,
+      SortOrder dataSortOrder,
+      FileFormat deleteFileFormat,
+      int[] equalityFieldIds,
+      Schema equalityDeleteRowSchema,
+      SortOrder equalityDeleteSortOrder,
+      Map<String, String> writerProperties,
+      S inputSchema,
+      S equalityDeleteInputSchema) {
+    this.table = table;
+    this.dataFileFormat = dataFileFormat;
+    this.inputType = inputType;
+    this.dataSchema = dataSchema;
+    this.dataSortOrder = dataSortOrder;
+    this.deleteFileFormat = deleteFileFormat;
+    this.equalityFieldIds = equalityFieldIds;
+    this.equalityDeleteRowSchema = equalityDeleteRowSchema;
+    this.equalityDeleteSortOrder = equalityDeleteSortOrder;
+    this.writerProperties = writerProperties != null ? writerProperties : ImmutableMap.of();
+    this.inputSchema = inputSchema;
+    this.equalityDeleteInputSchema = equalityDeleteInputSchema;
+  }
+
+  protected S inputSchema() {
+    return inputSchema;
+  }
+
+  protected S equalityDeleteInputSchema() {
+    return equalityDeleteInputSchema;
+  }
+
+  @Override
+  public DataWriter<T> newDataWriter(
+      EncryptedOutputFile file, PartitionSpec spec, StructLike partition) {
+    Preconditions.checkNotNull(dataSchema, "Data schema must not be null");
+    EncryptionKeyMetadata keyMetadata = file.keyMetadata();
+    Map<String, String> properties = table != null ? table.properties() : ImmutableMap.of();
+    MetricsConfig metricsConfig =
+        table != null ? MetricsConfig.forTable(table) : MetricsConfig.getDefault();
+
+    try {
+      FileWriterBuilder<DataWriter<T>, S> builder =
+          FormatModelRegistry.dataWriteBuilder(dataFileFormat, inputType, file);
+      return builder
+          .schema(dataSchema)
+          .engineSchema(inputSchema())
+          .setAll(properties)
+          .setAll(writerProperties)
+          .metricsConfig(metricsConfig)
+          .spec(spec)
+          .partition(partition)
+          .keyMetadata(keyMetadata)
+          .sortOrder(dataSortOrder)
+          .overwrite()
+          .build();
+    } catch (IOException e) {
+      throw new UncheckedIOException("Failed to create new data writer", e);
+    }
+  }
+
+  @Override
+  public EqualityDeleteWriter<T> newEqualityDeleteWriter(
+      EncryptedOutputFile file, PartitionSpec spec, StructLike partition) {
+    Preconditions.checkNotNull(equalityDeleteRowSchema, "Equality delete schema must not be null");

Review Comment:
   same as above



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Moving GenericFileWriterFactory to the new FormatModel API [iceberg]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #15334:
URL: https://github.com/apache/iceberg/pull/15334#discussion_r2812754263


##########
data/src/main/java/org/apache/iceberg/data/BaseFileWriterFactory.java:
##########
@@ -75,13 +81,6 @@ protected BaseFileWriterFactory(
     this.positionDeleteRowSchema = null;
   }
 
-  /**
-   * @deprecated This constructor is deprecated as of version 1.11.0 and will be removed in 1.12.0.
-   *     Position deletes that include row data are no longer supported. Use {@link

Review Comment:
   > Position deletes that include row data are no longer supported
   
   is this still true? i noticed its missing in the new deprecation message



##########
data/src/main/java/org/apache/iceberg/data/GenericFileWriterFactory.java:
##########
@@ -107,62 +130,163 @@ public class GenericFileWriterFactory extends BaseFileWriterFactory<Record> {
     super(
         table,
         dataFileFormat,
+        Record.class,
         dataSchema,
         dataSortOrder,
         deleteFileFormat,
         equalityFieldIds,
         equalityDeleteRowSchema,
         equalityDeleteSortOrder,
-        positionDeleteRowSchema);
+        ImmutableMap.of(),
+        dataSchema,
+        equalityDeleteRowSchema);
+    this.table = table;
+    this.format = dataFileFormat;
+    this.positionDeleteRowSchema = positionDeleteRowSchema;
   }
 
   static Builder builderFor(Table table) {
     return new Builder(table);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.10.0, will be removed in 1.11.0. It won't be called starting 1.10.0 as the

Review Comment:
   trying to wrap my head around the timing aspect for this comment...
   1.10.x is already released. This PR will be included in 1.11.x (hopefully). Is it ok to say that its been deprecated since 1.10.0 since it has not been marked as deprecated in 1.10.x? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Moving GenericFileWriterFactory to the new FormatModel API [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #15334:
URL: https://github.com/apache/iceberg/pull/15334#discussion_r2816106588


##########
data/src/main/java/org/apache/iceberg/data/GenericFileWriterFactory.java:
##########
@@ -107,64 +132,169 @@ public class GenericFileWriterFactory extends BaseFileWriterFactory<Record> {
     super(
         table,
         dataFileFormat,
+        Record.class,
         dataSchema,
         dataSortOrder,
         deleteFileFormat,
         equalityFieldIds,
         equalityDeleteRowSchema,
         equalityDeleteSortOrder,
-        positionDeleteRowSchema);
+        ImmutableMap.of(),
+        dataSchema,
+        equalityDeleteRowSchema);
+    this.table = table;
+    this.format = dataFileFormat;
+    this.positionDeleteRowSchema = positionDeleteRowSchema;
   }
 
   static Builder builderFor(Table table) {
     return new Builder(table);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting in 1.11.0 as
+   *     the configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureDataWrite(Avro.DataWriteBuilder builder) {
     builder.createWriterFunc(DataWriter::create);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting in 1.11.0 as
+   *     the configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureEqualityDelete(Avro.DeleteWriteBuilder builder) {
     builder.createWriterFunc(DataWriter::create);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting in 1.11.0 as
+   *     the configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configurePositionDelete(Avro.DeleteWriteBuilder builder) {
     builder.createWriterFunc(DataWriter::create);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting in 1.11.0 as
+   *     the configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureDataWrite(Parquet.DataWriteBuilder builder) {
     builder.createWriterFunc(GenericParquetWriter::create);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting in 1.11.0 as
+   *     the configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureEqualityDelete(Parquet.DeleteWriteBuilder builder) {
     builder.createWriterFunc(GenericParquetWriter::create);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting in 1.11.0 as
+   *     the configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configurePositionDelete(Parquet.DeleteWriteBuilder builder) {
     builder.createWriterFunc(GenericParquetWriter::create);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting in 1.11.0 as
+   *     the configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureDataWrite(ORC.DataWriteBuilder builder) {
     builder.createWriterFunc(GenericOrcWriter::buildWriter);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting in 1.11.0 as
+   *     the configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureEqualityDelete(ORC.DeleteWriteBuilder builder) {
     builder.createWriterFunc(GenericOrcWriter::buildWriter);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting in 1.11.0 as
+   *     the configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configurePositionDelete(ORC.DeleteWriteBuilder builder) {
     builder.createWriterFunc(GenericOrcWriter::buildWriter);
   }
 
+  @Override
+  public PositionDeleteWriter<Record> newPositionDeleteWriter(
+      EncryptedOutputFile file, PartitionSpec spec, StructLike partition) {
+    if (positionDeleteRowSchema == null) {
+      return super.newPositionDeleteWriter(file, spec, partition);
+    } else {
+      LOG.warn(
+          "Deprecated feature used. Position delete row schema is used to create the position delete writer.");
+      Map<String, String> properties = table == null ? ImmutableMap.of() : table.properties();
+      MetricsConfig metricsConfig =
+          table == null
+              ? MetricsConfig.forPositionDelete()
+              : MetricsConfig.forPositionDelete(table);
+
+      try {
+        return switch (format) {
+          case AVRO ->
+              Avro.writeDeletes(file)
+                  .setAll(properties)
+                  .setAll(writerProperties)
+                  .metricsConfig(metricsConfig)
+                  .createWriterFunc(DataWriter::create)
+                  .withPartition(partition)
+                  .overwrite()
+                  .rowSchema(positionDeleteRowSchema)
+                  .withSpec(spec)
+                  .withKeyMetadata(file.keyMetadata())
+                  .buildPositionWriter();
+          case ORC ->
+              ORC.writeDeletes(file)
+                  .setAll(properties)
+                  .setAll(writerProperties)
+                  .metricsConfig(metricsConfig)
+                  .createWriterFunc(GenericOrcWriter::buildWriter)
+                  .withPartition(partition)
+                  .overwrite()
+                  .rowSchema(positionDeleteRowSchema)
+                  .withSpec(spec)
+                  .withKeyMetadata(file.keyMetadata())
+                  .buildPositionWriter();
+          case PARQUET ->
+              Parquet.writeDeletes(file)
+                  .setAll(properties)
+                  .setAll(writerProperties)
+                  .metricsConfig(metricsConfig)
+                  .createWriterFunc(GenericParquetWriter::create)
+                  .withPartition(partition)
+                  .overwrite()
+                  .metricsConfig(metricsConfig)

Review Comment:
   metricsConfig is set twice here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Moving GenericFileWriterFactory to the new FormatModel API [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #15334:
URL: https://github.com/apache/iceberg/pull/15334#discussion_r2813141420


##########
data/src/main/java/org/apache/iceberg/data/BaseFileWriterFactory.java:
##########
@@ -40,7 +40,13 @@
 import org.apache.iceberg.parquet.Parquet;
 import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
 
-/** A base writer factory to be extended by query engine integrations. */
+/**
+ * A base writer factory to be extended by query engine integrations.
+ *
+ * @deprecated deprecated as of version 1.11.0 and will be removed in 1.12.0. Use {@link

Review Comment:
   ```suggestion
    * @deprecated since version 1.11.0 and will be removed in 1.12.0. Use {@link
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Moving GenericFileWriterFactory to the new FormatModel API [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15334:
URL: https://github.com/apache/iceberg/pull/15334#discussion_r2815538698


##########
data/src/main/java/org/apache/iceberg/data/RegistryBasedFileWriterFactory.java:
##########
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.io.UncheckedIOException;
+import java.util.Map;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.formats.FileWriterBuilder;
+import org.apache.iceberg.formats.FormatModelRegistry;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileWriterFactory;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+
+/**
+ * A base writer factory to be extended by query engine integrations.
+ *
+ * @param <T> row type
+ */
+public abstract class RegistryBasedFileWriterFactory<T, S>
+    implements FileWriterFactory<T>, Serializable {
+  private final Table table;
+  private final FileFormat dataFileFormat;
+  private final Class<T> inputType;
+  private final Schema dataSchema;
+  private final SortOrder dataSortOrder;
+  private final FileFormat deleteFileFormat;
+  private final int[] equalityFieldIds;
+  private final Schema equalityDeleteRowSchema;
+  private final SortOrder equalityDeleteSortOrder;
+  private final Map<String, String> writerProperties;
+  private final S inputSchema;
+  private final S equalityDeleteInputSchema;
+
+  protected RegistryBasedFileWriterFactory(
+      Table table,
+      FileFormat dataFileFormat,
+      Class<T> inputType,
+      Schema dataSchema,
+      SortOrder dataSortOrder,
+      FileFormat deleteFileFormat,
+      int[] equalityFieldIds,
+      Schema equalityDeleteRowSchema,
+      SortOrder equalityDeleteSortOrder,
+      Map<String, String> writerProperties,
+      S inputSchema,
+      S equalityDeleteInputSchema) {
+    this.table = table;
+    this.dataFileFormat = dataFileFormat;
+    this.inputType = inputType;
+    this.dataSchema = dataSchema;
+    this.dataSortOrder = dataSortOrder;
+    this.deleteFileFormat = deleteFileFormat;
+    this.equalityFieldIds = equalityFieldIds;
+    this.equalityDeleteRowSchema = equalityDeleteRowSchema;
+    this.equalityDeleteSortOrder = equalityDeleteSortOrder;
+    this.writerProperties = writerProperties != null ? writerProperties : ImmutableMap.of();
+    this.inputSchema = inputSchema;
+    this.equalityDeleteInputSchema = equalityDeleteInputSchema;
+  }
+
+  protected S inputSchema() {
+    return inputSchema;
+  }
+
+  protected S equalityDeleteInputSchema() {
+    return equalityDeleteInputSchema;
+  }
+
+  @Override
+  public DataWriter<T> newDataWriter(
+      EncryptedOutputFile file, PartitionSpec spec, StructLike partition) {
+    Preconditions.checkNotNull(dataSchema, "Data schema must not be null");
+    EncryptionKeyMetadata keyMetadata = file.keyMetadata();
+    Map<String, String> properties = table != null ? table.properties() : ImmutableMap.of();
+    MetricsConfig metricsConfig =
+        table != null ? MetricsConfig.forTable(table) : MetricsConfig.getDefault();
+
+    try {
+      FileWriterBuilder<DataWriter<T>, S> builder =
+          FormatModelRegistry.dataWriteBuilder(dataFileFormat, inputType, file);
+      return builder
+          .schema(dataSchema)
+          .engineSchema(inputSchema())
+          .setAll(properties)
+          .setAll(writerProperties)
+          .metricsConfig(metricsConfig)
+          .spec(spec)
+          .partition(partition)
+          .keyMetadata(keyMetadata)
+          .sortOrder(dataSortOrder)
+          .overwrite()
+          .build();
+    } catch (IOException e) {
+      throw new UncheckedIOException("Failed to create new data writer", e);
+    }
+  }
+
+  @Override
+  public EqualityDeleteWriter<T> newEqualityDeleteWriter(
+      EncryptedOutputFile file, PartitionSpec spec, StructLike partition) {
+    Preconditions.checkNotNull(equalityDeleteRowSchema, "Equality delete schema must not be null");

Review Comment:
   Done



##########
data/src/main/java/org/apache/iceberg/data/GenericFileWriterFactory.java:
##########
@@ -107,64 +130,159 @@ public class GenericFileWriterFactory extends BaseFileWriterFactory<Record> {
     super(
         table,
         dataFileFormat,
+        Record.class,
         dataSchema,
         dataSortOrder,
         deleteFileFormat,
         equalityFieldIds,
         equalityDeleteRowSchema,
         equalityDeleteSortOrder,
-        positionDeleteRowSchema);
+        ImmutableMap.of(),
+        dataSchema,
+        equalityDeleteRowSchema);
+    this.table = table;
+    this.format = dataFileFormat;
+    this.positionDeleteRowSchema = positionDeleteRowSchema;
   }
 
   static Builder builderFor(Table table) {
     return new Builder(table);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting 1.11.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureDataWrite(Avro.DataWriteBuilder builder) {
     builder.createWriterFunc(DataWriter::create);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting 1.11.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureEqualityDelete(Avro.DeleteWriteBuilder builder) {
     builder.createWriterFunc(DataWriter::create);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting 1.11.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configurePositionDelete(Avro.DeleteWriteBuilder builder) {
     builder.createWriterFunc(DataWriter::create);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting 1.11.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureDataWrite(Parquet.DataWriteBuilder builder) {
     builder.createWriterFunc(GenericParquetWriter::create);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting 1.11.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureEqualityDelete(Parquet.DeleteWriteBuilder builder) {
     builder.createWriterFunc(GenericParquetWriter::create);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting 1.11.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configurePositionDelete(Parquet.DeleteWriteBuilder builder) {
     builder.createWriterFunc(GenericParquetWriter::create);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting 1.11.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureDataWrite(ORC.DataWriteBuilder builder) {
     builder.createWriterFunc(GenericOrcWriter::buildWriter);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting 1.11.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureEqualityDelete(ORC.DeleteWriteBuilder builder) {
     builder.createWriterFunc(GenericOrcWriter::buildWriter);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting 1.11.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configurePositionDelete(ORC.DeleteWriteBuilder builder) {
     builder.createWriterFunc(GenericOrcWriter::buildWriter);
   }
 
+  @Override
+  public PositionDeleteWriter<Record> newPositionDeleteWriter(
+      EncryptedOutputFile file, PartitionSpec spec, StructLike partition) {
+    if (positionDeleteRowSchema == null) {
+      return super.newPositionDeleteWriter(file, spec, partition);
+    } else {
+      LOG.warn(
+          "Deprecated feature used. Position delete row schema is used to create the position delete writer.");
+      MetricsConfig metricsConfig =

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Moving GenericFileWriterFactory to the new FormatModel API [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15334:
URL: https://github.com/apache/iceberg/pull/15334#discussion_r2815537951


##########
data/src/main/java/org/apache/iceberg/data/GenericFileWriterFactory.java:
##########
@@ -107,64 +130,159 @@ public class GenericFileWriterFactory extends BaseFileWriterFactory<Record> {
     super(
         table,
         dataFileFormat,
+        Record.class,
         dataSchema,
         dataSortOrder,
         deleteFileFormat,
         equalityFieldIds,
         equalityDeleteRowSchema,
         equalityDeleteSortOrder,
-        positionDeleteRowSchema);
+        ImmutableMap.of(),
+        dataSchema,
+        equalityDeleteRowSchema);
+    this.table = table;
+    this.format = dataFileFormat;
+    this.positionDeleteRowSchema = positionDeleteRowSchema;
   }
 
   static Builder builderFor(Table table) {
     return new Builder(table);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting 1.11.0 as the

Review Comment:
   Fixed in every place



##########
data/src/main/java/org/apache/iceberg/data/RegistryBasedFileWriterFactory.java:
##########
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.io.UncheckedIOException;
+import java.util.Map;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.formats.FileWriterBuilder;
+import org.apache.iceberg.formats.FormatModelRegistry;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileWriterFactory;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+
+/**
+ * A base writer factory to be extended by query engine integrations.
+ *
+ * @param <T> row type
+ */
+public abstract class RegistryBasedFileWriterFactory<T, S>
+    implements FileWriterFactory<T>, Serializable {
+  private final Table table;
+  private final FileFormat dataFileFormat;
+  private final Class<T> inputType;
+  private final Schema dataSchema;
+  private final SortOrder dataSortOrder;
+  private final FileFormat deleteFileFormat;
+  private final int[] equalityFieldIds;
+  private final Schema equalityDeleteRowSchema;
+  private final SortOrder equalityDeleteSortOrder;
+  private final Map<String, String> writerProperties;
+  private final S inputSchema;
+  private final S equalityDeleteInputSchema;
+
+  protected RegistryBasedFileWriterFactory(
+      Table table,
+      FileFormat dataFileFormat,
+      Class<T> inputType,
+      Schema dataSchema,
+      SortOrder dataSortOrder,
+      FileFormat deleteFileFormat,
+      int[] equalityFieldIds,
+      Schema equalityDeleteRowSchema,
+      SortOrder equalityDeleteSortOrder,
+      Map<String, String> writerProperties,
+      S inputSchema,
+      S equalityDeleteInputSchema) {
+    this.table = table;
+    this.dataFileFormat = dataFileFormat;
+    this.inputType = inputType;
+    this.dataSchema = dataSchema;
+    this.dataSortOrder = dataSortOrder;
+    this.deleteFileFormat = deleteFileFormat;
+    this.equalityFieldIds = equalityFieldIds;
+    this.equalityDeleteRowSchema = equalityDeleteRowSchema;
+    this.equalityDeleteSortOrder = equalityDeleteSortOrder;
+    this.writerProperties = writerProperties != null ? writerProperties : ImmutableMap.of();
+    this.inputSchema = inputSchema;
+    this.equalityDeleteInputSchema = equalityDeleteInputSchema;
+  }
+
+  protected S inputSchema() {
+    return inputSchema;
+  }
+
+  protected S equalityDeleteInputSchema() {
+    return equalityDeleteInputSchema;
+  }
+
+  @Override
+  public DataWriter<T> newDataWriter(
+      EncryptedOutputFile file, PartitionSpec spec, StructLike partition) {
+    Preconditions.checkNotNull(dataSchema, "Data schema must not be null");

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Moving GenericFileWriterFactory to the new FormatModel API [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15334:
URL: https://github.com/apache/iceberg/pull/15334#discussion_r2815844372


##########
data/src/main/java/org/apache/iceberg/data/GenericFileWriterFactory.java:
##########
@@ -107,64 +130,159 @@ public class GenericFileWriterFactory extends BaseFileWriterFactory<Record> {
     super(
         table,
         dataFileFormat,
+        Record.class,
         dataSchema,
         dataSortOrder,
         deleteFileFormat,
         equalityFieldIds,
         equalityDeleteRowSchema,
         equalityDeleteSortOrder,
-        positionDeleteRowSchema);
+        ImmutableMap.of(),
+        dataSchema,
+        equalityDeleteRowSchema);
+    this.table = table;
+    this.format = dataFileFormat;
+    this.positionDeleteRowSchema = positionDeleteRowSchema;
   }
 
   static Builder builderFor(Table table) {
     return new Builder(table);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting 1.11.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureDataWrite(Avro.DataWriteBuilder builder) {
     builder.createWriterFunc(DataWriter::create);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting 1.11.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureEqualityDelete(Avro.DeleteWriteBuilder builder) {
     builder.createWriterFunc(DataWriter::create);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting 1.11.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configurePositionDelete(Avro.DeleteWriteBuilder builder) {
     builder.createWriterFunc(DataWriter::create);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting 1.11.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureDataWrite(Parquet.DataWriteBuilder builder) {
     builder.createWriterFunc(GenericParquetWriter::create);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting 1.11.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureEqualityDelete(Parquet.DeleteWriteBuilder builder) {
     builder.createWriterFunc(GenericParquetWriter::create);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting 1.11.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configurePositionDelete(Parquet.DeleteWriteBuilder builder) {
     builder.createWriterFunc(GenericParquetWriter::create);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting 1.11.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureDataWrite(ORC.DataWriteBuilder builder) {
     builder.createWriterFunc(GenericOrcWriter::buildWriter);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting 1.11.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureEqualityDelete(ORC.DeleteWriteBuilder builder) {
     builder.createWriterFunc(GenericOrcWriter::buildWriter);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting 1.11.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configurePositionDelete(ORC.DeleteWriteBuilder builder) {
     builder.createWriterFunc(GenericOrcWriter::buildWriter);
   }
 
+  @Override
+  public PositionDeleteWriter<Record> newPositionDeleteWriter(
+      EncryptedOutputFile file, PartitionSpec spec, StructLike partition) {
+    if (positionDeleteRowSchema == null) {
+      return super.newPositionDeleteWriter(file, spec, partition);
+    } else {
+      LOG.warn(
+          "Deprecated feature used. Position delete row schema is used to create the position delete writer.");
+      MetricsConfig metricsConfig =

Review Comment:
   I originally misunderstood this comment.
   Fixed now. Please check



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Moving GenericFileWriterFactory to the new FormatModel API [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15334:
URL: https://github.com/apache/iceberg/pull/15334#discussion_r2813919120


##########
data/src/main/java/org/apache/iceberg/data/GenericFileWriterFactory.java:
##########
@@ -107,62 +130,163 @@ public class GenericFileWriterFactory extends BaseFileWriterFactory<Record> {
     super(
         table,
         dataFileFormat,
+        Record.class,
         dataSchema,
         dataSortOrder,
         deleteFileFormat,
         equalityFieldIds,
         equalityDeleteRowSchema,
         equalityDeleteSortOrder,
-        positionDeleteRowSchema);
+        ImmutableMap.of(),
+        dataSchema,
+        equalityDeleteRowSchema);
+    this.table = table;
+    this.format = dataFileFormat;
+    this.positionDeleteRowSchema = positionDeleteRowSchema;
   }
 
   static Builder builderFor(Table table) {
     return new Builder(table);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.10.0, will be removed in 1.11.0. It won't be called starting 1.10.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureDataWrite(Avro.DataWriteBuilder builder) {
-    builder.createWriterFunc(DataWriter::create);
+    throwUnsupportedOperationException();
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.10.0, will be removed in 1.11.0. It won't be called starting 1.10.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureEqualityDelete(Avro.DeleteWriteBuilder builder) {
-    builder.createWriterFunc(DataWriter::create);
+    throwUnsupportedOperationException();
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.10.0, will be removed in 1.11.0. It won't be called starting 1.10.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configurePositionDelete(Avro.DeleteWriteBuilder builder) {
-    builder.createWriterFunc(DataWriter::create);
+    throwUnsupportedOperationException();
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.10.0, will be removed in 1.11.0. It won't be called starting 1.10.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureDataWrite(Parquet.DataWriteBuilder builder) {
-    builder.createWriterFunc(GenericParquetWriter::create);
+    throwUnsupportedOperationException();
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.10.0, will be removed in 1.11.0. It won't be called starting 1.10.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureEqualityDelete(Parquet.DeleteWriteBuilder builder) {
-    builder.createWriterFunc(GenericParquetWriter::create);
+    throwUnsupportedOperationException();
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.10.0, will be removed in 1.11.0. It won't be called starting 1.10.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configurePositionDelete(Parquet.DeleteWriteBuilder builder) {
-    builder.createWriterFunc(GenericParquetWriter::create);
+    throwUnsupportedOperationException();
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.10.0, will be removed in 1.11.0. It won't be called starting 1.10.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureDataWrite(ORC.DataWriteBuilder builder) {
-    builder.createWriterFunc(GenericOrcWriter::buildWriter);
+    throwUnsupportedOperationException();
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.10.0, will be removed in 1.11.0. It won't be called starting 1.10.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureEqualityDelete(ORC.DeleteWriteBuilder builder) {
-    builder.createWriterFunc(GenericOrcWriter::buildWriter);
+    throwUnsupportedOperationException();
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.10.0, will be removed in 1.11.0. It won't be called starting 1.10.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configurePositionDelete(ORC.DeleteWriteBuilder builder) {
-    builder.createWriterFunc(GenericOrcWriter::buildWriter);
+    throwUnsupportedOperationException();
+  }
+
+  private void throwUnsupportedOperationException() {
+    throw new UnsupportedOperationException(
+        "Method is deprecated and should not be called. "
+            + "Configuration is already done by the registry.");
+  }
+
+  @Override
+  public PositionDeleteWriter<Record> newPositionDeleteWriter(
+      EncryptedOutputFile file, PartitionSpec spec, StructLike partition) {
+    if (positionDeleteRowSchema == null) {
+      return super.newPositionDeleteWriter(file, spec, partition);
+    } else {
+      LOG.info(

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Moving GenericFileWriterFactory to the new FormatModel API [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15334:
URL: https://github.com/apache/iceberg/pull/15334#discussion_r2813918465


##########
data/src/main/java/org/apache/iceberg/data/GenericFileWriterFactory.java:
##########
@@ -107,62 +130,163 @@ public class GenericFileWriterFactory extends BaseFileWriterFactory<Record> {
     super(
         table,
         dataFileFormat,
+        Record.class,
         dataSchema,
         dataSortOrder,
         deleteFileFormat,
         equalityFieldIds,
         equalityDeleteRowSchema,
         equalityDeleteSortOrder,
-        positionDeleteRowSchema);
+        ImmutableMap.of(),
+        dataSchema,
+        equalityDeleteRowSchema);
+    this.table = table;
+    this.format = dataFileFormat;
+    this.positionDeleteRowSchema = positionDeleteRowSchema;
   }
 
   static Builder builderFor(Table table) {
     return new Builder(table);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.10.0, will be removed in 1.11.0. It won't be called starting 1.10.0 as the

Review Comment:
   Ehh. Thanks for catching



##########
data/src/main/java/org/apache/iceberg/data/BaseFileWriterFactory.java:
##########
@@ -40,7 +40,13 @@
 import org.apache.iceberg.parquet.Parquet;
 import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
 
-/** A base writer factory to be extended by query engine integrations. */
+/**
+ * A base writer factory to be extended by query engine integrations.
+ *
+ * @deprecated deprecated as of version 1.11.0 and will be removed in 1.12.0. Use {@link

Review Comment:
   Changed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Moving GenericFileWriterFactory to the new FormatModel API [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #15334:
URL: https://github.com/apache/iceberg/pull/15334#discussion_r2815135163


##########
data/src/main/java/org/apache/iceberg/data/GenericFileWriterFactory.java:
##########
@@ -107,64 +130,159 @@ public class GenericFileWriterFactory extends BaseFileWriterFactory<Record> {
     super(
         table,
         dataFileFormat,
+        Record.class,
         dataSchema,
         dataSortOrder,
         deleteFileFormat,
         equalityFieldIds,
         equalityDeleteRowSchema,
         equalityDeleteSortOrder,
-        positionDeleteRowSchema);
+        ImmutableMap.of(),
+        dataSchema,
+        equalityDeleteRowSchema);
+    this.table = table;
+    this.format = dataFileFormat;
+    this.positionDeleteRowSchema = positionDeleteRowSchema;
   }
 
   static Builder builderFor(Table table) {
     return new Builder(table);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting 1.11.0 as the

Review Comment:
   just a nit here on the messaging
   ```suggestion
      * @deprecated since 1.11.0, will be removed in 1.12.0. It won't be called starting in 1.11.0 as the
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Moving GenericFileWriterFactory to the new FormatModel API [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #15334:
URL: https://github.com/apache/iceberg/pull/15334#discussion_r2813154765


##########
data/src/main/java/org/apache/iceberg/data/GenericFileWriterFactory.java:
##########
@@ -107,62 +130,163 @@ public class GenericFileWriterFactory extends BaseFileWriterFactory<Record> {
     super(
         table,
         dataFileFormat,
+        Record.class,
         dataSchema,
         dataSortOrder,
         deleteFileFormat,
         equalityFieldIds,
         equalityDeleteRowSchema,
         equalityDeleteSortOrder,
-        positionDeleteRowSchema);
+        ImmutableMap.of(),
+        dataSchema,
+        equalityDeleteRowSchema);
+    this.table = table;
+    this.format = dataFileFormat;
+    this.positionDeleteRowSchema = positionDeleteRowSchema;
   }
 
   static Builder builderFor(Table table) {
     return new Builder(table);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.10.0, will be removed in 1.11.0. It won't be called starting 1.10.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureDataWrite(Avro.DataWriteBuilder builder) {
-    builder.createWriterFunc(DataWriter::create);
+    throwUnsupportedOperationException();
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.10.0, will be removed in 1.11.0. It won't be called starting 1.10.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureEqualityDelete(Avro.DeleteWriteBuilder builder) {
-    builder.createWriterFunc(DataWriter::create);
+    throwUnsupportedOperationException();
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.10.0, will be removed in 1.11.0. It won't be called starting 1.10.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configurePositionDelete(Avro.DeleteWriteBuilder builder) {
-    builder.createWriterFunc(DataWriter::create);
+    throwUnsupportedOperationException();
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.10.0, will be removed in 1.11.0. It won't be called starting 1.10.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureDataWrite(Parquet.DataWriteBuilder builder) {
-    builder.createWriterFunc(GenericParquetWriter::create);
+    throwUnsupportedOperationException();
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.10.0, will be removed in 1.11.0. It won't be called starting 1.10.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureEqualityDelete(Parquet.DeleteWriteBuilder builder) {
-    builder.createWriterFunc(GenericParquetWriter::create);
+    throwUnsupportedOperationException();
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.10.0, will be removed in 1.11.0. It won't be called starting 1.10.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configurePositionDelete(Parquet.DeleteWriteBuilder builder) {
-    builder.createWriterFunc(GenericParquetWriter::create);
+    throwUnsupportedOperationException();
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.10.0, will be removed in 1.11.0. It won't be called starting 1.10.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureDataWrite(ORC.DataWriteBuilder builder) {
-    builder.createWriterFunc(GenericOrcWriter::buildWriter);
+    throwUnsupportedOperationException();
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.10.0, will be removed in 1.11.0. It won't be called starting 1.10.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureEqualityDelete(ORC.DeleteWriteBuilder builder) {
-    builder.createWriterFunc(GenericOrcWriter::buildWriter);
+    throwUnsupportedOperationException();
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.10.0, will be removed in 1.11.0. It won't be called starting 1.10.0 as the
+   *     configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configurePositionDelete(ORC.DeleteWriteBuilder builder) {
-    builder.createWriterFunc(GenericOrcWriter::buildWriter);
+    throwUnsupportedOperationException();
+  }
+
+  private void throwUnsupportedOperationException() {
+    throw new UnsupportedOperationException(

Review Comment:
   should those methods be no-ops instead? Otherwise users will hit this exception when calling those methods. Typically when we deprecate something we keep/maintain the functionality without failing immediately



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Moving GenericFileWriterFactory to the new FormatModel API [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #15334:
URL: https://github.com/apache/iceberg/pull/15334#discussion_r2815168834


##########
data/src/main/java/org/apache/iceberg/data/RegistryBasedFileWriterFactory.java:
##########
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.data;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.io.UncheckedIOException;
+import java.util.Map;
+import org.apache.iceberg.FileFormat;
+import org.apache.iceberg.MetricsConfig;
+import org.apache.iceberg.PartitionSpec;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.SortOrder;
+import org.apache.iceberg.StructLike;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.deletes.EqualityDeleteWriter;
+import org.apache.iceberg.deletes.PositionDeleteWriter;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.encryption.EncryptionKeyMetadata;
+import org.apache.iceberg.formats.FileWriterBuilder;
+import org.apache.iceberg.formats.FormatModelRegistry;
+import org.apache.iceberg.io.DataWriter;
+import org.apache.iceberg.io.FileWriterFactory;
+import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+
+/**
+ * A base writer factory to be extended by query engine integrations.
+ *
+ * @param <T> row type
+ */
+public abstract class RegistryBasedFileWriterFactory<T, S>
+    implements FileWriterFactory<T>, Serializable {
+  private final Table table;
+  private final FileFormat dataFileFormat;
+  private final Class<T> inputType;
+  private final Schema dataSchema;
+  private final SortOrder dataSortOrder;
+  private final FileFormat deleteFileFormat;
+  private final int[] equalityFieldIds;
+  private final Schema equalityDeleteRowSchema;
+  private final SortOrder equalityDeleteSortOrder;
+  private final Map<String, String> writerProperties;
+  private final S inputSchema;
+  private final S equalityDeleteInputSchema;
+
+  protected RegistryBasedFileWriterFactory(
+      Table table,
+      FileFormat dataFileFormat,
+      Class<T> inputType,
+      Schema dataSchema,
+      SortOrder dataSortOrder,
+      FileFormat deleteFileFormat,
+      int[] equalityFieldIds,
+      Schema equalityDeleteRowSchema,
+      SortOrder equalityDeleteSortOrder,
+      Map<String, String> writerProperties,
+      S inputSchema,
+      S equalityDeleteInputSchema) {
+    this.table = table;
+    this.dataFileFormat = dataFileFormat;
+    this.inputType = inputType;
+    this.dataSchema = dataSchema;
+    this.dataSortOrder = dataSortOrder;
+    this.deleteFileFormat = deleteFileFormat;
+    this.equalityFieldIds = equalityFieldIds;
+    this.equalityDeleteRowSchema = equalityDeleteRowSchema;
+    this.equalityDeleteSortOrder = equalityDeleteSortOrder;
+    this.writerProperties = writerProperties != null ? writerProperties : ImmutableMap.of();
+    this.inputSchema = inputSchema;
+    this.equalityDeleteInputSchema = equalityDeleteInputSchema;
+  }
+
+  protected S inputSchema() {
+    return inputSchema;
+  }
+
+  protected S equalityDeleteInputSchema() {
+    return equalityDeleteInputSchema;
+  }
+
+  @Override
+  public DataWriter<T> newDataWriter(
+      EncryptedOutputFile file, PartitionSpec spec, StructLike partition) {
+    Preconditions.checkNotNull(dataSchema, "Data schema must not be null");

Review Comment:
   nit: I think `Preconditions.checkArgument()` is more appropriate here since it immediately indicates that it's about the argument being passed. Also I think we would want to align the error msg with the rest of the codebase to `Invalid data schema: null`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Data: Moving GenericFileWriterFactory to the new FormatModel API [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #15334:
URL: https://github.com/apache/iceberg/pull/15334#discussion_r2816201255


##########
data/src/main/java/org/apache/iceberg/data/GenericFileWriterFactory.java:
##########
@@ -107,64 +132,169 @@ public class GenericFileWriterFactory extends BaseFileWriterFactory<Record> {
     super(
         table,
         dataFileFormat,
+        Record.class,
         dataSchema,
         dataSortOrder,
         deleteFileFormat,
         equalityFieldIds,
         equalityDeleteRowSchema,
         equalityDeleteSortOrder,
-        positionDeleteRowSchema);
+        ImmutableMap.of(),
+        dataSchema,
+        equalityDeleteRowSchema);
+    this.table = table;
+    this.format = dataFileFormat;
+    this.positionDeleteRowSchema = positionDeleteRowSchema;
   }
 
   static Builder builderFor(Table table) {
     return new Builder(table);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting in 1.11.0 as
+   *     the configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureDataWrite(Avro.DataWriteBuilder builder) {
     builder.createWriterFunc(DataWriter::create);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting in 1.11.0 as
+   *     the configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureEqualityDelete(Avro.DeleteWriteBuilder builder) {
     builder.createWriterFunc(DataWriter::create);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting in 1.11.0 as
+   *     the configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configurePositionDelete(Avro.DeleteWriteBuilder builder) {
     builder.createWriterFunc(DataWriter::create);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting in 1.11.0 as
+   *     the configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureDataWrite(Parquet.DataWriteBuilder builder) {
     builder.createWriterFunc(GenericParquetWriter::create);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting in 1.11.0 as
+   *     the configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureEqualityDelete(Parquet.DeleteWriteBuilder builder) {
     builder.createWriterFunc(GenericParquetWriter::create);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting in 1.11.0 as
+   *     the configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configurePositionDelete(Parquet.DeleteWriteBuilder builder) {
     builder.createWriterFunc(GenericParquetWriter::create);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting in 1.11.0 as
+   *     the configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureDataWrite(ORC.DataWriteBuilder builder) {
     builder.createWriterFunc(GenericOrcWriter::buildWriter);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting in 1.11.0 as
+   *     the configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configureEqualityDelete(ORC.DeleteWriteBuilder builder) {
     builder.createWriterFunc(GenericOrcWriter::buildWriter);
   }
 
-  @Override
+  /**
+   * @deprecated Since 1.11.0, will be removed in 1.12.0. It won't be called starting in 1.11.0 as
+   *     the configuration is done by the {@link FormatModelRegistry}.
+   */
+  @Deprecated
   protected void configurePositionDelete(ORC.DeleteWriteBuilder builder) {
     builder.createWriterFunc(GenericOrcWriter::buildWriter);
   }
 
+  @Override
+  public PositionDeleteWriter<Record> newPositionDeleteWriter(
+      EncryptedOutputFile file, PartitionSpec spec, StructLike partition) {
+    if (positionDeleteRowSchema == null) {
+      return super.newPositionDeleteWriter(file, spec, partition);
+    } else {
+      LOG.warn(
+          "Deprecated feature used. Position delete row schema is used to create the position delete writer.");
+      Map<String, String> properties = table == null ? ImmutableMap.of() : table.properties();
+      MetricsConfig metricsConfig =
+          table == null
+              ? MetricsConfig.forPositionDelete()
+              : MetricsConfig.forPositionDelete(table);
+
+      try {
+        return switch (format) {
+          case AVRO ->
+              Avro.writeDeletes(file)
+                  .setAll(properties)
+                  .setAll(writerProperties)
+                  .metricsConfig(metricsConfig)
+                  .createWriterFunc(DataWriter::create)
+                  .withPartition(partition)
+                  .overwrite()
+                  .rowSchema(positionDeleteRowSchema)
+                  .withSpec(spec)
+                  .withKeyMetadata(file.keyMetadata())
+                  .buildPositionWriter();
+          case ORC ->
+              ORC.writeDeletes(file)
+                  .setAll(properties)
+                  .setAll(writerProperties)
+                  .metricsConfig(metricsConfig)
+                  .createWriterFunc(GenericOrcWriter::buildWriter)
+                  .withPartition(partition)
+                  .overwrite()
+                  .rowSchema(positionDeleteRowSchema)
+                  .withSpec(spec)
+                  .withKeyMetadata(file.keyMetadata())
+                  .buildPositionWriter();
+          case PARQUET ->
+              Parquet.writeDeletes(file)
+                  .setAll(properties)
+                  .setAll(writerProperties)
+                  .metricsConfig(metricsConfig)
+                  .createWriterFunc(GenericParquetWriter::create)
+                  .withPartition(partition)
+                  .overwrite()
+                  .metricsConfig(metricsConfig)

Review Comment:
   Fixed.
   Thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org