[GitHub] [iceberg] openinx commented on a change in pull request #1888: Core: Add BaseDeltaWriter

[GitBox <gi...@apache.org>]

openinx commented on a change in pull request #1888:
URL: https://github.com/apache/iceberg/pull/1888#discussion_r539047703



##########
File path: core/src/main/java/org/apache/iceberg/io/BaseTaskWriter.java
##########
@@ -75,6 +79,113 @@ public WriteResult complete() throws IOException {
         .build();
   }
 
+  /**
+   * Base delta writer to write both insert records and equality-deletes.
+   */
+  protected abstract class BaseDeltaWriter implements Closeable {
+    private RollingFileWriter dataWriter;
+    private RollingEqDeleteWriter eqDeleteWriter;
+    private SortedPosDeleteWriter<T> posDeleteWriter;
+    private StructLikeMap<PathOffset> insertedRowMap;
+
+    public BaseDeltaWriter(PartitionKey partition, Schema eqDeleteSchema) {
+      Preconditions.checkNotNull(eqDeleteSchema, "equality-delete schema could not be null.");
+
+      this.dataWriter = new RollingFileWriter(partition);
+
+      this.eqDeleteWriter = new RollingEqDeleteWriter(partition);
+      this.insertedRowMap = StructLikeMap.create(eqDeleteSchema.asStruct());
+
+      this.posDeleteWriter = new SortedPosDeleteWriter<>(appenderFactory, fileFactory, format, partition);
+    }
+
+    /**
+     * Make the generic data could be read as a {@link StructLike}.
+     */
+    protected abstract StructLike asStructLike(T data);
+
+    protected abstract StructLike asCopiedKey(T row);

Review comment:
       I think the above implementation of `asCopiedKey` may has problems, because in some compute engine --- for example flink,  it will use the singleton `RowDataWrapper` instance to implement the `asStructLike`,  then for both the old key and copied key, they are sharing the same `RowDataWrapper` which has wrapped the new `RowData`.   That is messing up the keys in `insertedRowMap`.
   
   Considered this issue,  I finally decided to abstract a separate `asCopiedKey` to clone a totally different key which won't share object values with the old one.  




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org