You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "ayushtkn (via GitHub)" <gi...@apache.org> on 2026/04/29 17:40:19 UTC

[PR] Fix reading encrypted Deletion Vectors [iceberg]

ayushtkn opened a new pull request, #16158:
URL: https://github.com/apache/iceberg/pull/16158

   Fixes: https://github.com/apache/iceberg/issues/16157


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Fix reading encrypted Deletion Vectors [iceberg]

Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on code in PR #16158:
URL: https://github.com/apache/iceberg/pull/16158#discussion_r3254152123


##########
spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/sql/TestTableEncryption.java:
##########
@@ -212,16 +212,27 @@ public void testInsertAndDelete() {
         expected,
         sql("SELECT * FROM %s ORDER BY id", tableName));
 
-    sql("DELETE FROM %s WHERE id < 4", tableName);
+    sql("DELETE FROM %s WHERE id < 3", tableName);
 
-    expected = ImmutableList.of(row(4L, "d", 4.0F), row(5L, "e", 5.0F), row(6L, "f", Float.NaN));
+    expected =
+        ImmutableList.of(
+            row(3L, "c", Float.NaN),
+            row(4L, "d", 4.0F),
+            row(5L, "e", 5.0F),
+            row(6L, "f", Float.NaN));
 
     assertEquals(
         "Should return all expected rows",
         expected,
         sql("SELECT * FROM %s ORDER BY id", tableName));
   }
 
+  @TestTemplate
+  public void testInsertAndDeleteMOR() {
+    sql("ALTER TABLE %s SET TBLPROPERTIES ('write.delete.mode'='merge-on-read')", tableName);
+    testInsertAndDelete();

Review Comment:
   Thanx @singhpk234 for the review!!!
   I added the assertion for DVs, Regarding the v2 tables, Encryption isn't supported for tables below V3. It was disabled via: https://github.com/apache/iceberg/commit/19b4bd024486d9d516d0e547e273419c1bc7074e



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Fix reading encrypted Deletion Vectors [iceberg]

Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on code in PR #16158:
URL: https://github.com/apache/iceberg/pull/16158#discussion_r3254150432


##########
spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/sql/TestTableEncryption.java:
##########
@@ -212,16 +212,27 @@ public void testInsertAndDelete() {
         expected,
         sql("SELECT * FROM %s ORDER BY id", tableName));
 
-    sql("DELETE FROM %s WHERE id < 4", tableName);
+    sql("DELETE FROM %s WHERE id < 3", tableName);
 
-    expected = ImmutableList.of(row(4L, "d", 4.0F), row(5L, "e", 5.0F), row(6L, "f", Float.NaN));
+    expected =
+        ImmutableList.of(
+            row(3L, "c", Float.NaN),
+            row(4L, "d", 4.0F),
+            row(5L, "e", 5.0F),
+            row(6L, "f", Float.NaN));
 
     assertEquals(
         "Should return all expected rows",
         expected,
         sql("SELECT * FROM %s ORDER BY id", tableName));
   }
 
+  @TestTemplate
+  public void testInsertAndDeleteMOR() {

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Fix reading encrypted Deletion Vectors [iceberg]

Posted by "ggershinsky (via GitHub)" <gi...@apache.org>.
ggershinsky commented on code in PR #16158:
URL: https://github.com/apache/iceberg/pull/16158#discussion_r3232497175


##########
core/src/main/java/org/apache/iceberg/DVUtil.java:
##########
@@ -189,7 +190,9 @@ private static List<DeleteFile> writeDVs(
       String dvOutputLocation,
       Map<String, Pair<PartitionSpec, StructLike>> partitions) {
     OutputFile dvOutputFile = fileIO.newOutputFile(dvOutputLocation);
-    try (DVFileWriter dvFileWriter = new BaseDVFileWriter(() -> dvOutputFile, path -> null)) {
+    try (DVFileWriter dvFileWriter =
+        new BaseDVFileWriter(
+            () -> EncryptionUtil.plainAsEncryptedOutput(dvOutputFile), path -> null)) {

Review Comment:
   this bypasses encryption



##########
core/src/main/java/org/apache/iceberg/DVUtil.java:
##########
@@ -189,7 +190,9 @@ private static List<DeleteFile> writeDVs(
       String dvOutputLocation,
       Map<String, Pair<PartitionSpec, StructLike>> partitions) {
     OutputFile dvOutputFile = fileIO.newOutputFile(dvOutputLocation);

Review Comment:
   if `fileIO` instance of `EncryptingFileIO`, then EncryptingOutputFile dvEncrOutputFile = ((EncryptingFileIO)fileIO).newEncryptingOutputFile(dvOutputLocation)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Fix reading encrypted Deletion Vectors [iceberg]

Posted by "ggershinsky (via GitHub)" <gi...@apache.org>.
ggershinsky commented on code in PR #16158:
URL: https://github.com/apache/iceberg/pull/16158#discussion_r3232514461


##########
core/src/main/java/org/apache/iceberg/DVUtil.java:
##########
@@ -189,7 +190,9 @@ private static List<DeleteFile> writeDVs(
       String dvOutputLocation,
       Map<String, Pair<PartitionSpec, StructLike>> partitions) {
     OutputFile dvOutputFile = fileIO.newOutputFile(dvOutputLocation);

Review Comment:
   if `fileIO` instance of `EncryptingFileIO`, then EncryptingOutputFile dvEncrOutputFile = ((EncryptingFileIO)fileIO).newEncryptingOutputFile(dvOutputLocation)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Fix reading encrypted Deletion Vectors [iceberg]

Posted by "ggershinsky (via GitHub)" <gi...@apache.org>.
ggershinsky commented on PR #16158:
URL: https://github.com/apache/iceberg/pull/16158#issuecomment-4458034990

   After experimenting with the code, I think this patch is the best solution. Thanks @ayushtkn .


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Fix reading encrypted Deletion Vectors [iceberg]

Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #16158:
URL: https://github.com/apache/iceberg/pull/16158#discussion_r3267559847


##########
core/src/main/java/org/apache/iceberg/deletes/BaseDVFileWriter.java:
##########
@@ -52,19 +55,19 @@ public class BaseDVFileWriter implements DVFileWriter {
   private static final String REFERENCED_DATA_FILE_KEY = "referenced-data-file";
   private static final String CARDINALITY_KEY = "cardinality";
 
-  private final Supplier<OutputFile> dvOutputFile;
+  private final Supplier<EncryptedOutputFile> dvOutputFile;
   private final Function<String, PositionDeleteIndex> loadPreviousDeletes;
   private final Map<String, Deletes> deletesByPath = Maps.newHashMap();
   private final Map<String, BlobMetadata> blobsByPath = Maps.newHashMap();
   private DeleteWriteResult result = null;
 
   public BaseDVFileWriter(
       OutputFileFactory fileFactory, Function<String, PositionDeleteIndex> loadPreviousDeletes) {
-    this(() -> fileFactory.newOutputFile().encryptingOutputFile(), loadPreviousDeletes);
+    this(fileFactory::newOutputFile, loadPreviousDeletes);
   }
 
   public BaseDVFileWriter(
-      Supplier<OutputFile> dvOutputFile,
+      Supplier<EncryptedOutputFile> dvOutputFile,

Review Comment:
   I'm a bit surprised revAPI didn't catch this since it's a breaking change unfortunately



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Fix reading encrypted Deletion Vectors [iceberg]

Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn closed pull request #16158: Fix reading encrypted Deletion Vectors
URL: https://github.com/apache/iceberg/pull/16158


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Fix reading encrypted Deletion Vectors [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #16158:
URL: https://github.com/apache/iceberg/pull/16158#issuecomment-4397091353

   cc: @ggershinsky 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Fix reading encrypted Deletion Vectors [iceberg]

Posted by "ggershinsky (via GitHub)" <gi...@apache.org>.
ggershinsky commented on code in PR #16158:
URL: https://github.com/apache/iceberg/pull/16158#discussion_r3232497175


##########
core/src/main/java/org/apache/iceberg/DVUtil.java:
##########
@@ -189,7 +190,9 @@ private static List<DeleteFile> writeDVs(
       String dvOutputLocation,
       Map<String, Pair<PartitionSpec, StructLike>> partitions) {
     OutputFile dvOutputFile = fileIO.newOutputFile(dvOutputLocation);
-    try (DVFileWriter dvFileWriter = new BaseDVFileWriter(() -> dvOutputFile, path -> null)) {
+    try (DVFileWriter dvFileWriter =
+        new BaseDVFileWriter(
+            () -> EncryptionUtil.plainAsEncryptedOutput(dvOutputFile), path -> null)) {

Review Comment:
   this bypasses encryption



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Fix reading encrypted Deletion Vectors [iceberg]

Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn closed pull request #16158: Core: Fix reading encrypted Deletion Vectors
URL: https://github.com/apache/iceberg/pull/16158


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Fix reading encrypted Deletion Vectors [iceberg]

Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #16158:
URL: https://github.com/apache/iceberg/pull/16158#discussion_r3254108123


##########
spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/sql/TestTableEncryption.java:
##########
@@ -212,16 +212,27 @@ public void testInsertAndDelete() {
         expected,
         sql("SELECT * FROM %s ORDER BY id", tableName));
 
-    sql("DELETE FROM %s WHERE id < 4", tableName);
+    sql("DELETE FROM %s WHERE id < 3", tableName);
 
-    expected = ImmutableList.of(row(4L, "d", 4.0F), row(5L, "e", 5.0F), row(6L, "f", Float.NaN));
+    expected =
+        ImmutableList.of(
+            row(3L, "c", Float.NaN),
+            row(4L, "d", 4.0F),
+            row(5L, "e", 5.0F),
+            row(6L, "f", Float.NaN));
 
     assertEquals(
         "Should return all expected rows",
         expected,
         sql("SELECT * FROM %s ORDER BY id", tableName));
   }
 
+  @TestTemplate
+  public void testInsertAndDeleteMOR() {
+    sql("ALTER TABLE %s SET TBLPROPERTIES ('write.delete.mode'='merge-on-read')", tableName);
+    testInsertAndDelete();

Review Comment:
   i would also additional assertions that dvs were there ? by loading the table using the catalog obj and checking its recent snapshot summary
   I understand test is gonna trigger this implicitly, though i would suggest being explicit would be ideal as we are strictly checking dvs here.
   
   Also what happens in v2 tables pos deletes and Eq delete, do you know ?



##########
core/src/main/java/org/apache/iceberg/DVUtil.java:
##########
@@ -188,8 +190,16 @@ private static List<DeleteFile> writeDVs(
       FileIO fileIO,
       String dvOutputLocation,
       Map<String, Pair<PartitionSpec, StructLike>> partitions) {
-    OutputFile dvOutputFile = fileIO.newOutputFile(dvOutputLocation);
-    try (DVFileWriter dvFileWriter = new BaseDVFileWriter(() -> dvOutputFile, path -> null)) {
+    EncryptedOutputFile dvEncryptedOutputFile;
+    if (fileIO instanceof EncryptingFileIO) {
+      dvEncryptedOutputFile = ((EncryptingFileIO) fileIO).newEncryptingOutputFile(dvOutputLocation);
+    } else {
+      dvEncryptedOutputFile =
+          EncryptionUtil.plainAsEncryptedOutput(fileIO.newOutputFile(dvOutputLocation));
+    }

Review Comment:
   ```suggestion
       if (fileIO instanceof EncryptingFileIO encryptingFileIO) {
         dvEncryptedOutputFile = encryptingFileIO.newEncryptingOutputFile(dvOutputLocation);
       } else {
         dvEncryptedOutputFile =
             EncryptionUtil.plainAsEncryptedOutput(fileIO.newOutputFile(dvOutputLocation));
   ```



##########
spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/sql/TestTableEncryption.java:
##########
@@ -212,16 +212,27 @@ public void testInsertAndDelete() {
         expected,
         sql("SELECT * FROM %s ORDER BY id", tableName));
 
-    sql("DELETE FROM %s WHERE id < 4", tableName);
+    sql("DELETE FROM %s WHERE id < 3", tableName);
 
-    expected = ImmutableList.of(row(4L, "d", 4.0F), row(5L, "e", 5.0F), row(6L, "f", Float.NaN));
+    expected =
+        ImmutableList.of(
+            row(3L, "c", Float.NaN),
+            row(4L, "d", 4.0F),
+            row(5L, "e", 5.0F),
+            row(6L, "f", Float.NaN));
 
     assertEquals(
         "Should return all expected rows",
         expected,
         sql("SELECT * FROM %s ORDER BY id", tableName));
   }
 
+  @TestTemplate
+  public void testInsertAndDeleteMOR() {

Review Comment:
   lets name the other test COW then ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Fix reading encrypted Deletion Vectors [iceberg]

Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on PR #16158:
URL: https://github.com/apache/iceberg/pull/16158#issuecomment-4489588415

   thanx @amogh-jahagirdar, I will close mine. Thanx everyone for the reviews here!!!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Fix reading encrypted Deletion Vectors [iceberg]

Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on PR #16158:
URL: https://github.com/apache/iceberg/pull/16158#issuecomment-4494876812

   Thank you @amogh-jahagirdar I missed the other PR :) ! Its really nice to see this being worked upon
   
   Thank you still @ayushtkn for your contribution and time, Thank you @ggershinsky for your review and time, please do review the other PR when you get some time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Fix reading encrypted Deletion Vectors [iceberg]

Posted by "ggershinsky (via GitHub)" <gi...@apache.org>.
ggershinsky commented on code in PR #16158:
URL: https://github.com/apache/iceberg/pull/16158#discussion_r3239652971


##########
spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/sql/TestTableEncryption.java:
##########
@@ -212,16 +212,22 @@ public void testInsertAndDelete() {
         expected,
         sql("SELECT * FROM %s ORDER BY id", tableName));
 
-    sql("DELETE FROM %s WHERE id < 4", tableName);
+    sql("DELETE FROM %s WHERE id < 5", tableName);
 
-    expected = ImmutableList.of(row(4L, "d", 4.0F), row(5L, "e", 5.0F), row(6L, "f", Float.NaN));
+    expected = ImmutableList.of(row(5L, "e", 5.0F), row(6L, "f", Float.NaN));
 
     assertEquals(
         "Should return all expected rows",
         expected,
         sql("SELECT * FROM %s ORDER BY id", tableName));
   }
 
+  @TestTemplate
+  public void testInsertAndDeleteMOR() {
+    sql("ALTER TABLE %s SET TBLPROPERTIES ('write.delete.mode'='merge-on-read')", tableName);
+    testInsertAndDelete();

Review Comment:
   The exception is not reproduced in my environment. I ran `git clone git@github.com:apache/iceberg.git`, changed the 8 lines in this class, and ran `./gradlew :iceberg-spark:iceberg-spark-4.1:test --tests org.apache.iceberg.spark.sql.TestTableEncryption`. The test passes ok. Am I doing anything wrong?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Fix reading encrypted Deletion Vectors [iceberg]

Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on code in PR #16158:
URL: https://github.com/apache/iceberg/pull/16158#discussion_r3240135195


##########
spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/sql/TestTableEncryption.java:
##########
@@ -212,16 +212,22 @@ public void testInsertAndDelete() {
         expected,
         sql("SELECT * FROM %s ORDER BY id", tableName));
 
-    sql("DELETE FROM %s WHERE id < 4", tableName);
+    sql("DELETE FROM %s WHERE id < 5", tableName);
 
-    expected = ImmutableList.of(row(4L, "d", 4.0F), row(5L, "e", 5.0F), row(6L, "f", Float.NaN));
+    expected = ImmutableList.of(row(5L, "e", 5.0F), row(6L, "f", Float.NaN));
 
     assertEquals(
         "Should return all expected rows",
         expected,
         sql("SELECT * FROM %s ORDER BY id", tableName));
   }
 
+  @TestTemplate
+  public void testInsertAndDeleteMOR() {
+    sql("ALTER TABLE %s SET TBLPROPERTIES ('write.delete.mode'='merge-on-read')", tableName);
+    testInsertAndDelete();

Review Comment:
   Some Spark thing, it fallbacks to COW, if there is a Metadata delete, not sure if it is bug in Spark or behaviour, I have changed the test.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Fix reading encrypted Deletion Vectors [iceberg]

Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on PR #16158:
URL: https://github.com/apache/iceberg/pull/16158#issuecomment-4489402236

   Hey FYI @ayushtkn @singhpk234 @ggershinsky I think there [was a bit older PR ](https://github.com/apache/iceberg/pull/15911)from @manuzhang that's now actively working on trying to address this issue, are there any objections to working off of that one? Sorry, looks like multiple people were working on this issue in parallel and I just came to know about this one too.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Fix reading encrypted Deletion Vectors [iceberg]

Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on PR #16158:
URL: https://github.com/apache/iceberg/pull/16158#issuecomment-4396661364

   @pvary can u help review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Fix reading encrypted Deletion Vectors [iceberg]

Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on code in PR #16158:
URL: https://github.com/apache/iceberg/pull/16158#discussion_r3240141803


##########
spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/sql/TestTableEncryption.java:
##########
@@ -212,16 +212,22 @@ public void testInsertAndDelete() {
         expected,
         sql("SELECT * FROM %s ORDER BY id", tableName));
 
-    sql("DELETE FROM %s WHERE id < 4", tableName);
+    sql("DELETE FROM %s WHERE id < 5", tableName);
 
-    expected = ImmutableList.of(row(4L, "d", 4.0F), row(5L, "e", 5.0F), row(6L, "f", Float.NaN));
+    expected = ImmutableList.of(row(5L, "e", 5.0F), row(6L, "f", Float.NaN));
 
     assertEquals(
         "Should return all expected rows",
         expected,
         sql("SELECT * FROM %s ORDER BY id", tableName));
   }
 
+  @TestTemplate
+  public void testInsertAndDeleteMOR() {
+    sql("ALTER TABLE %s SET TBLPROPERTIES ('write.delete.mode'='merge-on-read')", tableName);
+    testInsertAndDelete();

Review Comment:
   Can you check, I did this:
   ```
   ayushsaxena@Q3NW54Y0C5 iceberg % git reset HEAD~1
   Unstaged changes after reset:
   M       core/src/main/java/org/apache/iceberg/DVUtil.java
   M       core/src/main/java/org/apache/iceberg/deletes/BaseDVFileWriter.java
   M       spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/sql/TestTableEncryption.java
   M       spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/sql/TestTableEncryption.java
   ayushsaxena@Q3NW54Y0C5 iceberg % git checkout  core/src/main/java/org/apache/iceberg/DVUtil.java core/src/main/java/org/apache/iceberg/deletes/BaseDVFileWriter.java
   Updated 2 paths from the index
   ayushsaxena@Q3NW54Y0C5 iceberg % ./gradlew clean :iceberg-spark:iceberg-spark-4.1_2.13:test --tests "org.apache.iceberg.spark.sql.TestTableEncryption.testInsertAndDeleteMOR"
   Configuration on demand is an incubating feature.
   
   > Task :iceberg-api:testJar
   file '/Users/ayushsaxena/code/iceberg/build/iceberg-build.properties' will be copied to 'iceberg-build.properties', overwriting file '/Users/ayushsaxena/code/iceberg/api/build/resources/test/iceberg-build.properties', which has already been copied there.
   
   > Task :iceberg-spark:iceberg-spark-4.1_2.13:compileTestJava
   Note: Some input files use or override a deprecated API.
   Note: Recompile with -Xlint:deprecation for details.
   Note: Some input files use unchecked or unsafe operations.
   Note: Recompile with -Xlint:unchecked for details.
   
   > Task :iceberg-spark:iceberg-spark-4.1_2.13:test
   
   TestTableEncryption > testInsertAndDeleteMOR() > catalogName = testhive, implementation = org.apache.iceberg.spark.SparkCatalog, config = {type=hive, default-namespace=default, encryption.kms-impl=org.apache.iceberg.encryption.UnitestKMS} FAILED
       org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 9.0 failed 1 times, most recent failure: Lost task 0.0 in stage 9.0 (TID 9) (192.168.29.167 executor driver): java.lang.IllegalStateException: Null key metadata buffer
           at org.apache.iceberg.encryption.StandardKeyMetadata.castOrParse(StandardKeyMetadata.java:111)
           at org.apache.iceberg.encryption.StandardEncryptionManager$StandardDecryptedInputFile.keyMetadata(StandardEncryptionManager.java:302)
           at org.apache.iceberg.encryption.StandardEncryptionManager$StandardDecryptedInputFile.decrypted(StandardEncryptionManager.java:313)
           at org.apache.iceberg.encryption.StandardEncryptionManager$StandardDecryptedInputFile.location(StandardEncryptionManager.java:333)
           at org.apache.iceberg.encryption.EncryptingFileIO.bulkDecrypt(EncryptingFileIO.java:71)
           at org.apache.iceberg.spark.source.BaseReader.inputFiles(BaseReader.java:186)
           at org.apache.iceberg.spark.source.BaseReader.getInputFile(BaseReader.java:180)
           at org.apache.iceberg.spark.source.BatchDataReader.open(BatchDataReader.java:108)
           at org.apache.iceberg.spark.source.BatchDataReader.open(BatchDataReader.java:43)
           at org.apache.iceberg.spark.source.BaseReader.next(BaseReader.java:144)
           at org.apache.spark.sql.execution.datasources.v2.PartitionIterator.hasNext(DataSourceRDD.scala:148)
           at org.apache.spark.sql.execution.datasources.v2.MetricsIterator.hasNext(DataSourceRDD.scala:186)
           at org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.$anonfun$hasNext$1(DataSourceRDD.scala:72)
           at org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.$anonfun$hasNext$1$adapted(DataSourceRDD.scala:72)
           at scala.Option.exists(Option.scala:406)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Fix reading encrypted Deletion Vectors [iceberg]

Posted by "ggershinsky (via GitHub)" <gi...@apache.org>.
ggershinsky commented on PR #16158:
URL: https://github.com/apache/iceberg/pull/16158#issuecomment-4438905838

   Actually, there might be a simpler solution. Please ignore my previous comments. I'll experiment with another approach.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Fix reading encrypted Deletion Vectors [iceberg]

Posted by "ggershinsky (via GitHub)" <gi...@apache.org>.
ggershinsky commented on code in PR #16158:
URL: https://github.com/apache/iceberg/pull/16158#discussion_r3241161581


##########
spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/sql/TestTableEncryption.java:
##########
@@ -212,16 +212,22 @@ public void testInsertAndDelete() {
         expected,
         sql("SELECT * FROM %s ORDER BY id", tableName));
 
-    sql("DELETE FROM %s WHERE id < 4", tableName);
+    sql("DELETE FROM %s WHERE id < 5", tableName);
 
-    expected = ImmutableList.of(row(4L, "d", 4.0F), row(5L, "e", 5.0F), row(6L, "f", Float.NaN));
+    expected = ImmutableList.of(row(5L, "e", 5.0F), row(6L, "f", Float.NaN));
 
     assertEquals(
         "Should return all expected rows",
         expected,
         sql("SELECT * FROM %s ORDER BY id", tableName));
   }
 
+  @TestTemplate
+  public void testInsertAndDeleteMOR() {
+    sql("ALTER TABLE %s SET TBLPROPERTIES ('write.delete.mode'='merge-on-read')", tableName);
+    testInsertAndDelete();

Review Comment:
   Thanks, now it throws the exception. I'll have a look.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Fix reading encrypted Deletion Vectors [iceberg]

Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on code in PR #16158:
URL: https://github.com/apache/iceberg/pull/16158#discussion_r3240135195


##########
spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/sql/TestTableEncryption.java:
##########
@@ -212,16 +212,22 @@ public void testInsertAndDelete() {
         expected,
         sql("SELECT * FROM %s ORDER BY id", tableName));
 
-    sql("DELETE FROM %s WHERE id < 4", tableName);
+    sql("DELETE FROM %s WHERE id < 5", tableName);
 
-    expected = ImmutableList.of(row(4L, "d", 4.0F), row(5L, "e", 5.0F), row(6L, "f", Float.NaN));
+    expected = ImmutableList.of(row(5L, "e", 5.0F), row(6L, "f", Float.NaN));
 
     assertEquals(
         "Should return all expected rows",
         expected,
         sql("SELECT * FROM %s ORDER BY id", tableName));
   }
 
+  @TestTemplate
+  public void testInsertAndDeleteMOR() {
+    sql("ALTER TABLE %s SET TBLPROPERTIES ('write.delete.mode'='merge-on-read')", tableName);
+    testInsertAndDelete();

Review Comment:
   Some Spark thing, it fallbacks to MOR, if there is a Metadata delete, not sure if it is bug in Spark or behaviour, I have changed the test.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org