You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "ayushtkn (via GitHub)" <gi...@apache.org> on 2026/04/29 17:40:19 UTC
[PR] Fix reading encrypted Deletion Vectors [iceberg]
ayushtkn opened a new pull request, #16158:
URL: https://github.com/apache/iceberg/pull/16158
Fixes: https://github.com/apache/iceberg/issues/16157
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Fix reading encrypted Deletion Vectors [iceberg]
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on code in PR #16158:
URL: https://github.com/apache/iceberg/pull/16158#discussion_r3254152123
##########
spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/sql/TestTableEncryption.java:
##########
@@ -212,16 +212,27 @@ public void testInsertAndDelete() {
expected,
sql("SELECT * FROM %s ORDER BY id", tableName));
- sql("DELETE FROM %s WHERE id < 4", tableName);
+ sql("DELETE FROM %s WHERE id < 3", tableName);
- expected = ImmutableList.of(row(4L, "d", 4.0F), row(5L, "e", 5.0F), row(6L, "f", Float.NaN));
+ expected =
+ ImmutableList.of(
+ row(3L, "c", Float.NaN),
+ row(4L, "d", 4.0F),
+ row(5L, "e", 5.0F),
+ row(6L, "f", Float.NaN));
assertEquals(
"Should return all expected rows",
expected,
sql("SELECT * FROM %s ORDER BY id", tableName));
}
+ @TestTemplate
+ public void testInsertAndDeleteMOR() {
+ sql("ALTER TABLE %s SET TBLPROPERTIES ('write.delete.mode'='merge-on-read')", tableName);
+ testInsertAndDelete();
Review Comment:
Thanx @singhpk234 for the review!!!
I added the assertion for DVs, Regarding the v2 tables, Encryption isn't supported for tables below V3. It was disabled via: https://github.com/apache/iceberg/commit/19b4bd024486d9d516d0e547e273419c1bc7074e
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Fix reading encrypted Deletion Vectors [iceberg]
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on code in PR #16158:
URL: https://github.com/apache/iceberg/pull/16158#discussion_r3254150432
##########
spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/sql/TestTableEncryption.java:
##########
@@ -212,16 +212,27 @@ public void testInsertAndDelete() {
expected,
sql("SELECT * FROM %s ORDER BY id", tableName));
- sql("DELETE FROM %s WHERE id < 4", tableName);
+ sql("DELETE FROM %s WHERE id < 3", tableName);
- expected = ImmutableList.of(row(4L, "d", 4.0F), row(5L, "e", 5.0F), row(6L, "f", Float.NaN));
+ expected =
+ ImmutableList.of(
+ row(3L, "c", Float.NaN),
+ row(4L, "d", 4.0F),
+ row(5L, "e", 5.0F),
+ row(6L, "f", Float.NaN));
assertEquals(
"Should return all expected rows",
expected,
sql("SELECT * FROM %s ORDER BY id", tableName));
}
+ @TestTemplate
+ public void testInsertAndDeleteMOR() {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix reading encrypted Deletion Vectors [iceberg]
Posted by "ggershinsky (via GitHub)" <gi...@apache.org>.
ggershinsky commented on code in PR #16158:
URL: https://github.com/apache/iceberg/pull/16158#discussion_r3232497175
##########
core/src/main/java/org/apache/iceberg/DVUtil.java:
##########
@@ -189,7 +190,9 @@ private static List<DeleteFile> writeDVs(
String dvOutputLocation,
Map<String, Pair<PartitionSpec, StructLike>> partitions) {
OutputFile dvOutputFile = fileIO.newOutputFile(dvOutputLocation);
- try (DVFileWriter dvFileWriter = new BaseDVFileWriter(() -> dvOutputFile, path -> null)) {
+ try (DVFileWriter dvFileWriter =
+ new BaseDVFileWriter(
+ () -> EncryptionUtil.plainAsEncryptedOutput(dvOutputFile), path -> null)) {
Review Comment:
this bypasses encryption
##########
core/src/main/java/org/apache/iceberg/DVUtil.java:
##########
@@ -189,7 +190,9 @@ private static List<DeleteFile> writeDVs(
String dvOutputLocation,
Map<String, Pair<PartitionSpec, StructLike>> partitions) {
OutputFile dvOutputFile = fileIO.newOutputFile(dvOutputLocation);
Review Comment:
if `fileIO` instance of `EncryptingFileIO`, then EncryptingOutputFile dvEncrOutputFile = ((EncryptingFileIO)fileIO).newEncryptingOutputFile(dvOutputLocation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix reading encrypted Deletion Vectors [iceberg]
Posted by "ggershinsky (via GitHub)" <gi...@apache.org>.
ggershinsky commented on code in PR #16158:
URL: https://github.com/apache/iceberg/pull/16158#discussion_r3232514461
##########
core/src/main/java/org/apache/iceberg/DVUtil.java:
##########
@@ -189,7 +190,9 @@ private static List<DeleteFile> writeDVs(
String dvOutputLocation,
Map<String, Pair<PartitionSpec, StructLike>> partitions) {
OutputFile dvOutputFile = fileIO.newOutputFile(dvOutputLocation);
Review Comment:
if `fileIO` instance of `EncryptingFileIO`, then EncryptingOutputFile dvEncrOutputFile = ((EncryptingFileIO)fileIO).newEncryptingOutputFile(dvOutputLocation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Fix reading encrypted Deletion Vectors [iceberg]
Posted by "ggershinsky (via GitHub)" <gi...@apache.org>.
ggershinsky commented on PR #16158:
URL: https://github.com/apache/iceberg/pull/16158#issuecomment-4458034990
After experimenting with the code, I think this patch is the best solution. Thanks @ayushtkn .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Fix reading encrypted Deletion Vectors [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #16158:
URL: https://github.com/apache/iceberg/pull/16158#discussion_r3267559847
##########
core/src/main/java/org/apache/iceberg/deletes/BaseDVFileWriter.java:
##########
@@ -52,19 +55,19 @@ public class BaseDVFileWriter implements DVFileWriter {
private static final String REFERENCED_DATA_FILE_KEY = "referenced-data-file";
private static final String CARDINALITY_KEY = "cardinality";
- private final Supplier<OutputFile> dvOutputFile;
+ private final Supplier<EncryptedOutputFile> dvOutputFile;
private final Function<String, PositionDeleteIndex> loadPreviousDeletes;
private final Map<String, Deletes> deletesByPath = Maps.newHashMap();
private final Map<String, BlobMetadata> blobsByPath = Maps.newHashMap();
private DeleteWriteResult result = null;
public BaseDVFileWriter(
OutputFileFactory fileFactory, Function<String, PositionDeleteIndex> loadPreviousDeletes) {
- this(() -> fileFactory.newOutputFile().encryptingOutputFile(), loadPreviousDeletes);
+ this(fileFactory::newOutputFile, loadPreviousDeletes);
}
public BaseDVFileWriter(
- Supplier<OutputFile> dvOutputFile,
+ Supplier<EncryptedOutputFile> dvOutputFile,
Review Comment:
I'm a bit surprised revAPI didn't catch this since it's a breaking change unfortunately
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix reading encrypted Deletion Vectors [iceberg]
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn closed pull request #16158: Fix reading encrypted Deletion Vectors
URL: https://github.com/apache/iceberg/pull/16158
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix reading encrypted Deletion Vectors [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #16158:
URL: https://github.com/apache/iceberg/pull/16158#issuecomment-4397091353
cc: @ggershinsky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix reading encrypted Deletion Vectors [iceberg]
Posted by "ggershinsky (via GitHub)" <gi...@apache.org>.
ggershinsky commented on code in PR #16158:
URL: https://github.com/apache/iceberg/pull/16158#discussion_r3232497175
##########
core/src/main/java/org/apache/iceberg/DVUtil.java:
##########
@@ -189,7 +190,9 @@ private static List<DeleteFile> writeDVs(
String dvOutputLocation,
Map<String, Pair<PartitionSpec, StructLike>> partitions) {
OutputFile dvOutputFile = fileIO.newOutputFile(dvOutputLocation);
- try (DVFileWriter dvFileWriter = new BaseDVFileWriter(() -> dvOutputFile, path -> null)) {
+ try (DVFileWriter dvFileWriter =
+ new BaseDVFileWriter(
+ () -> EncryptionUtil.plainAsEncryptedOutput(dvOutputFile), path -> null)) {
Review Comment:
this bypasses encryption
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Fix reading encrypted Deletion Vectors [iceberg]
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn closed pull request #16158: Core: Fix reading encrypted Deletion Vectors
URL: https://github.com/apache/iceberg/pull/16158
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Fix reading encrypted Deletion Vectors [iceberg]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #16158:
URL: https://github.com/apache/iceberg/pull/16158#discussion_r3254108123
##########
spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/sql/TestTableEncryption.java:
##########
@@ -212,16 +212,27 @@ public void testInsertAndDelete() {
expected,
sql("SELECT * FROM %s ORDER BY id", tableName));
- sql("DELETE FROM %s WHERE id < 4", tableName);
+ sql("DELETE FROM %s WHERE id < 3", tableName);
- expected = ImmutableList.of(row(4L, "d", 4.0F), row(5L, "e", 5.0F), row(6L, "f", Float.NaN));
+ expected =
+ ImmutableList.of(
+ row(3L, "c", Float.NaN),
+ row(4L, "d", 4.0F),
+ row(5L, "e", 5.0F),
+ row(6L, "f", Float.NaN));
assertEquals(
"Should return all expected rows",
expected,
sql("SELECT * FROM %s ORDER BY id", tableName));
}
+ @TestTemplate
+ public void testInsertAndDeleteMOR() {
+ sql("ALTER TABLE %s SET TBLPROPERTIES ('write.delete.mode'='merge-on-read')", tableName);
+ testInsertAndDelete();
Review Comment:
i would also additional assertions that dvs were there ? by loading the table using the catalog obj and checking its recent snapshot summary
I understand test is gonna trigger this implicitly, though i would suggest being explicit would be ideal as we are strictly checking dvs here.
Also what happens in v2 tables pos deletes and Eq delete, do you know ?
##########
core/src/main/java/org/apache/iceberg/DVUtil.java:
##########
@@ -188,8 +190,16 @@ private static List<DeleteFile> writeDVs(
FileIO fileIO,
String dvOutputLocation,
Map<String, Pair<PartitionSpec, StructLike>> partitions) {
- OutputFile dvOutputFile = fileIO.newOutputFile(dvOutputLocation);
- try (DVFileWriter dvFileWriter = new BaseDVFileWriter(() -> dvOutputFile, path -> null)) {
+ EncryptedOutputFile dvEncryptedOutputFile;
+ if (fileIO instanceof EncryptingFileIO) {
+ dvEncryptedOutputFile = ((EncryptingFileIO) fileIO).newEncryptingOutputFile(dvOutputLocation);
+ } else {
+ dvEncryptedOutputFile =
+ EncryptionUtil.plainAsEncryptedOutput(fileIO.newOutputFile(dvOutputLocation));
+ }
Review Comment:
```suggestion
if (fileIO instanceof EncryptingFileIO encryptingFileIO) {
dvEncryptedOutputFile = encryptingFileIO.newEncryptingOutputFile(dvOutputLocation);
} else {
dvEncryptedOutputFile =
EncryptionUtil.plainAsEncryptedOutput(fileIO.newOutputFile(dvOutputLocation));
```
##########
spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/sql/TestTableEncryption.java:
##########
@@ -212,16 +212,27 @@ public void testInsertAndDelete() {
expected,
sql("SELECT * FROM %s ORDER BY id", tableName));
- sql("DELETE FROM %s WHERE id < 4", tableName);
+ sql("DELETE FROM %s WHERE id < 3", tableName);
- expected = ImmutableList.of(row(4L, "d", 4.0F), row(5L, "e", 5.0F), row(6L, "f", Float.NaN));
+ expected =
+ ImmutableList.of(
+ row(3L, "c", Float.NaN),
+ row(4L, "d", 4.0F),
+ row(5L, "e", 5.0F),
+ row(6L, "f", Float.NaN));
assertEquals(
"Should return all expected rows",
expected,
sql("SELECT * FROM %s ORDER BY id", tableName));
}
+ @TestTemplate
+ public void testInsertAndDeleteMOR() {
Review Comment:
lets name the other test COW then ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Fix reading encrypted Deletion Vectors [iceberg]
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on PR #16158:
URL: https://github.com/apache/iceberg/pull/16158#issuecomment-4489588415
thanx @amogh-jahagirdar, I will close mine. Thanx everyone for the reviews here!!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Fix reading encrypted Deletion Vectors [iceberg]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on PR #16158:
URL: https://github.com/apache/iceberg/pull/16158#issuecomment-4494876812
Thank you @amogh-jahagirdar I missed the other PR :) ! Its really nice to see this being worked upon
Thank you still @ayushtkn for your contribution and time, Thank you @ggershinsky for your review and time, please do review the other PR when you get some time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix reading encrypted Deletion Vectors [iceberg]
Posted by "ggershinsky (via GitHub)" <gi...@apache.org>.
ggershinsky commented on code in PR #16158:
URL: https://github.com/apache/iceberg/pull/16158#discussion_r3239652971
##########
spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/sql/TestTableEncryption.java:
##########
@@ -212,16 +212,22 @@ public void testInsertAndDelete() {
expected,
sql("SELECT * FROM %s ORDER BY id", tableName));
- sql("DELETE FROM %s WHERE id < 4", tableName);
+ sql("DELETE FROM %s WHERE id < 5", tableName);
- expected = ImmutableList.of(row(4L, "d", 4.0F), row(5L, "e", 5.0F), row(6L, "f", Float.NaN));
+ expected = ImmutableList.of(row(5L, "e", 5.0F), row(6L, "f", Float.NaN));
assertEquals(
"Should return all expected rows",
expected,
sql("SELECT * FROM %s ORDER BY id", tableName));
}
+ @TestTemplate
+ public void testInsertAndDeleteMOR() {
+ sql("ALTER TABLE %s SET TBLPROPERTIES ('write.delete.mode'='merge-on-read')", tableName);
+ testInsertAndDelete();
Review Comment:
The exception is not reproduced in my environment. I ran `git clone git@github.com:apache/iceberg.git`, changed the 8 lines in this class, and ran `./gradlew :iceberg-spark:iceberg-spark-4.1:test --tests org.apache.iceberg.spark.sql.TestTableEncryption`. The test passes ok. Am I doing anything wrong?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Fix reading encrypted Deletion Vectors [iceberg]
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on code in PR #16158:
URL: https://github.com/apache/iceberg/pull/16158#discussion_r3240135195
##########
spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/sql/TestTableEncryption.java:
##########
@@ -212,16 +212,22 @@ public void testInsertAndDelete() {
expected,
sql("SELECT * FROM %s ORDER BY id", tableName));
- sql("DELETE FROM %s WHERE id < 4", tableName);
+ sql("DELETE FROM %s WHERE id < 5", tableName);
- expected = ImmutableList.of(row(4L, "d", 4.0F), row(5L, "e", 5.0F), row(6L, "f", Float.NaN));
+ expected = ImmutableList.of(row(5L, "e", 5.0F), row(6L, "f", Float.NaN));
assertEquals(
"Should return all expected rows",
expected,
sql("SELECT * FROM %s ORDER BY id", tableName));
}
+ @TestTemplate
+ public void testInsertAndDeleteMOR() {
+ sql("ALTER TABLE %s SET TBLPROPERTIES ('write.delete.mode'='merge-on-read')", tableName);
+ testInsertAndDelete();
Review Comment:
Some Spark thing, it fallbacks to COW, if there is a Metadata delete, not sure if it is bug in Spark or behaviour, I have changed the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Fix reading encrypted Deletion Vectors [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on PR #16158:
URL: https://github.com/apache/iceberg/pull/16158#issuecomment-4489402236
Hey FYI @ayushtkn @singhpk234 @ggershinsky I think there [was a bit older PR ](https://github.com/apache/iceberg/pull/15911)from @manuzhang that's now actively working on trying to address this issue, are there any objections to working off of that one? Sorry, looks like multiple people were working on this issue in parallel and I just came to know about this one too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix reading encrypted Deletion Vectors [iceberg]
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on PR #16158:
URL: https://github.com/apache/iceberg/pull/16158#issuecomment-4396661364
@pvary can u help review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix reading encrypted Deletion Vectors [iceberg]
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on code in PR #16158:
URL: https://github.com/apache/iceberg/pull/16158#discussion_r3240141803
##########
spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/sql/TestTableEncryption.java:
##########
@@ -212,16 +212,22 @@ public void testInsertAndDelete() {
expected,
sql("SELECT * FROM %s ORDER BY id", tableName));
- sql("DELETE FROM %s WHERE id < 4", tableName);
+ sql("DELETE FROM %s WHERE id < 5", tableName);
- expected = ImmutableList.of(row(4L, "d", 4.0F), row(5L, "e", 5.0F), row(6L, "f", Float.NaN));
+ expected = ImmutableList.of(row(5L, "e", 5.0F), row(6L, "f", Float.NaN));
assertEquals(
"Should return all expected rows",
expected,
sql("SELECT * FROM %s ORDER BY id", tableName));
}
+ @TestTemplate
+ public void testInsertAndDeleteMOR() {
+ sql("ALTER TABLE %s SET TBLPROPERTIES ('write.delete.mode'='merge-on-read')", tableName);
+ testInsertAndDelete();
Review Comment:
Can you check, I did this:
```
ayushsaxena@Q3NW54Y0C5 iceberg % git reset HEAD~1
Unstaged changes after reset:
M core/src/main/java/org/apache/iceberg/DVUtil.java
M core/src/main/java/org/apache/iceberg/deletes/BaseDVFileWriter.java
M spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/sql/TestTableEncryption.java
M spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/sql/TestTableEncryption.java
ayushsaxena@Q3NW54Y0C5 iceberg % git checkout core/src/main/java/org/apache/iceberg/DVUtil.java core/src/main/java/org/apache/iceberg/deletes/BaseDVFileWriter.java
Updated 2 paths from the index
ayushsaxena@Q3NW54Y0C5 iceberg % ./gradlew clean :iceberg-spark:iceberg-spark-4.1_2.13:test --tests "org.apache.iceberg.spark.sql.TestTableEncryption.testInsertAndDeleteMOR"
Configuration on demand is an incubating feature.
> Task :iceberg-api:testJar
file '/Users/ayushsaxena/code/iceberg/build/iceberg-build.properties' will be copied to 'iceberg-build.properties', overwriting file '/Users/ayushsaxena/code/iceberg/api/build/resources/test/iceberg-build.properties', which has already been copied there.
> Task :iceberg-spark:iceberg-spark-4.1_2.13:compileTestJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
> Task :iceberg-spark:iceberg-spark-4.1_2.13:test
TestTableEncryption > testInsertAndDeleteMOR() > catalogName = testhive, implementation = org.apache.iceberg.spark.SparkCatalog, config = {type=hive, default-namespace=default, encryption.kms-impl=org.apache.iceberg.encryption.UnitestKMS} FAILED
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 9.0 failed 1 times, most recent failure: Lost task 0.0 in stage 9.0 (TID 9) (192.168.29.167 executor driver): java.lang.IllegalStateException: Null key metadata buffer
at org.apache.iceberg.encryption.StandardKeyMetadata.castOrParse(StandardKeyMetadata.java:111)
at org.apache.iceberg.encryption.StandardEncryptionManager$StandardDecryptedInputFile.keyMetadata(StandardEncryptionManager.java:302)
at org.apache.iceberg.encryption.StandardEncryptionManager$StandardDecryptedInputFile.decrypted(StandardEncryptionManager.java:313)
at org.apache.iceberg.encryption.StandardEncryptionManager$StandardDecryptedInputFile.location(StandardEncryptionManager.java:333)
at org.apache.iceberg.encryption.EncryptingFileIO.bulkDecrypt(EncryptingFileIO.java:71)
at org.apache.iceberg.spark.source.BaseReader.inputFiles(BaseReader.java:186)
at org.apache.iceberg.spark.source.BaseReader.getInputFile(BaseReader.java:180)
at org.apache.iceberg.spark.source.BatchDataReader.open(BatchDataReader.java:108)
at org.apache.iceberg.spark.source.BatchDataReader.open(BatchDataReader.java:43)
at org.apache.iceberg.spark.source.BaseReader.next(BaseReader.java:144)
at org.apache.spark.sql.execution.datasources.v2.PartitionIterator.hasNext(DataSourceRDD.scala:148)
at org.apache.spark.sql.execution.datasources.v2.MetricsIterator.hasNext(DataSourceRDD.scala:186)
at org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.$anonfun$hasNext$1(DataSourceRDD.scala:72)
at org.apache.spark.sql.execution.datasources.v2.DataSourceRDD$$anon$1.$anonfun$hasNext$1$adapted(DataSourceRDD.scala:72)
at scala.Option.exists(Option.scala:406)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix reading encrypted Deletion Vectors [iceberg]
Posted by "ggershinsky (via GitHub)" <gi...@apache.org>.
ggershinsky commented on PR #16158:
URL: https://github.com/apache/iceberg/pull/16158#issuecomment-4438905838
Actually, there might be a simpler solution. Please ignore my previous comments. I'll experiment with another approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Fix reading encrypted Deletion Vectors [iceberg]
Posted by "ggershinsky (via GitHub)" <gi...@apache.org>.
ggershinsky commented on code in PR #16158:
URL: https://github.com/apache/iceberg/pull/16158#discussion_r3241161581
##########
spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/sql/TestTableEncryption.java:
##########
@@ -212,16 +212,22 @@ public void testInsertAndDelete() {
expected,
sql("SELECT * FROM %s ORDER BY id", tableName));
- sql("DELETE FROM %s WHERE id < 4", tableName);
+ sql("DELETE FROM %s WHERE id < 5", tableName);
- expected = ImmutableList.of(row(4L, "d", 4.0F), row(5L, "e", 5.0F), row(6L, "f", Float.NaN));
+ expected = ImmutableList.of(row(5L, "e", 5.0F), row(6L, "f", Float.NaN));
assertEquals(
"Should return all expected rows",
expected,
sql("SELECT * FROM %s ORDER BY id", tableName));
}
+ @TestTemplate
+ public void testInsertAndDeleteMOR() {
+ sql("ALTER TABLE %s SET TBLPROPERTIES ('write.delete.mode'='merge-on-read')", tableName);
+ testInsertAndDelete();
Review Comment:
Thanks, now it throws the exception. I'll have a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix reading encrypted Deletion Vectors [iceberg]
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on code in PR #16158:
URL: https://github.com/apache/iceberg/pull/16158#discussion_r3240135195
##########
spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/sql/TestTableEncryption.java:
##########
@@ -212,16 +212,22 @@ public void testInsertAndDelete() {
expected,
sql("SELECT * FROM %s ORDER BY id", tableName));
- sql("DELETE FROM %s WHERE id < 4", tableName);
+ sql("DELETE FROM %s WHERE id < 5", tableName);
- expected = ImmutableList.of(row(4L, "d", 4.0F), row(5L, "e", 5.0F), row(6L, "f", Float.NaN));
+ expected = ImmutableList.of(row(5L, "e", 5.0F), row(6L, "f", Float.NaN));
assertEquals(
"Should return all expected rows",
expected,
sql("SELECT * FROM %s ORDER BY id", tableName));
}
+ @TestTemplate
+ public void testInsertAndDeleteMOR() {
+ sql("ALTER TABLE %s SET TBLPROPERTIES ('write.delete.mode'='merge-on-read')", tableName);
+ testInsertAndDelete();
Review Comment:
Some Spark thing, it fallbacks to MOR, if there is a Metadata delete, not sure if it is bug in Spark or behaviour, I have changed the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org