You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "aokolnychyi (via GitHub)" <gi...@apache.org> on 2023/11/07 23:54:06 UTC
[PR] Core: Support replacing delete manifests [iceberg]
aokolnychyi opened a new pull request, #9000:
URL: https://github.com/apache/iceberg/pull/9000
This PR enables delete manifest replacement, basis for delete manifest compaction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support replacing delete manifests [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #9000:
URL: https://github.com/apache/iceberg/pull/9000#discussion_r1388316731
##########
core/src/test/java/org/apache/iceberg/TestRewriteManifests.java:
##########
@@ -1105,6 +1108,530 @@ public void testRewriteManifestsOnBranchUnsupported() {
"Cannot commit to branch someBranch: org.apache.iceberg.BaseRewriteManifests does not support branch commits");
}
+ @Test
+ public void testRewriteDataManifestsPreservesDeletes() {
+ Assumptions.assumeThat(formatVersion).isGreaterThan(1);
+
+ Table table = load();
+
+ table.newAppend().appendFile(FILE_A).appendFile(FILE_B).commit();
Review Comment:
From here to 1131 could probably be refactored out into
Into something like "prepareTable(DataFiles, DeleteFiles)"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support replacing delete manifests [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #9000:
URL: https://github.com/apache/iceberg/pull/9000#discussion_r1388322693
##########
core/src/test/java/org/apache/iceberg/TestRewriteManifests.java:
##########
@@ -1105,6 +1108,530 @@ public void testRewriteManifestsOnBranchUnsupported() {
"Cannot commit to branch someBranch: org.apache.iceberg.BaseRewriteManifests does not support branch commits");
}
+ @Test
+ public void testRewriteDataManifestsPreservesDeletes() {
+ Assumptions.assumeThat(formatVersion).isGreaterThan(1);
+
+ Table table = load();
+
+ table.newAppend().appendFile(FILE_A).appendFile(FILE_B).commit();
Review Comment:
One challenge is sometimes we want 2 data commits, sometimes we want 2 delete commits, sometimes one, etc. Also, we capture the snapshot of each operation. What would be the return value of that method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support replacing delete manifests [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #9000:
URL: https://github.com/apache/iceberg/pull/9000#discussion_r1387199271
##########
core/src/test/java/org/apache/iceberg/TestRewriteManifests.java:
##########
@@ -1105,6 +1108,499 @@ public void testRewriteManifestsOnBranchUnsupported() {
"Cannot commit to branch someBranch: org.apache.iceberg.BaseRewriteManifests does not support branch commits");
}
+ @Test
+ public void testRewriteDataManifestsPreservesDeletes() {
+ Assumptions.assumeThat(formatVersion).isGreaterThan(1);
+
+ Table table = load();
+
+ table.newAppend().appendFile(FILE_A).appendFile(FILE_B).commit();
+
+ Snapshot appendSnapshot = table.currentSnapshot();
+ Assertions.assertThat(appendSnapshot.dataManifests(table.io())).hasSize(1);
+ Assertions.assertThat(appendSnapshot.deleteManifests(table.io())).isEmpty();
+
+ table.newRowDelta().addDeletes(FILE_A_DELETES).addDeletes(FILE_A2_DELETES).commit();
+
+ Snapshot deleteSnapshot = table.currentSnapshot();
+ Assertions.assertThat(deleteSnapshot.dataManifests(table.io())).hasSize(1);
+ Assertions.assertThat(deleteSnapshot.deleteManifests(table.io())).hasSize(1);
+
+ table.rewriteManifests().clusterBy(file -> file.path().toString()).commit();
+
+ Snapshot rewriteSnapshot = table.currentSnapshot();
+
+ validateSummary(rewriteSnapshot, 1, 1, 2, 2);
+
+ List<ManifestFile> dataManifests = sortedDataManifests(table.io(), rewriteSnapshot);
+ Assertions.assertThat(dataManifests).hasSize(2);
+ validateManifest(
+ dataManifests.get(0),
+ dataSeqs(appendSnapshot.sequenceNumber(), appendSnapshot.sequenceNumber()),
Review Comment:
Nit: Could put sequence number and snapshot ids in some local vars?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support replacing delete manifests [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #9000:
URL: https://github.com/apache/iceberg/pull/9000#discussion_r1387200273
##########
core/src/test/java/org/apache/iceberg/TestRewriteManifests.java:
##########
@@ -1105,6 +1108,499 @@ public void testRewriteManifestsOnBranchUnsupported() {
"Cannot commit to branch someBranch: org.apache.iceberg.BaseRewriteManifests does not support branch commits");
}
+ @Test
+ public void testRewriteDataManifestsPreservesDeletes() {
+ Assumptions.assumeThat(formatVersion).isGreaterThan(1);
+
+ Table table = load();
+
+ table.newAppend().appendFile(FILE_A).appendFile(FILE_B).commit();
+
+ Snapshot appendSnapshot = table.currentSnapshot();
+ Assertions.assertThat(appendSnapshot.dataManifests(table.io())).hasSize(1);
+ Assertions.assertThat(appendSnapshot.deleteManifests(table.io())).isEmpty();
+
+ table.newRowDelta().addDeletes(FILE_A_DELETES).addDeletes(FILE_A2_DELETES).commit();
+
+ Snapshot deleteSnapshot = table.currentSnapshot();
+ Assertions.assertThat(deleteSnapshot.dataManifests(table.io())).hasSize(1);
+ Assertions.assertThat(deleteSnapshot.deleteManifests(table.io())).hasSize(1);
+
+ table.rewriteManifests().clusterBy(file -> file.path().toString()).commit();
+
+ Snapshot rewriteSnapshot = table.currentSnapshot();
+
+ validateSummary(rewriteSnapshot, 1, 1, 2, 2);
+
+ List<ManifestFile> dataManifests = sortedDataManifests(table.io(), rewriteSnapshot);
+ Assertions.assertThat(dataManifests).hasSize(2);
+ validateManifest(
+ dataManifests.get(0),
+ dataSeqs(appendSnapshot.sequenceNumber(), appendSnapshot.sequenceNumber()),
Review Comment:
Will do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support replacing delete manifests [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi merged PR #9000:
URL: https://github.com/apache/iceberg/pull/9000
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support replacing delete manifests [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi closed pull request #9000: Core: Support replacing delete manifests
URL: https://github.com/apache/iceberg/pull/9000
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support replacing delete manifests [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #9000:
URL: https://github.com/apache/iceberg/pull/9000#discussion_r1385766686
##########
core/src/main/java/org/apache/iceberg/BaseRewriteManifests.java:
##########
@@ -242,7 +241,7 @@ private void performRewrite(List<ManifestFile> currentManifests) {
.executeWith(workerPool())
.run(
manifest -> {
- if (predicate != null && !predicate.test(manifest)) {
+ if (containsDeletes(manifest) || !matchesPredicate(manifest)) {
Review Comment:
Our `BaseRewriteManifests` can rewrite data manifests as well as simply replace manifests rewritten by an external process. Adding support for rewriting delete manifests is out of scope. The primary and much more scalable way of rewriting manifests is via an action. This PR focuses purely on the ability to replace delete manifests rewritten externally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support replacing delete manifests [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #9000:
URL: https://github.com/apache/iceberg/pull/9000#discussion_r1388316731
##########
core/src/test/java/org/apache/iceberg/TestRewriteManifests.java:
##########
@@ -1105,6 +1108,530 @@ public void testRewriteManifestsOnBranchUnsupported() {
"Cannot commit to branch someBranch: org.apache.iceberg.BaseRewriteManifests does not support branch commits");
}
+ @Test
+ public void testRewriteDataManifestsPreservesDeletes() {
+ Assumptions.assumeThat(formatVersion).isGreaterThan(1);
+
+ Table table = load();
+
+ table.newAppend().appendFile(FILE_A).appendFile(FILE_B).commit();
Review Comment:
From here to 1131 could probably be refactored out into
Into something like "prepareTable(DataFiles, DeleteFiles)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support replacing delete manifests [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #9000:
URL: https://github.com/apache/iceberg/pull/9000#discussion_r1387199381
##########
core/src/test/java/org/apache/iceberg/TestRewriteManifests.java:
##########
@@ -1105,6 +1108,499 @@ public void testRewriteManifestsOnBranchUnsupported() {
"Cannot commit to branch someBranch: org.apache.iceberg.BaseRewriteManifests does not support branch commits");
}
+ @Test
+ public void testRewriteDataManifestsPreservesDeletes() {
+ Assumptions.assumeThat(formatVersion).isGreaterThan(1);
+
+ Table table = load();
+
+ table.newAppend().appendFile(FILE_A).appendFile(FILE_B).commit();
+
+ Snapshot appendSnapshot = table.currentSnapshot();
+ Assertions.assertThat(appendSnapshot.dataManifests(table.io())).hasSize(1);
+ Assertions.assertThat(appendSnapshot.deleteManifests(table.io())).isEmpty();
+
+ table.newRowDelta().addDeletes(FILE_A_DELETES).addDeletes(FILE_A2_DELETES).commit();
+
+ Snapshot deleteSnapshot = table.currentSnapshot();
+ Assertions.assertThat(deleteSnapshot.dataManifests(table.io())).hasSize(1);
+ Assertions.assertThat(deleteSnapshot.deleteManifests(table.io())).hasSize(1);
+
+ table.rewriteManifests().clusterBy(file -> file.path().toString()).commit();
Review Comment:
It is `ContentFile$path` that returns `CharSequence`. The idea of this closure is to split manifest entries based on the content file path. This ensures the original manifest with 2 entires is split into 2 new manifests. It is not the path of the manifest.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support replacing delete manifests [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #9000:
URL: https://github.com/apache/iceberg/pull/9000#discussion_r1388159792
##########
core/src/test/java/org/apache/iceberg/TestRewriteManifests.java:
##########
@@ -1105,6 +1108,499 @@ public void testRewriteManifestsOnBranchUnsupported() {
"Cannot commit to branch someBranch: org.apache.iceberg.BaseRewriteManifests does not support branch commits");
}
+ @Test
+ public void testRewriteDataManifestsPreservesDeletes() {
+ Assumptions.assumeThat(formatVersion).isGreaterThan(1);
+
+ Table table = load();
+
+ table.newAppend().appendFile(FILE_A).appendFile(FILE_B).commit();
+
+ Snapshot appendSnapshot = table.currentSnapshot();
+ Assertions.assertThat(appendSnapshot.dataManifests(table.io())).hasSize(1);
+ Assertions.assertThat(appendSnapshot.deleteManifests(table.io())).isEmpty();
+
+ table.newRowDelta().addDeletes(FILE_A_DELETES).addDeletes(FILE_A2_DELETES).commit();
+
+ Snapshot deleteSnapshot = table.currentSnapshot();
+ Assertions.assertThat(deleteSnapshot.dataManifests(table.io())).hasSize(1);
+ Assertions.assertThat(deleteSnapshot.deleteManifests(table.io())).hasSize(1);
+
+ table.rewriteManifests().clusterBy(file -> file.path().toString()).commit();
Review Comment:
`CharSequence` has no defined `equals` and `hashCode`. You are right it would work without `toString` because `BaseFile` uses `String` internally but it is generally not safe to use `CharSequence` as a key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support replacing delete manifests [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #9000:
URL: https://github.com/apache/iceberg/pull/9000#discussion_r1388159792
##########
core/src/test/java/org/apache/iceberg/TestRewriteManifests.java:
##########
@@ -1105,6 +1108,499 @@ public void testRewriteManifestsOnBranchUnsupported() {
"Cannot commit to branch someBranch: org.apache.iceberg.BaseRewriteManifests does not support branch commits");
}
+ @Test
+ public void testRewriteDataManifestsPreservesDeletes() {
+ Assumptions.assumeThat(formatVersion).isGreaterThan(1);
+
+ Table table = load();
+
+ table.newAppend().appendFile(FILE_A).appendFile(FILE_B).commit();
+
+ Snapshot appendSnapshot = table.currentSnapshot();
+ Assertions.assertThat(appendSnapshot.dataManifests(table.io())).hasSize(1);
+ Assertions.assertThat(appendSnapshot.deleteManifests(table.io())).isEmpty();
+
+ table.newRowDelta().addDeletes(FILE_A_DELETES).addDeletes(FILE_A2_DELETES).commit();
+
+ Snapshot deleteSnapshot = table.currentSnapshot();
+ Assertions.assertThat(deleteSnapshot.dataManifests(table.io())).hasSize(1);
+ Assertions.assertThat(deleteSnapshot.deleteManifests(table.io())).hasSize(1);
+
+ table.rewriteManifests().clusterBy(file -> file.path().toString()).commit();
Review Comment:
`CharSequence` has no defined `equals` and `hashCode`. You are right it would work without `toString` because `BaseFile` uses `String` internally but it is generally not safe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support replacing delete manifests [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #9000:
URL: https://github.com/apache/iceberg/pull/9000#discussion_r1388322693
##########
core/src/test/java/org/apache/iceberg/TestRewriteManifests.java:
##########
@@ -1105,6 +1108,530 @@ public void testRewriteManifestsOnBranchUnsupported() {
"Cannot commit to branch someBranch: org.apache.iceberg.BaseRewriteManifests does not support branch commits");
}
+ @Test
+ public void testRewriteDataManifestsPreservesDeletes() {
+ Assumptions.assumeThat(formatVersion).isGreaterThan(1);
+
+ Table table = load();
+
+ table.newAppend().appendFile(FILE_A).appendFile(FILE_B).commit();
Review Comment:
One challenge is sometimes we want 2 data commits, sometimes we want 2 delete commits, sometimes one, etc. Also, we capture the snapshot of each operation. What would be the return value?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support replacing delete manifests [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi closed pull request #9000: Core: Support replacing delete manifests
URL: https://github.com/apache/iceberg/pull/9000
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support replacing delete manifests [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #9000:
URL: https://github.com/apache/iceberg/pull/9000#discussion_r1387199381
##########
core/src/test/java/org/apache/iceberg/TestRewriteManifests.java:
##########
@@ -1105,6 +1108,499 @@ public void testRewriteManifestsOnBranchUnsupported() {
"Cannot commit to branch someBranch: org.apache.iceberg.BaseRewriteManifests does not support branch commits");
}
+ @Test
+ public void testRewriteDataManifestsPreservesDeletes() {
+ Assumptions.assumeThat(formatVersion).isGreaterThan(1);
+
+ Table table = load();
+
+ table.newAppend().appendFile(FILE_A).appendFile(FILE_B).commit();
+
+ Snapshot appendSnapshot = table.currentSnapshot();
+ Assertions.assertThat(appendSnapshot.dataManifests(table.io())).hasSize(1);
+ Assertions.assertThat(appendSnapshot.deleteManifests(table.io())).isEmpty();
+
+ table.newRowDelta().addDeletes(FILE_A_DELETES).addDeletes(FILE_A2_DELETES).commit();
+
+ Snapshot deleteSnapshot = table.currentSnapshot();
+ Assertions.assertThat(deleteSnapshot.dataManifests(table.io())).hasSize(1);
+ Assertions.assertThat(deleteSnapshot.deleteManifests(table.io())).hasSize(1);
+
+ table.rewriteManifests().clusterBy(file -> file.path().toString()).commit();
Review Comment:
It is `ContentFile$path` that returns `CharSequence`. The idea of this closure is to split manifest entries based on the content file path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support replacing delete manifests [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #9000:
URL: https://github.com/apache/iceberg/pull/9000#discussion_r1387249948
##########
core/src/test/java/org/apache/iceberg/TestRewriteManifests.java:
##########
@@ -1105,6 +1108,499 @@ public void testRewriteManifestsOnBranchUnsupported() {
"Cannot commit to branch someBranch: org.apache.iceberg.BaseRewriteManifests does not support branch commits");
}
+ @Test
+ public void testRewriteDataManifestsPreservesDeletes() {
+ Assumptions.assumeThat(formatVersion).isGreaterThan(1);
+
+ Table table = load();
+
+ table.newAppend().appendFile(FILE_A).appendFile(FILE_B).commit();
+
+ Snapshot appendSnapshot = table.currentSnapshot();
+ Assertions.assertThat(appendSnapshot.dataManifests(table.io())).hasSize(1);
+ Assertions.assertThat(appendSnapshot.deleteManifests(table.io())).isEmpty();
+
+ table.newRowDelta().addDeletes(FILE_A_DELETES).addDeletes(FILE_A2_DELETES).commit();
+
+ Snapshot deleteSnapshot = table.currentSnapshot();
+ Assertions.assertThat(deleteSnapshot.dataManifests(table.io())).hasSize(1);
+ Assertions.assertThat(deleteSnapshot.deleteManifests(table.io())).hasSize(1);
+
+ table.rewriteManifests().clusterBy(file -> file.path().toString()).commit();
+
+ Snapshot rewriteSnapshot = table.currentSnapshot();
+
+ validateSummary(rewriteSnapshot, 1, 1, 2, 2);
+
+ List<ManifestFile> dataManifests = sortedDataManifests(table.io(), rewriteSnapshot);
+ Assertions.assertThat(dataManifests).hasSize(2);
+ validateManifest(
+ dataManifests.get(0),
+ dataSeqs(appendSnapshot.sequenceNumber(), appendSnapshot.sequenceNumber()),
Review Comment:
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support replacing delete manifests [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #9000:
URL: https://github.com/apache/iceberg/pull/9000#discussion_r1385766686
##########
core/src/main/java/org/apache/iceberg/BaseRewriteManifests.java:
##########
@@ -242,7 +241,7 @@ private void performRewrite(List<ManifestFile> currentManifests) {
.executeWith(workerPool())
.run(
manifest -> {
- if (predicate != null && !predicate.test(manifest)) {
+ if (containsDeletes(manifest) || !matchesPredicate(manifest)) {
Review Comment:
Our `BaseRewriteManifests` can rewrite data manifests as well as simply replace manifests rewritten by an external process. Adding support for directly rewriting delete manifests is out of scope. The primary and much more scalable way of rewriting manifests is via an action. This PR focuses purely on the ability to replace delete manifests rewritten externally that I will consume in the action.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support replacing delete manifests [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #9000:
URL: https://github.com/apache/iceberg/pull/9000#discussion_r1387199381
##########
core/src/test/java/org/apache/iceberg/TestRewriteManifests.java:
##########
@@ -1105,6 +1108,499 @@ public void testRewriteManifestsOnBranchUnsupported() {
"Cannot commit to branch someBranch: org.apache.iceberg.BaseRewriteManifests does not support branch commits");
}
+ @Test
+ public void testRewriteDataManifestsPreservesDeletes() {
+ Assumptions.assumeThat(formatVersion).isGreaterThan(1);
+
+ Table table = load();
+
+ table.newAppend().appendFile(FILE_A).appendFile(FILE_B).commit();
+
+ Snapshot appendSnapshot = table.currentSnapshot();
+ Assertions.assertThat(appendSnapshot.dataManifests(table.io())).hasSize(1);
+ Assertions.assertThat(appendSnapshot.deleteManifests(table.io())).isEmpty();
+
+ table.newRowDelta().addDeletes(FILE_A_DELETES).addDeletes(FILE_A2_DELETES).commit();
+
+ Snapshot deleteSnapshot = table.currentSnapshot();
+ Assertions.assertThat(deleteSnapshot.dataManifests(table.io())).hasSize(1);
+ Assertions.assertThat(deleteSnapshot.deleteManifests(table.io())).hasSize(1);
+
+ table.rewriteManifests().clusterBy(file -> file.path().toString()).commit();
Review Comment:
It is `ContentFile$path` that returns `CharSequence`. The idea of this closure is to split manifest entries based on the content file path. This ensures the original manifest with 2 entires is split into 2 new manifests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support replacing delete manifests [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #9000:
URL: https://github.com/apache/iceberg/pull/9000#discussion_r1387196142
##########
core/src/test/java/org/apache/iceberg/TestRewriteManifests.java:
##########
@@ -1105,6 +1108,499 @@ public void testRewriteManifestsOnBranchUnsupported() {
"Cannot commit to branch someBranch: org.apache.iceberg.BaseRewriteManifests does not support branch commits");
}
+ @Test
+ public void testRewriteDataManifestsPreservesDeletes() {
+ Assumptions.assumeThat(formatVersion).isGreaterThan(1);
+
+ Table table = load();
+
+ table.newAppend().appendFile(FILE_A).appendFile(FILE_B).commit();
+
+ Snapshot appendSnapshot = table.currentSnapshot();
+ Assertions.assertThat(appendSnapshot.dataManifests(table.io())).hasSize(1);
+ Assertions.assertThat(appendSnapshot.deleteManifests(table.io())).isEmpty();
+
+ table.newRowDelta().addDeletes(FILE_A_DELETES).addDeletes(FILE_A2_DELETES).commit();
+
+ Snapshot deleteSnapshot = table.currentSnapshot();
+ Assertions.assertThat(deleteSnapshot.dataManifests(table.io())).hasSize(1);
+ Assertions.assertThat(deleteSnapshot.deleteManifests(table.io())).hasSize(1);
+
+ table.rewriteManifests().clusterBy(file -> file.path().toString()).commit();
Review Comment:
nit: Paths are equivalent here too right? Just wondering about the string
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support replacing delete manifests [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #9000:
URL: https://github.com/apache/iceberg/pull/9000#discussion_r1387975982
##########
core/src/test/java/org/apache/iceberg/TestRewriteManifests.java:
##########
@@ -1105,6 +1108,499 @@ public void testRewriteManifestsOnBranchUnsupported() {
"Cannot commit to branch someBranch: org.apache.iceberg.BaseRewriteManifests does not support branch commits");
}
+ @Test
+ public void testRewriteDataManifestsPreservesDeletes() {
+ Assumptions.assumeThat(formatVersion).isGreaterThan(1);
+
+ Table table = load();
+
+ table.newAppend().appendFile(FILE_A).appendFile(FILE_B).commit();
+
+ Snapshot appendSnapshot = table.currentSnapshot();
+ Assertions.assertThat(appendSnapshot.dataManifests(table.io())).hasSize(1);
+ Assertions.assertThat(appendSnapshot.deleteManifests(table.io())).isEmpty();
+
+ table.newRowDelta().addDeletes(FILE_A_DELETES).addDeletes(FILE_A2_DELETES).commit();
+
+ Snapshot deleteSnapshot = table.currentSnapshot();
+ Assertions.assertThat(deleteSnapshot.dataManifests(table.io())).hasSize(1);
+ Assertions.assertThat(deleteSnapshot.deleteManifests(table.io())).hasSize(1);
+
+ table.rewriteManifests().clusterBy(file -> file.path().toString()).commit();
Review Comment:
I meant that "path.toString" and "path" would have equivalent effects in the closure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Support replacing delete manifests [iceberg]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on PR #9000:
URL: https://github.com/apache/iceberg/pull/9000#issuecomment-1804811157
Thank you, @singhpk234 @RussellSpitzer!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org