You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "fqaiser94 (via GitHub)" <gi...@apache.org> on 2024/03/03 17:07:37 UTC

[PR] Core: Support appending files with different specs [iceberg]

fqaiser94 opened a new pull request, #9860:
URL: https://github.com/apache/iceberg/pull/9860

   # What is the problem?
   
   Currently the `table.newAppend()` API expects users to provide Datafiles with the same PartitionSpec via `.appendFile()`. 
   Failure to do so [raises](https://github.com/apache/iceberg/blob/1a4f23bc0e6cda520ca815f2a245f5f21bfbc24f/core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java#L258) a `ValidationException("Invalid data file, expected spec id: %d", dataSpec.specId())`. 
   
   CMIIW but the Iceberg spec doesn't seem to impose any such restriction. 
   The only related restriction I could find was in the [manifests section](https://iceberg.apache.org/spec/#manifests) which says: 
   
   > A manifest stores files for a single partition spec. 
   
   We can easily work around this by writing multiple manifests, one for each spec for which files are being appended. 
   
   # Why is this change needed/valuable? 
   In the iceberg-kafka-connect project, we've seen that when users evolve the PartitionSpec of the table, often they'll end up in a situation where Datafiles with different PartitionSpecs might be inflight and committing these DataFiles together as part of the same snapshot becomes impossible due to the aforementioned `ValidationException`. 
   
   While we could work around this by committing DataFiles with different PartitionSpecs as separate snapshots, this makes it complex for us to correctly associate valuable (watermarking) metadata with each snapshot in the snapshot properties. In addition, it makes the table snapshot history unnecessarily longer. It would be more ideal if we could avoid these issues. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Support appending files with different specs [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #9860:
URL: https://github.com/apache/iceberg/pull/9860#discussion_r1521469812


##########
core/src/test/java/org/apache/iceberg/TestOverwrite.java:
##########
@@ -293,23 +332,178 @@ public void testValidatedOverwriteWithAppendOutsideOfDeleteMetrics() {
   }
 
   @Test
-  public void testValidatedOverwriteWithAppendSuccess() {
+  public void testValidatedOverwriteWithAppendInsideOfDeleteMetrics() {
+    // ensure the overwrite results in a merge
+    table.updateProperties().set(TableProperties.MANIFEST_MIN_MERGE_COUNT, "1").commit();
+
     TableMetadata base = TestTables.readMetadata(TABLE_NAME);
     long baseId =
         latestSnapshot(base, branch) == null ? -1 : latestSnapshot(base, branch).snapshotId();
 
+    DataFile file10To14WithMetrics =
+        DataFiles.builder(PARTITION_BY_DATE)
+            .withPath("/path/to/data-3.parquet")
+            .withFileSizeInBytes(0)
+            .withPartitionPath("date=2018-06-09")
+            .withMetrics(
+                new Metrics(
+                    5L,
+                    null, // no column sizes
+                    ImmutableMap.of(1, 5L, 2, 3L), // value count
+                    ImmutableMap.of(1, 0L, 2, 2L), // null count
+                    null,
+                    ImmutableMap.of(
+                        1, longToBuffer(5L), 3, stringToBuffer("2018-06-09")), // lower bounds
+                    ImmutableMap.of(
+                        1, longToBuffer(9L), 3, stringToBuffer("2018-06-09")) // upper bounds
+                    ))
+            .build();
+
     OverwriteFiles overwrite =
         table
             .newOverwrite()
-            .overwriteByRowFilter(and(equal("date", "2018-06-09"), lessThan("id", 20)))
-            .addFile(FILE_10_TO_14) // in 2018-06-09 matches and IDs are inside range
+            .overwriteByRowFilter(and(equal("date", "2018-06-09"), lessThan("id", 10)))
+            .addFile(
+                file10To14WithMetrics) // in 2018-06-09 and IDs are inside range based on metrics
             .validateAddedFilesMatchOverwriteFilter();
 
-    Assertions.assertThatThrownBy(() -> commit(table, overwrite, branch))
-        .isInstanceOf(ValidationException.class)
-        .hasMessageStartingWith("Cannot append file with rows that do not match filter");
+    commit(table, overwrite, branch);
 
-    Assert.assertEquals(
-        "Should not create a new snapshot", baseId, latestSnapshot(base, branch).snapshotId());
+    long overwriteId = latestSnapshot(table, branch).snapshotId();
+
+    assertThat(overwriteId).as("Should create a new snapshot").isNotEqualTo(baseId);
+    assertThat(latestSnapshot(table, branch).allManifests(table.io()).size())
+        .as("Table should have one manifest")
+        .isEqualTo(1);
+
+    validateManifestEntries(
+        latestSnapshot(table, branch).allManifests(table.io()).get(0),
+        ids(overwriteId, baseId, overwriteId),
+        files(FILE_10_TO_14, FILE_0_TO_4, FILE_5_TO_9),
+        statuses(Status.ADDED, Status.EXISTING, Status.DELETED));
+  }
+
+  @Test
+  public void testShouldFailValidationIfAnyFileIsInAPartitionNotMatchingOverwriteByRowFilter() {
+    // ensure the overwrite results in a merge
+    table.updateProperties().set(TableProperties.MANIFEST_MIN_MERGE_COUNT, "1").commit();
+
+    table.updateSpec().addField("id").commit();
+    PartitionSpec partitionByDateAndId =
+        PartitionSpec.builderFor(DATE_SCHEMA).withSpecId(1).identity("date").identity("id").build();
+    assertThat(table.spec())
+        .as("New spec should be partitioned by date and id.")
+        .isEqualTo(partitionByDateAndId);
+
+    TableMetadata base = TestTables.readMetadata(TABLE_NAME);
+    long baseId =
+        latestSnapshot(base, branch) == null ? -1 : latestSnapshot(base, branch).snapshotId();
+
+    String overwriteDate = "2018-06-09";
+    String notOverwriteDate = "2018-06-10";
+
+    ImmutableList.of(

Review Comment:
   ah sorry, I didn't realize that the class itself is already parameterized. I think the issue is that currently this is a JUnit4 test class and so you can't use JUnit5 parameterization for a single test method



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Support appending files with different specs [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #9860:
URL: https://github.com/apache/iceberg/pull/9860#discussion_r1510704394


##########
core/src/test/java/org/apache/iceberg/TestMergeAppend.java:
##########
@@ -92,6 +93,77 @@ public void testEmptyTableAppend() {
         statuses(Status.ADDED, Status.ADDED));
   }
 
+  @Test
+  public void testEmptyTableAppendFilesWithDifferentSpecs() {
+    Assert.assertEquals("Table should start empty", 0, listManifestFiles().size());

Review Comment:
   the community is in the process of migrating away from Junit4-style tests/assertions, so please use AssertJ-style assertions for new tests, as otherwise this adds additional overhead for the migration process. See also https://iceberg.apache.org/contribute/#assertj



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Support appending files with different specs [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #9860:
URL: https://github.com/apache/iceberg/pull/9860#discussion_r1519797012


##########
core/src/test/java/org/apache/iceberg/TestOverwrite.java:
##########
@@ -293,23 +332,178 @@ public void testValidatedOverwriteWithAppendOutsideOfDeleteMetrics() {
   }
 
   @Test
-  public void testValidatedOverwriteWithAppendSuccess() {
+  public void testValidatedOverwriteWithAppendInsideOfDeleteMetrics() {
+    // ensure the overwrite results in a merge
+    table.updateProperties().set(TableProperties.MANIFEST_MIN_MERGE_COUNT, "1").commit();
+
     TableMetadata base = TestTables.readMetadata(TABLE_NAME);
     long baseId =
         latestSnapshot(base, branch) == null ? -1 : latestSnapshot(base, branch).snapshotId();
 
+    DataFile file10To14WithMetrics =
+        DataFiles.builder(PARTITION_BY_DATE)
+            .withPath("/path/to/data-3.parquet")
+            .withFileSizeInBytes(0)
+            .withPartitionPath("date=2018-06-09")
+            .withMetrics(
+                new Metrics(
+                    5L,
+                    null, // no column sizes
+                    ImmutableMap.of(1, 5L, 2, 3L), // value count
+                    ImmutableMap.of(1, 0L, 2, 2L), // null count
+                    null,
+                    ImmutableMap.of(
+                        1, longToBuffer(5L), 3, stringToBuffer("2018-06-09")), // lower bounds
+                    ImmutableMap.of(
+                        1, longToBuffer(9L), 3, stringToBuffer("2018-06-09")) // upper bounds
+                    ))
+            .build();
+
     OverwriteFiles overwrite =
         table
             .newOverwrite()
-            .overwriteByRowFilter(and(equal("date", "2018-06-09"), lessThan("id", 20)))
-            .addFile(FILE_10_TO_14) // in 2018-06-09 matches and IDs are inside range
+            .overwriteByRowFilter(and(equal("date", "2018-06-09"), lessThan("id", 10)))
+            .addFile(
+                file10To14WithMetrics) // in 2018-06-09 and IDs are inside range based on metrics
             .validateAddedFilesMatchOverwriteFilter();
 
-    Assertions.assertThatThrownBy(() -> commit(table, overwrite, branch))
-        .isInstanceOf(ValidationException.class)
-        .hasMessageStartingWith("Cannot append file with rows that do not match filter");
+    commit(table, overwrite, branch);
 
-    Assert.assertEquals(
-        "Should not create a new snapshot", baseId, latestSnapshot(base, branch).snapshotId());
+    long overwriteId = latestSnapshot(table, branch).snapshotId();
+
+    assertThat(overwriteId).as("Should create a new snapshot").isNotEqualTo(baseId);
+    assertThat(latestSnapshot(table, branch).allManifests(table.io()).size())
+        .as("Table should have one manifest")
+        .isEqualTo(1);
+
+    validateManifestEntries(
+        latestSnapshot(table, branch).allManifests(table.io()).get(0),
+        ids(overwriteId, baseId, overwriteId),
+        files(FILE_10_TO_14, FILE_0_TO_4, FILE_5_TO_9),
+        statuses(Status.ADDED, Status.EXISTING, Status.DELETED));
+  }
+
+  @Test
+  public void testShouldFailValidationIfAnyFileIsInAPartitionNotMatchingOverwriteByRowFilter() {
+    // ensure the overwrite results in a merge
+    table.updateProperties().set(TableProperties.MANIFEST_MIN_MERGE_COUNT, "1").commit();
+
+    table.updateSpec().addField("id").commit();
+    PartitionSpec partitionByDateAndId =
+        PartitionSpec.builderFor(DATE_SCHEMA).withSpecId(1).identity("date").identity("id").build();
+    assertThat(table.spec())
+        .as("New spec should be partitioned by date and id.")
+        .isEqualTo(partitionByDateAndId);
+
+    TableMetadata base = TestTables.readMetadata(TABLE_NAME);
+    long baseId =
+        latestSnapshot(base, branch) == null ? -1 : latestSnapshot(base, branch).snapshotId();
+
+    String overwriteDate = "2018-06-09";
+    String notOverwriteDate = "2018-06-10";
+
+    ImmutableList.of(

Review Comment:
   this looks like it could be converted to a parameterized test via `@ParameterizedTest`. This would make the test easier to read/debug, since it would be clearer what's wrong if it fails with a particular input



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Support appending files with different specs [iceberg]

Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #9860:
URL: https://github.com/apache/iceberg/pull/9860#discussion_r1511996190


##########
core/src/test/java/org/apache/iceberg/TestMergeAppend.java:
##########
@@ -92,6 +94,74 @@ public void testEmptyTableAppend() {
         statuses(Status.ADDED, Status.ADDED));
   }
 
+  @Test
+  public void testEmptyTableAppendFilesWithDifferentSpecs() {

Review Comment:
   Can we add a test where validations will fail when there are multiple specs? I know we're exercising that for a single spec now but it'd be nice to see that we get the expected failures when there's multiple.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Support appending files with different specs [iceberg]

Posted by "fqaiser94 (via GitHub)" <gi...@apache.org>.
fqaiser94 commented on code in PR #9860:
URL: https://github.com/apache/iceberg/pull/9860#discussion_r1521453498


##########
core/src/test/java/org/apache/iceberg/TestOverwrite.java:
##########
@@ -293,23 +332,178 @@ public void testValidatedOverwriteWithAppendOutsideOfDeleteMetrics() {
   }
 
   @Test
-  public void testValidatedOverwriteWithAppendSuccess() {
+  public void testValidatedOverwriteWithAppendInsideOfDeleteMetrics() {
+    // ensure the overwrite results in a merge
+    table.updateProperties().set(TableProperties.MANIFEST_MIN_MERGE_COUNT, "1").commit();
+
     TableMetadata base = TestTables.readMetadata(TABLE_NAME);
     long baseId =
         latestSnapshot(base, branch) == null ? -1 : latestSnapshot(base, branch).snapshotId();
 
+    DataFile file10To14WithMetrics =
+        DataFiles.builder(PARTITION_BY_DATE)
+            .withPath("/path/to/data-3.parquet")
+            .withFileSizeInBytes(0)
+            .withPartitionPath("date=2018-06-09")
+            .withMetrics(
+                new Metrics(
+                    5L,
+                    null, // no column sizes
+                    ImmutableMap.of(1, 5L, 2, 3L), // value count
+                    ImmutableMap.of(1, 0L, 2, 2L), // null count
+                    null,
+                    ImmutableMap.of(
+                        1, longToBuffer(5L), 3, stringToBuffer("2018-06-09")), // lower bounds
+                    ImmutableMap.of(
+                        1, longToBuffer(9L), 3, stringToBuffer("2018-06-09")) // upper bounds
+                    ))
+            .build();
+
     OverwriteFiles overwrite =
         table
             .newOverwrite()
-            .overwriteByRowFilter(and(equal("date", "2018-06-09"), lessThan("id", 20)))
-            .addFile(FILE_10_TO_14) // in 2018-06-09 matches and IDs are inside range
+            .overwriteByRowFilter(and(equal("date", "2018-06-09"), lessThan("id", 10)))
+            .addFile(
+                file10To14WithMetrics) // in 2018-06-09 and IDs are inside range based on metrics
             .validateAddedFilesMatchOverwriteFilter();
 
-    Assertions.assertThatThrownBy(() -> commit(table, overwrite, branch))
-        .isInstanceOf(ValidationException.class)
-        .hasMessageStartingWith("Cannot append file with rows that do not match filter");
+    commit(table, overwrite, branch);
 
-    Assert.assertEquals(
-        "Should not create a new snapshot", baseId, latestSnapshot(base, branch).snapshotId());
+    long overwriteId = latestSnapshot(table, branch).snapshotId();
+
+    assertThat(overwriteId).as("Should create a new snapshot").isNotEqualTo(baseId);
+    assertThat(latestSnapshot(table, branch).allManifests(table.io()).size())
+        .as("Table should have one manifest")
+        .isEqualTo(1);
+
+    validateManifestEntries(
+        latestSnapshot(table, branch).allManifests(table.io()).get(0),
+        ids(overwriteId, baseId, overwriteId),
+        files(FILE_10_TO_14, FILE_0_TO_4, FILE_5_TO_9),
+        statuses(Status.ADDED, Status.EXISTING, Status.DELETED));
+  }
+
+  @Test
+  public void testShouldFailValidationIfAnyFileIsInAPartitionNotMatchingOverwriteByRowFilter() {
+    // ensure the overwrite results in a merge
+    table.updateProperties().set(TableProperties.MANIFEST_MIN_MERGE_COUNT, "1").commit();
+
+    table.updateSpec().addField("id").commit();
+    PartitionSpec partitionByDateAndId =
+        PartitionSpec.builderFor(DATE_SCHEMA).withSpecId(1).identity("date").identity("id").build();
+    assertThat(table.spec())
+        .as("New spec should be partitioned by date and id.")
+        .isEqualTo(partitionByDateAndId);
+
+    TableMetadata base = TestTables.readMetadata(TABLE_NAME);
+    long baseId =
+        latestSnapshot(base, branch) == null ? -1 : latestSnapshot(base, branch).snapshotId();
+
+    String overwriteDate = "2018-06-09";
+    String notOverwriteDate = "2018-06-10";
+
+    ImmutableList.of(

Review Comment:
   I agree but unfortunately, I'm not seeing any way to combine `@RunWith(Parameterized.class)` with `@ParmaterizedTest`. The two seem to clash :/ 
   
   <details>
     <summary>Code</summary>
   
   ```
   @ParameterizedTest
     @CsvSource({"2018-06-09,2018-06-10", "2018-06-10,2018-06-09"})
     public void testShouldFailValidationIfAnyFileIsInAPartitionNotMatchingOverwriteByRowFilter(
         String fileOldSpecDatePartition, String fileNewSpecDatePartition) {
       String overwriteDate = "2018-06-09";
   
       // ensure the overwrite results in a merge
       table.updateProperties().set(TableProperties.MANIFEST_MIN_MERGE_COUNT, "1").commit();
   
       table.updateSpec().addField("id").commit();
       PartitionSpec partitionByDateAndId =
           PartitionSpec.builderFor(DATE_SCHEMA).withSpecId(1).identity("date").identity("id").build();
       assertThat(table.spec())
           .as("New spec should be partitioned by date and id.")
           .isEqualTo(partitionByDateAndId);
   
       TableMetadata base = TestTables.readMetadata(TABLE_NAME);
       long baseId =
           latestSnapshot(base, branch) == null ? -1 : latestSnapshot(base, branch).snapshotId();
   
       DataFile fileOldSpec =
           DataFiles.builder(PARTITION_BY_DATE)
               .withPath("/path/to/data-1.parquet")
               .withPartitionPath(String.format("date=%s", fileOldSpecDatePartition))
               .withFileSizeInBytes(0)
               .withRecordCount(100)
               .build();
   
       DataFile fileNewSpec =
           DataFiles.builder(partitionByDateAndId)
               .withPath("/path/to/data-2.parquet")
               .withPartitionPath(String.format("date=%s/id=10", fileNewSpecDatePartition))
               .withFileSizeInBytes(0)
               .withRecordCount(100)
               .build();
   
       assertThatThrownBy(
               () ->
                   commit(
                       table,
                       table
                           .newOverwrite()
                           .overwriteByRowFilter(equal("date", overwriteDate))
                           .addFile(fileOldSpec)
                           .addFile(fileNewSpec)
                           .validateAddedFilesMatchOverwriteFilter(),
                       branch))
           .isInstanceOf(ValidationException.class)
           .hasMessageStartingWith("Cannot append file with rows that do not match filter");
   
       assertThat(latestSnapshot(table, branch).snapshotId())
           .as("Should not create a new snapshot")
           .isEqualTo(baseId);
     }
   ```
   
   </details>
   
   Error: 
   ```
   org.junit.jupiter.api.extension.ParameterResolutionException: No ParameterResolver registered for parameter [int arg0] in constructor [public org.apache.iceberg.TestOverwrite(int,java.lang.String)]
   ```
   
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Support appending files with different specs [iceberg]

Posted by "fqaiser94 (via GitHub)" <gi...@apache.org>.
fqaiser94 commented on PR #9860:
URL: https://github.com/apache/iceberg/pull/9860#issuecomment-2241204364

   > Really sorry for the delayed review on this @fqaiser94 I see this PR came up in discussion on the kafka commit coordination PR [#10351 (comment)](https://github.com/apache/iceberg/pull/10351#discussion_r1664602970).
   > 
   > I do remember the context of this PR and I just took another pass, at least from me it looks good. I'll leave it open for a bit in case @nastra @danielcweeks are interested, otherwise I'll merge.
   
   No problems, and thanks all for the reviews 😄 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Support appending files with different specs [iceberg]

Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #9860:
URL: https://github.com/apache/iceberg/pull/9860#discussion_r1511975142


##########
core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java:
##########
@@ -878,7 +886,7 @@ public Object updateEvent() {
 
   @SuppressWarnings("checkstyle:CyclomaticComplexity")
   private void cleanUncommittedAppends(Set<ManifestFile> committed) {
-    if (cachedNewDataManifests != null) {
+    if (!cachedNewDataManifests.isEmpty()) {

Review Comment:
   Any reason we need to change this? I see you covered all the cases in this class where we do explicit null checks and replaced them with empty checks so I think logically we're good. More so just curious if it's required for this change (also I think nulling it out would be quicker than clearing, since clear goes through very element and nulls it out but that's probably minor).



##########
core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java:
##########
@@ -220,9 +233,15 @@ protected boolean addsDeleteFiles() {
   /** Add a data file to the new snapshot. */
   protected void add(DataFile file) {
     Preconditions.checkNotNull(file, "Invalid data file: null");
-    setDataSpec(file);
-    addedFilesSummary.addedFile(dataSpec(), file);
+
+    PartitionSpec dataSpec = ops.current().spec(file.specId());
+    Preconditions.checkNotNull(
+        dataSpec, "Cannot find partition spec for data file: %s", file.path());

Review Comment:
   Could we include the partition spec ID in the failure message? Also could we make this `Preconditions.checkArgument(dataSpec != null)`? I know it was already a notNull earlier but that leads to unclear NPEs as opposed to this case which I think really is an illegal argument exception (a data file with an invalid spec)



##########
core/src/test/java/org/apache/iceberg/TestMergeAppend.java:
##########
@@ -92,6 +94,74 @@ public void testEmptyTableAppend() {
         statuses(Status.ADDED, Status.ADDED));
   }
 
+  @Test
+  public void testEmptyTableAppendFilesWithDifferentSpecs() {
+    assertThat(listManifestFiles()).as("Table should start empty").isEmpty();
+
+    TableMetadata base = readMetadata();
+    assertThat(base.currentSnapshot()).as("Should not have a current snapshot").isNull();
+    assertThat(base.lastSequenceNumber()).as("Last sequence number should be 0").isEqualTo(0);
+
+    table.updateSpec().addField("id").commit();
+    PartitionSpec newSpec = table.spec();
+
+    assertThat(table.specs().size()).as("Table should have 2 specs").isEqualTo(2);
+
+    DataFile fileOriginalSpec =
+        DataFiles.builder(SPEC)
+            .withPath("/path/to/data-a.parquet")
+            .withPartitionPath("data_bucket=0")
+            .withFileSizeInBytes(10)
+            .withRecordCount(1)
+            .build();
+
+    DataFile fileNewSpec =
+        DataFiles.builder(newSpec)
+            .withPath("/path/to/data-b.parquet")
+            .withPartitionPath("data_bucket=0/id=0")
+            .withFileSizeInBytes(10)
+            .withRecordCount(1)
+            .build();
+
+    Snapshot committedSnapshot =
+        commit(
+            table, table.newAppend().appendFile(fileOriginalSpec).appendFile(fileNewSpec), branch);
+
+    assertThat(committedSnapshot).as("Should create a snapshot").isNotNull();
+    V1Assert.assertEquals(
+        "Last sequence number should be 0", 0, table.ops().current().lastSequenceNumber());
+    V2Assert.assertEquals(
+        "Last sequence number should be 1", 1, table.ops().current().lastSequenceNumber());
+
+    assertThat(committedSnapshot.allManifests(table.io()).size())
+        .as("Should create 2 manifests for initial write, 1 manifest per spec")
+        .isEqualTo(2);
+
+    long snapshotId = committedSnapshot.snapshotId();
+
+    validateManifest(
+        committedSnapshot.allManifests(table.io()).stream()
+            .filter(m -> Objects.equals(m.partitionSpecId(), SPEC.specId()))
+            .findAny()
+            .get(),
+        dataSeqs(1L),
+        fileSeqs(1L),
+        ids(snapshotId),
+        files(fileOriginalSpec),
+        statuses(Status.ADDED));
+
+    validateManifest(
+        committedSnapshot.allManifests(table.io()).stream()
+            .filter(m -> Objects.equals(m.partitionSpecId(), newSpec.specId()))
+            .findAny()
+            .get(),
+        dataSeqs(1L),
+        fileSeqs(1L),
+        ids(snapshotId),
+        files(fileNewSpec),
+        statuses(Status.ADDED));

Review Comment:
   Nit: I think for this case you could just have a single validateManifest call and loop over the expected specs, something like:
   
   ```
   for (Map.Entry<Long, DataFile> expectedFilesBySpec : ImmutableMap.of(newSpec, fileNewSpec, spec.specId(), fileOriginalSpec).entrySet()) {
      validateManifest(
           committedSnapshot.allManifests(table.io()).stream()
               .filter(m -> Objects.equals(m.partitionSpecId(), expectedFilesBySpec.getKey()))
               .findAny()
               .get(),
           dataSeqs(1L),
           fileSeqs(1L),
           ids(snapshotId),
           files(expectedFilesBySpec.getValue()),
           statuses(Status.ADDED));
   }
   ```



##########
core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java:
##########
@@ -878,7 +886,7 @@ public Object updateEvent() {
 
   @SuppressWarnings("checkstyle:CyclomaticComplexity")
   private void cleanUncommittedAppends(Set<ManifestFile> committed) {
-    if (cachedNewDataManifests != null) {
+    if (!cachedNewDataManifests.isEmpty()) {

Review Comment:
   I see, we're doing the same thing for delete manifests as well. Should be fine. For context, I was particularly scrutinizing at this because this path was recently added as part of the 1.4.3 patch release to fix an issue where we were skipping added files as part of transaction retries: https://github.com/apache/iceberg/pull/9230 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Support appending files with different specs [iceberg]

Posted by "fqaiser94 (via GitHub)" <gi...@apache.org>.
fqaiser94 commented on PR #9860:
URL: https://github.com/apache/iceberg/pull/9860#issuecomment-2206964353

   Simplified the PR to just multi-partition-append use case. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Support appending files with different specs [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #9860:
URL: https://github.com/apache/iceberg/pull/9860#discussion_r1519789244


##########
core/src/test/java/org/apache/iceberg/TestOverwrite.java:
##########
@@ -271,6 +280,36 @@ public void testValidatedOverwriteWithAppendOutsideOfDelete() {
         "Should not create a new snapshot", baseId, latestSnapshot(table, branch).snapshotId());
   }
 
+  @Test
+  public void testValidatedOverwriteWithAppendSuccess() {
+    // ensure the overwrite results in a merge
+    table.updateProperties().set(TableProperties.MANIFEST_MIN_MERGE_COUNT, "1").commit();
+
+    TableMetadata base = TestTables.readMetadata(TABLE_NAME);
+    long baseId = latestSnapshot(base, branch).snapshotId();
+
+    OverwriteFiles overwrite =
+        table
+            .newOverwrite()
+            .overwriteByRowFilter(equal("date", "2018-06-09"))
+            .addFile(FILE_10_TO_14) // in 2018-06-09
+            .validateAddedFilesMatchOverwriteFilter();
+    commit(table, overwrite, branch);
+
+    long overwriteId = latestSnapshot(table, branch).snapshotId();
+
+    assertThat(overwriteId).as("Should create a new snapshot").isNotEqualTo(baseId);
+    assertThat(latestSnapshot(table, branch).allManifests(table.io()).size())

Review Comment:
   ```suggestion
       assertThat(latestSnapshot(table, branch).allManifests(table.io())).hasSize(1)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Support appending files with different specs [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #9860:
URL: https://github.com/apache/iceberg/pull/9860#discussion_r1519787882


##########
core/src/test/java/org/apache/iceberg/TestMergeAppend.java:
##########
@@ -92,6 +96,92 @@ public void testEmptyTableAppend() {
         statuses(Status.ADDED, Status.ADDED));
   }
 
+  @Test
+  public void testEmptyTableAppendFilesWithDifferentSpecs() {
+    assertThat(listManifestFiles()).as("Table should start empty").isEmpty();
+
+    TableMetadata base = readMetadata();
+    assertThat(base.currentSnapshot()).as("Should not have a current snapshot").isNull();
+    assertThat(base.lastSequenceNumber()).as("Last sequence number should be 0").isEqualTo(0);
+
+    table.updateSpec().addField("id").commit();
+    PartitionSpec newSpec = table.spec();
+
+    assertThat(table.specs().size()).as("Table should have 2 specs").isEqualTo(2);

Review Comment:
   ```suggestion
       assertThat(table.specs()).as("Table should have 2 specs").hasSize(2);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Support appending files with different specs [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #9860:
URL: https://github.com/apache/iceberg/pull/9860#discussion_r1510704394


##########
core/src/test/java/org/apache/iceberg/TestMergeAppend.java:
##########
@@ -92,6 +93,77 @@ public void testEmptyTableAppend() {
         statuses(Status.ADDED, Status.ADDED));
   }
 
+  @Test
+  public void testEmptyTableAppendFilesWithDifferentSpecs() {
+    Assert.assertEquals("Table should start empty", 0, listManifestFiles().size());

Review Comment:
   the community is in the process of [migrating away](https://github.com/orgs/apache/projects/305) from Junit4-style tests/assertions, so please use AssertJ-style assertions for new tests, as otherwise this adds additional overhead for the migration process. See also https://iceberg.apache.org/contribute/#assertj



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Support appending files with different specs [iceberg]

Posted by "fqaiser94 (via GitHub)" <gi...@apache.org>.
fqaiser94 commented on code in PR #9860:
URL: https://github.com/apache/iceberg/pull/9860#discussion_r1519687485


##########
core/src/test/java/org/apache/iceberg/TestMergeAppend.java:
##########
@@ -92,6 +94,74 @@ public void testEmptyTableAppend() {
         statuses(Status.ADDED, Status.ADDED));
   }
 
+  @Test
+  public void testEmptyTableAppendFilesWithDifferentSpecs() {

Review Comment:
   Good call; added `testDataSpecThrowsExceptionIfDataFilesWithDifferentSpecsAreAdded`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Support appending files with different specs [iceberg]

Posted by "fqaiser94 (via GitHub)" <gi...@apache.org>.
fqaiser94 commented on code in PR #9860:
URL: https://github.com/apache/iceberg/pull/9860#discussion_r1519703373


##########
core/src/main/java/org/apache/iceberg/BaseReplacePartitions.java:
##########
@@ -87,31 +87,40 @@ public BaseReplacePartitions toBranch(String branch) {
 
   @Override
   public void validate(TableMetadata currentMetadata, Snapshot parent) {
-    if (validateConflictingData) {
-      if (dataSpec().isUnpartitioned()) {
-        validateAddedDataFiles(
-            currentMetadata, startingSnapshotId, Expressions.alwaysTrue(), parent);
-      } else {
-        validateAddedDataFiles(currentMetadata, startingSnapshotId, replacedPartitions, parent);
-      }
-    }
-
-    if (validateConflictingDeletes) {
-      if (dataSpec().isUnpartitioned()) {
-        validateDeletedDataFiles(
-            currentMetadata, startingSnapshotId, Expressions.alwaysTrue(), parent);
-        validateNoNewDeleteFiles(
-            currentMetadata, startingSnapshotId, Expressions.alwaysTrue(), parent);
-      } else {
-        validateDeletedDataFiles(currentMetadata, startingSnapshotId, replacedPartitions, parent);
-        validateNoNewDeleteFiles(currentMetadata, startingSnapshotId, replacedPartitions, parent);
-      }
-    }
+    dataSpecs()
+        .forEach(
+            dataSpec -> {
+              if (validateConflictingData) {
+                if (dataSpec.isUnpartitioned()) {
+                  validateAddedDataFiles(
+                      currentMetadata, startingSnapshotId, Expressions.alwaysTrue(), parent);
+                } else {
+                  validateAddedDataFiles(
+                      currentMetadata, startingSnapshotId, replacedPartitions, parent);
+                }
+              }
+
+              if (validateConflictingDeletes) {
+                if (dataSpec.isUnpartitioned()) {
+                  validateDeletedDataFiles(
+                      currentMetadata, startingSnapshotId, Expressions.alwaysTrue(), parent);
+                  validateNoNewDeleteFiles(
+                      currentMetadata, startingSnapshotId, Expressions.alwaysTrue(), parent);
+                } else {
+                  validateDeletedDataFiles(
+                      currentMetadata, startingSnapshotId, replacedPartitions, parent);
+                  validateNoNewDeleteFiles(
+                      currentMetadata, startingSnapshotId, replacedPartitions, parent);
+                }
+              }
+            });
   }
 
   @Override
   public List<ManifestFile> apply(TableMetadata base, Snapshot snapshot) {
-    if (dataSpec().fields().size() <= 0) {
+    // TODO: I don't understand this, why delete all data? what if only part of the table is

Review Comment:
   Your explanation makes sense but I'm still a little lost when looking at the code. 
   
   Specifically, I don't understand how this [line of code](https://github.com/apache/iceberg/blob/131265d4ec79ccd114219b2989dd68694b473240/core/src/main/java/org/apache/iceberg/BaseReplacePartitions.java#L114) is correct? 
   It's presumably checking `dataSpec().fields().size() <= 0` to figure out if the current schema of the table is unpartitioned. 
   However, to my understanding, `dataSpec()` returns the spec of the files added, which is not necessarily the same as the current schema of the table. 
   I feel like the correct way to do this check would be to do something along the lines of  `base.spec().isPartitioned()`? 
   
   Would love to understand this more as this class in general is the only part of this PR I'm unsure about how to go about updating. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Support appending files with different specs [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #9860:
URL: https://github.com/apache/iceberg/pull/9860#discussion_r1511474175


##########
core/src/test/java/org/apache/iceberg/TestMergeAppend.java:
##########
@@ -92,6 +94,74 @@ public void testEmptyTableAppend() {
         statuses(Status.ADDED, Status.ADDED));
   }
 
+  @Test
+  public void testEmptyTableAppendFilesWithDifferentSpecs() {
+    assertThat(listManifestFiles().size()).as("Table should start empty").isEqualTo(0);

Review Comment:
   ```suggestion
       assertThat(listManifestFiles().isEmpty();
   ```
   or `.hasSize(0)`. Both are preferred, because they will actually print the content of the collection if the assertion ever fails, whereas `assertThat(listManifestFiles().size()).isEqualTo(xyz)` will only say `3 != xyz` without knowing what the content is



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Support appending files with different specs [iceberg]

Posted by "fqaiser94 (via GitHub)" <gi...@apache.org>.
fqaiser94 commented on code in PR #9860:
URL: https://github.com/apache/iceberg/pull/9860#discussion_r1519685957


##########
core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java:
##########
@@ -878,7 +886,7 @@ public Object updateEvent() {
 
   @SuppressWarnings("checkstyle:CyclomaticComplexity")
   private void cleanUncommittedAppends(Set<ManifestFile> committed) {
-    if (cachedNewDataManifests != null) {
+    if (!cachedNewDataManifests.isEmpty()) {

Review Comment:
   Yea, the reason for this change was just to be consistent with the delete-files approach. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Support appending files with different specs [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #9860:
URL: https://github.com/apache/iceberg/pull/9860#discussion_r1519787373


##########
core/src/test/java/org/apache/iceberg/TestMergeAppend.java:
##########
@@ -92,6 +96,92 @@ public void testEmptyTableAppend() {
         statuses(Status.ADDED, Status.ADDED));
   }
 
+  @Test
+  public void testEmptyTableAppendFilesWithDifferentSpecs() {
+    assertThat(listManifestFiles()).as("Table should start empty").isEmpty();
+
+    TableMetadata base = readMetadata();
+    assertThat(base.currentSnapshot()).as("Should not have a current snapshot").isNull();
+    assertThat(base.lastSequenceNumber()).as("Last sequence number should be 0").isEqualTo(0);
+
+    table.updateSpec().addField("id").commit();
+    PartitionSpec newSpec = table.spec();
+
+    assertThat(table.specs().size()).as("Table should have 2 specs").isEqualTo(2);
+
+    DataFile fileNewSpec =
+        DataFiles.builder(newSpec)
+            .withPath("/path/to/data-b.parquet")
+            .withPartitionPath("data_bucket=0/id=0")
+            .withFileSizeInBytes(10)
+            .withRecordCount(1)
+            .build();
+
+    Snapshot committedSnapshot =
+        commit(table, table.newAppend().appendFile(FILE_A).appendFile(fileNewSpec), branch);
+
+    assertThat(committedSnapshot).as("Should create a snapshot").isNotNull();
+    V1Assert.assertEquals(
+        "Last sequence number should be 0", 0, table.ops().current().lastSequenceNumber());
+    V2Assert.assertEquals(
+        "Last sequence number should be 1", 1, table.ops().current().lastSequenceNumber());
+
+    assertThat(committedSnapshot.allManifests(table.io()).size())
+        .as("Should create 2 manifests for initial write, 1 manifest per spec")
+        .isEqualTo(2);
+
+    long snapshotId = committedSnapshot.snapshotId();
+
+    ImmutableMap<Integer, DataFile> expectedFileBySpec =
+        ImmutableMap.of(SPEC.specId(), FILE_A, newSpec.specId(), fileNewSpec);
+
+    expectedFileBySpec.forEach(
+        (specId, expectedDataFile) -> {
+          ManifestFile manifestFileForSpecId =
+              committedSnapshot.allManifests(table.io()).stream()
+                  .filter(m -> Objects.equals(m.partitionSpecId(), specId))
+                  .findAny()
+                  .get();
+
+          validateManifest(
+              manifestFileForSpecId,
+              dataSeqs(1L),
+              fileSeqs(1L),
+              ids(snapshotId),
+              files(expectedDataFile),
+              statuses(Status.ADDED));
+        });
+  }
+
+  @Test
+  public void testDataSpecThrowsExceptionIfDataFilesWithDifferentSpecsAreAdded() {
+    assertThat(listManifestFiles()).as("Table should start empty").isEmpty();
+
+    TableMetadata base = readMetadata();
+    assertThat(base.currentSnapshot()).as("Should not have a current snapshot").isNull();
+    assertThat(base.lastSequenceNumber()).as("Last sequence number should be 0").isEqualTo(0);
+
+    table.updateSpec().addField("id").commit();
+    PartitionSpec newSpec = table.spec();
+
+    assertThat(table.specs().size()).as("Table should have 2 specs").isEqualTo(2);

Review Comment:
   ```suggestion
       assertThat(table.specs()).as("Table should have 2 specs").hasSize(2);
   ```
   the advantage here is that it will show the content of specs if the assertion ever fails, making stuff easier to debug



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Support appending files with different specs [iceberg]

Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #9860:
URL: https://github.com/apache/iceberg/pull/9860#discussion_r1511419189


##########
core/src/main/java/org/apache/iceberg/BaseReplacePartitions.java:
##########
@@ -87,31 +87,40 @@ public BaseReplacePartitions toBranch(String branch) {
 
   @Override
   public void validate(TableMetadata currentMetadata, Snapshot parent) {
-    if (validateConflictingData) {
-      if (dataSpec().isUnpartitioned()) {
-        validateAddedDataFiles(
-            currentMetadata, startingSnapshotId, Expressions.alwaysTrue(), parent);
-      } else {
-        validateAddedDataFiles(currentMetadata, startingSnapshotId, replacedPartitions, parent);
-      }
-    }
-
-    if (validateConflictingDeletes) {
-      if (dataSpec().isUnpartitioned()) {
-        validateDeletedDataFiles(
-            currentMetadata, startingSnapshotId, Expressions.alwaysTrue(), parent);
-        validateNoNewDeleteFiles(
-            currentMetadata, startingSnapshotId, Expressions.alwaysTrue(), parent);
-      } else {
-        validateDeletedDataFiles(currentMetadata, startingSnapshotId, replacedPartitions, parent);
-        validateNoNewDeleteFiles(currentMetadata, startingSnapshotId, replacedPartitions, parent);
-      }
-    }
+    dataSpecs()
+        .forEach(
+            dataSpec -> {
+              if (validateConflictingData) {
+                if (dataSpec.isUnpartitioned()) {
+                  validateAddedDataFiles(
+                      currentMetadata, startingSnapshotId, Expressions.alwaysTrue(), parent);
+                } else {
+                  validateAddedDataFiles(
+                      currentMetadata, startingSnapshotId, replacedPartitions, parent);
+                }
+              }
+
+              if (validateConflictingDeletes) {
+                if (dataSpec.isUnpartitioned()) {
+                  validateDeletedDataFiles(
+                      currentMetadata, startingSnapshotId, Expressions.alwaysTrue(), parent);
+                  validateNoNewDeleteFiles(
+                      currentMetadata, startingSnapshotId, Expressions.alwaysTrue(), parent);
+                } else {
+                  validateDeletedDataFiles(
+                      currentMetadata, startingSnapshotId, replacedPartitions, parent);
+                  validateNoNewDeleteFiles(
+                      currentMetadata, startingSnapshotId, replacedPartitions, parent);
+                }
+              }
+            });
   }
 
   @Override
   public List<ManifestFile> apply(TableMetadata base, Snapshot snapshot) {
-    if (dataSpec().fields().size() <= 0) {
+    // TODO: I don't understand this, why delete all data? what if only part of the table is

Review Comment:
   If the current schema of the table is unpartitioned, then a replace is replacing the whole table.  Historical partitions don't apply at that point since the replace is against the current schema.



##########
core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java:
##########
@@ -138,19 +138,34 @@ protected boolean isCaseSensitive() {
     return caseSensitive;
   }
 
+  /**
+   * @deprecated Use {@link #dataSpecs()} instead. This method only returns one of potentially
+   *     multiple {@link PartitionSpec} involved in this operation.
+   */
+  @Deprecated
   protected PartitionSpec dataSpec() {
+    Set<PartitionSpec> partitionSpecs = dataSpecs();

Review Comment:
   This assignment doesn't seem necessary:  `dataSpecs().iterator().next()`



##########
core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java:
##########
@@ -79,18 +80,18 @@ abstract class MergingSnapshotProducer<ThisT> extends SnapshotProducer<ThisT> {
   private final ManifestFilterManager<DeleteFile> deleteFilterManager;
 
   // update data
-  private final List<DataFile> newDataFiles = Lists.newArrayList();
+  private final Map<Integer, List<DataFile>> newDataFilesBySpec = Maps.newHashMap();
   private Long newDataFilesDataSequenceNumber;
   private final Map<Integer, List<DeleteFileHolder>> newDeleteFilesBySpec = Maps.newHashMap();
   private final List<ManifestFile> appendManifests = Lists.newArrayList();
   private final List<ManifestFile> rewrittenAppendManifests = Lists.newArrayList();
   private final SnapshotSummary.Builder addedFilesSummary = SnapshotSummary.builder();
   private final SnapshotSummary.Builder appendedManifestsSummary = SnapshotSummary.builder();
   private Expression deleteExpression = Expressions.alwaysFalse();
-  private PartitionSpec dataSpec;
+  private final Set<PartitionSpec> dataSpecs = Sets.newHashSet();

Review Comment:
   Can we drop this and just make the map `Map<PartitionSpec, List<DataFile>>` and use the keyset instead?



##########
core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java:
##########
@@ -138,19 +138,34 @@ protected boolean isCaseSensitive() {
     return caseSensitive;
   }
 
+  /**
+   * @deprecated Use {@link #dataSpecs()} instead. This method only returns one of potentially
+   *     multiple {@link PartitionSpec} involved in this operation.
+   */
+  @Deprecated
   protected PartitionSpec dataSpec() {
+    Set<PartitionSpec> partitionSpecs = dataSpecs();
+    return partitionSpecs.iterator().next();
+  }
+
+  protected Set<PartitionSpec> dataSpecs() {
     Preconditions.checkState(
-        dataSpec != null, "Cannot determine partition spec: no data files have been added");
-    // the spec is set when the write is started
-    return dataSpec;
+        !dataSpecs.isEmpty(), "Cannot determine partition specs: no data files have been added");
+    return dataSpecs;
   }
 
   protected Expression rowFilter() {
     return deleteExpression;
   }
 
   protected List<DataFile> addedDataFiles() {
-    return ImmutableList.copyOf(newDataFiles);
+    ImmutableList.Builder<DataFile> builder = ImmutableList.builder();
+    addedDataFilesBySpec().values().forEach(builder::addAll);

Review Comment:
   I think you can use values+flatmap here instead of using a builder



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Support appending files with different specs [iceberg]

Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar merged PR #9860:
URL: https://github.com/apache/iceberg/pull/9860


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Support appending files with different specs [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on code in PR #9860:
URL: https://github.com/apache/iceberg/pull/9860#discussion_r1519791560


##########
core/src/test/java/org/apache/iceberg/TestOverwrite.java:
##########
@@ -293,23 +332,178 @@ public void testValidatedOverwriteWithAppendOutsideOfDeleteMetrics() {
   }
 
   @Test
-  public void testValidatedOverwriteWithAppendSuccess() {
+  public void testValidatedOverwriteWithAppendInsideOfDeleteMetrics() {
+    // ensure the overwrite results in a merge
+    table.updateProperties().set(TableProperties.MANIFEST_MIN_MERGE_COUNT, "1").commit();
+
     TableMetadata base = TestTables.readMetadata(TABLE_NAME);
     long baseId =
         latestSnapshot(base, branch) == null ? -1 : latestSnapshot(base, branch).snapshotId();
 
+    DataFile file10To14WithMetrics =
+        DataFiles.builder(PARTITION_BY_DATE)
+            .withPath("/path/to/data-3.parquet")
+            .withFileSizeInBytes(0)
+            .withPartitionPath("date=2018-06-09")
+            .withMetrics(
+                new Metrics(
+                    5L,
+                    null, // no column sizes
+                    ImmutableMap.of(1, 5L, 2, 3L), // value count
+                    ImmutableMap.of(1, 0L, 2, 2L), // null count
+                    null,
+                    ImmutableMap.of(
+                        1, longToBuffer(5L), 3, stringToBuffer("2018-06-09")), // lower bounds
+                    ImmutableMap.of(
+                        1, longToBuffer(9L), 3, stringToBuffer("2018-06-09")) // upper bounds
+                    ))
+            .build();
+
     OverwriteFiles overwrite =
         table
             .newOverwrite()
-            .overwriteByRowFilter(and(equal("date", "2018-06-09"), lessThan("id", 20)))
-            .addFile(FILE_10_TO_14) // in 2018-06-09 matches and IDs are inside range
+            .overwriteByRowFilter(and(equal("date", "2018-06-09"), lessThan("id", 10)))
+            .addFile(
+                file10To14WithMetrics) // in 2018-06-09 and IDs are inside range based on metrics
             .validateAddedFilesMatchOverwriteFilter();
 
-    Assertions.assertThatThrownBy(() -> commit(table, overwrite, branch))
-        .isInstanceOf(ValidationException.class)
-        .hasMessageStartingWith("Cannot append file with rows that do not match filter");
+    commit(table, overwrite, branch);
 
-    Assert.assertEquals(
-        "Should not create a new snapshot", baseId, latestSnapshot(base, branch).snapshotId());
+    long overwriteId = latestSnapshot(table, branch).snapshotId();
+
+    assertThat(overwriteId).as("Should create a new snapshot").isNotEqualTo(baseId);
+    assertThat(latestSnapshot(table, branch).allManifests(table.io()).size())
+        .as("Table should have one manifest")
+        .isEqualTo(1);
+
+    validateManifestEntries(
+        latestSnapshot(table, branch).allManifests(table.io()).get(0),
+        ids(overwriteId, baseId, overwriteId),
+        files(FILE_10_TO_14, FILE_0_TO_4, FILE_5_TO_9),
+        statuses(Status.ADDED, Status.EXISTING, Status.DELETED));
+  }
+
+  @Test
+  public void testShouldFailValidationIfAnyFileIsInAPartitionNotMatchingOverwriteByRowFilter() {
+    // ensure the overwrite results in a merge
+    table.updateProperties().set(TableProperties.MANIFEST_MIN_MERGE_COUNT, "1").commit();
+
+    table.updateSpec().addField("id").commit();
+    PartitionSpec partitionByDateAndId =
+        PartitionSpec.builderFor(DATE_SCHEMA).withSpecId(1).identity("date").identity("id").build();
+    assertThat(table.spec())
+        .as("New spec should be partitioned by date and id.")
+        .isEqualTo(partitionByDateAndId);
+
+    TableMetadata base = TestTables.readMetadata(TABLE_NAME);
+    long baseId =
+        latestSnapshot(base, branch) == null ? -1 : latestSnapshot(base, branch).snapshotId();
+
+    String overwriteDate = "2018-06-09";
+    String notOverwriteDate = "2018-06-10";
+
+    ImmutableList.of(
+            ImmutableList.of(notOverwriteDate, overwriteDate),
+            ImmutableList.of(overwriteDate, notOverwriteDate))
+        .forEach(
+            datePartitions -> {
+              String datePartition1 = datePartitions.get(0);
+              String datePartition2 = datePartitions.get(1);
+
+              DataFile fileOldSpec =
+                  DataFiles.builder(PARTITION_BY_DATE)
+                      .withPath("/path/to/data-1.parquet")
+                      .withPartitionPath(String.format("date=%s", datePartition1))
+                      .withFileSizeInBytes(0)
+                      .withRecordCount(100)
+                      .build();
+
+              DataFile fileNewSpec =
+                  DataFiles.builder(partitionByDateAndId)
+                      .withPath("/path/to/data-2.parquet")
+                      .withPartitionPath(String.format("date=%s/id=10", datePartition2))
+                      .withFileSizeInBytes(0)
+                      .withRecordCount(100)
+                      .build();
+
+              assertThatThrownBy(
+                      () ->
+                          commit(
+                              table,
+                              table
+                                  .newOverwrite()
+                                  .overwriteByRowFilter(equal("date", overwriteDate))
+                                  .addFile(fileOldSpec)
+                                  .addFile(fileNewSpec)
+                                  .validateAddedFilesMatchOverwriteFilter(),
+                              branch))
+                  .isInstanceOf(ValidationException.class)
+                  .hasMessageStartingWith("Cannot append file with rows that do not match filter");
+
+              assertThat(latestSnapshot(table, branch).snapshotId())
+                  .as("Should not create a new snapshot")
+                  .isEqualTo(baseId);
+            });
+  }
+
+  @Test
+  public void testShouldPassValidationIfAllFilesAreInPartitionMatchingOverwriteByRowFilter() {
+    // ensure the overwrite results in a merge
+    table.updateProperties().set(TableProperties.MANIFEST_MIN_MERGE_COUNT, "1").commit();
+
+    table.updateSpec().addField("id").commit();
+    PartitionSpec partitionByDateAndId =
+        PartitionSpec.builderFor(DATE_SCHEMA).withSpecId(1).identity("date").identity("id").build();
+    assertThat(table.spec())
+        .as("New spec should be partitioned by date and id.")
+        .isEqualTo(partitionByDateAndId);
+
+    TableMetadata base = TestTables.readMetadata(TABLE_NAME);
+    long baseId =
+        latestSnapshot(base, branch) == null ? -1 : latestSnapshot(base, branch).snapshotId();
+
+    DataFile fileOldSpec =
+        DataFiles.builder(PARTITION_BY_DATE)
+            .withPath("/path/to/data-1.parquet")
+            .withPartitionPath("date=2018-06-09")
+            .withFileSizeInBytes(0)
+            .withRecordCount(100)
+            .build();
+
+    DataFile fileNewSpec =
+        DataFiles.builder(partitionByDateAndId)
+            .withPath("/path/to/data-2.parquet")
+            .withPartitionPath("date=2018-06-09/id=10")
+            .withFileSizeInBytes(0)
+            .withRecordCount(100)
+            .build();
+
+    commit(
+        table,
+        table
+            .newOverwrite()
+            .overwriteByRowFilter(equal("date", "2018-06-09"))
+            .addFile(fileOldSpec) // in 2018-06-09
+            .addFile(fileNewSpec) // in 2018-06-09
+            .validateAddedFilesMatchOverwriteFilter(),
+        branch);
+
+    long overwriteId = latestSnapshot(table, branch).snapshotId();
+
+    assertThat(overwriteId).as("Should create a new snapshot").isNotEqualTo(baseId);
+
+    List<ManifestFile> allManifests = latestSnapshot(table, branch).allManifests(table.io());
+    assertThat(allManifests.size())

Review Comment:
   ```suggestion
       assertThat(allManifests).as(..).hasSize(2)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Support appending files with different specs [iceberg]

Posted by "fqaiser94 (via GitHub)" <gi...@apache.org>.
fqaiser94 commented on code in PR #9860:
URL: https://github.com/apache/iceberg/pull/9860#discussion_r1521454344


##########
core/src/test/java/org/apache/iceberg/TestMergeAppend.java:
##########
@@ -92,6 +96,92 @@ public void testEmptyTableAppend() {
         statuses(Status.ADDED, Status.ADDED));
   }
 
+  @Test
+  public void testEmptyTableAppendFilesWithDifferentSpecs() {
+    assertThat(listManifestFiles()).as("Table should start empty").isEmpty();
+
+    TableMetadata base = readMetadata();
+    assertThat(base.currentSnapshot()).as("Should not have a current snapshot").isNull();
+    assertThat(base.lastSequenceNumber()).as("Last sequence number should be 0").isEqualTo(0);
+
+    table.updateSpec().addField("id").commit();
+    PartitionSpec newSpec = table.spec();
+
+    assertThat(table.specs().size()).as("Table should have 2 specs").isEqualTo(2);
+
+    DataFile fileNewSpec =
+        DataFiles.builder(newSpec)
+            .withPath("/path/to/data-b.parquet")
+            .withPartitionPath("data_bucket=0/id=0")
+            .withFileSizeInBytes(10)
+            .withRecordCount(1)
+            .build();
+
+    Snapshot committedSnapshot =
+        commit(table, table.newAppend().appendFile(FILE_A).appendFile(fileNewSpec), branch);
+
+    assertThat(committedSnapshot).as("Should create a snapshot").isNotNull();
+    V1Assert.assertEquals(
+        "Last sequence number should be 0", 0, table.ops().current().lastSequenceNumber());
+    V2Assert.assertEquals(
+        "Last sequence number should be 1", 1, table.ops().current().lastSequenceNumber());
+
+    assertThat(committedSnapshot.allManifests(table.io()).size())
+        .as("Should create 2 manifests for initial write, 1 manifest per spec")
+        .isEqualTo(2);
+
+    long snapshotId = committedSnapshot.snapshotId();
+
+    ImmutableMap<Integer, DataFile> expectedFileBySpec =
+        ImmutableMap.of(SPEC.specId(), FILE_A, newSpec.specId(), fileNewSpec);
+
+    expectedFileBySpec.forEach(
+        (specId, expectedDataFile) -> {
+          ManifestFile manifestFileForSpecId =
+              committedSnapshot.allManifests(table.io()).stream()
+                  .filter(m -> Objects.equals(m.partitionSpecId(), specId))
+                  .findAny()
+                  .get();
+
+          validateManifest(
+              manifestFileForSpecId,
+              dataSeqs(1L),
+              fileSeqs(1L),
+              ids(snapshotId),
+              files(expectedDataFile),
+              statuses(Status.ADDED));
+        });
+  }
+
+  @Test
+  public void testDataSpecThrowsExceptionIfDataFilesWithDifferentSpecsAreAdded() {
+    assertThat(listManifestFiles()).as("Table should start empty").isEmpty();
+
+    TableMetadata base = readMetadata();
+    assertThat(base.currentSnapshot()).as("Should not have a current snapshot").isNull();
+    assertThat(base.lastSequenceNumber()).as("Last sequence number should be 0").isEqualTo(0);
+
+    table.updateSpec().addField("id").commit();
+    PartitionSpec newSpec = table.spec();
+
+    assertThat(table.specs().size()).as("Table should have 2 specs").isEqualTo(2);

Review Comment:
   Makes sense, thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org