You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "deniskuzZ (via GitHub)" <gi...@apache.org> on 2025/03/14 18:01:10 UTC

[PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and null partition values [iceberg]

deniskuzZ opened a new pull request, #12528:
URL: https://github.com/apache/iceberg/pull/12528

   BugFix:
   
   Below 3 StructProjections are considered equal leading to key overwrite in PartitionMap;
   ````
   "ice_orc(company_id=null)"
   "ice_orc(company_id=null/dept_id=null)"
   "ice_orc(company_id=null/dept_id=null/team_id=null)"
   ````


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and NULL partition values [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #12528:
URL: https://github.com/apache/iceberg/pull/12528#discussion_r1999094564


##########
core/src/test/java/org/apache/iceberg/TestMetadataTableScansWithPartitionEvolution.java:
##########
@@ -243,6 +245,102 @@ public void testPartitionSpecEvolutionToUnpartitioned() throws IOException {
     }
   }
 
+  @TestTemplate
+  public void testPartitionSpecEvolutionNullValues() throws IOException {
+    Schema schema =
+        new Schema(
+            required(1, "company_id", Types.IntegerType.get()),
+            required(2, "dept_id", Types.IntegerType.get()),
+            required(3, "team_id", Types.IntegerType.get()));
+
+    table =
+        TestTables.create(
+            tableDir,
+            metadataDir,
+            "nulltest",
+            schema,
+            PartitionSpec.builderFor(schema).identity("company_id").build(),
+            SortOrder.unsorted(),
+            formatVersion);
+    table.newFastAppend().appendFile(newDataFile("company_id=__HIVE_DEFAULT_PARTITION__")).commit();
+
+    table.updateSpec().addField("dept_id").commit();
+    table
+        .newFastAppend()
+        .appendFile(
+            newDataFile(
+                "company_id=__HIVE_DEFAULT_PARTITION__" + "/dept_id=__HIVE_DEFAULT_PARTITION__"))
+        .commit();
+
+    table.updateSpec().addField("team_id").commit();
+    table
+        .newFastAppend()
+        .appendFile(
+            newDataFile(
+                "company_id=__HIVE_DEFAULT_PARTITION__"

Review Comment:
   Is there a reason to do this with partition paths instead of structs? I generally would like to avoid using strings if at all possible



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and NULL partition values [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12528:
URL: https://github.com/apache/iceberg/pull/12528#issuecomment-2843782074

   This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and NULL partition values [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12528:
URL: https://github.com/apache/iceberg/pull/12528#discussion_r2095692340


##########
core/src/main/java/org/apache/iceberg/PartitionsTable.java:
##########
@@ -290,27 +297,25 @@ void update(ContentFile<?> file, Snapshot snapshot) {
       if (snapshot != null) {
         long snapshotCommitTime = snapshot.timestampMillis() * 1000;
         if (this.lastUpdatedAt == null || snapshotCommitTime > this.lastUpdatedAt) {
+          this.specId = file.specId();
+

Review Comment:
   I don't get this change.
   
   Why is it good to only use the `file.specId` then the times are ordered?
   IIRC there is no guarantees on the values of the timestamps, so this could cause unexpected behavior



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and null partition values [iceberg]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #12528:
URL: https://github.com/apache/iceberg/pull/12528#discussion_r1996252807


##########
api/src/main/java/org/apache/iceberg/types/Comparators.java:
##########
@@ -108,6 +109,15 @@ public int compare(StructLike o1, StructLike o2) {
         return 0;
       }
 
+      if (o1 instanceof StructProjection && o2 instanceof StructProjection) {

Review Comment:
   Updated the PR to compare the number of projected fields.
   
   The issue is that 2 projections with null and missing field values are considered equal. That leads to incorrect PartitionMap construction: 
   
   Below PartitionProjections are equal, so in PartitionMap we'll have just 1 random entry:
   ````
   "ice_orc(company_id=null)"
   "ice_orc(company_id=null/dept_id=null)"
   "ice_orc(company_id=null/dept_id=null/team_id=null)"
   ````



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and null partition values [iceberg]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #12528:
URL: https://github.com/apache/iceberg/pull/12528#discussion_r1996398329


##########
api/src/main/java/org/apache/iceberg/types/Comparators.java:
##########
@@ -108,6 +109,15 @@ public int compare(StructLike o1, StructLike o2) {
         return 0;
       }
 
+      if (o1 instanceof StructProjection && o2 instanceof StructProjection) {

Review Comment:
   an alternative solution could be replacing `NULL` values in a struct with `__HIVE_DEFAULT_PARTITION__`
   ````
   StructLike partitionKey = newPartitionKey(file.partition(), "__HIVE_DEFAULT_PARTITION__");
   StructLike partition = PartitionUtil.coercePartition(
       partitionType, table.specs().get(file.specId()), file.partition());
   partitions.get(partition).update(file, snapshot);
   ````
   however, I don't think it's a better approach since this hack should be applied in multiple places and could potentially lead to new issues. 
   
   I believe that PartitionProjections with MISSING and NULL fields should not be considered equal. If all tests pass under the assumption that they should be, this assumption may not be accurate.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and null partition values [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #12528:
URL: https://github.com/apache/iceberg/pull/12528#discussion_r1996373199


##########
api/src/main/java/org/apache/iceberg/types/Comparators.java:
##########
@@ -108,6 +109,15 @@ public int compare(StructLike o1, StructLike o2) {
         return 0;
       }
 
+      if (o1 instanceof StructProjection && o2 instanceof StructProjection) {

Review Comment:
   I think that's true, but that we should fix it in the partition list evaluation and not in the base equality method.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and null partition values [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #12528:
URL: https://github.com/apache/iceberg/pull/12528#discussion_r1996184450


##########
api/src/main/java/org/apache/iceberg/types/Comparators.java:
##########
@@ -108,6 +109,15 @@ public int compare(StructLike o1, StructLike o2) {
         return 0;
       }
 
+      if (o1 instanceof StructProjection && o2 instanceof StructProjection) {

Review Comment:
   This is a pretty big behavior change and I'm not sure it's the right place to fix this issue. The previous behavior created a comparator which only cared about those fields which are passed in the Constructor, this changes that to instead look at the raw lengths of the projections instead.
   
   Can you explain a little more about the bug you are trying to fix and write some test cases which are currently failing to demonstrate the issue?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and NULL partition values [iceberg]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #12528:
URL: https://github.com/apache/iceberg/pull/12528#discussion_r1999259243


##########
core/src/test/java/org/apache/iceberg/TestMetadataTableScansWithPartitionEvolution.java:
##########
@@ -243,6 +245,102 @@ public void testPartitionSpecEvolutionToUnpartitioned() throws IOException {
     }
   }
 
+  @TestTemplate
+  public void testPartitionSpecEvolutionNullValues() throws IOException {
+    Schema schema =
+        new Schema(
+            required(1, "company_id", Types.IntegerType.get()),
+            required(2, "dept_id", Types.IntegerType.get()),
+            required(3, "team_id", Types.IntegerType.get()));
+
+    table =
+        TestTables.create(
+            tableDir,
+            metadataDir,
+            "nulltest",
+            schema,
+            PartitionSpec.builderFor(schema).identity("company_id").build(),
+            SortOrder.unsorted(),
+            formatVersion);
+    table.newFastAppend().appendFile(newDataFile("company_id=__HIVE_DEFAULT_PARTITION__")).commit();
+
+    table.updateSpec().addField("dept_id").commit();
+    table
+        .newFastAppend()
+        .appendFile(
+            newDataFile(
+                "company_id=__HIVE_DEFAULT_PARTITION__" + "/dept_id=__HIVE_DEFAULT_PARTITION__"))
+        .commit();
+
+    table.updateSpec().addField("team_id").commit();
+    table
+        .newFastAppend()
+        .appendFile(
+            newDataFile(
+                "company_id=__HIVE_DEFAULT_PARTITION__"

Review Comment:
   note, this test class extends from TestBase that doesn't support `StructLike partition`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and NULL partition values [iceberg]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #12528:
URL: https://github.com/apache/iceberg/pull/12528#discussion_r2131824235


##########
core/src/main/java/org/apache/iceberg/PartitionsTable.java:
##########
@@ -290,27 +297,25 @@ void update(ContentFile<?> file, Snapshot snapshot) {
       if (snapshot != null) {
         long snapshotCommitTime = snapshot.timestampMillis() * 1000;
         if (this.lastUpdatedAt == null || snapshotCommitTime > this.lastUpdatedAt) {
+          this.specId = file.specId();
+

Review Comment:
   thanks for the review, @pvary! That is required to pick the latest specId in case or rename.
   
   see `TestMetadataTableScansWithPartitionEvolution#testPartitionSpecRenameFields()`
   
   **_specId=0_**
   ````
   PartitionSpec.builderFor(schema).identity("data").identity("category").build(),
   ````
   execute partCol rename
   **_specId=1_**
   ````
   table.updateSpec().renameField("category", "category_another_name").commit();
   ````
   assertPartitions("data=c1/category_another_name=d1", "data=c2/category_another_name=d2");



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and NULL partition values [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12528:
URL: https://github.com/apache/iceberg/pull/12528#discussion_r2131959065


##########
core/src/main/java/org/apache/iceberg/PartitionsTable.java:
##########
@@ -238,26 +246,25 @@ private class PartitionsScan extends StaticTableScan {
     }
   }
 
-  static class PartitionMap {
-    private final StructLikeMap<Partition> partitions;
-    private final Types.StructType keyType;
+  private static class PartitionComparator implements Comparator<StructLike> {
+    private Comparator<StructLike> comparator;
 
-    PartitionMap(Types.StructType type) {
-      this.partitions = StructLikeMap.create(type);
-      this.keyType = type;
+    private PartitionComparator(Types.StructType struct) {
+      this.comparator = Comparators.forType(struct);
     }
 
-    Partition get(StructLike key) {
-      Partition partition = partitions.get(key);
-      if (partition == null) {
-        partition = new Partition(key, keyType);
-        partitions.put(key, partition);
+    @Override
+    public int compare(StructLike o1, StructLike o2) {
+      if (o1 instanceof StructProjection && o2 instanceof StructProjection) {
+        int cmp =
+            Integer.compare(
+                ((StructProjection) o1).projectedFields(),
+                ((StructProjection) o2).projectedFields());
+        if (cmp != 0) {
+          return cmp;
+        }
       }
-      return partition;
-    }
-
-    Iterable<Partition> all() {
-      return partitions.values();
+      return comparator.compare(o1, o2);

Review Comment:
   nit: newline before this



##########
core/src/main/java/org/apache/iceberg/PartitionsTable.java:
##########
@@ -290,27 +297,25 @@ void update(ContentFile<?> file, Snapshot snapshot) {
       if (snapshot != null) {
         long snapshotCommitTime = snapshot.timestampMillis() * 1000;
         if (this.lastUpdatedAt == null || snapshotCommitTime > this.lastUpdatedAt) {
+          this.specId = file.specId();
+

Review Comment:
   Thx



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and NULL partition values [iceberg]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #12528:
URL: https://github.com/apache/iceberg/pull/12528#discussion_r1999234522


##########
core/src/test/java/org/apache/iceberg/TestMetadataTableScansWithPartitionEvolution.java:
##########
@@ -243,6 +245,102 @@ public void testPartitionSpecEvolutionToUnpartitioned() throws IOException {
     }
   }
 
+  @TestTemplate
+  public void testPartitionSpecEvolutionNullValues() throws IOException {
+    Schema schema =
+        new Schema(
+            required(1, "company_id", Types.IntegerType.get()),
+            required(2, "dept_id", Types.IntegerType.get()),
+            required(3, "team_id", Types.IntegerType.get()));
+
+    table =
+        TestTables.create(
+            tableDir,
+            metadataDir,
+            "nulltest",
+            schema,
+            PartitionSpec.builderFor(schema).identity("company_id").build(),
+            SortOrder.unsorted(),
+            formatVersion);
+    table.newFastAppend().appendFile(newDataFile("company_id=__HIVE_DEFAULT_PARTITION__")).commit();
+
+    table.updateSpec().addField("dept_id").commit();
+    table
+        .newFastAppend()
+        .appendFile(
+            newDataFile(
+                "company_id=__HIVE_DEFAULT_PARTITION__" + "/dept_id=__HIVE_DEFAULT_PARTITION__"))
+        .commit();
+
+    table.updateSpec().addField("team_id").commit();
+    table
+        .newFastAppend()
+        .appendFile(
+            newDataFile(
+                "company_id=__HIVE_DEFAULT_PARTITION__"

Review Comment:
   no reason, I followed the style used in this test class: https://github.com/apache/iceberg/pull/12528/files#diff-31a7139557d5a20af8245ddf3f9a4ff125094cd5d4f50b19e08104fecbabf98dR64-R66



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and null partition values [iceberg]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #12528:
URL: https://github.com/apache/iceberg/pull/12528#discussion_r1996252807


##########
api/src/main/java/org/apache/iceberg/types/Comparators.java:
##########
@@ -108,6 +109,15 @@ public int compare(StructLike o1, StructLike o2) {
         return 0;
       }
 
+      if (o1 instanceof StructProjection && o2 instanceof StructProjection) {

Review Comment:
   Updated the PR to compare the number of projected fields.
   
   The issue is that 2 projections with null and missing field values are considered equals. That leads to incorrect PartitionMap construction: 
   
   Below PartitionProjections are equal, so in PartitionMap we'll have just 1 random entry:
   ````
   "ice_orc(company_id=null)"
   "ice_orc(company_id=null/dept_id=null)"
   "ice_orc(company_id=null/dept_id=null/team_id=null)"
   ````



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and NULL partition values [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12528:
URL: https://github.com/apache/iceberg/pull/12528#discussion_r2131959732


##########
core/src/main/java/org/apache/iceberg/PartitionsTable.java:
##########
@@ -290,27 +297,25 @@ void update(ContentFile<?> file, Snapshot snapshot) {
       if (snapshot != null) {
         long snapshotCommitTime = snapshot.timestampMillis() * 1000;
         if (this.lastUpdatedAt == null || snapshotCommitTime > this.lastUpdatedAt) {
+          this.specId = file.specId();
+
           this.lastUpdatedAt = snapshotCommitTime;
           this.lastUpdatedSnapshotId = snapshot.snapshotId();
         }
       }
-

Review Comment:
   please revert this deletion



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and NULL partition values [iceberg]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #12528:
URL: https://github.com/apache/iceberg/pull/12528#discussion_r2131976899


##########
core/src/test/java/org/apache/iceberg/TestTables.java:
##########
@@ -59,10 +59,22 @@ public static TestTable create(
       PartitionSpec spec,
       SortOrder sortOrder,
       int formatVersion) {
+

Review Comment:
   removed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and NULL partition values [iceberg]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #12528:
URL: https://github.com/apache/iceberg/pull/12528#issuecomment-2985483024

   Thank you, @pvary and @RussellSpitzer!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and NULL partition values [iceberg]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #12528:
URL: https://github.com/apache/iceberg/pull/12528#discussion_r1999234522


##########
core/src/test/java/org/apache/iceberg/TestMetadataTableScansWithPartitionEvolution.java:
##########
@@ -243,6 +245,102 @@ public void testPartitionSpecEvolutionToUnpartitioned() throws IOException {
     }
   }
 
+  @TestTemplate
+  public void testPartitionSpecEvolutionNullValues() throws IOException {
+    Schema schema =
+        new Schema(
+            required(1, "company_id", Types.IntegerType.get()),
+            required(2, "dept_id", Types.IntegerType.get()),
+            required(3, "team_id", Types.IntegerType.get()));
+
+    table =
+        TestTables.create(
+            tableDir,
+            metadataDir,
+            "nulltest",
+            schema,
+            PartitionSpec.builderFor(schema).identity("company_id").build(),
+            SortOrder.unsorted(),
+            formatVersion);
+    table.newFastAppend().appendFile(newDataFile("company_id=__HIVE_DEFAULT_PARTITION__")).commit();
+
+    table.updateSpec().addField("dept_id").commit();
+    table
+        .newFastAppend()
+        .appendFile(
+            newDataFile(
+                "company_id=__HIVE_DEFAULT_PARTITION__" + "/dept_id=__HIVE_DEFAULT_PARTITION__"))
+        .commit();
+
+    table.updateSpec().addField("team_id").commit();
+    table
+        .newFastAppend()
+        .appendFile(
+            newDataFile(
+                "company_id=__HIVE_DEFAULT_PARTITION__"

Review Comment:
   no reason, I followed the style used in this test class: 
   https://github.com/apache/iceberg/pull/12528/files#diff-31a7139557d5a20af8245ddf3f9a4ff125094cd5d4f50b19e08104fecbabf98dR64-R66



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and null partition values [iceberg]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #12528:
URL: https://github.com/apache/iceberg/pull/12528#discussion_r1996708396


##########
api/src/main/java/org/apache/iceberg/types/Comparators.java:
##########
@@ -108,6 +109,15 @@ public int compare(StructLike o1, StructLike o2) {
         return 0;
       }
 
+      if (o1 instanceof StructProjection && o2 instanceof StructProjection) {

Review Comment:
   @RussellSpitzer as per your recommendation, created a localized fix for PartitionsTable. Please take a look. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and NULL partition values [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #12528:
URL: https://github.com/apache/iceberg/pull/12528#issuecomment-2985237855

   Since there were no new comments in the last 2 weeks, I have merged the change.
   Thanks @deniskuzZ for the PR and @RussellSpitzer for the review!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and null partition values [iceberg]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #12528:
URL: https://github.com/apache/iceberg/pull/12528#discussion_r1996398329


##########
api/src/main/java/org/apache/iceberg/types/Comparators.java:
##########
@@ -108,6 +109,15 @@ public int compare(StructLike o1, StructLike o2) {
         return 0;
       }
 
+      if (o1 instanceof StructProjection && o2 instanceof StructProjection) {

Review Comment:
   an alternative solution could be replacing `NULL` values in a struct with `__HIVE_DEFAULT_PARTITION__`
   ````
   StructLike partitionKey = newPartitionKey(file.partition(), "__HIVE_DEFAULT_PARTITION__");
   StructLike partition = PartitionUtil.coercePartition(
       partitionType, table.specs().get(file.specId()), file.partition());
   partitions.get(partition).update(file, snapshot);
   ````
   however, I don't think it's a better approach since this hack should be applied in multiple places and could potentially lead to new issues. 
   
   I believe that PartitionProjections containing VOID and NULL fields should not be considered equal. If all tests pass under the assumption that they should be, this assumption may not be accurate.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and NULL partition values [iceberg]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #12528:
URL: https://github.com/apache/iceberg/pull/12528#discussion_r2131824235


##########
core/src/main/java/org/apache/iceberg/PartitionsTable.java:
##########
@@ -290,27 +297,25 @@ void update(ContentFile<?> file, Snapshot snapshot) {
       if (snapshot != null) {
         long snapshotCommitTime = snapshot.timestampMillis() * 1000;
         if (this.lastUpdatedAt == null || snapshotCommitTime > this.lastUpdatedAt) {
+          this.specId = file.specId();
+

Review Comment:
   thanks for the review, @pvary! Need to check cause it doesn't seem right.  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and NULL partition values [iceberg]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #12528:
URL: https://github.com/apache/iceberg/pull/12528#issuecomment-2765912091

   hi @RussellSpitzer, do you know if anyone else could assist with the review? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and NULL partition values [iceberg]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #12528:
URL: https://github.com/apache/iceberg/pull/12528#discussion_r1999259243


##########
core/src/test/java/org/apache/iceberg/TestMetadataTableScansWithPartitionEvolution.java:
##########
@@ -243,6 +245,102 @@ public void testPartitionSpecEvolutionToUnpartitioned() throws IOException {
     }
   }
 
+  @TestTemplate
+  public void testPartitionSpecEvolutionNullValues() throws IOException {
+    Schema schema =
+        new Schema(
+            required(1, "company_id", Types.IntegerType.get()),
+            required(2, "dept_id", Types.IntegerType.get()),
+            required(3, "team_id", Types.IntegerType.get()));
+
+    table =
+        TestTables.create(
+            tableDir,
+            metadataDir,
+            "nulltest",
+            schema,
+            PartitionSpec.builderFor(schema).identity("company_id").build(),
+            SortOrder.unsorted(),
+            formatVersion);
+    table.newFastAppend().appendFile(newDataFile("company_id=__HIVE_DEFAULT_PARTITION__")).commit();
+
+    table.updateSpec().addField("dept_id").commit();
+    table
+        .newFastAppend()
+        .appendFile(
+            newDataFile(
+                "company_id=__HIVE_DEFAULT_PARTITION__" + "/dept_id=__HIVE_DEFAULT_PARTITION__"))
+        .commit();
+
+    table.updateSpec().addField("team_id").commit();
+    table
+        .newFastAppend()
+        .appendFile(
+            newDataFile(
+                "company_id=__HIVE_DEFAULT_PARTITION__"

Review Comment:
   note, this test class extends from TestBase that doesn't support `StructLike partition`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and null partition values [iceberg]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #12528:
URL: https://github.com/apache/iceberg/pull/12528#discussion_r1996398329


##########
api/src/main/java/org/apache/iceberg/types/Comparators.java:
##########
@@ -108,6 +109,15 @@ public int compare(StructLike o1, StructLike o2) {
         return 0;
       }
 
+      if (o1 instanceof StructProjection && o2 instanceof StructProjection) {

Review Comment:
   an alternative solution could be replacing `NULL` values in a struct with `__HIVE_DEFAULT_PARTITION__`
   ````
   StructLike partitionKey = newPartitionKey(file.partition(), "__HIVE_DEFAULT_PARTITION__");
   StructLike partition = PartitionUtil.coercePartition(
       partitionType, table.specs().get(file.specId()), file.partition());
   partitions.get(partition).update(file, snapshot);
   ````
   however, I don't think it's a better approach since this hack should be applied in multiple places and could potentially lead to new issues. 
   
   I believe that PartitionProjections with NON_EXISTING and NULL fields should not be considered equal. If all tests pass under the assumption that they should be, this assumption may not be accurate.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and NULL partition values [iceberg]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #12528:
URL: https://github.com/apache/iceberg/pull/12528#discussion_r1999259243


##########
core/src/test/java/org/apache/iceberg/TestMetadataTableScansWithPartitionEvolution.java:
##########
@@ -243,6 +245,102 @@ public void testPartitionSpecEvolutionToUnpartitioned() throws IOException {
     }
   }
 
+  @TestTemplate
+  public void testPartitionSpecEvolutionNullValues() throws IOException {
+    Schema schema =
+        new Schema(
+            required(1, "company_id", Types.IntegerType.get()),
+            required(2, "dept_id", Types.IntegerType.get()),
+            required(3, "team_id", Types.IntegerType.get()));
+
+    table =
+        TestTables.create(
+            tableDir,
+            metadataDir,
+            "nulltest",
+            schema,
+            PartitionSpec.builderFor(schema).identity("company_id").build(),
+            SortOrder.unsorted(),
+            formatVersion);
+    table.newFastAppend().appendFile(newDataFile("company_id=__HIVE_DEFAULT_PARTITION__")).commit();
+
+    table.updateSpec().addField("dept_id").commit();
+    table
+        .newFastAppend()
+        .appendFile(
+            newDataFile(
+                "company_id=__HIVE_DEFAULT_PARTITION__" + "/dept_id=__HIVE_DEFAULT_PARTITION__"))
+        .commit();
+
+    table.updateSpec().addField("team_id").commit();
+    table
+        .newFastAppend()
+        .appendFile(
+            newDataFile(
+                "company_id=__HIVE_DEFAULT_PARTITION__"

Review Comment:
   note, this test class uses TestTables.TestTable wrapper that doesn't support `StructLike partition`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and NULL partition values [iceberg]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #12528:
URL: https://github.com/apache/iceberg/pull/12528#issuecomment-2854332139

   hi @pvary, could you please help move this forward? Thank you! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and NULL partition values [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12528:
URL: https://github.com/apache/iceberg/pull/12528#discussion_r2131963783


##########
core/src/main/java/org/apache/iceberg/util/StructLikeMap.java:
##########
@@ -20,28 +20,36 @@
 
 import java.util.AbstractMap;
 import java.util.Collection;
+import java.util.Comparator;
 import java.util.Map;
 import java.util.Set;
 import java.util.function.Function;
+import java.util.function.Supplier;
 import org.apache.iceberg.StructLike;
 import org.apache.iceberg.relocated.com.google.common.collect.Maps;
 import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Comparators;
 import org.apache.iceberg.types.Types;
 
 public class StructLikeMap<T> extends AbstractMap<StructLike, T> implements Map<StructLike, T> {
 
+  public static <T> StructLikeMap<T> create(

Review Comment:
   Add javadoc please



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and NULL partition values [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12528:
URL: https://github.com/apache/iceberg/pull/12528#discussion_r2095707435


##########
core/src/test/java/org/apache/iceberg/TestTables.java:
##########
@@ -118,9 +131,17 @@ private static TestTable createTable(
       throw new AlreadyExistsException("Table %s already exists at location: %s", name, temp);
     }
 
-    ops.commit(
-        null,
-        newTableMetadata(schema, spec, sortOrder, temp.toString(), properties, formatVersion));
+    TableMetadata metadata =
+        newTableMetadata(schema, spec, sortOrder, temp.toString(), properties, formatVersion);
+
+    if (metaTemp != null) {
+      metadata =
+          TableMetadata.buildFrom(metadata)
+              .discardChanges()
+              .withMetadataLocation(metaTemp.toString())
+              .build();
+    }
+    ops.commit(null, metadata);

Review Comment:
   nit newline



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and NULL partition values [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #12528:
URL: https://github.com/apache/iceberg/pull/12528#discussion_r2131973253


##########
core/src/test/java/org/apache/iceberg/TestTables.java:
##########
@@ -59,10 +59,22 @@ public static TestTable create(
       PartitionSpec spec,
       SortOrder sortOrder,
       int formatVersion) {
+

Review Comment:
   nit: why is this newline?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and NULL partition values [iceberg]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #12528:
URL: https://github.com/apache/iceberg/pull/12528#discussion_r2131824235


##########
core/src/main/java/org/apache/iceberg/PartitionsTable.java:
##########
@@ -290,27 +297,25 @@ void update(ContentFile<?> file, Snapshot snapshot) {
       if (snapshot != null) {
         long snapshotCommitTime = snapshot.timestampMillis() * 1000;
         if (this.lastUpdatedAt == null || snapshotCommitTime > this.lastUpdatedAt) {
+          this.specId = file.specId();
+

Review Comment:
   thanks for the review, @pvary! That is required to pick the latest specId in case or rename.
   
   see `TestMetadataTableScansWithPartitionEvolution#testPartitionSpecRenameFields()`
   
   ````
   specId=0
   PartitionSpec.builderFor(schema).identity("data").identity("category").build(),
   ````
   execute partCol rename
   ````
   specId=1
   table.updateSpec().renameField("category", "category_another_name").commit();
   ````
   assertPartitions("data=c1/category_another_name=d1", "data=c2/category_another_name=d2");



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and NULL partition values [iceberg]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #12528:
URL: https://github.com/apache/iceberg/pull/12528#discussion_r2131824235


##########
core/src/main/java/org/apache/iceberg/PartitionsTable.java:
##########
@@ -290,27 +297,25 @@ void update(ContentFile<?> file, Snapshot snapshot) {
       if (snapshot != null) {
         long snapshotCommitTime = snapshot.timestampMillis() * 1000;
         if (this.lastUpdatedAt == null || snapshotCommitTime > this.lastUpdatedAt) {
+          this.specId = file.specId();
+

Review Comment:
   thanks for the review, @pvary! That is required to pick the latest specId in case or rename.
   
   see TestMetadataTableScansWithPartitionEvolution#testPartitionSpecRenameFields()
   
   ````
   specId=0
   PartitionSpec.builderFor(schema).identity("data").identity("category").build(),
   ````
   execute partCol rename
   ````
   specId=1
   table.updateSpec().renameField("category", "category_another_name").commit();
   ````
   assertPartitions("data=c1/category_another_name=d1", "data=c2/category_another_name=d2");



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and null partition values [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #12528:
URL: https://github.com/apache/iceberg/pull/12528#discussion_r1996345959


##########
api/src/main/java/org/apache/iceberg/types/Comparators.java:
##########
@@ -108,6 +109,15 @@ public int compare(StructLike o1, StructLike o2) {
         return 0;
       }
 
+      if (o1 instanceof StructProjection && o2 instanceof StructProjection) {

Review Comment:
   I think this is going to break other things, because most other places in the code we do want those partitions to be equal



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and NULL partition values [iceberg]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #12528:
URL: https://github.com/apache/iceberg/pull/12528#discussion_r1996398329


##########
api/src/main/java/org/apache/iceberg/types/Comparators.java:
##########
@@ -108,6 +109,15 @@ public int compare(StructLike o1, StructLike o2) {
         return 0;
       }
 
+      if (o1 instanceof StructProjection && o2 instanceof StructProjection) {

Review Comment:
   an alternative solution could be replacing `NULL` values in a struct with `__HIVE_DEFAULT_PARTITION__`
   ````
   StructLike partitionKey = newPartitionKey(file.partition(), "__HIVE_DEFAULT_PARTITION__");
   StructLike partition = PartitionUtil.coercePartition(
       partitionType, table.specs().get(file.specId()), file.partition());
   partitions.get(partition).update(file, snapshot);
   ````
   however, I don't think it's a better approach since this hack should be applied in multiple places and could potentially lead to new issues. 
   
   I believe that PartitionProjections with NON_EXISTING and NULL fields should not be considered equal. If all tests pass under the assumption that they should be, this assumption may not be accurate.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and NULL partition values [iceberg]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #12528:
URL: https://github.com/apache/iceberg/pull/12528#discussion_r2131863748


##########
core/src/main/java/org/apache/iceberg/PartitionsTable.java:
##########
@@ -290,27 +297,25 @@ void update(ContentFile<?> file, Snapshot snapshot) {
       if (snapshot != null) {
         long snapshotCommitTime = snapshot.timestampMillis() * 1000;
         if (this.lastUpdatedAt == null || snapshotCommitTime > this.lastUpdatedAt) {
+          this.specId = file.specId();
+

Review Comment:
   without this fix master branch code returns incorrect result. 
   ````
   "data=c1/category=d1", "data=c2/category=d2"
   ````



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and NULL partition values [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary merged PR #12528:
URL: https://github.com/apache/iceberg/pull/12528


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and NULL partition values [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on PR #12528:
URL: https://github.com/apache/iceberg/pull/12528#issuecomment-2730063569

   @szehon-ho Adding you since you were in this code most recently


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and NULL partition values [iceberg]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #12528:
URL: https://github.com/apache/iceberg/pull/12528#discussion_r1999329563


##########
core/src/test/java/org/apache/iceberg/TestMetadataTableScansWithPartitionEvolution.java:
##########
@@ -243,6 +245,102 @@ public void testPartitionSpecEvolutionToUnpartitioned() throws IOException {
     }
   }
 
+  @TestTemplate
+  public void testPartitionSpecEvolutionNullValues() throws IOException {
+    Schema schema =
+        new Schema(
+            required(1, "company_id", Types.IntegerType.get()),
+            required(2, "dept_id", Types.IntegerType.get()),
+            required(3, "team_id", Types.IntegerType.get()));
+
+    table =
+        TestTables.create(
+            tableDir,
+            metadataDir,
+            "nulltest",
+            schema,
+            PartitionSpec.builderFor(schema).identity("company_id").build(),
+            SortOrder.unsorted(),
+            formatVersion);
+    table.newFastAppend().appendFile(newDataFile("company_id=__HIVE_DEFAULT_PARTITION__")).commit();
+
+    table.updateSpec().addField("dept_id").commit();
+    table
+        .newFastAppend()
+        .appendFile(
+            newDataFile(
+                "company_id=__HIVE_DEFAULT_PARTITION__" + "/dept_id=__HIVE_DEFAULT_PARTITION__"))
+        .commit();
+
+    table.updateSpec().addField("team_id").commit();
+    table
+        .newFastAppend()
+        .appendFile(
+            newDataFile(
+                "company_id=__HIVE_DEFAULT_PARTITION__"

Review Comment:
   refactored to use struct



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and null partition values [iceberg]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #12528:
URL: https://github.com/apache/iceberg/pull/12528#discussion_r1996343029


##########
api/src/main/java/org/apache/iceberg/types/Comparators.java:
##########
@@ -108,6 +109,15 @@ public int compare(StructLike o1, StructLike o2) {
         return 0;
       }
 
+      if (o1 instanceof StructProjection && o2 instanceof StructProjection) {

Review Comment:
   added test: `TestMetadataTableScansWithPartitionEvolution#testPartitionSpecEvolutionNullValues`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and null partition values [iceberg]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #12528:
URL: https://github.com/apache/iceberg/pull/12528#discussion_r1996347542


##########
api/src/main/java/org/apache/iceberg/types/Comparators.java:
##########
@@ -108,6 +109,15 @@ public int compare(StructLike o1, StructLike o2) {
         return 0;
       }
 
+      if (o1 instanceof StructProjection && o2 instanceof StructProjection) {

Review Comment:
   let's see, but right now that leads to invalid partition list evaluation



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: PartitionsTable#partitions returns incomplete list in case of partition evolution and NULL partition values [iceberg]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #12528:
URL: https://github.com/apache/iceberg/pull/12528#issuecomment-2742973079

   hi @szehon-ho, could you please take a look? thank you! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org