[GitHub] [iceberg] rdblue commented on a change in pull request #3411: Fixes read metadata failed after dropped partition

[GitBox <gi...@apache.org>]

rdblue commented on a change in pull request #3411:
URL: https://github.com/apache/iceberg/pull/3411#discussion_r823186537



##########
File path: core/src/main/java/org/apache/iceberg/Partitioning.java
##########
@@ -210,7 +211,8 @@ public static StructType partitionType(Table table) {
     }
 
     Map<Integer, PartitionField> fieldMap = Maps.newHashMap();
-    List<NestedField> structFields = Lists.newArrayList();
+    Map<Integer, Type> typeMap = Maps.newHashMap();
+    Map<Integer, String> nameMap = Maps.newHashMap();

Review comment:
       I think we have two cases to solve this problem. The combined partition struct type is used in both metadata tables and when we need to project the _partition field for certain queries. For queries that use the _partition field, I think adding a suffix to make the names unique is the right approach. That's internal so it doesn't really matter what we rename to, as long as we get the projections that produce the partition tuple for a given spec right.
   
   For metadata tables, we expect the field names to match the partition names. A simple example is that we use identity partitions using the original column name. So it would be weird to partition by `category` and need to query `category_r1003` in the metadata table. I think that the right thing to do for metadata tables is to have a separate way to produce the combined struct that produces only one partition column per name.
   
   In @aokolnychyi's example, `1001: data` should be present and `1000: data` should not be used for metadata tables.
   
   This is also an area where we may want to bring back compatible partition columns. There's no reason why Anton's example couldn't detect that `1000: data` was in an old spec and reuse the ID to avoid this problem.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org