[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #5665: Core: Correctly project the partition fields

[GitBox <gi...@apache.org>]

aokolnychyi commented on code in PR #5665:
URL: https://github.com/apache/iceberg/pull/5665#discussion_r960123754


##########
spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/SparkDataFile.java:
##########
@@ -84,10 +93,31 @@ public SparkDataFile(Types.StructType type, StructType sparkType) {
     sortOrderIdPosition = positions.get("sort_order_id");
   }
 
+  private void wrapPartitionSpec(GenericRowWithSchema specRow) {
+    // We get all the partition fields, but want to project to the current one
+    StructType wrappedPartitionStruct = specRow.schema();
+
+    if (!wrappedPartitionStruct.equals(currentWrappedPartitionStruct)) {
+      this.currentWrappedPartitionStruct = wrappedPartitionStruct;
+
+      // The original IDs are lost in translation, therefore we apply the ones that we know

Review Comment:
   Or we could handle the projection before even wrapping `Row` into `SparkDataFile`. We will need to try out these options and see what makes more sense.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org