You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "aokolnychyi (via GitHub)" <gi...@apache.org> on 2023/07/13 19:25:08 UTC
[GitHub] [iceberg] aokolnychyi commented on a diff in pull request #8059: Spark 3.4: Fix rewrite_position_deletes for certain partition types
aokolnychyi commented on code in PR #8059:
URL: https://github.com/apache/iceberg/pull/8059#discussion_r1262972615
##########
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/actions/SparkBinPackPositionDeletesRewriter.java:
##########
@@ -138,4 +142,30 @@ private Dataset<Row> dataFiles(Types.StructType partitionType, StructLike partit
return SparkTableUtil.loadMetadataTable(spark, table(), MetadataTableType.DATA_FILES);
}
}
+
+ /**
+ * If necessary, convert the partition data (from avro) to its logical type, to use in the spark
+ * queries. See https://iceberg.apache.org/spec/#avro.
+ *
+ * @param value partition object
+ * @param typeId iceberg type
+ * @return object converted to its logical type
+ */
+ private Object convertPartitionValue(Object value, Type.TypeID typeId) {
+ switch (typeId) {
Review Comment:
I understand there is a type mismatch but don't we have to convert internal Iceberg values for a given type to a Spark value? Why Avro?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org