You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "cashmand (via GitHub)" <gi...@apache.org> on 2024/12/18 20:52:36 UTC
Re: [PR] [SPARK-50614][SQL] Add Variant shredding support for Parquet [spark]
cashmand commented on code in PR #49234:
URL: https://github.com/apache/spark/pull/49234#discussion_r1890821529
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetWriteSupport.scala:
##########
@@ -95,7 +100,19 @@ class ParquetWriteSupport extends WriteSupport[InternalRow] with Logging {
override def init(configuration: Configuration): WriteContext = {
val schemaString = configuration.get(ParquetWriteSupport.SPARK_ROW_SCHEMA)
+ val shreddedSchemaString = configuration.get(ParquetWriteSupport.SPARK_VARIANT_SHREDDING_SCHEMA)
this.schema = StructType.fromString(schemaString)
+ // If shreddingSchemaString is provided, we use that everywhere in the writer, except for
+ // setting the spark schema in the Parquet metadata. If it isn't provided, it means that there
+ // are no shredded Variant columns, so it is identical to this.schema.
+ this.shreddedSchema = if (shreddedSchemaString == null) {
+ this.schema
+ } else {
+ val v = StructType.fromString(shreddedSchemaString)
+ // A bit awkwardly, the schema string doesn't include metadata to identify which struct
Review Comment:
I'm open to suggestions, but from reading the code, it didn't look like the metadata could be passed through the DDL representation, and I couldn't think of an alternative approach to identify the shredding for each Variant field. Technically, for this PR I guess I could have gotten away with constructing the shredded schema in this code directly from the SQLConf, but in the future, I think we'll want to do it based on table properties and/or buffered data in the task, and I don't think it would make sense for that to go here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org