You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "cashmand (via GitHub)" <gi...@apache.org> on 2024/12/18 20:23:50 UTC

[PR] Add shredding support [spark]

cashmand opened a new pull request, #49234:
URL: https://github.com/apache/spark/pull/49234

   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
     2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
     3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
     4. Be sure to keep the PR description updated to reflect all changes.
     5. Please write your PR title to summarize what this PR proposes.
     6. If possible, provide a concise example to reproduce the issue for a faster review.
     7. If you want to add a new configuration, please read the guideline first for naming configurations in
        'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
     8. If you want to add or modify an error type or message, please read the guideline first in
        'common/utils/src/main/resources/error/README.md'.
   -->
   
   ### What changes were proposed in this pull request?
   
   Adds support for shredding in the Parquet writer code. Currently, the only way to enable shredding is through a SQLConf that provides the schema to use for shredding. This doesn't make sense as a user API, and is added only for testing. The exact API for Spark to determine a shredding schema is still TBD, but likely candidates are to infer it at the task level by inspecting the first few rows of data, or add an API to specify the schema for a given column. Either way, the code in this PR would basically be unchanged, it would just use a different mechanism to provide the schema.
   
   ### Why are the changes needed?
   
   Needed for Variant shredding support.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No, the feature is new in Spark 4.0, and is currently disabled, and only usable as a test feature.
   
   ### How was this patch tested?
   
   Added a unit test suite.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   No.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50614][SQL] Add Variant shredding support for Parquet [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #49234:
URL: https://github.com/apache/spark/pull/49234#discussion_r1896372350


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -4644,6 +4644,22 @@ object SQLConf {
       .booleanConf
       .createWithDefault(false)
 
+  val VARIANT_WRITE_SHREDDING_ENABLED =
+    buildConf("spark.sql.variant.writeShredding.enabled")
+      .internal()
+      .doc("When true, the Parquet reader is allowed to write shredded variant. ")

Review Comment:
   It's ok to have a kill switch.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50614][SQL] Add Variant shredding support for Parquet [spark]

Posted by "cashmand (via GitHub)" <gi...@apache.org>.
cashmand commented on code in PR #49234:
URL: https://github.com/apache/spark/pull/49234#discussion_r1899610920


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetUtils.scala:
##########
@@ -420,6 +420,22 @@ object ParquetUtils extends Logging {
     statistics.getNumNulls;
   }
 
+  // Replaces each VariantType in the schema with the corresponding type in the shredding schema.
+  // Used for testing, where we force a single shredding schema for all Variant fields.
+  // Does not touch Variant fields nested in arrays, maps, or UDTs.
+  private def replaceVariantTypes(schema: StructType, shreddingSchema: StructType): StructType = {
+    val newFields = schema.fields.zip(shreddingSchema.fields).map {
+      case (field, shreddingField) =>
+        field.dataType match {
+          case s: StructType =>
+            field.copy(dataType = replaceVariantTypes(s, shreddingSchema))

Review Comment:
   Hi @Zouxxyy, I tend to agree with @cloud-fan's comment. My preference would be to stick with the current approach for now. I think it should be entirely contained within the Parquet writer code, so if we decide to extend the VariantType later, I don't think it would be hard to change this part of the code to use a different approach.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50614][SQL] Add Variant shredding support for Parquet [spark]

Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on code in PR #49234:
URL: https://github.com/apache/spark/pull/49234#discussion_r1897070048


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetUtils.scala:
##########
@@ -420,6 +420,22 @@ object ParquetUtils extends Logging {
     statistics.getNumNulls;
   }
 
+  // Replaces each VariantType in the schema with the corresponding type in the shredding schema.
+  // Used for testing, where we force a single shredding schema for all Variant fields.
+  // Does not touch Variant fields nested in arrays, maps, or UDTs.
+  private def replaceVariantTypes(schema: StructType, shreddingSchema: StructType): StructType = {
+    val newFields = schema.fields.zip(shreddingSchema.fields).map {
+      case (field, shreddingField) =>
+        field.dataType match {
+          case s: StructType =>
+            field.copy(dataType = replaceVariantTypes(s, shreddingSchema))

Review Comment:
   Have you considered adding a property like `shreddedSchema` in `VariantType`? This way, we won't need to determine how this struct type was from. Currently, it seems that this is judged through __VARIANT_WRITE_SHREDDING_KEY."



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50614][SQL] Add Variant shredding support for Parquet [spark]

Posted by "cashmand (via GitHub)" <gi...@apache.org>.
cashmand commented on code in PR #49234:
URL: https://github.com/apache/spark/pull/49234#discussion_r1896803568


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetWriteSupport.scala:
##########
@@ -95,7 +100,19 @@ class ParquetWriteSupport extends WriteSupport[InternalRow] with Logging {
 
   override def init(configuration: Configuration): WriteContext = {
     val schemaString = configuration.get(ParquetWriteSupport.SPARK_ROW_SCHEMA)
+    val shreddedSchemaString = configuration.get(ParquetWriteSupport.SPARK_VARIANT_SHREDDING_SCHEMA)
     this.schema = StructType.fromString(schemaString)
+    // If shreddingSchemaString is provided, we use that everywhere in the writer, except for
+    // setting the spark schema in the Parquet metadata. If it isn't provided, it means that there
+    // are no shredded Variant columns, so it is identical to this.schema.
+    this.shreddedSchema = if (shreddedSchemaString == null) {
+      this.schema
+    } else {
+      val v = StructType.fromString(shreddedSchemaString)
+      // A bit awkwardly, the schema string doesn't include metadata to identify which struct

Review Comment:
   I updated the PR to clean this up. I also moved the code to add/check the metadata into `SparkShreddingUtils`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50614][SQL] Add Variant shredding support for Parquet [spark]

Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #49234:
URL: https://github.com/apache/spark/pull/49234#issuecomment-3258858285

   @cashmand  Hi, does Spark fully support shredding now, and will the variant schema be built automatically?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50614][SQL] Add Variant shredding support for Parquet [spark]

Posted by "cashmand (via GitHub)" <gi...@apache.org>.
cashmand commented on PR #49234:
URL: https://github.com/apache/spark/pull/49234#issuecomment-3258996970

   > @cashmand Hi, does Spark fully support shredding now, and will the variant schema be built automatically?
   
   Hi @eldenmoon, I believe the shredding read and write support is compliant with the spec, but currently the only way to actually write shredded data is using the `spark.sql.variant.forceShreddingSchemaForTest` conf added in this PR, so we still need a mechanism to provide a shredding schema to the parquet writer outside of tests.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50614][SQL] Add Variant shredding support for Parquet [spark]

Posted by "cashmand (via GitHub)" <gi...@apache.org>.
cashmand commented on code in PR #49234:
URL: https://github.com/apache/spark/pull/49234#discussion_r1894360846


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -4644,6 +4644,22 @@ object SQLConf {
       .booleanConf
       .createWithDefault(false)
 
+  val VARIANT_WRITE_SHREDDING_ENABLED =
+    buildConf("spark.sql.variant.writeShredding.enabled")
+      .internal()
+      .doc("When true, the Parquet reader is allowed to write shredded variant. ")

Review Comment:
   The intent of this conf is to be a global kill switch for the write shredding feature, and that we'll pass a write-specific conf to actually control how we shred - e.g. have the shredding determined by sampling data in the task, or specify a shredding schema for each Variant column.
   
   Right now, the conf only has an effect if `VARIANT_FORCE_SHREDDING_SCHEMA_FOR_TEST` is also set to a non-empty value. I could remove it if we don't think there's any value to having a global kill switch like this. @cloud-fan, any opinion?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50614][SQL] Add Variant shredding support for Parquet [spark]

Posted by "cashmand (via GitHub)" <gi...@apache.org>.
cashmand commented on code in PR #49234:
URL: https://github.com/apache/spark/pull/49234#discussion_r1890821529


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetWriteSupport.scala:
##########
@@ -95,7 +100,19 @@ class ParquetWriteSupport extends WriteSupport[InternalRow] with Logging {
 
   override def init(configuration: Configuration): WriteContext = {
     val schemaString = configuration.get(ParquetWriteSupport.SPARK_ROW_SCHEMA)
+    val shreddedSchemaString = configuration.get(ParquetWriteSupport.SPARK_VARIANT_SHREDDING_SCHEMA)
     this.schema = StructType.fromString(schemaString)
+    // If shreddingSchemaString is provided, we use that everywhere in the writer, except for
+    // setting the spark schema in the Parquet metadata. If it isn't provided, it means that there
+    // are no shredded Variant columns, so it is identical to this.schema.
+    this.shreddedSchema = if (shreddedSchemaString == null) {
+      this.schema
+    } else {
+      val v = StructType.fromString(shreddedSchemaString)
+      // A bit awkwardly, the schema string doesn't include metadata to identify which struct

Review Comment:
   I'm open to suggestions, but from reading the code, it didn't look like the metadata could be passed through the DDL representation, and I couldn't think of an alternative approach to identify the shredding for each Variant field. Technically, for this PR I guess I could have gotten away with constructing the shredded schema in this code directly from the SQLConf, but in the future, I think we'll want to do it based on table properties and/or buffered data in the task, and I don't think it would make sense for that to go here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50614][SQL] Add Variant shredding support for Parquet [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #49234:
URL: https://github.com/apache/spark/pull/49234#discussion_r1898188783


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetUtils.scala:
##########
@@ -420,6 +420,22 @@ object ParquetUtils extends Logging {
     statistics.getNumNulls;
   }
 
+  // Replaces each VariantType in the schema with the corresponding type in the shredding schema.
+  // Used for testing, where we force a single shredding schema for all Variant fields.
+  // Does not touch Variant fields nested in arrays, maps, or UDTs.
+  private def replaceVariantTypes(schema: StructType, shreddingSchema: StructType): StructType = {
+    val newFields = schema.fields.zip(shreddingSchema.fields).map {
+      case (field, shreddingField) =>
+        field.dataType match {
+          case s: StructType =>
+            field.copy(dataType = replaceVariantTypes(s, shreddingSchema))

Review Comment:
   shredding only matters for read/write and I think it's weird to put the info in the `VariantType` which can also be consumed by SQL functions/operators. The current approach looks cleaner to me.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50614][SQL] Add Variant shredding support for Parquet [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #49234:
URL: https://github.com/apache/spark/pull/49234#issuecomment-2567279749

   thanks, merging to master!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50614][SQL] Add Variant shredding support for Parquet [spark]

Posted by "cashmand (via GitHub)" <gi...@apache.org>.
cashmand commented on code in PR #49234:
URL: https://github.com/apache/spark/pull/49234#discussion_r1896772605


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetWriteSupport.scala:
##########
@@ -95,7 +100,19 @@ class ParquetWriteSupport extends WriteSupport[InternalRow] with Logging {
 
   override def init(configuration: Configuration): WriteContext = {
     val schemaString = configuration.get(ParquetWriteSupport.SPARK_ROW_SCHEMA)
+    val shreddedSchemaString = configuration.get(ParquetWriteSupport.SPARK_VARIANT_SHREDDING_SCHEMA)
     this.schema = StructType.fromString(schemaString)
+    // If shreddingSchemaString is provided, we use that everywhere in the writer, except for
+    // setting the spark schema in the Parquet metadata. If it isn't provided, it means that there
+    // are no shredded Variant columns, so it is identical to this.schema.
+    this.shreddedSchema = if (shreddedSchemaString == null) {
+      this.schema
+    } else {
+      val v = StructType.fromString(shreddedSchemaString)
+      // A bit awkwardly, the schema string doesn't include metadata to identify which struct

Review Comment:
   Oh, thanks. I had looked at the `json` method in StructType, and forgot that the metadata is in StructField, not StructType, so I missed that it is writing the metadata there. Let me clean this up.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50614][SQL] Add Variant shredding support for Parquet [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #49234:
URL: https://github.com/apache/spark/pull/49234#discussion_r1896357397


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -4644,6 +4644,22 @@ object SQLConf {
       .booleanConf
       .createWithDefault(false)
 
+  val VARIANT_WRITE_SHREDDING_ENABLED =
+    buildConf("spark.sql.variant.writeShredding.enabled")
+      .internal()
+      .doc("When true, the Parquet reader is allowed to write shredded variant. ")

Review Comment:
   ```suggestion
         .doc("When true, the Parquet writer is allowed to write shredded variant. ")
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50614][SQL] Add Variant shredding support for Parquet [spark]

Posted by "gene-db (via GitHub)" <gi...@apache.org>.
gene-db commented on code in PR #49234:
URL: https://github.com/apache/spark/pull/49234#discussion_r1894318392


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -4644,6 +4644,22 @@ object SQLConf {
       .booleanConf
       .createWithDefault(false)
 
+  val VARIANT_WRITE_SHREDDING_ENABLED =
+    buildConf("spark.sql.variant.writeShredding.enabled")
+      .internal()
+      .doc("When true, the Parquet reader is allowed to write shredded variant. ")

Review Comment:
   Does this mean Delta is controlled some other way?



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala:
##########
@@ -772,6 +780,13 @@ private[sql] object ParquetSchemaConverter {
   val EMPTY_MESSAGE: MessageType =
     Types.buildMessage().named(ParquetSchemaConverter.SPARK_PARQUET_SCHEMA_NAME)
 
+  // Used to annotate as metadata on the struct that replaces a VariantType when shredding.
+  val VARIANT_WRITE_SHREDDING_KEY: String = "__VARIANT_WRITE_SHREDDING_KEY"
+
+  def isVariantShreddingStruct(s: StructType): Boolean = {
+    s.fields.length > 0 && s.fields.forall(_.metadata.contains(VARIANT_WRITE_SHREDDING_KEY))

Review Comment:
   Every field contains the shredding key? What happens only some have it?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50614][SQL] Add Variant shredding support for Parquet [spark]

Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on code in PR #49234:
URL: https://github.com/apache/spark/pull/49234#discussion_r1899915615


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetUtils.scala:
##########
@@ -420,6 +420,22 @@ object ParquetUtils extends Logging {
     statistics.getNumNulls;
   }
 
+  // Replaces each VariantType in the schema with the corresponding type in the shredding schema.
+  // Used for testing, where we force a single shredding schema for all Variant fields.
+  // Does not touch Variant fields nested in arrays, maps, or UDTs.
+  private def replaceVariantTypes(schema: StructType, shreddingSchema: StructType): StructType = {
+    val newFields = schema.fields.zip(shreddingSchema.fields).map {
+      case (field, shreddingField) =>
+        field.dataType match {
+          case s: StructType =>
+            field.copy(dataType = replaceVariantTypes(s, shreddingSchema))

Review Comment:
   Got it, It does look much more concise for now, looking forward to its completion!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50614][SQL] Add Variant shredding support for Parquet [spark]

Posted by "cashmand (via GitHub)" <gi...@apache.org>.
cashmand commented on code in PR #49234:
URL: https://github.com/apache/spark/pull/49234#discussion_r1894357728


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala:
##########
@@ -772,6 +780,13 @@ private[sql] object ParquetSchemaConverter {
   val EMPTY_MESSAGE: MessageType =
     Types.buildMessage().named(ParquetSchemaConverter.SPARK_PARQUET_SCHEMA_NAME)
 
+  // Used to annotate as metadata on the struct that replaces a VariantType when shredding.
+  val VARIANT_WRITE_SHREDDING_KEY: String = "__VARIANT_WRITE_SHREDDING_KEY"
+
+  def isVariantShreddingStruct(s: StructType): Boolean = {
+    s.fields.length > 0 && s.fields.forall(_.metadata.contains(VARIANT_WRITE_SHREDDING_KEY))

Review Comment:
   I put it on all or none in `updateSchemaForVariantShredding`, so that shouldn't really happen. We could check and fail here if there's an inconsistency. I could also just put it on the first field in the struct, if you prefer. I don't think there's any real need to put it on all of them.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50614][SQL] Add Variant shredding support for Parquet [spark]

Posted by "cashmand (via GitHub)" <gi...@apache.org>.
cashmand commented on code in PR #49234:
URL: https://github.com/apache/spark/pull/49234#discussion_r1897960988


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetUtils.scala:
##########
@@ -420,6 +420,22 @@ object ParquetUtils extends Logging {
     statistics.getNumNulls;
   }
 
+  // Replaces each VariantType in the schema with the corresponding type in the shredding schema.
+  // Used for testing, where we force a single shredding schema for all Variant fields.
+  // Does not touch Variant fields nested in arrays, maps, or UDTs.
+  private def replaceVariantTypes(schema: StructType, shreddingSchema: StructType): StructType = {
+    val newFields = schema.fields.zip(shreddingSchema.fields).map {
+      case (field, shreddingField) =>
+        field.dataType match {
+          case s: StructType =>
+            field.copy(dataType = replaceVariantTypes(s, shreddingSchema))

Review Comment:
   Hi @Zouxxyy, that's actually what I tried initially. I could go back to that approach if there's a consensus that it's better, but I felt that it had some downsides. Specifically, the approach I took was to add an optional `shreddingSchema` property to the VariantType class, but leave the `case object VariantType` to represent a VariantType with no shredding (similar to what's been done for StringType with default collation), to avoid having to modify all of the existing code that pattern matches to `VariantType`.
   
   I didn't think this approach added much value - it's not very clear what it means for the type to have a shredding schema outside of the specific context of the couple of writer classes that care about it, and it felt like it was just pushing writer-specific code into the core of the type system. E.g. when serializing the type to a string, should we include the shredding schema? I think the answer is that we'd want to when sending it to ParquetWriteSupport, but not when writing it to the parquet file (since it's not part of the Spark type when reading it back from Parquet). It seemed better to resolve these ambiguities explicitly in the relevant writer code, rather than making them part of the data type definition.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50614][SQL] Add Variant shredding support for Parquet [spark]

Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on code in PR #49234:
URL: https://github.com/apache/spark/pull/49234#discussion_r1897969339


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetUtils.scala:
##########
@@ -420,6 +420,22 @@ object ParquetUtils extends Logging {
     statistics.getNumNulls;
   }
 
+  // Replaces each VariantType in the schema with the corresponding type in the shredding schema.
+  // Used for testing, where we force a single shredding schema for all Variant fields.
+  // Does not touch Variant fields nested in arrays, maps, or UDTs.
+  private def replaceVariantTypes(schema: StructType, shreddingSchema: StructType): StructType = {
+    val newFields = schema.fields.zip(shreddingSchema.fields).map {
+      case (field, shreddingField) =>
+        field.dataType match {
+          case s: StructType =>
+            field.copy(dataType = replaceVariantTypes(s, shreddingSchema))

Review Comment:
   @cashmand Thank you for your reply. We have also implemented the shredding logic internally in Apache Paimon. Specifically, we use `VariantType` along with an optional `ShreddSchema`. In particular, on the write side, we calculate the `ShreddSchema` based on the table's shredding props and then set it into `VariantType`, which is passed to our writer (currently Parquet, but it could be ORC in the future, etc.).
   
   On the read side, we calculate the required schema based on the scan filter and projection, which is also written into the optional attribute of VariantType, and passed to our reader. Combined with the schema of the Parquet variant, we generate the trimmed read ShreddSchema. I think that shredding is an extension of the variant type; when our writer sees, oh, there exists a ShreddSchema, the writer immediately knows that shredding needs to be performed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50614][SQL] Add Variant shredding support for Parquet [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #49234:
URL: https://github.com/apache/spark/pull/49234#discussion_r1896373092


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetWriteSupport.scala:
##########
@@ -95,7 +100,19 @@ class ParquetWriteSupport extends WriteSupport[InternalRow] with Logging {
 
   override def init(configuration: Configuration): WriteContext = {
     val schemaString = configuration.get(ParquetWriteSupport.SPARK_ROW_SCHEMA)
+    val shreddedSchemaString = configuration.get(ParquetWriteSupport.SPARK_VARIANT_SHREDDING_SCHEMA)
     this.schema = StructType.fromString(schemaString)
+    // If shreddingSchemaString is provided, we use that everywhere in the writer, except for
+    // setting the spark schema in the Parquet metadata. If it isn't provided, it means that there
+    // are no shredded Variant columns, so it is identical to this.schema.
+    this.shreddedSchema = if (shreddedSchemaString == null) {
+      this.schema
+    } else {
+      val v = StructType.fromString(shreddedSchemaString)
+      // A bit awkwardly, the schema string doesn't include metadata to identify which struct

Review Comment:
   Shall we use data type JSON string?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50614][SQL] Add Variant shredding support for Parquet [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #49234: [SPARK-50614][SQL] Add Variant shredding support for Parquet
URL: https://github.com/apache/spark/pull/49234


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org