You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "zhztheplayer (via GitHub)" <gi...@apache.org> on 2025/06/16 17:46:07 UTC

[PR] [SPARK-52492][SQL] Make InMemoryRelation.convertToColumnarIfPossible customizable [spark]

zhztheplayer opened a new pull request, #51189:
URL: https://github.com/apache/spark/pull/51189

   https://issues.apache.org/jira/browse/SPARK-52492?filter=-6&jql=project%20%3D%20SPARK%20and%20created%20%3E%3D%20-1w%20order%20by%20created%20DESC
   
   InMemoryRelation.convertToColumnarIfPossible is highly coupled with vanilla Spark's columnar processing logic. It unwraps the input columnar plan by removing the topmost ColumnarToRowExec, the assumes that the outcome RDD after this process can be recognized by the user-customized cache serializer.
   
   But sometimes this assertion is invalid. As in the Apache Gluten project, we may continue distiguishing plans that are all have `supportsColumnar=true` with different columnar batch types. So even the topmost `ColumnarToRowExec` is removed, we still don't know whether the columnar RDD unwrapped can be accepted by Gluten's cache serializer (assuming it only handles one certain type of columnar batch or something).
   
   So in Gluten we had a rule to workaround the logic in InMemoryRelation.convertToColumnarIfPossible: https://github.com/apache/incubator-gluten/blob/c6461b4e0c7d3022a31fa832aeab588b1a3200e6/gluten-substrait/src/main/scala/org/apache/gluten/extension/columnar/MiscColumnarRules.scala#L192-L217. This is the best way we had thought about to get through the issue but it's still not elegant, especially the rule is even caller-sensitive to determine whether it's called in the caching planning process or not.
   
   This PR moves `InMemoryRelation.convertToColumnarIfPossible` to as a public API of `CachedBatchSerializer` so that plugins like Gluten could have the relevant logic customized for their own catch serializers.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-52492][SQL] Make InMemoryRelation.convertToColumnarIfPossible customizable [spark]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on code in PR #51189:
URL: https://github.com/apache/spark/pull/51189#discussion_r2150620291


##########
sql/core/src/main/scala/org/apache/spark/sql/columnar/CachedBatchSerializer.scala:
##########
@@ -58,6 +60,39 @@ trait CachedBatchSerializer extends Serializable {
    */
   def supportsColumnarInput(schema: Seq[Attribute]): Boolean
 
+  /**
+   * Attempt to convert a query plan to its columnar equivalence for columnar caching.
+   * Called on the query plan that is about to be cached once [[supportsColumnarInput]] returns
+   * true on its output schema.
+   *
+   * The default implementation works by stripping the topmost columnar-to-row transition to
+   * expose the columnar-based plan to the serializer.
+   *
+   * @param plan The plan to convert.
+   * @return The output plan. Could either be a columnar plan if the input plan is convertible, or
+   *         the input plan unchanged if no viable conversion can be done.
+   */
+  @Since("4.0.1")

Review Comment:
   Thanks. Addressed and also updated `Target Version/s` field in the JIRA ticket.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-52492][SQL] Make InMemoryRelation.convertToColumnarIfPossible customizable [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #51189:
URL: https://github.com/apache/spark/pull/51189#discussion_r2150587606


##########
sql/core/src/main/scala/org/apache/spark/sql/columnar/CachedBatchSerializer.scala:
##########
@@ -58,6 +60,39 @@ trait CachedBatchSerializer extends Serializable {
    */
   def supportsColumnarInput(schema: Seq[Attribute]): Boolean
 
+  /**
+   * Attempt to convert a query plan to its columnar equivalence for columnar caching.
+   * Called on the query plan that is about to be cached once [[supportsColumnarInput]] returns
+   * true on its output schema.
+   *
+   * The default implementation works by stripping the topmost columnar-to-row transition to
+   * expose the columnar-based plan to the serializer.
+   *
+   * @param plan The plan to convert.
+   * @return The output plan. Could either be a columnar plan if the input plan is convertible, or
+   *         the input plan unchanged if no viable conversion can be done.
+   */
+  @Since("4.0.1")

Review Comment:
   According to the Apache Spark backporting policy, this should be 4.1.0 because only bug-fixes are allowed for `branch-4.0`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-52492][SQL] Make InMemoryRelation.convertToColumnarIfPossible customizable [spark]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on PR #51189:
URL: https://github.com/apache/spark/pull/51189#issuecomment-2983778193

   @yaooqinn Thanks for amending the fix!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-52492][SQL] Make InMemoryRelation.convertToColumnarIfPossible customizable [spark]

Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on code in PR #51189:
URL: https://github.com/apache/spark/pull/51189#discussion_r2153569938


##########
sql/core/src/main/scala/org/apache/spark/sql/columnar/CachedBatchSerializer.scala:
##########
@@ -58,6 +60,39 @@ trait CachedBatchSerializer extends Serializable {
    */
   def supportsColumnarInput(schema: Seq[Attribute]): Boolean
 
+  /**
+   * Attempt to convert a query plan to its columnar equivalence for columnar caching.
+   * Called on the query plan that is about to be cached once [[supportsColumnarInput]] returns
+   * true on its output schema.
+   *
+   * The default implementation works by stripping the topmost columnar-to-row transition to
+   * expose the columnar-based plan to the serializer.
+   *
+   * @param plan The plan to convert.
+   * @return The output plan. Could either be a columnar plan if the input plan is convertible, or
+   *         the input plan unchanged if no viable conversion can be done.
+   */
+  @Since("4.1.0")

Review Comment:
   ```suggestion
     @DeveloperApi
     @Since("4.1.0")
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-52492][SQL] Make InMemoryRelation.convertToColumnarIfPossible customizable [spark]

Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on PR #51189:
URL: https://github.com/apache/spark/pull/51189#issuecomment-2986328230

   Merged to master, thank you @zhztheplayer @dongjoon-hyun 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-52492][SQL] Make InMemoryRelation.convertToColumnarIfPossible customizable [spark]

Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn closed pull request #51189: [SPARK-52492][SQL] Make InMemoryRelation.convertToColumnarIfPossible customizable
URL: https://github.com/apache/spark/pull/51189


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org