You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Hongze Zhang (Jira)" <ji...@apache.org> on 2025/06/16 06:55:00 UTC

[jira] [Created] (SPARK-52484) Remove child.supportsColumnar assertion in ColumnarToRowExec

Hongze Zhang created SPARK-52484:
------------------------------------

             Summary: Remove child.supportsColumnar assertion in ColumnarToRowExec
                 Key: SPARK-52484
                 URL: https://issues.apache.org/jira/browse/SPARK-52484
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 4.0.0
            Reporter: Hongze Zhang
             Fix For: 4.0.1


Starting from https://github.com/apache/spark/pull/25264, an assertion assert(child.supportsColumnar) was introduced to fail the instantiation of ColumnarToRowExec when the child query plan doesn't support columnar processing.

However this check could become a blocker for 3rd plugins when an intermediate state of the query plan is needed, e.g., in the Apache Gluten project, a C2R node on top of a dummy row-based query plan node could sometimes be generated as an intermediate state upon the final query plan. This plan doesn't have to be executed but should be created without error. Hence, Gluten had to [fake a task context|https://github.com/apache/incubator-gluten/blob/0a1b5c28678653242ab0fd7b28ebba1dca43ccb1/gluten-core/src/main/scala/org/apache/gluten/extension/columnar/enumerated/planner/plan/GlutenPlanModel.scala#L50-L55] to bypass the assertion, though it's indeed a workaround.

This change is to remove the assertion to ensure some flexibilities to the 3rd plugins. This should be no harm for the upstream Apache Spark, because the query execution will still be failed by [this error|https://github.com/apache/spark/blob/5d0b2f41794bf4dd25b3ce19bc4f634082b40876/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlan.scala#L343-L351] without this assertion on an illegal query plan.

This could also make ColumnarToRowExec align with RowToColumnarExec - the later doesn't do this check during instantiation.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org