You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Kent Yao (Jira)" <ji...@apache.org> on 2025/06/18 03:34:00 UTC
[jira] [Resolved] (SPARK-52484) Skip child.supportsColumnar assertion from driver side in ColumnarToRowExec
[ https://issues.apache.org/jira/browse/SPARK-52484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kent Yao resolved SPARK-52484.
------------------------------
Fix Version/s: 4.1.0
Resolution: Fixed
Issue resolved by pull request 51183
[https://github.com/apache/spark/pull/51183]
> Skip child.supportsColumnar assertion from driver side in ColumnarToRowExec
> ---------------------------------------------------------------------------
>
> Key: SPARK-52484
> URL: https://issues.apache.org/jira/browse/SPARK-52484
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.0.0
> Reporter: Hongze Zhang
> Assignee: Hongze Zhang
> Priority: Major
> Fix For: 4.1.0
>
>
> Starting from https://github.com/apache/spark/pull/25264, an assertion assert(child.supportsColumnar) was introduced to fail the instantiation of ColumnarToRowExec when the child query plan doesn't support columnar processing.
> However this check could become a blocker for 3rd plugins when an intermediate state of the query plan is needed, e.g., in the Apache Gluten project, a C2R node on top of a dummy row-based query plan node could sometimes be generated as an intermediate state upon the final query plan. This plan doesn't have to be executed but should be created without error. Hence, Gluten had to [fake a task context|https://github.com/apache/incubator-gluten/blob/0a1b5c28678653242ab0fd7b28ebba1dca43ccb1/gluten-core/src/main/scala/org/apache/gluten/extension/columnar/enumerated/planner/plan/GlutenPlanModel.scala#L50-L55] to bypass the assertion, though it's indeed a workaround.
> This change is to remove the assertion to ensure some flexibilities to the 3rd plugins. This should be no harm for the upstream Apache Spark, because the query execution will still be failed by [this error|https://github.com/apache/spark/blob/5d0b2f41794bf4dd25b3ce19bc4f634082b40876/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlan.scala#L343-L351] without this assertion on an illegal query plan.
> This could also make ColumnarToRowExec align with RowToColumnarExec - the later doesn't do this check during instantiation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org