You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "shubhangi priya (Jira)" <ji...@apache.org> on 2021/06/29 14:15:00 UTC
[jira] [Commented] (SPARK-35881) [SQL] AQE does not support
columnar execution for the final query stage
[ https://issues.apache.org/jira/browse/SPARK-35881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17371416#comment-17371416 ]
shubhangi priya commented on SPARK-35881:
-----------------------------------------
[~andygrove]
If the issue is still unresolved, Can I work upon this?
> [SQL] AQE does not support columnar execution for the final query stage
> -----------------------------------------------------------------------
>
> Key: SPARK-35881
> URL: https://issues.apache.org/jira/browse/SPARK-35881
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.2.0
> Reporter: Andy Grove
> Priority: Major
>
> In AdaptiveSparkPlanExec, a query is broken down into stages and these stages are executed until the entire query has been executed. These stages can be row-based or columnar. However, the final stage, produced by the private getFinalPhysicalPlan method is always assumed to be row-based. The only way to execute the final stage is by calling the various doExecute methods on AdaptiveSparkPlanExec, and doExecuteColumnar is not implemented. The supportsColumnar method also always returns false.
> In the RAPIDS Accelerator for Apache Spark, we currently call the private getFinalPhysicalPlan method using reflection and then determine if that plan is columnar or not, and then call the appropriate doExecute method, bypassing the doExecute methods on AdaptiveSparkPlanExec. We would like a supported mechanism for executing a columnar AQE plan so that we do not need to use reflection.
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org