You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2020/07/24 15:01:00 UTC
[jira] [Commented] (SPARK-32430) Allow plugins to inject rules into
AQE query stage preparation
[ https://issues.apache.org/jira/browse/SPARK-32430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17164481#comment-17164481 ]
Apache Spark commented on SPARK-32430:
--------------------------------------
User 'andygrove' has created a pull request for this issue:
https://github.com/apache/spark/pull/29224
> Allow plugins to inject rules into AQE query stage preparation
> --------------------------------------------------------------
>
> Key: SPARK-32430
> URL: https://issues.apache.org/jira/browse/SPARK-32430
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Thomas Graves
> Priority: Major
>
> Provide a generic mechanism for plugins to inject rules into the AQE "query prep" stage that happens before query stage creation.
> this goes along with https://issues.apache.org/jira/browse/SPARK-32332 where the current AQE implementation doesn't allow for users to properly extend it for columnar processing.
> The issue here is that we create new query stages but we do not have access to the parent plan of the new query stage so certain things can not be determined because you have to know what the parent did. With this change it would allow you to add TAGs to be able to figure out what is going on.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org