You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Chris O'Hara (JIRA)" <ji...@apache.org> on 2018/08/06 23:20:00 UTC
[jira] [Updated] (SPARK-25037) plan.transformAllExpressions doesn't
transform expressions in subquery plans
[ https://issues.apache.org/jira/browse/SPARK-25037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris O'Hara updated SPARK-25037:
---------------------------------
Description:
Given the following LogicalPlan, containing a SubqueryAlias and SubqueryExpression:
{code:java}
scala> val plan = spark.sql("SELECT 1 bar FROM (SELECT 1 foo) WHERE foo IN (SELECT 1 foo)").queryExecution.logical
plan: org.apache.spark.sql.catalyst.plans.logical.LogicalPlan =
'Project [1 AS bar#29]
+- 'Filter 'foo IN (list#31 [])
: +- Project [1 AS foo#30]
: +- OneRowRelation
+- SubqueryAlias __auto_generated_subquery_name
+- Project [1 AS foo#28]
+- OneRowRelation
{code}
The following transformation should replace all instances of lit(1) with lit(2):
{code:java}
scala> plan.transformAllExpressions { case l @ Literal(1, _) => l.copy(value = 2) }
res0: plan.type =
'Project [2 AS bar#29]
+- 'Filter 'foo IN (list#31 [])
: +- Project [1 AS foo#30]
: +- OneRowRelation
+- SubqueryAlias __auto_generated_subquery_name
+- Project [2 AS foo#28]
+- OneRowRelation
{code}
Instead, the nested SubqueryExpression plan is not transformed.
The expected output is:
{code:java}
'Project [2 AS bar#29]
+- 'Filter 'foo IN (list#31 [])
: +- Project [2 AS foo#30]
: +- OneRowRelation
+- SubqueryAlias __auto_generated_subquery_name
+- Project [2 AS foo#28]
+- OneRowRelation
{code}
was:
Given the following LogicalPlan, containing a SubqueryAlias and SubqueryExpression:
{code:java}
scala> val plan = spark.sql("SELECT 1 bar FROM (SELECT 1 foo) WHERE foo IN (SELECT 1 foo)").queryExecution.logical
plan: org.apache.spark.sql.catalyst.plans.logical.LogicalPlan =
'Project [1 AS bar#29]
+- 'Filter 'foo IN (list#31 [])
: +- Project [1 AS foo#30]
: +- OneRowRelation
+- SubqueryAlias __auto_generated_subquery_name
+- Project [1 AS foo#28]
+- OneRowRelation
{code}
The following transformation should replace all instances of lit(1) with lit(2):
{code:java}
scala> plan.transformAllExpressions { case l @ Literal(1, _) => l.copy(value = 2) }
res0: plan.type =
'Project [2 AS bar#29]
+- 'Filter 'foo IN (list#31 [])
: +- Project [1 AS foo#30]
: +- OneRowRelation
+- SubqueryAlias __auto_generated_subquery_name
+- Project [2 AS foo#28]
+- OneRowRelation
{code}
Instead, the nested SubqueryExpression plan is not transformed.
The expected output is:
{code:java}
'Project [2 AS bar#29]
+- 'Filter 'foo IN (list#31 [])
: +- Project [2 AS foo#30]
: +- OneRowRelation
+- SubqueryAlias __auto_generated_subquery_name
+- Project [2 AS foo#28]
+- OneRowRelation
{code}
> plan.transformAllExpressions doesn't transform expressions in subquery plans
> ----------------------------------------------------------------------------
>
> Key: SPARK-25037
> URL: https://issues.apache.org/jira/browse/SPARK-25037
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.3.1
> Reporter: Chris O'Hara
> Priority: Minor
>
> Given the following LogicalPlan, containing a SubqueryAlias and SubqueryExpression:
> {code:java}
> scala> val plan = spark.sql("SELECT 1 bar FROM (SELECT 1 foo) WHERE foo IN (SELECT 1 foo)").queryExecution.logical
> plan: org.apache.spark.sql.catalyst.plans.logical.LogicalPlan =
> 'Project [1 AS bar#29]
> +- 'Filter 'foo IN (list#31 [])
> : +- Project [1 AS foo#30]
> : +- OneRowRelation
> +- SubqueryAlias __auto_generated_subquery_name
> +- Project [1 AS foo#28]
> +- OneRowRelation
> {code}
> The following transformation should replace all instances of lit(1) with lit(2):
> {code:java}
> scala> plan.transformAllExpressions { case l @ Literal(1, _) => l.copy(value = 2) }
> res0: plan.type =
> 'Project [2 AS bar#29]
> +- 'Filter 'foo IN (list#31 [])
> : +- Project [1 AS foo#30]
> : +- OneRowRelation
> +- SubqueryAlias __auto_generated_subquery_name
> +- Project [2 AS foo#28]
> +- OneRowRelation
> {code}
> Instead, the nested SubqueryExpression plan is not transformed.
> The expected output is:
> {code:java}
> 'Project [2 AS bar#29]
> +- 'Filter 'foo IN (list#31 [])
> : +- Project [2 AS foo#30]
> : +- OneRowRelation
> +- SubqueryAlias __auto_generated_subquery_name
> +- Project [2 AS foo#28]
> +- OneRowRelation
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org