You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2021/06/01 20:34:00 UTC
[jira] [Assigned] (SPARK-35580) Support subexpression elimination
for higher order functions
[ https://issues.apache.org/jira/browse/SPARK-35580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-35580:
------------------------------------
Assignee: (was: Apache Spark)
> Support subexpression elimination for higher order functions
> ------------------------------------------------------------
>
> Key: SPARK-35580
> URL: https://issues.apache.org/jira/browse/SPARK-35580
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.1.1
> Reporter: Adam Binford
> Priority: Major
>
> Currently higher order functions are not candidates for subexpression elimination. This is because all higher order functions have different semantic hashes, due to "exprId" and "value" in "NamedLambdaVariable". These always are unique, so the semanticHash of a NamedLambdaVariable is always unique. Also, [https://github.com/apache/spark/pull/32424] might throw a wrench in things some too, depending on how you define your expressions the name could be different.
> {code:java}
> scala> var d = transform($"a", x => x + 1)
> d: org.apache.spark.sql.Column = transform(a, lambdafunction((x_2 + 1), x_2))
> scala> var e = transform($"a", x => x + 1)
> e: org.apache.spark.sql.Column = transform(a, lambdafunction((x_3 + 1), x_3))
> scala> struct(d.alias("1"), d.alias("2")).expr
> res9: org.apache.spark.sql.catalyst.expressions.Expression = struct(NamePlaceholder, transform('a, lambdafunction((lambda 'x_2 + 1), lambda 'x_2, false)) AS 1#4, NamePlaceholder, transform('a, lambdafunction((lambda 'x_2 + 1), lambda 'x_2, false)) AS 2#5)
> scala> struct(d.alias("1"), e.alias("2")).expr
> res10: org.apache.spark.sql.catalyst.expressions.Expression = struct(NamePlaceholder, transform('a, lambdafunction((lambda 'x_2 + 1), lambda 'x_2, false)) AS 1#6, NamePlaceholder, transform('a, lambdafunction((lambda 'x_3 + 1), lambda 'x_3, false)) AS 2#7)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org