[jira] [Resolved] (SPARK-32903) GeneratePredicate should be able to eliminate common sub-expressions

["Wenchen Fan (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-32903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Wenchen Fan resolved SPARK-32903.
---------------------------------
    Fix Version/s: 3.1.0
       Resolution: Fixed

Issue resolved by pull request 29776
[https://github.com/apache/spark/pull/29776]

> GeneratePredicate should be able to eliminate common sub-expressions
> --------------------------------------------------------------------
>
>                 Key: SPARK-32903
>                 URL: https://issues.apache.org/jira/browse/SPARK-32903
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 3.1.0
>            Reporter: L. C. Hsieh
>            Assignee: L. C. Hsieh
>            Priority: Major
>             Fix For: 3.1.0
>
>
> Both {{GenerateMutableProjection}} and {{GenerateUnsafeProjection}}, such codegen objects can eliminate common sub-expressions. But {{GeneratePredicate}} currently doesn't do it.
> We encounter a customer issue that a Filter pushed down through a Project causes performance issue, compared with not pushed down case. The issue is one expression used in Filter predicates are run many times. Due to the complex schema, the query nodes are not wholestage codegen, so it runs {{Filter.doExecute}} and then call {{GeneratePredicate}}. The common expression was run many time and became performance bottleneck. {{GeneratePredicate}} should be able to eliminate common sub-expressions for such case.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org