[jira] [Commented] (SPARK-32903) GeneratePredicate should be able to eliminate common sub-expressions

["Apache Spark (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-32903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17197223#comment-17197223 ] 

Apache Spark commented on SPARK-32903:
--------------------------------------

User 'viirya' has created a pull request for this issue:
https://github.com/apache/spark/pull/29776

> GeneratePredicate should be able to eliminate common sub-expressions
> --------------------------------------------------------------------
>
>                 Key: SPARK-32903
>                 URL: https://issues.apache.org/jira/browse/SPARK-32903
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 3.1.0
>            Reporter: L. C. Hsieh
>            Assignee: L. C. Hsieh
>            Priority: Major
>
> Both {{GenerateMutableProjection}} and {{GenerateUnsafeProjection}}, such codegen objects can eliminate common sub-expressions. But {{GeneratePredicate}} currently doesn't do it.
> We encounter a customer issue that a Filter pushed down through a Project causes performance issue, compared with not pushed down case. The issue is one expression used in Filter predicates are run many times. Due to the complex schema, the query nodes are not wholestage codegen, so it runs {{Filter.doExecute}} and then call {{GeneratePredicate}}. The common expression was run many time and became performance bottleneck. {{GeneratePredicate}} should be able to eliminate common sub-expressions for such case.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org