[jira] [Created] (SPARK-32903) GeneratePredicate should be able to eliminate common sub-expressions

["L. C. Hsieh (Jira)" <ji...@apache.org>]

L. C. Hsieh created SPARK-32903:
-----------------------------------

             Summary: GeneratePredicate should be able to eliminate common sub-expressions
                 Key: SPARK-32903
                 URL: https://issues.apache.org/jira/browse/SPARK-32903
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 3.1.0
            Reporter: L. C. Hsieh
            Assignee: L. C. Hsieh


Both {{GenerateMutableProjection}} and {{GenerateUnsafeProjection}}, such codegen objects can eliminate common sub-expressions. But {{GeneratePredicate}} currently doesn't do it.

We encounter a customer issue that a Filter pushed down through a Project causes performance issue, compared with not pushed down case. The issue is one expression used in Filter predicates are run many times. Due to the complex schema, the query nodes are not wholestage codegen, so it runs {{Filter.doExecute}} and then call {{GeneratePredicate}}. The common expression was run many time and became performance bottleneck. {{GeneratePredicate}} should be able to eliminate common sub-expressions for such case.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org