[jira] [Updated] (SPARK-27604) Enhance constant and constraint propagation

["Peter Toth (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-27604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Peter Toth updated SPARK-27604:
-------------------------------
    Description: 
There is a bug in constant propagation due to null handling:

{{SELECT * FROM t WHERE NOT(c = 1 AND c + 1 = 1)}} returns those rows where {{c}} is null due to {{1 + 1 = 1}} propagation

There is some room for improvement as constant propagation could allow substitution of deterministic expressions (instead of attributes only) to constants and substitutions in other than equal predicates.

  was:
There is some room for improvement in terms of how we can use constraints to simplify expressions and maybe what new expressions we can infer.

Here are some examples what the current implementation can do in terms of expression simplification:
{code:java}
SELECT * FROM table WHERE i = 5 AND j = i + 3      =>  SELECT * FROM table WHERE i = 5 AND j = 8
{code}

And here are some that it could with some improvement:
{code:java}
SELECT * FROM table WHERE i <= 5 AND i = 5         =>  SELECT * FROM table WHERE i = 5
SELECT * FROM table WHERE i < j AND ... AND i = j  =>  SELECT * FROM table WHERE false{code}

I also think that `ConstantPropagation` and `InferFiltersFromConstraints` functionality somewhat overlaps and with some refactoring we could come up with a better solution.



> Enhance constant and constraint propagation
> -------------------------------------------
>
>                 Key: SPARK-27604
>                 URL: https://issues.apache.org/jira/browse/SPARK-27604
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 3.0.0
>            Reporter: Peter Toth
>            Priority: Major
>
> There is a bug in constant propagation due to null handling:
> {{SELECT * FROM t WHERE NOT(c = 1 AND c + 1 = 1)}} returns those rows where {{c}} is null due to {{1 + 1 = 1}} propagation
> There is some room for improvement as constant propagation could allow substitution of deterministic expressions (instead of attributes only) to constants and substitutions in other than equal predicates.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org