[jira] [Assigned] (SPARK-19665) Improve constraint propagation

["Apache Spark (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-19665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Apache Spark reassigned SPARK-19665:
------------------------------------

    Assignee: Apache Spark

> Improve constraint propagation
> ------------------------------
>
>                 Key: SPARK-19665
>                 URL: https://issues.apache.org/jira/browse/SPARK-19665
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 2.1.0
>            Reporter: Liang-Chi Hsieh
>            Assignee: Apache Spark
>
> If there are aliased expression in the projection, we propagate constraints by completely expanding the original constraints with aliases.
> This expanding costs much computation time when the number of aliases increases.
> Another issue is we actually don't need the additional constraints at most of time. For example, if there is a constraint "a > b", and "a" is aliased to "c" and "d". When we use this constraint in filtering, we don't need all constraints "a > b", "c > b", "d > b". We only need "a > b" because if it is false, it is guaranteed that all other constraints are false too.
> Fully expanding all constraints at all the time makes iterative ML algorithms where a ML pipeline with many stages runs very slow.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org