[jira] [Commented] (SPARK-18130) Don't add inferred redundant isnotnull condition from constraints

["Apache Spark (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-18130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15610410#comment-15610410 ] 

Apache Spark commented on SPARK-18130:
--------------------------------------

User 'viirya' has created a pull request for this issue:
https://github.com/apache/spark/pull/15653

> Don't add inferred redundant isnotnull condition from constraints
> -----------------------------------------------------------------
>
>                 Key: SPARK-18130
>                 URL: https://issues.apache.org/jira/browse/SPARK-18130
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>            Reporter: Liang-Chi Hsieh
>            Priority: Minor
>
> In Optimizer, we will infer additional conditions from Filter's constraints which include IsNotNull. Some inferred IsNotNulls are useful because they can be short circuit for other predicates which refer the same attributes. But for an IsNotNull condition, we also infer other IsNotNull from it. Those IsNotNulls are redundant. We don't need to include them.
> E.g., for a condition like IsNotNull(a + Rand()), an additional IsNotNull(a) will be added into Filter's condition. So we need to evaluate IsNotNull(a + Rand()) && IsNotNull(a). Actually we don't need IsNotNull(a) because it is guaranteed by the original IsNotNull(a + Rand()).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org