You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Michael Armbrust (JIRA)" <ji...@apache.org> on 2016/04/01 22:01:25 UTC
[jira] [Resolved] (SPARK-13995) Extract correct IsNotNull
constraints for Expression
[ https://issues.apache.org/jira/browse/SPARK-13995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Armbrust resolved SPARK-13995.
--------------------------------------
Resolution: Fixed
Fix Version/s: 2.0.0
Issue resolved by pull request 11809
[https://github.com/apache/spark/pull/11809]
> Extract correct IsNotNull constraints for Expression
> ----------------------------------------------------
>
> Key: SPARK-13995
> URL: https://issues.apache.org/jira/browse/SPARK-13995
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Reporter: Liang-Chi Hsieh
> Fix For: 2.0.0
>
>
> We infer relative `IsNotNull` constraints from logical plan's expressions in `constructIsNotNullConstraints` now. However, we don't consider the case of (nested) `Cast`.
> For example:
> val tr = LocalRelation('a.int, 'b.long)
> val plan = tr.where('a.attr === 'b.attr).analyze
> Then, the plan's constraints will have `IsNotNull(Cast(resolveColumn(tr, "a"), LongType))`, instead of `IsNotNull(resolveColumn(tr, "a"))`. This PR fixes it.
> Besides, as `IsNotNull` constraints are most useful for `Attribute`, we should do recursing through any `Expression` that is null intolerant and construct `IsNotNull` constraints for all `Attribute`s under these Expressions.
> For example, consider the following constraints:
> val df = Seq((1,2,3)).toDF("a", "b", "c")
> df.where("a + b = c").queryExecution.analyzed.constraints
> The inferred isnotnull constraints should be isnotnull(a), isnotnull(b), isnotnull(c), instead of isnotnull(a + c) and isnotnull(c).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org