You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Yin Huai (JIRA)" <ji...@apache.org> on 2016/03/07 21:05:40 UTC
[jira] [Updated] (SPARK-13495) Add Null Filters in the query plan
for Filters/Joins based on their data constraints
[ https://issues.apache.org/jira/browse/SPARK-13495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yin Huai updated SPARK-13495:
-----------------------------
Assignee: Sameer Agarwal
> Add Null Filters in the query plan for Filters/Joins based on their data constraints
> ------------------------------------------------------------------------------------
>
> Key: SPARK-13495
> URL: https://issues.apache.org/jira/browse/SPARK-13495
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Reporter: Sameer Agarwal
> Assignee: Sameer Agarwal
> Fix For: 2.0.0
>
>
> We should add an optimizer rule that attempts to eliminate reading (unnecessary) NULL values if they are not required for correctness by inserting isNotNull filters is the query plan. These filters should be inserted beneath existing Filters and Join operators and are inferred based on their data constraints.
> For example, if we have filter on a = 10, we know that null values will not pass this predicate. So, we can add a IsNotNull below it.
> cc [~yhuai] [~nongli]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org