[jira] [Resolved] (SPARK-20678) Ndv for columns not in filter condition should also be updated

["Wenchen Fan (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-20678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Wenchen Fan resolved SPARK-20678.
---------------------------------
       Resolution: Fixed
    Fix Version/s: 2.3.0
                   2.2.1

Issue resolved by pull request 17918
[https://github.com/apache/spark/pull/17918]

> Ndv for columns not in filter condition should also be updated
> --------------------------------------------------------------
>
>                 Key: SPARK-20678
>                 URL: https://issues.apache.org/jira/browse/SPARK-20678
>             Project: Spark
>          Issue Type: Sub-task
>          Components: SQL
>    Affects Versions: 2.2.0
>            Reporter: Zhenhua Wang
>             Fix For: 2.2.1, 2.3.0
>
>
> In filter estimation, we update column stats for those columns in filter condition. However, if the number of rows decreases after the filter (i.e. the overall selectivity is less than 1), we need to update (scale down) the number of distinct values (NDV) for all columns, no matter they are in filter conditions or not.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org