You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by mi...@apache.org on 2018/08/17 15:48:23 UTC
[4/4] impala git commit: IMPALA-7140 fix: fix NDV calculation for
partition columns in LocalFsTable
IMPALA-7140 fix: fix NDV calculation for partition columns in LocalFsTable
For partition columns, we were previously calculating the NDV by adding
the count of partitions plus the count of null-partitions. However, all
of the null partitions have the same value (NULL) so they only
contribute one distinct value, regardless of how many there might be.
This fixes an existing test case in test_compute_stats so doesn't
include new tests.
Change-Id: I0a86750d52dcd744ace030a3c1ec70510d054acf
Reviewed-on: http://gerrit.cloudera.org:8080/11230
Reviewed-by: Vuk Ercegovac <ve...@cloudera.com>
Reviewed-by: Bharath Vissapragada <bh...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/impala/repo
Commit: http://git-wip-us.apache.org/repos/asf/impala/commit/199c0f2f
Tree: http://git-wip-us.apache.org/repos/asf/impala/tree/199c0f2f
Diff: http://git-wip-us.apache.org/repos/asf/impala/diff/199c0f2f
Branch: refs/heads/master
Commit: 199c0f2f9f5ef96e232c53579a21009242ea866c
Parents: 7c0450d
Author: Todd Lipcon <to...@apache.org>
Authored: Tue Aug 14 16:38:43 2018 -0700
Committer: Todd Lipcon <to...@apache.org>
Committed: Fri Aug 17 00:54:41 2018 +0000
----------------------------------------------------------------------
.../main/java/org/apache/impala/catalog/local/LocalFsTable.java | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/impala/blob/199c0f2f/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
----------------------------------------------------------------------
diff --git a/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java b/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
index 7743949..f19adfd 100644
--- a/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
+++ b/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
@@ -520,7 +520,8 @@ public class LocalFsTable extends LocalTable implements FeFsTable {
ColumnStats stats = getColumns().get(i).getStats();
int nonNullParts = partitionValueMap_.get(i).size();
int nullParts = nullPartitionIds_.get(i).size();
- stats.setNumDistinctValues(nonNullParts + nullParts);
+ stats.setNumDistinctValues(nonNullParts + (nullParts > 0 ? 1 : 0));
+
// TODO(todd): this calculation ends up setting the num_nulls stat
// to the number of partitions with null rows, not the number of rows.
// However, it maintains the existing behavior from HdfsTable.