You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@hive.apache.org by "Prasanth Jayachandran (JIRA)" <ji...@apache.org> on 2015/02/13 21:22:11 UTC

[jira] [Created] (HIVE-9689) Store distinct value estimator's bit vectors in metastore

Prasanth Jayachandran created HIVE-9689:
-------------------------------------------

             Summary: Store distinct value estimator's bit vectors in metastore
                 Key: HIVE-9689
                 URL: https://issues.apache.org/jira/browse/HIVE-9689
             Project: Hive
          Issue Type: New Feature
            Reporter: Prasanth Jayachandran


Hive currently uses PCSA (Probabilistic Counting and Stochastic Averaging) algorithm to determine distinct cardinality. The NDV value determined from the UDF is stored in the metastore instead of the actual bit vectors. This makes it impossible to estimation the overall NDV across all the partition (or selected partitions). We should ideally store the bitvectors in the metastore and do server side merging of the bitvectors. Also we could replace the current PCSA algorithm in favour of HyperLogLog if space is a constraint. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)