[jira] [Commented] (PHOENIX-418) Support approximate COUNT DISTINCT

["Ethan Wang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PHOENIX-418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108332#comment-16108332 ] 

Ethan Wang commented on PHOENIX-418:
------------------------------------

+1 on 
{quote}select count(distinct name)
from person
APPROXIMATE ()

select count(distinct name)
from person
APPROXIMATE (ALGORITHM 'hll')

select count(distinct name)
from person
APPROXIMATE (ALGORITHM 'ABC' WITHIN 10 PERCENT)

select count(distinct name) APPROXIMATE ()
from person

select count(distinct name) APPROXIMATE (ALGORITHM 'hll')
from person

select count(distinct name)
  APPROXIMATE (ALGORITHM 'ABC' WITHIN 10 PERCENT)
from person {quote}


The current patch that is in progress basically align with this suggestion, supporting both 
APPROX_COUNT_DISTINCT(xxx) 
and 
select count(name) from person APPROXIMATE ('hll')

Also, for hyperLogLog algorithm, suggesting PHOENIX to be as similar as Druid(CALCITE-1588), in that:
1, not including accuracy option for user to specify.
2, hard coded the two parameters during HLL initialization. (i.e., the precision value for the normal set and the precision value for the sparse set)


> Support approximate COUNT DISTINCT
> ----------------------------------
>
>                 Key: PHOENIX-418
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-418
>             Project: Phoenix
>          Issue Type: Task
>            Reporter: James Taylor
>            Assignee: Ethan Wang
>              Labels: gsoc2016
>
> Support an "approximation" of count distinct to prevent having to hold on to all distinct values (since this will not scale well when the number of distinct values is huge). The Apache Drill folks have had some interesting discussions on this [here](http://mail-archives.apache.org/mod_mbox/incubator-drill-dev/201306.mbox/%3CJIRA.12650169.1369931282407.88049.1370645900553%40arcas%3E). They recommend using  [Welford's method](http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance_Online_algorithm). I'm open to having a config option that uses exact versus approximate. I don't have experience implementing an approximate implementation, so I'm not sure how much state is required to keep on the server and return to the client (other than realizing it'd be much less that returning all distinct values and their counts).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)