[jira] [Commented] (IMPALA-10463) Implement ds_theta_sketch() and ds_theta_estimate() functions

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-10463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17286368#comment-17286368 ] 

ASF subversion and git services commented on IMPALA-10463:
----------------------------------------------------------

Commit 65c6a81ed908084da2175e97664c0f76ded9aed8 in impala's branch refs/heads/master from Fucun Chu
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=65c6a81 ]

IMPALA-10463: Implement ds_theta_sketch() and ds_theta_estimate() functions

These functions can be used to get cardinality estimates of data
using Theta algorithm from Apache DataSketches. ds_theta_sketch()
receives a dataset, e.g. a column from a table, and returns a
serialized Theta sketch in string format. This can be written to a
table or be fed directly to ds_theta_estimate() that returns the
cardinality estimate for that sketch.

Similar to the HLL sketch, the primary use-case for the Theta sketch
is for counting distinct values as a stream, and then merging
multiple sketches together for a total distinct count.

For more details about Apache DataSketches' Theta see:
https://datasketches.apache.org/docs/Theta/ThetaSketchFramework.html

Testing:
 - Added some tests running estimates for small datasets where the
   amount of data is small enough to get the correct results.
 - Ran manual tests on tpch25_parquet.lineitem to compare perfomance
   with ds_hll_*. ds_theta_* is faster than ds_hll_* on the original
   data, the difference is around 1%-10%. ds_hll_estimate() is faster
   than ds_theta_estimate() on existing sketch. HLL and Theta gives
   closer estimate except for string. see IMPALA-10464.

Change-Id: I14f24c16b815eec75cf90bb92c8b8b0363dcbfbc
Reviewed-on: http://gerrit.cloudera.org:8080/17008
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>


> Implement ds_theta_sketch() and ds_theta_estimate() functions
> -------------------------------------------------------------
>
>                 Key: IMPALA-10463
>                 URL: https://issues.apache.org/jira/browse/IMPALA-10463
>             Project: IMPALA
>          Issue Type: New Feature
>          Components: Backend, Frontend
>            Reporter: Fucun Chu
>            Assignee: Fucun Chu
>            Priority: Major
>             Fix For: Impala 4.0
>
>
> These should be built-in functions that use DataSketches functionality that was integrated by [IMPALA-10440|https://issues.apache.org/jira/browse/IMPALA-10440].
> ds_theta_sketch() should receive a primitive expression and return a sketch.ds_theta_estimate() should receive a sketch and return a primitive that is the cardinality estimate for that set of data provided to the sketch.
> Usage:
> select ds_theta_estimate(ds_theta_sketch(col_name)) from table_name;Returns a cardinality estimate for that particular column.
> Hive change that introduced the same: https://issues.apache.org/jira/browse/HIVE-22940
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/DataSketchesFunctions.java]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org