You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2020/06/04 18:02:00 UTC
[jira] [Commented] (IMPALA-9632) Implement ds_hll_sketch() and
ds_hll_estimate() functions
[ https://issues.apache.org/jira/browse/IMPALA-9632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17126120#comment-17126120 ]
ASF subversion and git services commented on IMPALA-9632:
---------------------------------------------------------
Commit 3713d5db8dcac540ce0b5cb45974054ca87792db in impala's branch refs/heads/master from Gabor Kaszab
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=3713d5d ]
IMPALA-9820: Pull Datasketches-5 HLL MurmurHash fix
There is a bug in DataSketches HLL MurmurHash where long strings are
over-read resulting a cardinality estimate that is more than 15% off
from the correct cardinality number. A recent upstream fix in Apache
DataSketches addresses this issue and this patch pulls it to Impala.
https://issues.apache.org/jira/browse/DATASKETCHES-5
Testing:
- I used ds_hll_sketch() and ds_hll_estimate() functions from
IMPALA-9632 to trigger DataSketches HLL functionality.
- Ran DataSketches HLL on lineitem.l_comment in TPCH25_parquet to
reproduce the issue. The symptom was that the actual result was
around 15% off from the correct cardinality result (~69M vs 79M).
- After applying this fix re-running the query gives much closer
results, usually under 3% error range.
Change-Id: I84d73fce1e7a197c1f8fb49404b58ed9bb0b843d
Reviewed-on: http://gerrit.cloudera.org:8080/16026
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Implement ds_hll_sketch() and ds_hll_estimate() functions
> ---------------------------------------------------------
>
> Key: IMPALA-9632
> URL: https://issues.apache.org/jira/browse/IMPALA-9632
> Project: IMPALA
> Issue Type: New Feature
> Components: Backend, Frontend
> Reporter: Gabor Kaszab
> Assignee: Gabor Kaszab
> Priority: Major
>
> These should be built-in functions that use DataSketches functionality that was integrated by IMPALA-9631.
> ds_hll_sketch() should receive a primitive expression and return a sketch.
> ds_hll_estimate() should receive a sketch and return a primitive that is the cardinality estimate for that set of data provided to the sketch.
> Usage:
> select ds_hll_estimate(ds_hll_sketch(col_name)) from table_name;
> Returns a cardinality estimate (similarly to ndv() ) for that particular column.
> Hive change that introduced the same: https://issues.apache.org/jira/browse/HIVE-22940
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org