[jira] [Commented] (IMPALA-10520) Implement ds_theta_intersect() builtin function

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-10520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17300683#comment-17300683 ] 

ASF subversion and git services commented on IMPALA-10520:
----------------------------------------------------------

Commit 0d22e89df452d3e21912b60c72a087a3e8c38057 in impala's branch refs/heads/master from Fucun Chu
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=0d22e89 ]

IMPALA-10520: Implement ds_theta_intersect() function

This function receives a set of serialized Apache DataSketches Theta
sketches produced by ds_theta_sketch() and intersects them into a
single sketch.

An example usage is to create a sketch for each partition of a table,
write these sketches to a separate table and intersect them to get
estimates based on the partitions the user is interested in related
sketches. E.g.:
  SELECT
      ds_theta_estimate(ds_theta_intersect(sketch_col))
  FROM sketch_tbl
  WHERE partition_col=1 OR partition_col=5;

Testing:
  - Apart from the automated tests I added to this patch I also
    tested ds_theta_intersect() on a bigger dataset to check that
    serialization, deserialization and merging steps work well. I
    took TPCH25.linelitem, created a number of sketches with grouping
    by l_shipdate and called ds_theta_intersect() on those sketches

Change-Id: I80e68c2151c4604f0386d3dfb004c82b10293f97
Reviewed-on: http://gerrit.cloudera.org:8080/17088
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>


> Implement ds_theta_intersect() builtin function
> -----------------------------------------------
>
>                 Key: IMPALA-10520
>                 URL: https://issues.apache.org/jira/browse/IMPALA-10520
>             Project: IMPALA
>          Issue Type: New Feature
>          Components: Backend, Frontend
>            Reporter: Fucun Chu
>            Assignee: Fucun Chu
>            Priority: Major
>             Fix For: Impala 4.0
>
>
> ds_theta_intersect() is an aggregate function that accepts a sketch and produces a single sketch, which is the intersection of the received sketches.
> Example from Hive:
> {code:java}
> create temporary table sketch_intermediate (category char(1), sketch binary);
> insert into sketch_intermediate select category, ds_theta_sketch(id) from sketch_input group by category;
> select ds_theta_estimate(ds_theta_intersect(sketch)) from sketch_intermediate;{code}
> Some test data for the example:
> {code:java}
> create temporary table sketch_input (id int, category char(1));
> insert into table sketch_input values
>  (1, 'a'), (2, 'a'), (3, 'a'), (4, 'a'), (5, 'a'), (6, 'a'), (7, 'a'), (8, 'a'), (9, 'a'), (10, 'a'),
>  (6, 'b'), (7, 'b'), (8, 'b'), (9, 'b'), (10, 'b'), (11, 'b'), (12, 'b'), (13, 'b'), (14, 'b'), (15, 'b');{code}
> Approximate result:
> {code:java}
> 5.0{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org