[jira] [Commented] (IMPALA-10020) Implement ds_kll_cdf() function

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-10020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17185348#comment-17185348 ] 

ASF subversion and git services commented on IMPALA-10020:
----------------------------------------------------------

Commit 28d94851b1418ddd57e4c046646f7a2df0e702d7 in impala's branch refs/heads/master from Gabor Kaszab
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=28d9485 ]

IMPALA-10020: Implement ds_kll_cdf_as_string() function

This is the support for Cumulative Distribution Function (CDF) from
Apache DataSketches KLL algorithm collection. It receives a serialized
KLL sketch and one or more float values to represent ranges in the
sketched values.
E.g. [1, 5, 10] will mean the following ranges:
(-inf, 1), (-inf, 5), (-inf, 10), (-inf, +inf)
Returns a comma separated string where each value in the string is a
number in the range of [0,1] and shows that what percentage of the
data is in the particular ranges.

Note, ds_kll_cdf() should return an Array of doubles as the result but
with that we have to wait for the complex type support. Until, we
provide ds_kll_cdf_as_string() that can be deprecated once we
have array support. Tracking Jira for returning complex types from
functions is IMPALA-9520.

Example:
select ds_kll_cdf_as_string(ds_kll_sketch(float_col), 2, 4, 10)
from alltypes;
+----------------------------------------------------------+
| ds_kll_cdf_as_string(ds_kll_sketch(float_col), 2, 4, 10) |
+----------------------------------------------------------+
| 0.2,0.401644,1,1                                         |
+----------------------------------------------------------+

Change-Id: I77e6afc4556ad05a295b89f6d06c2e4a6bb2cf82
Reviewed-on: http://gerrit.cloudera.org:8080/16359
Reviewed-by: Gabor Kaszab <ga...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>


> Implement ds_kll_cdf() function
> -------------------------------
>
>                 Key: IMPALA-10020
>                 URL: https://issues.apache.org/jira/browse/IMPALA-10020
>             Project: IMPALA
>          Issue Type: New Feature
>          Components: Backend, Frontend
>            Reporter: Gabor Kaszab
>            Assignee: Gabor Kaszab
>            Priority: Major
>             Fix For: Impala 4.0
>
>
> Requirements for ds_kll_cdf() (Cumulative Distribution Function):
>  - Receives a serialized KLL sketch in BINARY type (in Impala it can be STRING as long as we don't have BINARY) as first parameter.
>  - Receives one or more float values to create ranges from the sketched data.
>  - In Hive the return type is an array of doubles. However, Impala can't return complex types from functions at this point so we have to find some alternative approaches to implement this function. Follow whatever solution came up in https://issues.apache.org/jira/browse/IMPALA-9962
> An example:
> {code:java}
> select ds_kll_cdf(sketch_col, 1, 2, 3, 4) from sketches_table;
> {code}
> This will generate the following ranges: (-inf, 1), (-inf,2), (-inf,3), (-inf,4), (-inf,+inf)
> In Hive, the result would have an array of 5 doubles for the 5 ranges, where each number gives the probability between [0,1] that an item will fall into the particular range. Or in other words a ratio of items belonging to that range.
> Taking input values such as: 1,2,3,4,5
> {code:java}
> select ds_kll_cdf(f, 1, 3, 4, 5, 10) from kll_sketches;
> +----------------------------+
> |            _c0             |
> +----------------------------+
> | [0.0,0.4,0.6,0.8,1.0,1.0]  |
> +----------------------------+
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org