You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Gabor Kaszab (Jira)" <ji...@apache.org> on 2020/08/11 13:28:00 UTC
[jira] [Updated] (IMPALA-10019) Implement ds_kll_pmf() function
[ https://issues.apache.org/jira/browse/IMPALA-10019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gabor Kaszab updated IMPALA-10019:
----------------------------------
Description:
Requirements for ds_kll_pmf() (Probability Mass Function):
- Receives a serialized KLL sketch in BINARY type (in Impala it can be STRING as long as we don't have BINARY) as first parameter.
- Receives one or more float values to create ranges from the sketched data.
- In Hive the return type is an array of doubles. However, Impala can't return complex types from functions at this point so we have to find some alternative approaches to implement this function. Follow whatever solution came up in https://issues.apache.org/jira/browse/IMPALA-9962
An example:
{code:java}
select ds_kll_pmf(sketch_col, 1, 2, 3, 4) from sketches_table;
{code}
This will generate the following ranges: (-inf, 1), [1,2), [2,3), [3,4), [4,+inf)
In Hive, the result would have an array of 5 doubles for the 5 ranges, where each number gives the probability between [0,1] that an item will fall into the particular range. Or in other words a ratio of items belonging to that range.
Taking input values such as: 1,2,3,4,5
{code:java}
select ds_kll_pmf(f, 1, 3, 4, 5, 10) from kll_sketches;
+----------------------------+
| _c0 |
+----------------------------+
| [0.0,0.4,0.2,0.2,0.2,0.0] |
+----------------------------+
{code}
was:
Requirements for ds_kll_pmf() (Probability Mass Function):
- Receives a serialized KLL sketch in BINARY type (in Impala it can be STRING as long as we don't have BINARY) as first parameter.
- Receives one or more double values to create ranges from the sketched data.
- In Hive the return type is an array of doubles. However, Impala can't return complex types from functions at this point so we have to find some alternative approaches to implement this function. Follow whatever solution came up in https://issues.apache.org/jira/browse/IMPALA-9962
An example:
{code:java}
select ds_kll_pmf(sketch_col, 1, 2, 3, 4) from sketches_table;
{code}
This will generate the following ranges: (-inf, 1), [1,2), [2,3), [3,4), [4,+inf)
In Hive, the result would have an array of 5 doubles for the 5 ranges, where each number gives the probability between [0,1] that an item will fall into the particular range. Or in other words a ratio of items belonging to that range.
Taking input values such as: 1,2,3,4,5
{code:java}
select ds_kll_pmf(f, 1, 3, 4, 5, 10) from kll_sketches;
+----------------------------+
| _c0 |
+----------------------------+
| [0.0,0.4,0.2,0.2,0.2,0.0] |
+----------------------------+
{code}
> Implement ds_kll_pmf() function
> -------------------------------
>
> Key: IMPALA-10019
> URL: https://issues.apache.org/jira/browse/IMPALA-10019
> Project: IMPALA
> Issue Type: New Feature
> Reporter: Gabor Kaszab
> Assignee: Gabor Kaszab
> Priority: Major
>
> Requirements for ds_kll_pmf() (Probability Mass Function):
> - Receives a serialized KLL sketch in BINARY type (in Impala it can be STRING as long as we don't have BINARY) as first parameter.
> - Receives one or more float values to create ranges from the sketched data.
> - In Hive the return type is an array of doubles. However, Impala can't return complex types from functions at this point so we have to find some alternative approaches to implement this function. Follow whatever solution came up in https://issues.apache.org/jira/browse/IMPALA-9962
> An example:
> {code:java}
> select ds_kll_pmf(sketch_col, 1, 2, 3, 4) from sketches_table;
> {code}
> This will generate the following ranges: (-inf, 1), [1,2), [2,3), [3,4), [4,+inf)
> In Hive, the result would have an array of 5 doubles for the 5 ranges, where each number gives the probability between [0,1] that an item will fall into the particular range. Or in other words a ratio of items belonging to that range.
> Taking input values such as: 1,2,3,4,5
> {code:java}
> select ds_kll_pmf(f, 1, 3, 4, 5, 10) from kll_sketches;
> +----------------------------+
> | _c0 |
> +----------------------------+
> | [0.0,0.4,0.2,0.2,0.2,0.0] |
> +----------------------------+
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org