You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Gabor Kaszab (Code Review)" <ge...@cloudera.org> on 2020/07/24 14:32:34 UTC
[Impala-ASF-CR] IMPALA-9959: Implement ds kll sketch() and ds kll quantile() functions
Gabor Kaszab has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16235
Change subject: IMPALA-9959: Implement ds_kll_sketch() and ds_kll_quantile() functions
......................................................................
IMPALA-9959: Implement ds_kll_sketch() and ds_kll_quantile() functions
Change-Id: I11de5fe10bb5d0dd42fb4ee45c4f21cb31963e52
---
M be/src/exprs/aggregate-functions-ir.cc
M be/src/exprs/aggregate-functions.h
M be/src/exprs/datasketches-common.cc
M be/src/exprs/datasketches-common.h
M be/src/exprs/datasketches-functions-ir.cc
M be/src/exprs/datasketches-functions.h
M common/function-registry/impala_functions.py
M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java
M testdata/data/README
A testdata/data/kll_sketches_from_hive.parquet
M tests/query_test/test_datasketches.py
11 files changed, 173 insertions(+), 20 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/16235/1
--
To view, visit http://gerrit.cloudera.org:8080/16235
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I11de5fe10bb5d0dd42fb4ee45c4f21cb31963e52
Gerrit-Change-Number: 16235
Gerrit-PatchSet: 1
Gerrit-Owner: Gabor Kaszab <ga...@cloudera.com>
[Impala-ASF-CR] IMPALA-9959: Implement ds kll sketch() and ds kll quantile() functions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16235 )
Change subject: IMPALA-9959: Implement ds_kll_sketch() and ds_kll_quantile() functions
......................................................................
Patch Set 9: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/16235
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11de5fe10bb5d0dd42fb4ee45c4f21cb31963e52
Gerrit-Change-Number: 16235
Gerrit-PatchSet: 9
Gerrit-Owner: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 31 Jul 2020 09:29:00 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9959: Implement ds kll sketch() and ds kll quantile() functions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16235 )
Change subject: IMPALA-9959: Implement ds_kll_sketch() and ds_kll_quantile() functions
......................................................................
Patch Set 1:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/6703/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/16235
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11de5fe10bb5d0dd42fb4ee45c4f21cb31963e52
Gerrit-Change-Number: 16235
Gerrit-PatchSet: 1
Gerrit-Owner: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 24 Jul 2020 15:01:41 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9959: Implement ds kll sketch() and ds kll quantile() functions
Posted by "Gabor Kaszab (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16235
to look at the new patch set (#2).
Change subject: IMPALA-9959: Implement ds_kll_sketch() and ds_kll_quantile() functions
......................................................................
IMPALA-9959: Implement ds_kll_sketch() and ds_kll_quantile() functions
Change-Id: I11de5fe10bb5d0dd42fb4ee45c4f21cb31963e52
---
M be/src/exprs/aggregate-functions-ir.cc
M be/src/exprs/aggregate-functions.h
M be/src/exprs/datasketches-common.cc
M be/src/exprs/datasketches-common.h
M be/src/exprs/datasketches-functions-ir.cc
M be/src/exprs/datasketches-functions.h
M common/function-registry/impala_functions.py
M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java
M testdata/data/README
A testdata/data/kll_sketches_from_hive.parquet
A testdata/workloads/functional-query/queries/QueryTest/datasketches-kll.test
M tests/query_test/test_datasketches.py
12 files changed, 319 insertions(+), 20 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/16235/2
--
To view, visit http://gerrit.cloudera.org:8080/16235
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I11de5fe10bb5d0dd42fb4ee45c4f21cb31963e52
Gerrit-Change-Number: 16235
Gerrit-PatchSet: 2
Gerrit-Owner: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-9959: Implement ds kll sketch() and ds kll quantile() functions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16235 )
Change subject: IMPALA-9959: Implement ds_kll_sketch() and ds_kll_quantile() functions
......................................................................
Patch Set 7:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/6738/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/16235
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11de5fe10bb5d0dd42fb4ee45c4f21cb31963e52
Gerrit-Change-Number: 16235
Gerrit-PatchSet: 7
Gerrit-Owner: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 30 Jul 2020 10:51:01 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9959: Implement ds kll sketch() and ds kll quantile() functions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16235 )
Change subject: IMPALA-9959: Implement ds_kll_sketch() and ds_kll_quantile() functions
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6721/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16235
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11de5fe10bb5d0dd42fb4ee45c4f21cb31963e52
Gerrit-Change-Number: 16235
Gerrit-PatchSet: 5
Gerrit-Owner: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 28 Jul 2020 08:12:17 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9959: Implement ds kll sketch() and ds kll quantile() functions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16235 )
Change subject: IMPALA-9959: Implement ds_kll_sketch() and ds_kll_quantile() functions
......................................................................
IMPALA-9959: Implement ds_kll_sketch() and ds_kll_quantile() functions
ds_kll_sketch() is an aggregate function that receives a float
parameter (e.g. a float column of a table) and returns a serialized
Apache DataSketches KLL sketch of the input data set wrapped into
STRING type. This sketch can be saved into a table or view and later
used for quantile approximations. ds_kll_quantile() receives two
parameters: a STRING parameter that contains a serialized KLL sketch
and a DOUBLE that represents the rank of the quantile in the range of
[0,1]. E.g. rank=0.1 means the approximate value in the sketch where
10% of the sketched items are less than or equals to this value.
Testing:
- Added automated tests on small data sets to check the basic
functionality of sketching and getting a quantile approximate.
- Tested on TPCH25_parquet.lineitem to check that sketching and
approximating works on bigger scale as well where serialize/merge
phases are also required. On this scale the error range of the
quantile approximation is within 1-1.5%
Change-Id: I11de5fe10bb5d0dd42fb4ee45c4f21cb31963e52
Reviewed-on: http://gerrit.cloudera.org:8080/16235
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/exprs/aggregate-functions-ir.cc
M be/src/exprs/aggregate-functions.h
M be/src/exprs/datasketches-common.cc
M be/src/exprs/datasketches-common.h
M be/src/exprs/datasketches-functions-ir.cc
M be/src/exprs/datasketches-functions.h
M common/function-registry/impala_functions.py
M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java
M testdata/data/README
A testdata/data/kll_sketches_from_hive.parquet
A testdata/workloads/functional-query/queries/QueryTest/datasketches-kll.test
M tests/query_test/test_datasketches.py
12 files changed, 333 insertions(+), 22 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/16235
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I11de5fe10bb5d0dd42fb4ee45c4f21cb31963e52
Gerrit-Change-Number: 16235
Gerrit-PatchSet: 10
Gerrit-Owner: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-9959: Implement ds kll sketch() and ds kll quantile() functions
Posted by "Gabor Kaszab (Code Review)" <ge...@cloudera.org>.
Hello Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16235
to look at the new patch set (#7).
Change subject: IMPALA-9959: Implement ds_kll_sketch() and ds_kll_quantile() functions
......................................................................
IMPALA-9959: Implement ds_kll_sketch() and ds_kll_quantile() functions
ds_kll_sketch() is an aggregate function that receives a float
parameter (e.g. a float column of a table) and returns a serialized
Apache DataSketches KLL sketch of the input data set wrapped into
STRING type. This sketch can be saved into a table or view and later
used for quantile approximations. ds_kll_quantile() receives two
parameters: a STRING parameter that contains a serialized KLL sketch
and a DOUBLE that represents the rank of the quantile in the range of
[0,1]. E.g. rank=0.1 means the approximate value in the sketch where
10% of the sketched items are less than or equals to this value.
Testing:
- Added automated tests on small data sets to check the basic
functionality of sketching and getting a quantile approximate.
- Tested on TPCH25_parquet.lineitem to check that sketching and
approximating works on bigger scale as well where serialize/merge
phases are also required. On this scale the error range of the
quantile approximation is within 1-1.5%
Change-Id: I11de5fe10bb5d0dd42fb4ee45c4f21cb31963e52
---
M be/src/exprs/aggregate-functions-ir.cc
M be/src/exprs/aggregate-functions.h
M be/src/exprs/datasketches-common.cc
M be/src/exprs/datasketches-common.h
M be/src/exprs/datasketches-functions-ir.cc
M be/src/exprs/datasketches-functions.h
M common/function-registry/impala_functions.py
M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java
M testdata/data/README
A testdata/data/kll_sketches_from_hive.parquet
A testdata/workloads/functional-query/queries/QueryTest/datasketches-kll.test
M tests/query_test/test_datasketches.py
12 files changed, 333 insertions(+), 22 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/16235/7
--
To view, visit http://gerrit.cloudera.org:8080/16235
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I11de5fe10bb5d0dd42fb4ee45c4f21cb31963e52
Gerrit-Change-Number: 16235
Gerrit-PatchSet: 7
Gerrit-Owner: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-9959: Implement ds kll sketch() and ds kll quantile() functions
Posted by "Gabor Kaszab (Code Review)" <ge...@cloudera.org>.
Hello Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16235
to look at the new patch set (#6).
Change subject: IMPALA-9959: Implement ds_kll_sketch() and ds_kll_quantile() functions
......................................................................
IMPALA-9959: Implement ds_kll_sketch() and ds_kll_quantile() functions
ds_kll_sketch() is an aggregate function that receives a float parameter (e.g.
a float column of a table) and returns a serialized Apache DataSketches KLL
sketch of the input data set wrapped into STRING type. This sketch can be saved
into a table or view and later used for quantile approximations.
ds_kll_quantile() receives two parameters: a STRING parameter that contains a
serialized KLL sketch and a DOUBLE that represents the rank of the quantile in
the range of [0,1]. E.g. rank=0.1 means the approximate value in the sketch
where 10% of the sketched items are less than or equals to this value.
Testing:
- Added automated tests on small data sets to check the basic functionality
of sketching and getting a quantile approximate.
- Tested on TPCH25_parquet.lineitem to check that sketching and approximating
works on bigger scale as well where serialize/merge phases are also
required. On this scale the error range of the quantile approximation is
within 1-1.5%
Change-Id: I11de5fe10bb5d0dd42fb4ee45c4f21cb31963e52
---
M be/src/exprs/aggregate-functions-ir.cc
M be/src/exprs/aggregate-functions.h
M be/src/exprs/datasketches-common.cc
M be/src/exprs/datasketches-common.h
M be/src/exprs/datasketches-functions-ir.cc
M be/src/exprs/datasketches-functions.h
M common/function-registry/impala_functions.py
M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java
M testdata/data/README
A testdata/data/kll_sketches_from_hive.parquet
A testdata/workloads/functional-query/queries/QueryTest/datasketches-kll.test
M tests/query_test/test_datasketches.py
12 files changed, 319 insertions(+), 20 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/16235/6
--
To view, visit http://gerrit.cloudera.org:8080/16235
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I11de5fe10bb5d0dd42fb4ee45c4f21cb31963e52
Gerrit-Change-Number: 16235
Gerrit-PatchSet: 6
Gerrit-Owner: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-9959: Implement ds kll sketch() and ds kll quantile() functions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16235 )
Change subject: IMPALA-9959: Implement ds_kll_sketch() and ds_kll_quantile() functions
......................................................................
Patch Set 9: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/16235
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11de5fe10bb5d0dd42fb4ee45c4f21cb31963e52
Gerrit-Change-Number: 16235
Gerrit-PatchSet: 9
Gerrit-Owner: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 31 Jul 2020 14:34:48 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9959: Implement ds kll sketch() and ds kll quantile() functions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16235 )
Change subject: IMPALA-9959: Implement ds_kll_sketch() and ds_kll_quantile() functions
......................................................................
Patch Set 8:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6740/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16235
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11de5fe10bb5d0dd42fb4ee45c4f21cb31963e52
Gerrit-Change-Number: 16235
Gerrit-PatchSet: 8
Gerrit-Owner: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 30 Jul 2020 13:43:18 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9959: Implement ds kll sketch() and ds kll quantile() functions
Posted by "Gabor Kaszab (Code Review)" <ge...@cloudera.org>.
Hello Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16235
to look at the new patch set (#5).
Change subject: IMPALA-9959: Implement ds_kll_sketch() and ds_kll_quantile() functions
......................................................................
IMPALA-9959: Implement ds_kll_sketch() and ds_kll_quantile() functions
Change-Id: I11de5fe10bb5d0dd42fb4ee45c4f21cb31963e52
---
M be/src/exprs/aggregate-functions-ir.cc
M be/src/exprs/aggregate-functions.h
M be/src/exprs/datasketches-common.cc
M be/src/exprs/datasketches-common.h
M be/src/exprs/datasketches-functions-ir.cc
M be/src/exprs/datasketches-functions.h
M common/function-registry/impala_functions.py
M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java
M testdata/data/README
A testdata/data/kll_sketches_from_hive.parquet
A testdata/workloads/functional-query/queries/QueryTest/datasketches-kll.test
M tests/query_test/test_datasketches.py
12 files changed, 319 insertions(+), 20 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/16235/5
--
To view, visit http://gerrit.cloudera.org:8080/16235
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I11de5fe10bb5d0dd42fb4ee45c4f21cb31963e52
Gerrit-Change-Number: 16235
Gerrit-PatchSet: 5
Gerrit-Owner: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-9959: Implement ds kll sketch() and ds kll quantile() functions
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/16235 )
Change subject: IMPALA-9959: Implement ds_kll_sketch() and ds_kll_quantile() functions
......................................................................
Patch Set 6: Code-Review+1
(4 comments)
Great work! I have some nits + a question about exceptions.
http://gerrit.cloudera.org:8080/#/c/16235/6//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/16235/6//COMMIT_MSG@9
PS6, Line 9: ds_kll_sketch() is an aggregate function that receives a float parameter (e.g.
nit: wrap at 72 chars
http://gerrit.cloudera.org:8080/#/c/16235/6/be/src/exprs/aggregate-functions-ir.cc
File be/src/exprs/aggregate-functions-ir.cc:
http://gerrit.cloudera.org:8080/#/c/16235/6/be/src/exprs/aggregate-functions-ir.cc@1618
PS6, Line 1618: std
nit: could add "using std::string" + same for stringstream. This is already done for several std:: types in this file
http://gerrit.cloudera.org:8080/#/c/16235/6/be/src/exprs/datasketches-functions-ir.cc
File be/src/exprs/datasketches-functions-ir.cc:
http://gerrit.cloudera.org:8080/#/c/16235/6/be/src/exprs/datasketches-functions-ir.cc@50
PS6, Line 50: return sketch.get_quantile(rank.val);
Do you know if the datasketches code uses exceptions? I am wondering if there can be an error here, e.g. if a sketch with 0 elements was serialized. If the datasketches library uses exceptions in scenarios we can not exclude, then we should put these functions into try/catch blocks.
http://gerrit.cloudera.org:8080/#/c/16235/6/be/src/exprs/datasketches-functions.h
File be/src/exprs/datasketches-functions.h:
http://gerrit.cloudera.org:8080/#/c/16235/6/be/src/exprs/datasketches-functions.h@33
PS6, Line 33: distint
typo: distinct
--
To view, visit http://gerrit.cloudera.org:8080/16235
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11de5fe10bb5d0dd42fb4ee45c4f21cb31963e52
Gerrit-Change-Number: 16235
Gerrit-PatchSet: 6
Gerrit-Owner: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 29 Jul 2020 13:33:01 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9959: Implement ds kll sketch() and ds kll quantile() functions
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/16235 )
Change subject: IMPALA-9959: Implement ds_kll_sketch() and ds_kll_quantile() functions
......................................................................
Patch Set 8: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/16235
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11de5fe10bb5d0dd42fb4ee45c4f21cb31963e52
Gerrit-Change-Number: 16235
Gerrit-PatchSet: 8
Gerrit-Owner: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 30 Jul 2020 19:34:16 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9959: Implement ds kll sketch() and ds kll quantile() functions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16235 )
Change subject: IMPALA-9959: Implement ds_kll_sketch() and ds_kll_quantile() functions
......................................................................
Patch Set 4:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6187/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/16235
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11de5fe10bb5d0dd42fb4ee45c4f21cb31963e52
Gerrit-Change-Number: 16235
Gerrit-PatchSet: 4
Gerrit-Owner: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 28 Jul 2020 06:21:05 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9959: Implement ds kll sketch() and ds kll quantile() functions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16235 )
Change subject: IMPALA-9959: Implement ds_kll_sketch() and ds_kll_quantile() functions
......................................................................
Patch Set 9:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6201/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/16235
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11de5fe10bb5d0dd42fb4ee45c4f21cb31963e52
Gerrit-Change-Number: 16235
Gerrit-PatchSet: 9
Gerrit-Owner: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 31 Jul 2020 09:29:01 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9959: Implement ds kll sketch() and ds kll quantile() functions
Posted by "Gabor Kaszab (Code Review)" <ge...@cloudera.org>.
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/16235 )
Change subject: IMPALA-9959: Implement ds_kll_sketch() and ds_kll_quantile() functions
......................................................................
Patch Set 7:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/16235/6//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/16235/6//COMMIT_MSG@9
PS6, Line 9: ds_kll_sketch() is an aggregate function that receives a float
> nit: wrap at 72 chars
Done
http://gerrit.cloudera.org:8080/#/c/16235/6/be/src/exprs/aggregate-functions-ir.cc
File be/src/exprs/aggregate-functions-ir.cc:
http://gerrit.cloudera.org:8080/#/c/16235/6/be/src/exprs/aggregate-functions-ir.cc@1618
PS6, Line 1618: rin
> nit: could add "using std::string" + same for stringstream. This is already
Done
http://gerrit.cloudera.org:8080/#/c/16235/6/be/src/exprs/datasketches-functions-ir.cc
File be/src/exprs/datasketches-functions-ir.cc:
http://gerrit.cloudera.org:8080/#/c/16235/6/be/src/exprs/datasketches-functions-ir.cc@50
PS6, Line 50: LogSketchDeserializationError(ctx);
> Do you know if the datasketches code uses exceptions? I am wondering if the
Good point! In fact here we are safe as we can get invalid_argument exception if rank is not in [0,1] but I check it above. Some other exceptions are thrown if the internal state of the kll_sketch is off, that is not possible to happen but still it doesn't hurt to add a try-catch around this call.
Additionally, DeserializeDsSketch() covers for invalid_arguments error, but I might add another catch block to be on the safe side.
http://gerrit.cloudera.org:8080/#/c/16235/6/be/src/exprs/datasketches-functions.h
File be/src/exprs/datasketches-functions.h:
http://gerrit.cloudera.org:8080/#/c/16235/6/be/src/exprs/datasketches-functions.h@33
PS6, Line 33: distinc
> typo: distinct
Done
--
To view, visit http://gerrit.cloudera.org:8080/16235
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11de5fe10bb5d0dd42fb4ee45c4f21cb31963e52
Gerrit-Change-Number: 16235
Gerrit-PatchSet: 7
Gerrit-Owner: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 30 Jul 2020 10:31:18 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9959: Implement ds kll sketch() and ds kll quantile() functions
Posted by "Gabor Kaszab (Code Review)" <ge...@cloudera.org>.
Hello Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16235
to look at the new patch set (#8).
Change subject: IMPALA-9959: Implement ds_kll_sketch() and ds_kll_quantile() functions
......................................................................
IMPALA-9959: Implement ds_kll_sketch() and ds_kll_quantile() functions
ds_kll_sketch() is an aggregate function that receives a float
parameter (e.g. a float column of a table) and returns a serialized
Apache DataSketches KLL sketch of the input data set wrapped into
STRING type. This sketch can be saved into a table or view and later
used for quantile approximations. ds_kll_quantile() receives two
parameters: a STRING parameter that contains a serialized KLL sketch
and a DOUBLE that represents the rank of the quantile in the range of
[0,1]. E.g. rank=0.1 means the approximate value in the sketch where
10% of the sketched items are less than or equals to this value.
Testing:
- Added automated tests on small data sets to check the basic
functionality of sketching and getting a quantile approximate.
- Tested on TPCH25_parquet.lineitem to check that sketching and
approximating works on bigger scale as well where serialize/merge
phases are also required. On this scale the error range of the
quantile approximation is within 1-1.5%
Change-Id: I11de5fe10bb5d0dd42fb4ee45c4f21cb31963e52
---
M be/src/exprs/aggregate-functions-ir.cc
M be/src/exprs/aggregate-functions.h
M be/src/exprs/datasketches-common.cc
M be/src/exprs/datasketches-common.h
M be/src/exprs/datasketches-functions-ir.cc
M be/src/exprs/datasketches-functions.h
M common/function-registry/impala_functions.py
M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java
M testdata/data/README
A testdata/data/kll_sketches_from_hive.parquet
A testdata/workloads/functional-query/queries/QueryTest/datasketches-kll.test
M tests/query_test/test_datasketches.py
12 files changed, 333 insertions(+), 22 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/16235/8
--
To view, visit http://gerrit.cloudera.org:8080/16235
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I11de5fe10bb5d0dd42fb4ee45c4f21cb31963e52
Gerrit-Change-Number: 16235
Gerrit-PatchSet: 8
Gerrit-Owner: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-9959: Implement ds kll sketch() and ds kll quantile() functions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16235 )
Change subject: IMPALA-9959: Implement ds_kll_sketch() and ds_kll_quantile() functions
......................................................................
Patch Set 2:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/6704/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/16235
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11de5fe10bb5d0dd42fb4ee45c4f21cb31963e52
Gerrit-Change-Number: 16235
Gerrit-PatchSet: 2
Gerrit-Owner: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 24 Jul 2020 15:04:31 +0000
Gerrit-HasComments: No