You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "xiangfu0 (via GitHub)" <gi...@apache.org> on 2023/07/29 00:28:38 UTC
[GitHub] [pinot] xiangfu0 opened a new pull request, #11216: V2 mv aggregation
xiangfu0 opened a new pull request, #11216:
URL: https://github.com/apache/pinot/pull/11216
Support multi-value aggregation functions:
- countMV
- sumMV
- minMV
- maxMV
- avgMV
- minMaxRangeMV
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] xiangfu0 merged pull request #11216: [multistage] Support multi-value aggregation functions
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 merged PR #11216:
URL: https://github.com/apache/pinot/pull/11216
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on a diff in pull request #11216: [multistage] Support multi-value aggregation functions
Posted by "walterddr (via GitHub)" <gi...@apache.org>.
walterddr commented on code in PR #11216:
URL: https://github.com/apache/pinot/pull/11216#discussion_r1279587937
##########
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/MultiStageEngineIntegrationTest.java:
##########
@@ -128,12 +128,55 @@ public void testMultiValueColumnSelectionQuery()
testQueryWithMatchingRowCount(pinotQuery, h2Query);
}
+ @Test(dataProvider = "useBothQueryEngines")
+ public void testMultiValueColumnAggregationQuery(boolean useMultiStageQueryEngine)
+ throws Exception {
+ setUseMultiStageQueryEngine(useMultiStageQueryEngine);
+
+ String[] multiValueFunctions = new String[]{
+ "sumMV", "countMV", "minMV", "maxMV", "avgMV", "minMaxRangeMV", "distinctCountMV", "distinctCountBitmapMV",
+ "distinctCountHLLMV", "distinctSumMV", "distinctAvgMV"
+ };
+ double[] expectedResults = new double[]{
+ -5.421344202E9, 577725, -9999.0, 16271.0, -9383.95292223809, 26270.0, 312, 312, 328, 3954484.0,
+ 12674.628205128205
+ };
+
+ Assert.assertEquals(multiValueFunctions.length, expectedResults.length);
+
+ for (int i = 0; i < multiValueFunctions.length; i++) {
+ String pinotQuery = String.format("SELECT %s(DivAirportIDs) FROM mytable", multiValueFunctions[i]);
+ JsonNode jsonNode = postQuery(pinotQuery);
+ Assert.assertEquals(jsonNode.get("resultTable").get("rows").get(0).get(0).asDouble(), expectedResults[i]);
+ }
+
+ String pinotQuery = "SELECT percentileMV(DivAirportIDs, 99) FROM mytable";
+ JsonNode jsonNode = postQuery(pinotQuery);
+ Assert.assertEquals(jsonNode.get("resultTable").get("rows").get(0).get(0).asDouble(), 13433.0);
+
+ pinotQuery = "SELECT percentileEstMV(DivAirportIDs, 99) FROM mytable";
+ jsonNode = postQuery(pinotQuery);
+ Assert.assertTrue(jsonNode.get("resultTable").get("rows").get(0).get(0).asDouble() > 13000);
+ Assert.assertTrue(jsonNode.get("resultTable").get("rows").get(0).get(0).asDouble() < 14000);
+
+ pinotQuery = "SELECT percentileTDigestMV(DivAirportIDs, 99) FROM mytable";
+ jsonNode = postQuery(pinotQuery);
+ Assert.assertTrue(jsonNode.get("resultTable").get("rows").get(0).get(0).asDouble() > 13000);
+ Assert.assertTrue(jsonNode.get("resultTable").get("rows").get(0).get(0).asDouble() < 14000);
+
+ pinotQuery = "SELECT percentileKLLMV(DivAirportIDs, 99) FROM mytable";
Review Comment:
can we add tests for both the 2 and 3 arg version of the same function
(same applies to other functions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on a diff in pull request #11216: [multistage] Support multi-value aggregation functions
Posted by "walterddr (via GitHub)" <gi...@apache.org>.
walterddr commented on code in PR #11216:
URL: https://github.com/apache/pinot/pull/11216#discussion_r1279580999
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/AggregationFunctionType.java:
##########
@@ -150,25 +150,56 @@ public enum AggregationFunctionType {
STUNION("STUnion"),
// Aggregation functions for multi-valued columns
- COUNTMV("countMV"),
- MINMV("minMV"),
- MAXMV("maxMV"),
- SUMMV("sumMV"),
- AVGMV("avgMV"),
- MINMAXRANGEMV("minMaxRangeMV"),
- DISTINCTCOUNTMV("distinctCountMV"),
- DISTINCTCOUNTBITMAPMV("distinctCountBitmapMV"),
- DISTINCTCOUNTHLLMV("distinctCountHLLMV"),
- DISTINCTCOUNTRAWHLLMV("distinctCountRawHLLMV"),
- DISTINCTSUMMV("distinctSumMV"),
- DISTINCTAVGMV("distinctAvgMV"),
- PERCENTILEMV("percentileMV"),
- PERCENTILEESTMV("percentileEstMV"),
- PERCENTILERAWESTMV("percentileRawEstMV"),
- PERCENTILETDIGESTMV("percentileTDigestMV"),
- PERCENTILERAWTDIGESTMV("percentileRawTDigestMV"),
- PERCENTILEKLLMV("percentileKLLMV"),
- PERCENTILERAWKLLMV("percentileRawKLLMV"),
+ COUNTMV("countMV", null, SqlKind.OTHER_FUNCTION, SqlFunctionCategory.USER_DEFINED_FUNCTION,
Review Comment:
can we add unit-tests in the json resources?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #11216: [multistage] Support multi-value aggregation functions
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #11216:
URL: https://github.com/apache/pinot/pull/11216#issuecomment-1656501832
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11216?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#11216](https://app.codecov.io/gh/apache/pinot/pull/11216?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (4636dee) into [master](https://app.codecov.io/gh/apache/pinot/commit/05989559e06d448d23d55147672345ede624381d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (0598955) will **increase** coverage by `0.00%`.
> Report is 1 commits behind head on master.
> The diff coverage is `0.00%`.
```diff
@@ Coverage Diff @@
## master #11216 +/- ##
=========================================
Coverage 0.11% 0.11%
=========================================
Files 2227 2172 -55
Lines 119594 117047 -2547
Branches 18099 17787 -312
=========================================
Hits 137 137
+ Misses 119437 116890 -2547
Partials 20 20
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1temurin11 | `?` | |
| integration1temurin17 | `?` | |
| integration1temurin20 | `?` | |
| integration2temurin11 | `?` | |
| integration2temurin20 | `?` | |
| unittests1temurin11 | `?` | |
| unittests1temurin17 | `?` | |
| unittests1temurin20 | `?` | |
| unittests2temurin11 | `?` | |
| unittests2temurin17 | `?` | |
| unittests2temurin20 | `0.11% <0.00%> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files Changed](https://app.codecov.io/gh/apache/pinot/pull/11216?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...che/pinot/segment/spi/AggregationFunctionType.java](https://app.codecov.io/gh/apache/pinot/pull/11216?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL0FnZ3JlZ2F0aW9uRnVuY3Rpb25UeXBlLmphdmE=) | `0.00% <0.00%> (ø)` | |
... and [57 files with indirect coverage changes](https://app.codecov.io/gh/apache/pinot/pull/11216/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org