You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "ankitsultana (via GitHub)" <gi...@apache.org> on 2023/10/08 05:41:57 UTC
[PR] [experiment / poc] Dictionary Based Group-By [pinot]
ankitsultana opened a new pull request, #11758:
URL: https://github.com/apache/pinot/pull/11758
This is a experimental PR mainly for perf testing and PoC. Discussion can be done in this issue:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [experiment / poc] Dictionary Based Group-By [pinot]
Posted by "wirybeaver (via GitHub)" <gi...@apache.org>.
wirybeaver commented on code in PR #11758:
URL: https://github.com/apache/pinot/pull/11758#discussion_r1412993320
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/json/ImmutableJsonIndexReader.java:
##########
@@ -101,6 +103,23 @@ public MutableRoaringBitmap getMatchingDocIds(String filterString) {
}
}
+ public ImmutableRoaringBitmap getDocIds(int dictId) {
+ ImmutableRoaringBitmap flattenedDocIds = _invertedIndex.getDocIds(dictId);
+ MutableRoaringBitmap result = new MutableRoaringBitmap();
+ flattenedDocIds.stream().forEach(f -> result.add(getDocId(f)));
+ return result;
+ }
+
+ public Pair<Object[], Integer> getValues(String key) {
+ int startIndex = _dictionary.indexOf(key + JsonIndexCreator.KEY_VALUE_SEPARATOR);
Review Comment:
I realized that the current json indexing store the whole key/value pair as a sorted string table without compression even though the key is the common prefix when I see this two lines of code :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [experiment / poc] Dictionary Based Group-By [pinot]
Posted by "ankitsultana (via GitHub)" <gi...@apache.org>.
ankitsultana commented on PR #11758:
URL: https://github.com/apache/pinot/pull/11758#issuecomment-2116307570
We never got around to prioritizing this. Closing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [experiment / poc] Dictionary Based Group-By [pinot]
Posted by "ankitsultana (via GitHub)" <gi...@apache.org>.
ankitsultana closed pull request #11758: [experiment / poc] Dictionary Based Group-By
URL: https://github.com/apache/pinot/pull/11758
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [experiment / poc] Dictionary Based Group-By [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #11758:
URL: https://github.com/apache/pinot/pull/11758#issuecomment-1751935203
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11758?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#11758](https://app.codecov.io/gh/apache/pinot/pull/11758?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (7bcf791) into [master](https://app.codecov.io/gh/apache/pinot/commit/ae1681273c4929587729d71c6dc9526c62ca1fcc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (ae16812) will **decrease** coverage by `48.72%`.
> Report is 23 commits behind head on master.
> The diff coverage is `0.00%`.
```diff
@@ Coverage Diff @@
## master #11758 +/- ##
=============================================
- Coverage 63.11% 14.39% -48.72%
+ Complexity 1117 201 -916
=============================================
Files 2342 2347 +5
Lines 125802 126095 +293
Branches 19336 19398 +62
=============================================
- Hits 79395 18149 -61246
- Misses 40745 106410 +65665
+ Partials 5662 1536 -4126
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/11758/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [integration](https://app.codecov.io/gh/apache/pinot/pull/11758/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/11758/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/11758/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/11758/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-17](https://app.codecov.io/gh/apache/pinot/pull/11758/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-20](https://app.codecov.io/gh/apache/pinot/pull/11758/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.39% <0.00%> (-48.56%)` | :arrow_down: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/11758/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.39% <0.00%> (-48.72%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/11758/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.39% <0.00%> (-48.72%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/11758/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/11758/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.39% <0.00%> (-0.09%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files](https://app.codecov.io/gh/apache/pinot/pull/11758?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...gment/index/readers/BitmapInvertedIndexReader.java](https://app.codecov.io/gh/apache/pinot/pull/11758?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L3JlYWRlcnMvQml0bWFwSW52ZXJ0ZWRJbmRleFJlYWRlci5qYXZh) | `0.00% <ø> (-100.00%)` | :arrow_down: |
| [...ot/segment/spi/index/creator/JsonIndexCreator.java](https://app.codecov.io/gh/apache/pinot/pull/11758?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2luZGV4L2NyZWF0b3IvSnNvbkluZGV4Q3JlYXRvci5qYXZh) | `0.00% <ø> (-66.67%)` | :arrow_down: |
| [...m/function/JsonExtractScalarTransformFunction.java](https://app.codecov.io/gh/apache/pinot/pull/11758?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vSnNvbkV4dHJhY3RTY2FsYXJUcmFuc2Zvcm1GdW5jdGlvbi5qYXZh) | `0.00% <0.00%> (-54.07%)` | :arrow_down: |
| [...re/common/inverted/InvertedDataFetcherFactory.java](https://app.codecov.io/gh/apache/pinot/pull/11758?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vaW52ZXJ0ZWQvSW52ZXJ0ZWREYXRhRmV0Y2hlckZhY3RvcnkuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...t/index/readers/json/ImmutableJsonIndexReader.java](https://app.codecov.io/gh/apache/pinot/pull/11758?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L3JlYWRlcnMvanNvbi9JbW11dGFibGVKc29uSW5kZXhSZWFkZXIuamF2YQ==) | `0.00% <0.00%> (-75.37%)` | :arrow_down: |
| [...aggregation/groupby/InvertedGroupKeyGenerator.java](https://app.codecov.io/gh/apache/pinot/pull/11758?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9hZ2dyZWdhdGlvbi9ncm91cGJ5L0ludmVydGVkR3JvdXBLZXlHZW5lcmF0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [.../common/inverted/InvertedJsonIndexDataFetcher.java](https://app.codecov.io/gh/apache/pinot/pull/11758?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vaW52ZXJ0ZWQvSW52ZXJ0ZWRKc29uSW5kZXhEYXRhRmV0Y2hlci5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...core/common/inverted/InvertedIndexDataFetcher.java](https://app.codecov.io/gh/apache/pinot/pull/11758?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vaW52ZXJ0ZWQvSW52ZXJ0ZWRJbmRleERhdGFGZXRjaGVyLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...local/realtime/impl/json/MutableJsonIndexImpl.java](https://app.codecov.io/gh/apache/pinot/pull/11758?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9pbXBsL2pzb24vTXV0YWJsZUpzb25JbmRleEltcGwuamF2YQ==) | `0.00% <0.00%> (-88.00%)` | :arrow_down: |
| [...va/org/apache/pinot/core/plan/GroupByPlanNode.java](https://app.codecov.io/gh/apache/pinot/pull/11758?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0dyb3VwQnlQbGFuTm9kZS5qYXZh) | `0.00% <0.00%> (-56.42%)` | :arrow_down: |
| ... and [1 more](https://app.codecov.io/gh/apache/pinot/pull/11758?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | |
... and [1533 files with indirect coverage changes](https://app.codecov.io/gh/apache/pinot/pull/11758/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org