You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/06/16 00:59:32 UTC
[GitHub] [incubator-pinot] GSharayu opened a new pull request #7063: Add support for range index rule recommendation(#7034)
GSharayu opened a new pull request #7063:
URL: https://github.com/apache/incubator-pinot/pull/7063
Currently,
As part of pinot automated onboarding, we support automated inverted index and sorted index recommendations.
Wiki Doc: https://cwiki.apache.org/confluence/display/PINOT/Automated+Inverted+Index+Recommendation+for+Pinot
This issue will add an automated support to recommend range index in pinot.
Issue #7034
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-commenter edited a comment on pull request #7063: Add support for range index rule recommendation(#7034)
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7063:
URL: https://github.com/apache/incubator-pinot/pull/7063#issuecomment-861983097
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/7063?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7063](https://codecov.io/gh/apache/incubator-pinot/pull/7063?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (019b6cf) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/f7732082dde93a590488d1e5db2b583050c9fab1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f773208) will **increase** coverage by `0.03%`.
> The diff coverage is `85.96%`.
> :exclamation: Current head 019b6cf differs from pull request most recent head 955e47c. Consider uploading reports for the commit 955e47c to get more accurate results
[](https://codecov.io/gh/apache/incubator-pinot/pull/7063?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7063 +/- ##
============================================
+ Coverage 73.64% 73.67% +0.03%
Complexity 91 91
============================================
Files 1480 1482 +2
Lines 72871 72926 +55
Branches 10483 10493 +10
============================================
+ Hits 53664 53728 +64
+ Misses 15734 15727 -7
+ Partials 3473 3471 -2
```
| Flag | Coverage Δ | |
|---|---|---|
| integration | `41.86% <0.00%> (-0.06%)` | :arrow_down: |
| unittests | `65.58% <85.96%> (+<0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/7063?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ommender/rules/io/params/RecommenderConstants.java](https://codecov.io/gh/apache/incubator-pinot/pull/7063/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9yZWNvbW1lbmRlci9ydWxlcy9pby9wYXJhbXMvUmVjb21tZW5kZXJDb25zdGFudHMuamF2YQ==) | `16.66% <0.00%> (-1.52%)` | :arrow_down: |
| [...ommender/rules/io/params/RangeIndexRuleParams.java](https://codecov.io/gh/apache/incubator-pinot/pull/7063/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9yZWNvbW1lbmRlci9ydWxlcy9pby9wYXJhbXMvUmFuZ2VJbmRleFJ1bGVQYXJhbXMuamF2YQ==) | `40.00% <40.00%> (ø)` | |
| [...t/controller/recommender/rules/RulesToExecute.java](https://codecov.io/gh/apache/incubator-pinot/pull/7063/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9yZWNvbW1lbmRlci9ydWxlcy9SdWxlc1RvRXhlY3V0ZS5qYXZh) | `78.87% <66.66%> (-1.13%)` | :arrow_down: |
| [...troller/recommender/rules/impl/RangeIndexRule.java](https://codecov.io/gh/apache/incubator-pinot/pull/7063/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9yZWNvbW1lbmRlci9ydWxlcy9pbXBsL1JhbmdlSW5kZXhSdWxlLmphdmE=) | `95.23% <95.23%> (ø)` | |
| [.../pinot/controller/recommender/io/InputManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/7063/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9yZWNvbW1lbmRlci9pby9JbnB1dE1hbmFnZXIuamF2YQ==) | `91.53% <100.00%> (+0.06%)` | :arrow_up: |
| [...es/impl/NoDictionaryOnHeapDictionaryJointRule.java](https://codecov.io/gh/apache/incubator-pinot/pull/7063/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9yZWNvbW1lbmRlci9ydWxlcy9pbXBsL05vRGljdGlvbmFyeU9uSGVhcERpY3Rpb25hcnlKb2ludFJ1bGUuamF2YQ==) | `93.33% <100.00%> (-0.07%)` | :arrow_down: |
| [...a/manager/realtime/RealtimeSegmentDataManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/7063/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVTZWdtZW50RGF0YU1hbmFnZXIuamF2YQ==) | `50.00% <0.00%> (-50.00%)` | :arrow_down: |
| [...readers/constant/ConstantMVForwardIndexReader.java](https://codecov.io/gh/apache/incubator-pinot/pull/7063/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L3JlYWRlcnMvY29uc3RhbnQvQ29uc3RhbnRNVkZvcndhcmRJbmRleFJlYWRlci5qYXZh) | `14.28% <0.00%> (-28.58%)` | :arrow_down: |
| [...ot/segment/local/customobject/MinMaxRangePair.java](https://codecov.io/gh/apache/incubator-pinot/pull/7063/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9jdXN0b21vYmplY3QvTWluTWF4UmFuZ2VQYWlyLmphdmE=) | `75.86% <0.00%> (-24.14%)` | :arrow_down: |
| [...cheduler/resources/PolicyBasedResourceManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/7063/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9zY2hlZHVsZXIvcmVzb3VyY2VzL1BvbGljeUJhc2VkUmVzb3VyY2VNYW5hZ2VyLmphdmE=) | `75.00% <0.00%> (-18.75%)` | :arrow_down: |
| ... and [25 more](https://codecov.io/gh/apache/incubator-pinot/pull/7063/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/7063?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/7063?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f773208...955e47c](https://codecov.io/gh/apache/incubator-pinot/pull/7063?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] GSharayu commented on a change in pull request #7063: Add support for range index rule recommendation(#7034)
Posted by GitBox <gi...@apache.org>.
GSharayu commented on a change in pull request #7063:
URL: https://github.com/apache/incubator-pinot/pull/7063#discussion_r655537060
##########
File path: pinot-controller/src/main/java/org/apache/pinot/controller/recommender/rules/impl/NoDictionaryOnHeapDictionaryJointRule.java
##########
@@ -85,7 +85,7 @@ public void run() {
noDictCols.removeAll(_output.getIndexConfig().getInvertedIndexColumns());
noDictCols.remove(_output.getIndexConfig().getSortedColumn());
noDictCols.removeAll(_output.getIndexConfig()
- .getRangeIndexColumns()); // TODO: Remove this after range index is implemented for no-dictionary
+ .getRangeIndexColumns());
Review comment:
updated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-commenter edited a comment on pull request #7063: Add support for range index rule recommendation(#7034)
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7063:
URL: https://github.com/apache/incubator-pinot/pull/7063#issuecomment-861983097
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/7063?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7063](https://codecov.io/gh/apache/incubator-pinot/pull/7063?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (019b6cf) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/f7732082dde93a590488d1e5db2b583050c9fab1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f773208) will **decrease** coverage by `8.05%`.
> The diff coverage is `85.96%`.
> :exclamation: Current head 019b6cf differs from pull request most recent head 955e47c. Consider uploading reports for the commit 955e47c to get more accurate results
[](https://codecov.io/gh/apache/incubator-pinot/pull/7063?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7063 +/- ##
============================================
- Coverage 73.64% 65.58% -8.06%
Complexity 91 91
============================================
Files 1480 1482 +2
Lines 72871 72926 +55
Branches 10483 10493 +10
============================================
- Hits 53664 47832 -5832
- Misses 15734 21734 +6000
+ Partials 3473 3360 -113
```
| Flag | Coverage Δ | |
|---|---|---|
| integration | `?` | |
| unittests | `65.58% <85.96%> (+<0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/7063?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ommender/rules/io/params/RecommenderConstants.java](https://codecov.io/gh/apache/incubator-pinot/pull/7063/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9yZWNvbW1lbmRlci9ydWxlcy9pby9wYXJhbXMvUmVjb21tZW5kZXJDb25zdGFudHMuamF2YQ==) | `16.66% <0.00%> (-1.52%)` | :arrow_down: |
| [...ommender/rules/io/params/RangeIndexRuleParams.java](https://codecov.io/gh/apache/incubator-pinot/pull/7063/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9yZWNvbW1lbmRlci9ydWxlcy9pby9wYXJhbXMvUmFuZ2VJbmRleFJ1bGVQYXJhbXMuamF2YQ==) | `40.00% <40.00%> (ø)` | |
| [...t/controller/recommender/rules/RulesToExecute.java](https://codecov.io/gh/apache/incubator-pinot/pull/7063/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9yZWNvbW1lbmRlci9ydWxlcy9SdWxlc1RvRXhlY3V0ZS5qYXZh) | `78.87% <66.66%> (-1.13%)` | :arrow_down: |
| [...troller/recommender/rules/impl/RangeIndexRule.java](https://codecov.io/gh/apache/incubator-pinot/pull/7063/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9yZWNvbW1lbmRlci9ydWxlcy9pbXBsL1JhbmdlSW5kZXhSdWxlLmphdmE=) | `95.23% <95.23%> (ø)` | |
| [.../pinot/controller/recommender/io/InputManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/7063/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9yZWNvbW1lbmRlci9pby9JbnB1dE1hbmFnZXIuamF2YQ==) | `91.53% <100.00%> (+0.06%)` | :arrow_up: |
| [...es/impl/NoDictionaryOnHeapDictionaryJointRule.java](https://codecov.io/gh/apache/incubator-pinot/pull/7063/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9yZWNvbW1lbmRlci9ydWxlcy9pbXBsL05vRGljdGlvbmFyeU9uSGVhcERpY3Rpb25hcnlKb2ludFJ1bGUuamF2YQ==) | `93.33% <100.00%> (-0.07%)` | :arrow_down: |
| [...a/org/apache/pinot/minion/metrics/MinionMeter.java](https://codecov.io/gh/apache/incubator-pinot/pull/7063/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vbWV0cmljcy9NaW5pb25NZXRlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/BrokerQueryPhase.java](https://codecov.io/gh/apache/incubator-pinot/pull/7063/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Ccm9rZXJRdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/minion/metrics/MinionQueryPhase.java](https://codecov.io/gh/apache/incubator-pinot/pull/7063/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vbWV0cmljcy9NaW5pb25RdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pache/pinot/common/utils/grpc/GrpcQueryClient.java](https://codecov.io/gh/apache/incubator-pinot/pull/7063/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvZ3JwYy9HcnBjUXVlcnlDbGllbnQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [355 more](https://codecov.io/gh/apache/incubator-pinot/pull/7063/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/7063?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/7063?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [f773208...955e47c](https://codecov.io/gh/apache/incubator-pinot/pull/7063?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-commenter commented on pull request #7063: Add support for range index rule recommendation(#7034)
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #7063:
URL: https://github.com/apache/incubator-pinot/pull/7063#issuecomment-861983097
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/7063?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7063](https://codecov.io/gh/apache/incubator-pinot/pull/7063?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (86dfb9e) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/0a020b4558c815490b01516fb41556e6f9420406?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0a020b4) will **decrease** coverage by `31.76%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/incubator-pinot/pull/7063?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7063 +/- ##
=============================================
- Coverage 73.59% 41.83% -31.77%
+ Complexity 91 7 -84
=============================================
Files 1477 1479 +2
Lines 72800 72861 +61
Branches 10471 10483 +12
=============================================
- Hits 53576 30480 -23096
- Misses 15761 39838 +24077
+ Partials 3463 2543 -920
```
| Flag | Coverage Δ | |
|---|---|---|
| integration | `41.83% <0.00%> (-0.03%)` | :arrow_down: |
| unittests | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/7063?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../pinot/controller/recommender/io/InputManager.java](https://codecov.io/gh/apache/incubator-pinot/pull/7063/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9yZWNvbW1lbmRlci9pby9JbnB1dE1hbmFnZXIuamF2YQ==) | `0.00% <0.00%> (-91.47%)` | :arrow_down: |
| [...t/controller/recommender/rules/RulesToExecute.java](https://codecov.io/gh/apache/incubator-pinot/pull/7063/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9yZWNvbW1lbmRlci9ydWxlcy9SdWxlc1RvRXhlY3V0ZS5qYXZh) | `0.00% <0.00%> (-80.00%)` | :arrow_down: |
| [...es/impl/NoDictionaryOnHeapDictionaryJointRule.java](https://codecov.io/gh/apache/incubator-pinot/pull/7063/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9yZWNvbW1lbmRlci9ydWxlcy9pbXBsL05vRGljdGlvbmFyeU9uSGVhcERpY3Rpb25hcnlKb2ludFJ1bGUuamF2YQ==) | `0.00% <0.00%> (-93.40%)` | :arrow_down: |
| [...troller/recommender/rules/impl/RangeIndexRule.java](https://codecov.io/gh/apache/incubator-pinot/pull/7063/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9yZWNvbW1lbmRlci9ydWxlcy9pbXBsL1JhbmdlSW5kZXhSdWxlLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...ommender/rules/io/params/RangeIndexRuleParams.java](https://codecov.io/gh/apache/incubator-pinot/pull/7063/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9yZWNvbW1lbmRlci9ydWxlcy9pby9wYXJhbXMvUmFuZ2VJbmRleFJ1bGVQYXJhbXMuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...ommender/rules/io/params/RecommenderConstants.java](https://codecov.io/gh/apache/incubator-pinot/pull/7063/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9yZWNvbW1lbmRlci9ydWxlcy9pby9wYXJhbXMvUmVjb21tZW5kZXJDb25zdGFudHMuamF2YQ==) | `0.00% <0.00%> (-18.19%)` | :arrow_down: |
| [...c/main/java/org/apache/pinot/common/tier/Tier.java](https://codecov.io/gh/apache/incubator-pinot/pull/7063/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdGllci9UaWVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...ava/org/apache/pinot/spi/data/MetricFieldSpec.java](https://codecov.io/gh/apache/incubator-pinot/pull/7063/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9NZXRyaWNGaWVsZFNwZWMuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...va/org/apache/pinot/spi/utils/BigDecimalUtils.java](https://codecov.io/gh/apache/incubator-pinot/pull/7063/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQmlnRGVjaW1hbFV0aWxzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...java/org/apache/pinot/common/tier/TierFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/7063/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdGllci9UaWVyRmFjdG9yeS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [933 more](https://codecov.io/gh/apache/incubator-pinot/pull/7063/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/7063?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/7063?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [0a020b4...86dfb9e](https://codecov.io/gh/apache/incubator-pinot/pull/7063?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia merged pull request #7063: Add support for range index rule recommendation(#7034)
Posted by GitBox <gi...@apache.org>.
siddharthteotia merged pull request #7063:
URL: https://github.com/apache/incubator-pinot/pull/7063
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] jackjlli commented on a change in pull request #7063: Add support for range index rule recommendation(#7034)
Posted by GitBox <gi...@apache.org>.
jackjlli commented on a change in pull request #7063:
URL: https://github.com/apache/incubator-pinot/pull/7063#discussion_r654704283
##########
File path: pinot-controller/src/main/java/org/apache/pinot/controller/recommender/rules/impl/RangeIndexRule.java
##########
@@ -0,0 +1,117 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.controller.recommender.rules.impl;
+
+import com.google.common.util.concurrent.AtomicDouble;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.common.request.context.FilterContext;
+import org.apache.pinot.common.request.context.predicate.Predicate;
+import org.apache.pinot.controller.recommender.io.ConfigManager;
+import org.apache.pinot.controller.recommender.io.InputManager;
+import org.apache.pinot.controller.recommender.rules.AbstractRule;
+import org.apache.pinot.controller.recommender.rules.io.params.RangeIndexRuleParams;
+import org.apache.pinot.controller.recommender.rules.utils.FixedLenBitset;
+import org.apache.pinot.core.query.request.context.QueryContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Create range-index for columns used in <, > and between predicate
+ * Skip the inverted and sorted index columns if recommended
+ */
+public class RangeIndexRule extends AbstractRule {
+ private final Logger LOGGER = LoggerFactory.getLogger(RangeIndexRule.class);
+ private final RangeIndexRuleParams _params;
+
+ public RangeIndexRule(InputManager input, ConfigManager output) {
+ super(input, output);
+ _params = input.getRangeIndexRuleParams();
+ }
+
+ @Override
+ public void run() {
+ int numCols = _input.getNumCols();
+ double[] weights = new double[numCols];
+ AtomicDouble totalWeight = new AtomicDouble(0);
+
+ // For each query, find out the range columns
+ // and accumulate the (weighted) frequencies
+ _input.getParsedQueries().forEach(query -> {
+ Double weight = _input.getQueryWeight(query);
+ totalWeight.addAndGet(weight);
+ FixedLenBitset fixedLenBitset = parseQuery(_input.getQueryContext(query));
+ LOGGER.debug("fixedLenBitset {}", fixedLenBitset);
+ for (Integer i : fixedLenBitset.getOffsets()) {
+ weights[i] += weight;
+ }
+ });
+ LOGGER.debug("Weight: {}, Total {}", weights, totalWeight);
+
+ for (int i = 0; i < numCols; i++) {
+ String colName = _input.intToColName(i);
+ // This checks if column is not already recommended for inverted index or sorted index
+ // As currently, only numeric columns are selected in range index creation, we will skip non numeric columns
+ if (((weights[i] / totalWeight.get()) > _params.THRESHOLD_MIN_PERCENT_RANGE_INDEX) &&
+ !_output.getIndexConfig().getSortedColumn().equals(colName) &&
+ !_output.getIndexConfig().getInvertedIndexColumns().contains(colName) &&
+ _input.getFieldType(colName).isNumeric()) {
+ _output.getIndexConfig().getRangeIndexColumns().add(colName);
+ }
+ }
+ }
+
+ public FixedLenBitset parseQuery(QueryContext queryContext) {
+ if (queryContext.getFilter() == null) {
+ return FixedLenBitset.IMMUTABLE_EMPTY_SET;
+ }
+
+ LOGGER.trace("Parsing Where Clause: {}", queryContext.getFilter().toString());
Review comment:
We might not need log in trace level.
##########
File path: pinot-controller/src/main/java/org/apache/pinot/controller/recommender/rules/impl/RangeIndexRule.java
##########
@@ -0,0 +1,117 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.controller.recommender.rules.impl;
+
+import com.google.common.util.concurrent.AtomicDouble;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.common.request.context.FilterContext;
+import org.apache.pinot.common.request.context.predicate.Predicate;
+import org.apache.pinot.controller.recommender.io.ConfigManager;
+import org.apache.pinot.controller.recommender.io.InputManager;
+import org.apache.pinot.controller.recommender.rules.AbstractRule;
+import org.apache.pinot.controller.recommender.rules.io.params.RangeIndexRuleParams;
+import org.apache.pinot.controller.recommender.rules.utils.FixedLenBitset;
+import org.apache.pinot.core.query.request.context.QueryContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Create range-index for columns used in <, > and between predicate
+ * Skip the inverted and sorted index columns if recommended
+ */
+public class RangeIndexRule extends AbstractRule {
+ private final Logger LOGGER = LoggerFactory.getLogger(RangeIndexRule.class);
+ private final RangeIndexRuleParams _params;
+
+ public RangeIndexRule(InputManager input, ConfigManager output) {
+ super(input, output);
+ _params = input.getRangeIndexRuleParams();
+ }
+
+ @Override
+ public void run() {
+ int numCols = _input.getNumCols();
+ double[] weights = new double[numCols];
+ AtomicDouble totalWeight = new AtomicDouble(0);
+
+ // For each query, find out the range columns
+ // and accumulate the (weighted) frequencies
+ _input.getParsedQueries().forEach(query -> {
+ Double weight = _input.getQueryWeight(query);
+ totalWeight.addAndGet(weight);
+ FixedLenBitset fixedLenBitset = parseQuery(_input.getQueryContext(query));
+ LOGGER.debug("fixedLenBitset {}", fixedLenBitset);
+ for (Integer i : fixedLenBitset.getOffsets()) {
+ weights[i] += weight;
+ }
+ });
+ LOGGER.debug("Weight: {}, Total {}", weights, totalWeight);
Review comment:
Set it to `info` level since it's printed only once?
##########
File path: pinot-controller/src/main/java/org/apache/pinot/controller/recommender/rules/impl/NoDictionaryOnHeapDictionaryJointRule.java
##########
@@ -85,7 +85,7 @@ public void run() {
noDictCols.removeAll(_output.getIndexConfig().getInvertedIndexColumns());
noDictCols.remove(_output.getIndexConfig().getSortedColumn());
noDictCols.removeAll(_output.getIndexConfig()
- .getRangeIndexColumns()); // TODO: Remove this after range index is implemented for no-dictionary
+ .getRangeIndexColumns());
Review comment:
nit: merge this line to the previous line?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on pull request #7063: Add support for range index rule recommendation(#7034)
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on pull request #7063:
URL: https://github.com/apache/incubator-pinot/pull/7063#issuecomment-865314382
We can get this in since basis stuff is done but 2 follow-ups are must
- Remove the limitation that if the column already has inverted/sorted index, we will not recommend range index on it. It is perfectly fine to have multiple indexes on the same column. We should not make the algorithm restrictive by checking for that and instead let the query pattern tell us if it is needed.
- Test needs to be improved. We need to account for following cases as well
-- Same column is present multiple times in the same query and different queries with =, IN and range predicates. This will anyway be needed when you add support for the above
-- Same column is present multiple times in the same query and different queries with RANGE predicates to alter the weight of the predicate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia merged pull request #7063: Add support for range index rule recommendation(#7034)
Posted by GitBox <gi...@apache.org>.
siddharthteotia merged pull request #7063:
URL: https://github.com/apache/incubator-pinot/pull/7063
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] GSharayu commented on a change in pull request #7063: Add support for range index rule recommendation(#7034)
Posted by GitBox <gi...@apache.org>.
GSharayu commented on a change in pull request #7063:
URL: https://github.com/apache/incubator-pinot/pull/7063#discussion_r655537183
##########
File path: pinot-controller/src/main/java/org/apache/pinot/controller/recommender/rules/impl/RangeIndexRule.java
##########
@@ -0,0 +1,117 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.controller.recommender.rules.impl;
+
+import com.google.common.util.concurrent.AtomicDouble;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.common.request.context.FilterContext;
+import org.apache.pinot.common.request.context.predicate.Predicate;
+import org.apache.pinot.controller.recommender.io.ConfigManager;
+import org.apache.pinot.controller.recommender.io.InputManager;
+import org.apache.pinot.controller.recommender.rules.AbstractRule;
+import org.apache.pinot.controller.recommender.rules.io.params.RangeIndexRuleParams;
+import org.apache.pinot.controller.recommender.rules.utils.FixedLenBitset;
+import org.apache.pinot.core.query.request.context.QueryContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Create range-index for columns used in <, > and between predicate
+ * Skip the inverted and sorted index columns if recommended
+ */
+public class RangeIndexRule extends AbstractRule {
+ private final Logger LOGGER = LoggerFactory.getLogger(RangeIndexRule.class);
+ private final RangeIndexRuleParams _params;
+
+ public RangeIndexRule(InputManager input, ConfigManager output) {
+ super(input, output);
+ _params = input.getRangeIndexRuleParams();
+ }
+
+ @Override
+ public void run() {
+ int numCols = _input.getNumCols();
+ double[] weights = new double[numCols];
+ AtomicDouble totalWeight = new AtomicDouble(0);
+
+ // For each query, find out the range columns
+ // and accumulate the (weighted) frequencies
+ _input.getParsedQueries().forEach(query -> {
+ Double weight = _input.getQueryWeight(query);
+ totalWeight.addAndGet(weight);
+ FixedLenBitset fixedLenBitset = parseQuery(_input.getQueryContext(query));
+ LOGGER.debug("fixedLenBitset {}", fixedLenBitset);
+ for (Integer i : fixedLenBitset.getOffsets()) {
+ weights[i] += weight;
+ }
+ });
+ LOGGER.debug("Weight: {}, Total {}", weights, totalWeight);
Review comment:
updated!
##########
File path: pinot-controller/src/main/java/org/apache/pinot/controller/recommender/rules/impl/RangeIndexRule.java
##########
@@ -0,0 +1,117 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.controller.recommender.rules.impl;
+
+import com.google.common.util.concurrent.AtomicDouble;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.common.request.context.FilterContext;
+import org.apache.pinot.common.request.context.predicate.Predicate;
+import org.apache.pinot.controller.recommender.io.ConfigManager;
+import org.apache.pinot.controller.recommender.io.InputManager;
+import org.apache.pinot.controller.recommender.rules.AbstractRule;
+import org.apache.pinot.controller.recommender.rules.io.params.RangeIndexRuleParams;
+import org.apache.pinot.controller.recommender.rules.utils.FixedLenBitset;
+import org.apache.pinot.core.query.request.context.QueryContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Create range-index for columns used in <, > and between predicate
+ * Skip the inverted and sorted index columns if recommended
+ */
+public class RangeIndexRule extends AbstractRule {
+ private final Logger LOGGER = LoggerFactory.getLogger(RangeIndexRule.class);
+ private final RangeIndexRuleParams _params;
+
+ public RangeIndexRule(InputManager input, ConfigManager output) {
+ super(input, output);
+ _params = input.getRangeIndexRuleParams();
+ }
+
+ @Override
+ public void run() {
+ int numCols = _input.getNumCols();
+ double[] weights = new double[numCols];
+ AtomicDouble totalWeight = new AtomicDouble(0);
+
+ // For each query, find out the range columns
+ // and accumulate the (weighted) frequencies
+ _input.getParsedQueries().forEach(query -> {
+ Double weight = _input.getQueryWeight(query);
+ totalWeight.addAndGet(weight);
+ FixedLenBitset fixedLenBitset = parseQuery(_input.getQueryContext(query));
+ LOGGER.debug("fixedLenBitset {}", fixedLenBitset);
+ for (Integer i : fixedLenBitset.getOffsets()) {
+ weights[i] += weight;
+ }
+ });
+ LOGGER.debug("Weight: {}, Total {}", weights, totalWeight);
+
+ for (int i = 0; i < numCols; i++) {
+ String colName = _input.intToColName(i);
+ // This checks if column is not already recommended for inverted index or sorted index
+ // As currently, only numeric columns are selected in range index creation, we will skip non numeric columns
+ if (((weights[i] / totalWeight.get()) > _params.THRESHOLD_MIN_PERCENT_RANGE_INDEX) &&
+ !_output.getIndexConfig().getSortedColumn().equals(colName) &&
+ !_output.getIndexConfig().getInvertedIndexColumns().contains(colName) &&
+ _input.getFieldType(colName).isNumeric()) {
+ _output.getIndexConfig().getRangeIndexColumns().add(colName);
+ }
+ }
+ }
+
+ public FixedLenBitset parseQuery(QueryContext queryContext) {
+ if (queryContext.getFilter() == null) {
+ return FixedLenBitset.IMMUTABLE_EMPTY_SET;
+ }
+
+ LOGGER.trace("Parsing Where Clause: {}", queryContext.getFilter().toString());
Review comment:
updated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] siddharthteotia commented on pull request #7063: Add support for range index rule recommendation(#7034)
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on pull request #7063:
URL: https://github.com/apache/incubator-pinot/pull/7063#issuecomment-865314382
We can get this in since basis stuff is done but 2 follow-ups are must
- Remove the limitation that if the column already has inverted/sorted index, we will not recommend range index on it. It is perfectly fine to have multiple indexes on the same column. We should not make the algorithm restrictive by checking for that and instead let the query pattern tell us if it is needed.
- Test needs to be improved. We need to account for following cases as well
-- Same column is present multiple times in the same query and different queries with =, IN and range predicates. This will anyway be needed when you add support for the above
-- Same column is present multiple times in the same query and different queries with RANGE predicates to alter the weight of the predicate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-commenter edited a comment on pull request #7063: Add support for range index rule recommendation(#7034)
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7063:
URL: https://github.com/apache/incubator-pinot/pull/7063#issuecomment-861983097
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] GSharayu commented on a change in pull request #7063: Add support for range index rule recommendation(#7034)
Posted by GitBox <gi...@apache.org>.
GSharayu commented on a change in pull request #7063:
URL: https://github.com/apache/incubator-pinot/pull/7063#discussion_r655537060
##########
File path: pinot-controller/src/main/java/org/apache/pinot/controller/recommender/rules/impl/NoDictionaryOnHeapDictionaryJointRule.java
##########
@@ -85,7 +85,7 @@ public void run() {
noDictCols.removeAll(_output.getIndexConfig().getInvertedIndexColumns());
noDictCols.remove(_output.getIndexConfig().getSortedColumn());
noDictCols.removeAll(_output.getIndexConfig()
- .getRangeIndexColumns()); // TODO: Remove this after range index is implemented for no-dictionary
+ .getRangeIndexColumns());
Review comment:
updated!
##########
File path: pinot-controller/src/main/java/org/apache/pinot/controller/recommender/rules/impl/RangeIndexRule.java
##########
@@ -0,0 +1,117 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.controller.recommender.rules.impl;
+
+import com.google.common.util.concurrent.AtomicDouble;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.common.request.context.FilterContext;
+import org.apache.pinot.common.request.context.predicate.Predicate;
+import org.apache.pinot.controller.recommender.io.ConfigManager;
+import org.apache.pinot.controller.recommender.io.InputManager;
+import org.apache.pinot.controller.recommender.rules.AbstractRule;
+import org.apache.pinot.controller.recommender.rules.io.params.RangeIndexRuleParams;
+import org.apache.pinot.controller.recommender.rules.utils.FixedLenBitset;
+import org.apache.pinot.core.query.request.context.QueryContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Create range-index for columns used in <, > and between predicate
+ * Skip the inverted and sorted index columns if recommended
+ */
+public class RangeIndexRule extends AbstractRule {
+ private final Logger LOGGER = LoggerFactory.getLogger(RangeIndexRule.class);
+ private final RangeIndexRuleParams _params;
+
+ public RangeIndexRule(InputManager input, ConfigManager output) {
+ super(input, output);
+ _params = input.getRangeIndexRuleParams();
+ }
+
+ @Override
+ public void run() {
+ int numCols = _input.getNumCols();
+ double[] weights = new double[numCols];
+ AtomicDouble totalWeight = new AtomicDouble(0);
+
+ // For each query, find out the range columns
+ // and accumulate the (weighted) frequencies
+ _input.getParsedQueries().forEach(query -> {
+ Double weight = _input.getQueryWeight(query);
+ totalWeight.addAndGet(weight);
+ FixedLenBitset fixedLenBitset = parseQuery(_input.getQueryContext(query));
+ LOGGER.debug("fixedLenBitset {}", fixedLenBitset);
+ for (Integer i : fixedLenBitset.getOffsets()) {
+ weights[i] += weight;
+ }
+ });
+ LOGGER.debug("Weight: {}, Total {}", weights, totalWeight);
Review comment:
updated!
##########
File path: pinot-controller/src/main/java/org/apache/pinot/controller/recommender/rules/impl/RangeIndexRule.java
##########
@@ -0,0 +1,117 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.controller.recommender.rules.impl;
+
+import com.google.common.util.concurrent.AtomicDouble;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.common.request.context.FilterContext;
+import org.apache.pinot.common.request.context.predicate.Predicate;
+import org.apache.pinot.controller.recommender.io.ConfigManager;
+import org.apache.pinot.controller.recommender.io.InputManager;
+import org.apache.pinot.controller.recommender.rules.AbstractRule;
+import org.apache.pinot.controller.recommender.rules.io.params.RangeIndexRuleParams;
+import org.apache.pinot.controller.recommender.rules.utils.FixedLenBitset;
+import org.apache.pinot.core.query.request.context.QueryContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Create range-index for columns used in <, > and between predicate
+ * Skip the inverted and sorted index columns if recommended
+ */
+public class RangeIndexRule extends AbstractRule {
+ private final Logger LOGGER = LoggerFactory.getLogger(RangeIndexRule.class);
+ private final RangeIndexRuleParams _params;
+
+ public RangeIndexRule(InputManager input, ConfigManager output) {
+ super(input, output);
+ _params = input.getRangeIndexRuleParams();
+ }
+
+ @Override
+ public void run() {
+ int numCols = _input.getNumCols();
+ double[] weights = new double[numCols];
+ AtomicDouble totalWeight = new AtomicDouble(0);
+
+ // For each query, find out the range columns
+ // and accumulate the (weighted) frequencies
+ _input.getParsedQueries().forEach(query -> {
+ Double weight = _input.getQueryWeight(query);
+ totalWeight.addAndGet(weight);
+ FixedLenBitset fixedLenBitset = parseQuery(_input.getQueryContext(query));
+ LOGGER.debug("fixedLenBitset {}", fixedLenBitset);
+ for (Integer i : fixedLenBitset.getOffsets()) {
+ weights[i] += weight;
+ }
+ });
+ LOGGER.debug("Weight: {}, Total {}", weights, totalWeight);
+
+ for (int i = 0; i < numCols; i++) {
+ String colName = _input.intToColName(i);
+ // This checks if column is not already recommended for inverted index or sorted index
+ // As currently, only numeric columns are selected in range index creation, we will skip non numeric columns
+ if (((weights[i] / totalWeight.get()) > _params.THRESHOLD_MIN_PERCENT_RANGE_INDEX) &&
+ !_output.getIndexConfig().getSortedColumn().equals(colName) &&
+ !_output.getIndexConfig().getInvertedIndexColumns().contains(colName) &&
+ _input.getFieldType(colName).isNumeric()) {
+ _output.getIndexConfig().getRangeIndexColumns().add(colName);
+ }
+ }
+ }
+
+ public FixedLenBitset parseQuery(QueryContext queryContext) {
+ if (queryContext.getFilter() == null) {
+ return FixedLenBitset.IMMUTABLE_EMPTY_SET;
+ }
+
+ LOGGER.trace("Parsing Where Clause: {}", queryContext.getFilter().toString());
Review comment:
updated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org