You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/12/09 16:00:46 UTC
[GitHub] [pinot] richardstartin opened a new pull request #7885: make index creator provision interceptible
richardstartin opened a new pull request #7885:
URL: https://github.com/apache/pinot/pull/7885
This allows index creation to be intercepted, so that the current static logic in `SegmentIndexCreator` can be extended or overridden. This is achieved by introducing a new interface `IndexCreatorProvider` which provides various new index creators from an `IndexCreationContext` which bundles all information about index creation. External users can register a _decorator_ which can enhance or entirely replace the default index creator provision logic. Typically, a registered decorator should pattern match the `IndexCreatorContext` for very specific cases, and return a custom implementation only in those cases, and merely delegate to the default provisioning logic otherwise.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7885: make index creator provision interceptible
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#issuecomment-990031441
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7885](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a28f516) into [master](https://codecov.io/gh/apache/pinot/commit/86278194233781d8001498f0c2be4a8a3299cd9d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8627819) will **decrease** coverage by `0.00%`.
> The diff coverage is `84.95%`.
[](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7885 +/- ##
============================================
- Coverage 65.10% 65.10% -0.01%
- Complexity 4078 4092 +14
============================================
Files 1538 1541 +3
Lines 79998 80128 +130
Branches 12036 12036
============================================
+ Hits 52085 52169 +84
- Misses 24187 24235 +48
+ Partials 3726 3724 -2
```
| Flag | Coverage Δ | |
|---|---|---|
| unittests1 | `68.52% <84.95%> (+0.07%)` | :arrow_up: |
| unittests2 | `14.38% <0.00%> (-0.09%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/pinot/core/minion/RawIndexConverter.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9taW5pb24vUmF3SW5kZXhDb252ZXJ0ZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...t/index/loader/invertedindex/JsonIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0pzb25JbmRleEhhbmRsZXIuamF2YQ==) | `72.58% <57.14%> (-0.31%)` | :arrow_down: |
| [...not/segment/spi/creator/IndexCreatorProviders.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2NyZWF0b3IvSW5kZXhDcmVhdG9yUHJvdmlkZXJzLmphdmE=) | `69.23% <69.23%> (ø)` | |
| [...ment/creator/impl/DefaultIndexCreatorProvider.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9EZWZhdWx0SW5kZXhDcmVhdG9yUHJvdmlkZXIuamF2YQ==) | `69.44% <69.44%> (ø)` | |
| [...ment/creator/impl/SegmentColumnarIndexCreator.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9TZWdtZW50Q29sdW1uYXJJbmRleENyZWF0b3IuamF2YQ==) | `87.01% <88.23%> (+2.62%)` | :arrow_up: |
| [...nt/index/loader/invertedindex/FSTIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0ZTVEluZGV4SGFuZGxlci5qYXZh) | `86.79% <100.00%> (+4.10%)` | :arrow_up: |
| [...ent/index/loader/invertedindex/H3IndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0gzSW5kZXhIYW5kbGVyLmphdmE=) | `74.24% <100.00%> (+1.66%)` | :arrow_up: |
| [...dex/loader/invertedindex/InvertedIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0ludmVydGVkSW5kZXhIYW5kbGVyLmphdmE=) | `94.23% <100.00%> (+0.35%)` | :arrow_up: |
| [...t/index/loader/invertedindex/TextIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L1RleHRJbmRleEhhbmRsZXIuamF2YQ==) | `86.76% <100.00%> (+0.40%)` | :arrow_up: |
| [...inot/segment/spi/creator/IndexCreationContext.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2NyZWF0b3IvSW5kZXhDcmVhdGlvbkNvbnRleHQuamF2YQ==) | `100.00% <100.00%> (ø)` | |
| ... and [10 more](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [8627819...a28f516](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7885: make index creator provision interceptible
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#issuecomment-990031441
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7885](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (4cc35e3) into [master](https://codecov.io/gh/apache/pinot/commit/86278194233781d8001498f0c2be4a8a3299cd9d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8627819) will **decrease** coverage by `0.03%`.
> The diff coverage is `81.70%`.
[](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7885 +/- ##
============================================
- Coverage 65.10% 65.07% -0.04%
- Complexity 4078 4089 +11
============================================
Files 1538 1541 +3
Lines 79998 80079 +81
Branches 12036 12038 +2
============================================
+ Hits 52085 52113 +28
- Misses 24187 24239 +52
- Partials 3726 3727 +1
```
| Flag | Coverage Δ | |
|---|---|---|
| unittests1 | `68.47% <81.70%> (+0.01%)` | :arrow_up: |
| unittests2 | `14.40% <0.00%> (-0.07%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/pinot/core/minion/RawIndexConverter.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9taW5pb24vUmF3SW5kZXhDb252ZXJ0ZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...not/segment/spi/creator/IndexCreatorProviders.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2NyZWF0b3IvSW5kZXhDcmVhdG9yUHJvdmlkZXJzLmphdmE=) | `69.23% <69.23%> (ø)` | |
| [...ment/creator/impl/DefaultIndexCreatorProvider.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9EZWZhdWx0SW5kZXhDcmVhdG9yUHJvdmlkZXIuamF2YQ==) | `73.17% <73.17%> (ø)` | |
| [...ment/creator/impl/SegmentColumnarIndexCreator.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9TZWdtZW50Q29sdW1uYXJJbmRleENyZWF0b3IuamF2YQ==) | `86.76% <88.88%> (+2.37%)` | :arrow_up: |
| [...inot/segment/spi/creator/IndexCreationContext.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2NyZWF0b3IvSW5kZXhDcmVhdGlvbkNvbnRleHQuamF2YQ==) | `100.00% <100.00%> (ø)` | |
| [...er/validation/BrokerResourceValidationManager.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci92YWxpZGF0aW9uL0Jyb2tlclJlc291cmNlVmFsaWRhdGlvbk1hbmFnZXIuamF2YQ==) | `25.00% <0.00%> (-56.25%)` | :arrow_down: |
| [...r/validation/RealtimeSegmentValidationManager.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci92YWxpZGF0aW9uL1JlYWx0aW1lU2VnbWVudFZhbGlkYXRpb25NYW5hZ2VyLmphdmE=) | `37.14% <0.00%> (-34.29%)` | :arrow_down: |
| [...e/pinot/core/transport/InstanceRequestHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS90cmFuc3BvcnQvSW5zdGFuY2VSZXF1ZXN0SGFuZGxlci5qYXZh) | `53.16% <0.00%> (-3.80%)` | :arrow_down: |
| [...gregation/function/StUnionAggregationFunction.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9hZ2dyZWdhdGlvbi9mdW5jdGlvbi9TdFVuaW9uQWdncmVnYXRpb25GdW5jdGlvbi5qYXZh) | `73.52% <0.00%> (-2.95%)` | :arrow_down: |
| [.../pinot/core/query/scheduler/PriorityScheduler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9zY2hlZHVsZXIvUHJpb3JpdHlTY2hlZHVsZXIuamF2YQ==) | `80.82% <0.00%> (-2.74%)` | :arrow_down: |
| ... and [10 more](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [8627819...4cc35e3](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r768314829
##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/IndexCreatorProviders.java
##########
@@ -0,0 +1,143 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.creator;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.lang.invoke.MethodType;
+import java.util.concurrent.atomic.AtomicReference;
+import org.apache.pinot.segment.spi.index.creator.BloomFilterCreator;
+import org.apache.pinot.segment.spi.index.creator.CombinedInvertedIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.DictionaryBasedInvertedIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.GeoSpatialIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.JsonIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.TextIndexCreator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Plugin registration point to allow index creation logic to be swapped out.
+ * Plugins should not reimplement Pinot's default index creation logic.
+ * Users provide an override to Pinot's index creation logic. This is simplified
+ * by extending {@see IndexCreatorProviders.Default}
+ */
+public final class IndexCreatorProviders {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(IndexCreatorProviders.class);
+
+ static final IndexCreatorProvider DEFAULT = defaultProvider();
+ private static final AtomicReference<IndexCreatorProvider> REGISTRATION = new AtomicReference<>(DEFAULT);
+
+ private IndexCreatorProviders() {
+ }
+
+ /**
+ * The caller provides a decorator to wrap the default provider, which allows plugins to create
+ * a delegation chain.
+ * @param provider index creation override
+ * @return true if this is the first invocation and the provider has not yet been used.
+ */
+ public static boolean registerProvider(IndexCreatorProvider provider) {
+ return REGISTRATION.compareAndSet(DEFAULT, provider);
+ }
+
+ /**
+ * Obtain the registered index creator provider. If the user has provided an override, then it will be used instead.
+ * If the user has not provided an override yet, then this action will prevent them from doing so.
+ * @return the global index provision logic.
+ */
+ public static IndexCreatorProvider getIndexCreatorProvider() {
+ return Holder.PROVIDER;
+ }
+
+ private static final class Holder {
+ public static final IndexCreatorProvider PROVIDER = REGISTRATION.get();
+ }
+
+ private static IndexCreatorProvider defaultProvider() {
+ // use MethodHandle to break circular dependency and keep implementation details encapsulated within
+ // pinot-segment-local
+ String className = "org.apache.pinot.segment.local.segment.creator.impl.DefaultIndexCreatorProvider";
+ try {
+ Class<?> clazz = Class.forName(className, false, IndexCreatorProviders.class.getClassLoader());
+ return (IndexCreatorProvider) MethodHandles.publicLookup()
+ .findConstructor(clazz, MethodType.methodType(void.class)).invoke();
+ } catch (Throwable implausible) {
+ LOGGER.error("could not construct MethodHandle for {}", className,
+ implausible);
+ throw new IllegalStateException(implausible);
+ }
+ }
+
+ /**
+ * Extend this class to override index creation
+ */
+ public static class Default implements IndexCreatorProvider {
Review comment:
IIUC, this is the one that allows to mix ? So for example, someone might want to use existing forward index, dictionary and everything else but might want to plugin a different inverted index ? So, this can be sub-classed and `newInvertedIndexCreator` can be overridden to get custom behavior. The sub-class can then be registered ? Is this understanding correct ?
Also, can we try to add one test as an example that can come handy for devs (who in future might change this or related code). May not be needed if example usage pattern is going to be covered in docs later on?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r768311496
##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/DefaultIndexCreatorProvider.java
##########
@@ -0,0 +1,266 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.Map;
+import java.util.Objects;
+import org.apache.pinot.segment.local.io.writer.impl.BaseChunkSVForwardIndexWriter;
+import org.apache.pinot.segment.local.segment.creator.impl.bloom.OnHeapGuavaBloomFilterCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.MultiValueFixedByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.MultiValueUnsortedForwardIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.MultiValueVarByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueFixedByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueSortedForwardIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueUnsortedForwardIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueVarByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.BitSlicedRangeIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.OffHeapBitmapInvertedIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.OnHeapBitmapInvertedIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.RangeIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.geospatial.OffHeapH3IndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.geospatial.OnHeapH3IndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.json.OffHeapJsonIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.json.OnHeapJsonIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.text.LuceneFSTIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.text.LuceneTextIndexCreator;
+import org.apache.pinot.segment.local.utils.nativefst.NativeFSTIndexCreator;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.BloomFilterCreator;
+import org.apache.pinot.segment.spi.index.creator.CombinedInvertedIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.DictionaryBasedInvertedIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.GeoSpatialIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.JsonIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.TextIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.H3IndexResolution;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.config.table.FieldConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+public final class DefaultIndexCreatorProvider implements IndexCreatorProvider {
+
+ @Override
+ public ForwardIndexCreator newForwardIndexCreator(IndexCreationContext context)
+ throws Exception {
+ if (context.isRawColumn()) {
+ boolean deriveNumDocsPerChunk =
+ shouldDeriveNumDocsPerChunk(context.getFieldSpec().getName(), context.getColumnProperties());
+ int writerVersion = rawIndexWriterVersion(context.getFieldSpec().getName(), context.getColumnProperties());
+ if (context.getFieldSpec().isSingleValueField()) {
+ return getRawIndexCreatorForSVColumn(context.getIndexDir(), context.getChunkCompressionType(),
+ context.getFieldSpec().getName(), context.getFieldSpec().getDataType().getStoredType(),
+ context.getTotalDocs(), context.getLengthOfLongestEntry(), deriveNumDocsPerChunk, writerVersion);
+ } else {
+ return getRawIndexCreatorForMVColumn(context.getIndexDir(), context.getChunkCompressionType(),
+ context.getFieldSpec().getName(), context.getFieldSpec().getDataType().getStoredType(),
+ context.getTotalDocs(), context.getMaxNumberOfMultiValueElements(), deriveNumDocsPerChunk, writerVersion,
+ context.getMaxRowLengthInBytes());
+ }
+ } else {
+ if (context.getFieldSpec().isSingleValueField()) {
+ if (context.isSorted()) {
+ return new SingleValueSortedForwardIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(),
+ context.getCardinality());
+ } else {
+ return new SingleValueUnsortedForwardIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(),
+ context.getCardinality(), context.getTotalDocs());
+ }
+ } else {
+ return new MultiValueUnsortedForwardIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(),
+ context.getCardinality(), context.getTotalDocs(), context.getTotalNumberOfEntries());
+ }
+ }
+ }
+
+ @Override
+ public DictionaryBasedInvertedIndexCreator newInvertedIndexCreator(IndexCreationContext context)
+ throws IOException {
+ if (context.isOnHeap()) {
+ return new OnHeapBitmapInvertedIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(),
+ context.getCardinality());
+ } else {
+ return new OffHeapBitmapInvertedIndexCreator(context.getIndexDir(), context.getFieldSpec(),
+ context.getCardinality(), context.getTotalDocs(), context.getTotalNumberOfEntries());
+ }
+ }
+
+ @Override
+ public JsonIndexCreator newJsonIndexCreator(IndexCreationContext context)
+ throws IOException {
+ Preconditions.checkState(context.getFieldSpec().isSingleValueField(),
+ "Json index is currently only supported on single-value columns");
+ Preconditions.checkState(context.getFieldSpec().getDataType().getStoredType() == FieldSpec.DataType.STRING,
+ "Json index is currently only supported on STRING columns");
+ return context.isOnHeap() ? new OnHeapJsonIndexCreator(context.getIndexDir(), context.getFieldSpec().getName())
+ : new OffHeapJsonIndexCreator(context.getIndexDir(), context.getFieldSpec().getName());
+ }
+
+ @Override
+ public TextIndexCreator newTextIndexCreator(IndexCreationContext context)
+ throws IOException {
+ // Initialize text index creator
+ Preconditions.checkState(context.getFieldSpec().getDataType().getStoredType() == FieldSpec.DataType.STRING,
+ "Text index is currently only supported on STRING type columns");
+ return new LuceneTextIndexCreator(context.getFieldSpec().getName(), context.getIndexDir(),
+ true /* commitOnClose */);
+ }
+
+ @Override
+ public TextIndexCreator newFSTIndexCreator(IndexCreationContext context)
+ throws IOException {
+ Preconditions.checkState(context.getFieldSpec().isSingleValueField(),
+ "FST index is currently only supported on single-value columns");
+ Preconditions.checkState(context.getFieldSpec().getDataType().getStoredType() == FieldSpec.DataType.STRING,
+ "FST index is currently only supported on STRING type columns");
+ Preconditions.checkState(!context.isRawColumn(),
+ "FST index is currently only supported on dictionary-encoded columns");
+ String[] sortedValues = (String[]) context.getSortedUniqueElementsArray();
+ if (context.getFstType() == FSTType.NATIVE) {
+ return new NativeFSTIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(), sortedValues);
+ } else {
+ return new LuceneFSTIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(), sortedValues);
+ }
+ }
+
+ @Override
+ public GeoSpatialIndexCreator newGeoSpatialIndexCreator(IndexCreationContext context)
+ throws IOException {
+ Preconditions.checkState(context.getFieldSpec().isSingleValueField(),
+ "H3 index is currently only supported on single-value columns");
+ Preconditions.checkState(context.getFieldSpec().getDataType().getStoredType() == FieldSpec.DataType.BYTES,
+ "H3 index is currently only supported on BYTES columns");
+ H3IndexResolution resolution = Objects.requireNonNull(context.getH3IndexConfig()).getResolution();
+ return context.isOnHeap() ? new OnHeapH3IndexCreator(context.getIndexDir(), context.getFieldSpec().getName(),
+ resolution) : new OffHeapH3IndexCreator(context.getIndexDir(), context.getFieldSpec().getName(), resolution);
+ }
+
+ public static boolean shouldDeriveNumDocsPerChunk(String columnName,
Review comment:
Ignore. Saw the changes in `SegmentColumnarIndexCreator` after this and this is no longer a question
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r770373299
##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/IndexCreationContext.java
##########
@@ -0,0 +1,481 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.creator;
+
+import java.io.File;
+import java.util.Map;
+import java.util.Objects;
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.index.creator.H3IndexConfig;
+import org.apache.pinot.spi.config.table.BloomFilterConfig;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+/**
+ * Provides parameters for constructing indexes via {@see IndexCreatorProvider}.
+ * The responsibility for ensuring that the correct parameters for a particular
+ * index type lies with the caller.
+ */
+public interface IndexCreationContext {
+
+ FieldSpec getFieldSpec();
+
+ File getIndexDir();
+
+ boolean isOnHeap();
+
+ int getLengthOfLongestEntry();
+
+ int getMaxNumberOfMultiValueElements();
+
+ int getMaxRowLengthInBytes();
+
+ boolean isSorted();
+
+ int getCardinality();
+
+ int getTotalNumberOfEntries();
+
+ int getTotalDocs();
+
+ boolean hasDictionary();
+
+ BloomFilter forBloomFilter(BloomFilterConfig bloomFilterConfig);
+
+ Forward forForwardIndex(ChunkCompressionType chunkCompressionType,
+ @Nullable Map<String, Map<String, String>> columnProperties);
+
+ Geospatial forGeospatialIndex(H3IndexConfig h3IndexConfig);
+
+ Inverted forInvertedIndex();
+
+ Json forJsonIndex();
+
+ Range forRangeIndex(ColumnMetadata columnMetadata, int rangeIndexVersion);
+
+ Text forTextIndex(FSTType fstType, String[] sortedUniqueElementsArray);
Review comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r768319018
##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/IndexCreationContext.java
##########
@@ -0,0 +1,256 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.creator;
+
+import java.io.File;
+import java.util.Map;
+import java.util.Objects;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.index.creator.H3IndexConfig;
+import org.apache.pinot.spi.config.table.BloomFilterConfig;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+public class IndexCreationContext {
Review comment:
(nit) brief javadoc please ?
Also to understand the end state better, this is not customizable right ? `SegmentColumnarIndexCreator` will pick up the pluggable creator and then build this context from indexCreationInfo / stats and the invoke the creator's APIs or there is more to it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r768251912
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/minion/RawIndexConverter.java
##########
@@ -206,7 +206,7 @@ private void convertColumn(FieldSpec fieldSpec)
DataType storedType = dictionary.getValueType();
int numDocs = _originalSegmentMetadata.getTotalDocs();
int lengthOfLongestEntry = _originalSegmentMetadata.getColumnMetadataFor(columnName).getColumnMaxLength();
- try (ForwardIndexCreator rawIndexCreator = SegmentColumnarIndexCreator
+ try (ForwardIndexCreator rawIndexCreator = DefaultIndexCreatorProvider
Review comment:
This seems unsolved
##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/IndexCreatorProvider.java
##########
@@ -0,0 +1,28 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.creator;
+
+/**
+ * Plugin interface to abstract index creation.
+ */
+public interface IndexCreatorProvider
Review comment:
Should we consider putting all methods into this single interface instead of splitting them into providers for each index type since this is the pluggable unit?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r769810136
##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/IndexCreationContext.java
##########
@@ -0,0 +1,261 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.creator;
+
+import java.io.File;
+import java.util.Map;
+import java.util.Objects;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.index.creator.H3IndexConfig;
+import org.apache.pinot.spi.config.table.BloomFilterConfig;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+/**
+ * Provides parameters for constructing indexes via {@see IndexCreatorProvider}.
+ * The responsibility for ensuring that the correct parameters for a particular
+ * index type lies with the caller.
+ */
+public class IndexCreationContext {
+
+ public static final class Builder {
+ private File _indexDir;
+ private Map<String, Map<String, String>> _columnProperties;
+ private FSTType _fstType;
+ private H3IndexConfig _h3IndexConfig;
+ private int _lengthOfLongestEntry;
+ private int _maxNumberOfMultiValueElements;
+ private int _maxRowLengthInBytes;
+ private Object _sortedUniqueElementsArray;
+ private boolean _onHeap = false;
+ private ChunkCompressionType _chunkCompressionType;
+ public BloomFilterConfig _bloomFilterConfig;
+ private int _rangeIndexVersion;
+ private ColumnMetadata _columnMetadata;
Review comment:
I agree, I was thinking about reverting the change to standardise on `ColumnMetadata`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#issuecomment-990031441
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7885](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (24b64ff) into [master](https://codecov.io/gh/apache/pinot/commit/fb12a509056b4e36724126cef9a344a5e1e89bb1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (fb12a50) will **decrease** coverage by `6.36%`.
> The diff coverage is `81.74%`.
[](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7885 +/- ##
============================================
- Coverage 71.34% 64.97% -6.37%
- Complexity 4087 4102 +15
============================================
Files 1587 1545 -42
Lines 82071 80333 -1738
Branches 12267 12066 -201
============================================
- Hits 58553 52198 -6355
- Misses 19550 24404 +4854
+ Partials 3968 3731 -237
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `68.38% <81.74%> (+0.05%)` | :arrow_up: |
| unittests2 | `14.36% <0.00%> (-0.09%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/pinot/core/minion/RawIndexConverter.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9taW5pb24vUmF3SW5kZXhDb252ZXJ0ZXIuamF2YQ==) | `0.00% <0.00%> (-56.61%)` | :arrow_down: |
| [...not/segment/spi/creator/IndexCreatorProviders.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2NyZWF0b3IvSW5kZXhDcmVhdG9yUHJvdmlkZXJzLmphdmE=) | `40.90% <40.90%> (ø)` | |
| [...ment/creator/impl/DefaultIndexCreatorProvider.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9EZWZhdWx0SW5kZXhDcmVhdG9yUHJvdmlkZXIuamF2YQ==) | `65.27% <65.27%> (ø)` | |
| [...t/index/loader/invertedindex/JsonIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0pzb25JbmRleEhhbmRsZXIuamF2YQ==) | `73.33% <66.66%> (+0.45%)` | :arrow_up: |
| [.../index/loader/invertedindex/RangeIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L1JhbmdlSW5kZXhIYW5kbGVyLmphdmE=) | `44.44% <85.71%> (+3.44%)` | :arrow_up: |
| [...ment/creator/impl/SegmentColumnarIndexCreator.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9TZWdtZW50Q29sdW1uYXJJbmRleENyZWF0b3IuamF2YQ==) | `87.22% <90.00%> (+2.83%)` | :arrow_up: |
| [...ocal/segment/index/loader/IndexHandlerFactory.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9JbmRleEhhbmRsZXJGYWN0b3J5LmphdmE=) | `100.00% <100.00%> (ø)` | |
| [...t/index/loader/bloomfilter/BloomFilterHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9ibG9vbWZpbHRlci9CbG9vbUZpbHRlckhhbmRsZXIuamF2YQ==) | `78.33% <100.00%> (+0.74%)` | :arrow_up: |
| [...nt/index/loader/invertedindex/FSTIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0ZTVEluZGV4SGFuZGxlci5qYXZh) | `86.79% <100.00%> (+4.10%)` | :arrow_up: |
| [...ent/index/loader/invertedindex/H3IndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0gzSW5kZXhIYW5kbGVyLmphdmE=) | `73.84% <100.00%> (+1.26%)` | :arrow_up: |
| ... and [377 more](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [fb12a50...24b64ff](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r768723722
##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/IndexCreatorProviders.java
##########
@@ -0,0 +1,143 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.creator;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.lang.invoke.MethodType;
+import java.util.concurrent.atomic.AtomicReference;
+import org.apache.pinot.segment.spi.index.creator.BloomFilterCreator;
+import org.apache.pinot.segment.spi.index.creator.CombinedInvertedIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.DictionaryBasedInvertedIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.GeoSpatialIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.JsonIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.TextIndexCreator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Plugin registration point to allow index creation logic to be swapped out.
+ * Plugins should not reimplement Pinot's default index creation logic.
+ * Users provide an override to Pinot's index creation logic. This is simplified
+ * by extending {@see IndexCreatorProviders.Default}
+ */
+public final class IndexCreatorProviders {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(IndexCreatorProviders.class);
+
+ static final IndexCreatorProvider DEFAULT = defaultProvider();
+ private static final AtomicReference<IndexCreatorProvider> REGISTRATION = new AtomicReference<>(DEFAULT);
+
+ private IndexCreatorProviders() {
+ }
+
+ /**
+ * The caller provides a decorator to wrap the default provider, which allows plugins to create
+ * a delegation chain.
+ * @param provider index creation override
+ * @return true if this is the first invocation and the provider has not yet been used.
+ */
+ public static boolean registerProvider(IndexCreatorProvider provider) {
+ return REGISTRATION.compareAndSet(DEFAULT, provider);
+ }
+
+ /**
+ * Obtain the registered index creator provider. If the user has provided an override, then it will be used instead.
+ * If the user has not provided an override yet, then this action will prevent them from doing so.
+ * @return the global index provision logic.
+ */
+ public static IndexCreatorProvider getIndexCreatorProvider() {
+ return Holder.PROVIDER;
+ }
+
+ private static final class Holder {
+ public static final IndexCreatorProvider PROVIDER = REGISTRATION.get();
+ }
+
+ private static IndexCreatorProvider defaultProvider() {
+ // use MethodHandle to break circular dependency and keep implementation details encapsulated within
+ // pinot-segment-local
+ String className = "org.apache.pinot.segment.local.segment.creator.impl.DefaultIndexCreatorProvider";
+ try {
+ Class<?> clazz = Class.forName(className, false, IndexCreatorProviders.class.getClassLoader());
+ return (IndexCreatorProvider) MethodHandles.publicLookup()
+ .findConstructor(clazz, MethodType.methodType(void.class)).invoke();
+ } catch (Throwable implausible) {
+ LOGGER.error("could not construct MethodHandle for {}", className,
+ implausible);
+ throw new IllegalStateException(implausible);
+ }
+ }
+
+ /**
+ * Extend this class to override index creation
+ */
+ public static class Default implements IndexCreatorProvider {
Review comment:
Added a test which should make the mechanism (and motivation) fairly clear: https://github.com/apache/pinot/pull/7885/commits/3ad1746f94f2c82a1347dc084beba25d5fb8e95a
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r769704585
##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/TextIndexCreatorProvider.java
##########
@@ -0,0 +1,37 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.creator;
+
+import java.io.IOException;
+import org.apache.pinot.segment.spi.index.creator.TextIndexCreator;
+
+
+public interface TextIndexCreatorProvider {
+
+ /**
+ * Creates a {@see TextIndexCreator} from information about index creation.
+ * This allows a plugin to pattern match index creation information to select
+ * an appropriate implementation.
+ * @param context context about the index creation.
+ * @return a {@see ForwardIndexCreator}
+ * @throws IOException whenever something goes wrong matching or constructing the creator
+ */
+ TextIndexCreator newFSTIndexCreator(IndexCreationContext context)
Review comment:
This was purposefully excluded as something which doesn't make sense to override.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin edited a comment on pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
richardstartin edited a comment on pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#issuecomment-995617069
@siddharthteotia there is now only one method for creating text indexes now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7885: make index creator provision interceptible
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#issuecomment-990031441
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7885](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (874335b) into [master](https://codecov.io/gh/apache/pinot/commit/86278194233781d8001498f0c2be4a8a3299cd9d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8627819) will **decrease** coverage by `6.18%`.
> The diff coverage is `84.46%`.
[](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7885 +/- ##
============================================
- Coverage 71.37% 65.19% -6.19%
- Complexity 4082 4100 +18
============================================
Files 1583 1541 -42
Lines 81868 80130 -1738
Branches 12239 12038 -201
============================================
- Hits 58436 52241 -6195
- Misses 19470 24162 +4692
+ Partials 3962 3727 -235
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `68.57% <84.46%> (+0.10%)` | :arrow_up: |
| unittests2 | `14.44% <0.00%> (-0.06%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/pinot/core/minion/RawIndexConverter.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9taW5pb24vUmF3SW5kZXhDb252ZXJ0ZXIuamF2YQ==) | `0.00% <0.00%> (-56.61%)` | :arrow_down: |
| [...ment/creator/impl/DefaultIndexCreatorProvider.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9EZWZhdWx0SW5kZXhDcmVhdG9yUHJvdmlkZXIuamF2YQ==) | `66.25% <66.25%> (ø)` | |
| [...t/index/loader/invertedindex/JsonIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0pzb25JbmRleEhhbmRsZXIuamF2YQ==) | `73.33% <66.66%> (+0.45%)` | :arrow_up: |
| [...not/segment/spi/creator/IndexCreatorProviders.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2NyZWF0b3IvSW5kZXhDcmVhdG9yUHJvdmlkZXJzLmphdmE=) | `69.23% <69.23%> (ø)` | |
| [.../index/loader/invertedindex/RangeIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L1JhbmdlSW5kZXhIYW5kbGVyLmphdmE=) | `44.44% <85.71%> (+3.44%)` | :arrow_up: |
| [...ment/creator/impl/SegmentColumnarIndexCreator.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9TZWdtZW50Q29sdW1uYXJJbmRleENyZWF0b3IuamF2YQ==) | `87.22% <90.00%> (+2.83%)` | :arrow_up: |
| [...ocal/segment/index/loader/IndexHandlerFactory.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9JbmRleEhhbmRsZXJGYWN0b3J5LmphdmE=) | `100.00% <100.00%> (ø)` | |
| [...t/index/loader/bloomfilter/BloomFilterHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9ibG9vbWZpbHRlci9CbG9vbUZpbHRlckhhbmRsZXIuamF2YQ==) | `78.33% <100.00%> (+0.74%)` | :arrow_up: |
| [...nt/index/loader/invertedindex/FSTIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0ZTVEluZGV4SGFuZGxlci5qYXZh) | `86.79% <100.00%> (+4.10%)` | :arrow_up: |
| [...ent/index/loader/invertedindex/H3IndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0gzSW5kZXhIYW5kbGVyLmphdmE=) | `73.84% <100.00%> (+1.26%)` | :arrow_up: |
| ... and [374 more](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [8627819...874335b](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7885: make index creator provision interceptible
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#issuecomment-990031441
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7885](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (874335b) into [master](https://codecov.io/gh/apache/pinot/commit/86278194233781d8001498f0c2be4a8a3299cd9d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8627819) will **decrease** coverage by `2.80%`.
> The diff coverage is `84.46%`.
[](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7885 +/- ##
============================================
- Coverage 71.37% 68.57% -2.81%
+ Complexity 4082 4020 -62
============================================
Files 1583 1192 -391
Lines 81868 59307 -22561
Branches 12239 9153 -3086
============================================
- Hits 58436 40670 -17766
+ Misses 19470 15770 -3700
+ Partials 3962 2867 -1095
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `68.57% <84.46%> (+0.10%)` | :arrow_up: |
| unittests2 | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/pinot/core/minion/RawIndexConverter.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9taW5pb24vUmF3SW5kZXhDb252ZXJ0ZXIuamF2YQ==) | `0.00% <0.00%> (-56.61%)` | :arrow_down: |
| [...ment/creator/impl/DefaultIndexCreatorProvider.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9EZWZhdWx0SW5kZXhDcmVhdG9yUHJvdmlkZXIuamF2YQ==) | `66.25% <66.25%> (ø)` | |
| [...t/index/loader/invertedindex/JsonIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0pzb25JbmRleEhhbmRsZXIuamF2YQ==) | `73.33% <66.66%> (+0.45%)` | :arrow_up: |
| [...not/segment/spi/creator/IndexCreatorProviders.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2NyZWF0b3IvSW5kZXhDcmVhdG9yUHJvdmlkZXJzLmphdmE=) | `69.23% <69.23%> (ø)` | |
| [.../index/loader/invertedindex/RangeIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L1JhbmdlSW5kZXhIYW5kbGVyLmphdmE=) | `44.44% <85.71%> (+3.44%)` | :arrow_up: |
| [...ment/creator/impl/SegmentColumnarIndexCreator.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9TZWdtZW50Q29sdW1uYXJJbmRleENyZWF0b3IuamF2YQ==) | `87.22% <90.00%> (+2.83%)` | :arrow_up: |
| [...ocal/segment/index/loader/IndexHandlerFactory.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9JbmRleEhhbmRsZXJGYWN0b3J5LmphdmE=) | `100.00% <100.00%> (ø)` | |
| [...t/index/loader/bloomfilter/BloomFilterHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9ibG9vbWZpbHRlci9CbG9vbUZpbHRlckhhbmRsZXIuamF2YQ==) | `78.33% <100.00%> (+0.74%)` | :arrow_up: |
| [...nt/index/loader/invertedindex/FSTIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0ZTVEluZGV4SGFuZGxlci5qYXZh) | `86.79% <100.00%> (+4.10%)` | :arrow_up: |
| [...ent/index/loader/invertedindex/H3IndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0gzSW5kZXhIYW5kbGVyLmphdmE=) | `73.84% <100.00%> (+1.26%)` | :arrow_up: |
| ... and [617 more](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [8627819...874335b](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
richardstartin commented on pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#issuecomment-995047736
> If needed in future, would it be possible to extend these interfaces to allow for creation of multiple indices on a column or would it lead to breaking existing index plugins? For example, I could create an H3 index and a quad-tree index on a geospatial column, or a flat index and a tree index on a JSON column and then depending upon the type of query being evaluated, pick one out of the two available indices to evaluate the query.
great question. No, the ability to create multiple indexes on a column is not limited by this PR: that logic stays in SegmentColumnarIndexCreator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#issuecomment-990031441
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7885](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9dfc034) into [master](https://codecov.io/gh/apache/pinot/commit/fb12a509056b4e36724126cef9a344a5e1e89bb1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (fb12a50) will **decrease** coverage by `0.05%`.
> The diff coverage is `83.89%`.
[](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7885 +/- ##
============================================
- Coverage 71.34% 71.28% -0.06%
- Complexity 4087 4103 +16
============================================
Files 1587 1592 +5
Lines 82071 82321 +250
Branches 12267 12272 +5
============================================
+ Hits 58553 58686 +133
- Misses 19550 19668 +118
+ Partials 3968 3967 -1
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `29.02% <2.68%> (-0.25%)` | :arrow_down: |
| integration2 | `27.63% <0.00%> (-0.02%)` | :arrow_down: |
| unittests1 | `68.39% <81.75%> (+0.05%)` | :arrow_up: |
| unittests2 | `14.34% <0.00%> (-0.11%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...not/segment/spi/creator/IndexCreatorProviders.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2NyZWF0b3IvSW5kZXhDcmVhdG9yUHJvdmlkZXJzLmphdmE=) | `52.38% <52.38%> (ø)` | |
| [...t/index/loader/invertedindex/JsonIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0pzb25JbmRleEhhbmRsZXIuamF2YQ==) | `73.33% <66.66%> (+0.45%)` | :arrow_up: |
| [...ment/creator/impl/DefaultIndexCreatorProvider.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9EZWZhdWx0SW5kZXhDcmVhdG9yUHJvdmlkZXIuamF2YQ==) | `67.14% <67.14%> (ø)` | |
| [.../index/loader/invertedindex/RangeIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L1JhbmdlSW5kZXhIYW5kbGVyLmphdmE=) | `44.44% <85.71%> (+3.44%)` | :arrow_up: |
| [...ment/creator/impl/SegmentColumnarIndexCreator.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9TZWdtZW50Q29sdW1uYXJJbmRleENyZWF0b3IuamF2YQ==) | `87.02% <87.17%> (+2.63%)` | :arrow_up: |
| [...inot/segment/spi/creator/IndexCreationContext.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2NyZWF0b3IvSW5kZXhDcmVhdGlvbkNvbnRleHQuamF2YQ==) | `94.11% <94.11%> (ø)` | |
| [...rg/apache/pinot/core/minion/RawIndexConverter.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9taW5pb24vUmF3SW5kZXhDb252ZXJ0ZXIuamF2YQ==) | `57.79% <100.00%> (+1.19%)` | :arrow_up: |
| [...nverttorawindex/ConvertToRawIndexTaskExecutor.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvY29udmVydHRvcmF3aW5kZXgvQ29udmVydFRvUmF3SW5kZXhUYXNrRXhlY3V0b3IuamF2YQ==) | `100.00% <100.00%> (ø)` | |
| [...ocal/segment/index/loader/IndexHandlerFactory.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9JbmRleEhhbmRsZXJGYWN0b3J5LmphdmE=) | `100.00% <100.00%> (ø)` | |
| [...t/index/loader/bloomfilter/BloomFilterHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9ibG9vbWZpbHRlci9CbG9vbUZpbHRlckhhbmRsZXIuamF2YQ==) | `78.33% <100.00%> (+0.74%)` | :arrow_up: |
| ... and [48 more](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [fb12a50...9dfc034](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r770353392
##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java
##########
@@ -179,126 +167,71 @@ public void init(SegmentGeneratorConfig segmentCreationSpec, SegmentIndexCreatio
}
String columnName = fieldSpec.getName();
- DataType storedType = fieldSpec.getDataType().getStoredType();
- ColumnIndexCreationInfo indexCreationInfo = indexCreationInfoMap.get(columnName);
- Preconditions.checkNotNull(indexCreationInfo, "Missing index creation info for column: %s", columnName);
- boolean dictEnabledColumn = createDictionaryForColumn(indexCreationInfo, segmentCreationSpec, fieldSpec);
-
+ ColumnIndexCreationInfo columnIndexCreationInfo = indexCreationInfoMap.get(columnName);
+ Preconditions.checkNotNull(columnIndexCreationInfo, "Missing index creation info for column: %s", columnName);
+ boolean dictEnabledColumn = createDictionaryForColumn(columnIndexCreationInfo, segmentCreationSpec, fieldSpec);
+ Preconditions.checkState(dictEnabledColumn || !invertedIndexColumns.contains(columnName),
+ "Cannot create inverted index for raw index column: %s", columnName);
+ ColumnMetadata columnMetadata = ColumnMetadataImpl.builder()
+ .setCardinality(columnIndexCreationInfo.getDistinctValueCount())
+ .setHasDictionary(dictEnabledColumn)
+ .setFieldSpec(fieldSpec)
+ .setTotalDocs(segmentIndexCreationInfo.getTotalDocs())
+ .setTotalNumberOfEntries(columnIndexCreationInfo.getTotalNumberOfEntries())
+ .setSorted(columnIndexCreationInfo.isSorted())
+ .build();
+ IndexCreationContext context = IndexCreationContext.builder()
+ .withIndexDir(_indexDir)
+ .withColumnMetadata(columnMetadata)
+ .withSegmentGeneratorConfig(segmentCreationSpec)
+ .withColumnIndexCreationInfo(columnIndexCreationInfo)
+ .withFSTType(_config.getFSTIndexType())
+ .withH3IndexConfig(h3IndexConfigs.get(columnName))
+ .withCompressionType(dictEnabledColumn ? null : getColumnCompressionType(segmentCreationSpec, fieldSpec))
+ .build();
+ // Initialize forward index creator
+ _forwardIndexCreatorMap.put(columnName, _indexCreatorProvider.newForwardIndexCreator(context));
+
+ // Initialize inverted index creator; skip creating inverted index if sorted
+ if (invertedIndexColumns.contains(columnName) && !columnIndexCreationInfo.isSorted()) {
+ _invertedIndexCreatorMap.put(columnName, _indexCreatorProvider.newInvertedIndexCreator(context));
+ }
if (dictEnabledColumn) {
// Create dictionary-encoded index
-
// Initialize dictionary creator
SegmentDictionaryCreator dictionaryCreator =
- new SegmentDictionaryCreator(indexCreationInfo.getSortedUniqueElementsArray(), fieldSpec, _indexDir,
- indexCreationInfo.isUseVarLengthDictionary());
+ new SegmentDictionaryCreator(columnIndexCreationInfo.getSortedUniqueElementsArray(), fieldSpec, _indexDir,
+ columnIndexCreationInfo.isUseVarLengthDictionary());
_dictionaryCreatorMap.put(columnName, dictionaryCreator);
-
// Create dictionary
try {
dictionaryCreator.build();
} catch (Exception e) {
LOGGER.error("Error building dictionary for field: {}, cardinality: {}, number of bytes per entry: {}",
- fieldSpec.getName(), indexCreationInfo.getDistinctValueCount(), dictionaryCreator.getNumBytesPerEntry());
+ fieldSpec.getName(), columnIndexCreationInfo.getDistinctValueCount(),
+ dictionaryCreator.getNumBytesPerEntry());
throw e;
}
-
- // Initialize forward index creator
- int cardinality = indexCreationInfo.getDistinctValueCount();
- if (fieldSpec.isSingleValueField()) {
- if (indexCreationInfo.isSorted()) {
- _forwardIndexCreatorMap.put(columnName,
- new SingleValueSortedForwardIndexCreator(_indexDir, columnName, cardinality));
- } else {
- _forwardIndexCreatorMap.put(columnName,
- new SingleValueUnsortedForwardIndexCreator(_indexDir, columnName, cardinality, _totalDocs));
- }
- } else {
- _forwardIndexCreatorMap.put(columnName,
- new MultiValueUnsortedForwardIndexCreator(_indexDir, columnName, cardinality, _totalDocs,
- indexCreationInfo.getTotalNumberOfEntries()));
- }
-
- // Initialize inverted index creator; skip creating inverted index if sorted
- if (invertedIndexColumns.contains(columnName) && !indexCreationInfo.isSorted()) {
- if (segmentCreationSpec.isOnHeap()) {
- _invertedIndexCreatorMap.put(columnName,
- new OnHeapBitmapInvertedIndexCreator(_indexDir, columnName, cardinality));
- } else {
- _invertedIndexCreatorMap.put(columnName,
- new OffHeapBitmapInvertedIndexCreator(_indexDir, fieldSpec, cardinality, _totalDocs,
- indexCreationInfo.getTotalNumberOfEntries()));
- }
- }
- } else {
- // Create raw index
- Preconditions.checkState(!invertedIndexColumns.contains(columnName),
- "Cannot create inverted index for raw index column: %s", columnName);
-
- ChunkCompressionType compressionType = getColumnCompressionType(segmentCreationSpec, fieldSpec);
-
- // Initialize forward index creator
- boolean deriveNumDocsPerChunk =
- shouldDeriveNumDocsPerChunk(columnName, segmentCreationSpec.getColumnProperties());
- int writerVersion = rawIndexWriterVersion(columnName, segmentCreationSpec.getColumnProperties());
- if (fieldSpec.isSingleValueField()) {
- _forwardIndexCreatorMap.put(columnName,
- getRawIndexCreatorForSVColumn(_indexDir, compressionType, columnName, storedType, _totalDocs,
- indexCreationInfo.getLengthOfLongestEntry(), deriveNumDocsPerChunk, writerVersion));
- } else {
- _forwardIndexCreatorMap.put(columnName,
- getRawIndexCreatorForMVColumn(_indexDir, compressionType, columnName, storedType, _totalDocs,
- indexCreationInfo.getMaxNumberOfMultiValueElements(), deriveNumDocsPerChunk, writerVersion,
- indexCreationInfo.getMaxRowLengthInBytes()));
- }
}
if (textIndexColumns.contains(columnName)) {
// Initialize text index creator
- Preconditions.checkState(storedType == DataType.STRING,
+ Preconditions.checkState(fieldSpec.getDataType().getStoredType() == FieldSpec.DataType.STRING,
"Text index is currently only supported on STRING type columns");
_textIndexCreatorMap.put(columnName,
new LuceneTextIndexCreator(columnName, _indexDir, true /* commitOnClose */));
Review comment:
OK I will make this distinction in the interface
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r768449397
##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/IndexCreatorProvider.java
##########
@@ -0,0 +1,28 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.creator;
+
+/**
+ * Plugin interface to abstract index creation.
+ */
+public interface IndexCreatorProvider
Review comment:
I prefer it this way because it allows the code in pinot-segment-local to request as little functionality as it needs - in order to construct a forward index I shouldn't need the power to create a bloom filter. These types can be composed as generic intersection types when more than one service is required. In the future, we may want to change the implementations used within core, and shouldn't need to implement bloom filter provision to construct an object which needs to create forward indexes. The fact that there is only one override at the SPI level is an implementation detail of the override mechanism, and pinot-segment-local/pinot-core should not be aware of this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#issuecomment-990031441
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7885](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (989f68b) into [master](https://codecov.io/gh/apache/pinot/commit/fb12a509056b4e36724126cef9a344a5e1e89bb1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (fb12a50) will **decrease** coverage by `6.32%`.
> The diff coverage is `81.53%`.
[](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7885 +/- ##
============================================
- Coverage 71.34% 65.02% -6.33%
- Complexity 4087 4107 +20
============================================
Files 1587 1545 -42
Lines 82071 80341 -1730
Branches 12267 12066 -201
============================================
- Hits 58553 52239 -6314
- Misses 19550 24379 +4829
+ Partials 3968 3723 -245
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `68.45% <81.53%> (+0.12%)` | :arrow_up: |
| unittests2 | `14.34% <0.00%> (-0.10%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/pinot/core/minion/RawIndexConverter.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9taW5pb24vUmF3SW5kZXhDb252ZXJ0ZXIuamF2YQ==) | `0.00% <0.00%> (-56.61%)` | :arrow_down: |
| [...not/segment/spi/creator/IndexCreatorProviders.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2NyZWF0b3IvSW5kZXhDcmVhdG9yUHJvdmlkZXJzLmphdmE=) | `40.90% <40.90%> (ø)` | |
| [...ment/creator/impl/DefaultIndexCreatorProvider.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9EZWZhdWx0SW5kZXhDcmVhdG9yUHJvdmlkZXIuamF2YQ==) | `66.25% <66.25%> (ø)` | |
| [...t/index/loader/invertedindex/JsonIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0pzb25JbmRleEhhbmRsZXIuamF2YQ==) | `73.33% <66.66%> (+0.45%)` | :arrow_up: |
| [.../index/loader/invertedindex/RangeIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L1JhbmdlSW5kZXhIYW5kbGVyLmphdmE=) | `44.44% <85.71%> (+3.44%)` | :arrow_up: |
| [...ment/creator/impl/SegmentColumnarIndexCreator.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9TZWdtZW50Q29sdW1uYXJJbmRleENyZWF0b3IuamF2YQ==) | `87.22% <90.00%> (+2.83%)` | :arrow_up: |
| [...ocal/segment/index/loader/IndexHandlerFactory.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9JbmRleEhhbmRsZXJGYWN0b3J5LmphdmE=) | `100.00% <100.00%> (ø)` | |
| [...t/index/loader/bloomfilter/BloomFilterHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9ibG9vbWZpbHRlci9CbG9vbUZpbHRlckhhbmRsZXIuamF2YQ==) | `78.33% <100.00%> (+0.74%)` | :arrow_up: |
| [...nt/index/loader/invertedindex/FSTIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0ZTVEluZGV4SGFuZGxlci5qYXZh) | `86.79% <100.00%> (+4.10%)` | :arrow_up: |
| [...ent/index/loader/invertedindex/H3IndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0gzSW5kZXhIYW5kbGVyLmphdmE=) | `73.84% <100.00%> (+1.26%)` | :arrow_up: |
| ... and [380 more](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [fb12a50...989f68b](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#issuecomment-990031441
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7885](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d4a8970) into [master](https://codecov.io/gh/apache/pinot/commit/fb12a509056b4e36724126cef9a344a5e1e89bb1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (fb12a50) will **decrease** coverage by `1.25%`.
> The diff coverage is `81.53%`.
[](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7885 +/- ##
============================================
- Coverage 71.34% 70.09% -1.26%
- Complexity 4087 4102 +15
============================================
Files 1587 1590 +3
Lines 82071 82212 +141
Branches 12267 12269 +2
============================================
- Hits 58553 57623 -930
- Misses 19550 20643 +1093
+ Partials 3968 3946 -22
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `27.58% <0.00%> (-0.07%)` | :arrow_down: |
| unittests1 | `68.39% <81.53%> (+0.06%)` | :arrow_up: |
| unittests2 | `14.36% <0.00%> (-0.09%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/pinot/core/minion/RawIndexConverter.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9taW5pb24vUmF3SW5kZXhDb252ZXJ0ZXIuamF2YQ==) | `0.00% <0.00%> (-56.61%)` | :arrow_down: |
| [...not/segment/spi/creator/IndexCreatorProviders.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2NyZWF0b3IvSW5kZXhDcmVhdG9yUHJvdmlkZXJzLmphdmE=) | `40.90% <40.90%> (ø)` | |
| [...ment/creator/impl/DefaultIndexCreatorProvider.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9EZWZhdWx0SW5kZXhDcmVhdG9yUHJvdmlkZXIuamF2YQ==) | `66.25% <66.25%> (ø)` | |
| [...t/index/loader/invertedindex/JsonIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0pzb25JbmRleEhhbmRsZXIuamF2YQ==) | `73.33% <66.66%> (+0.45%)` | :arrow_up: |
| [.../index/loader/invertedindex/RangeIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L1JhbmdlSW5kZXhIYW5kbGVyLmphdmE=) | `44.44% <85.71%> (+3.44%)` | :arrow_up: |
| [...ment/creator/impl/SegmentColumnarIndexCreator.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9TZWdtZW50Q29sdW1uYXJJbmRleENyZWF0b3IuamF2YQ==) | `87.22% <90.00%> (+2.83%)` | :arrow_up: |
| [...ocal/segment/index/loader/IndexHandlerFactory.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9JbmRleEhhbmRsZXJGYWN0b3J5LmphdmE=) | `100.00% <100.00%> (ø)` | |
| [...t/index/loader/bloomfilter/BloomFilterHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9ibG9vbWZpbHRlci9CbG9vbUZpbHRlckhhbmRsZXIuamF2YQ==) | `78.33% <100.00%> (+0.74%)` | :arrow_up: |
| [...nt/index/loader/invertedindex/FSTIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0ZTVEluZGV4SGFuZGxlci5qYXZh) | `86.79% <100.00%> (+4.10%)` | :arrow_up: |
| [...ent/index/loader/invertedindex/H3IndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0gzSW5kZXhIYW5kbGVyLmphdmE=) | `73.84% <100.00%> (+1.26%)` | :arrow_up: |
| ... and [119 more](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [fb12a50...d4a8970](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r770166077
##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/IndexCreationContext.java
##########
@@ -0,0 +1,481 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.creator;
+
+import java.io.File;
+import java.util.Map;
+import java.util.Objects;
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.index.creator.H3IndexConfig;
+import org.apache.pinot.spi.config.table.BloomFilterConfig;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+/**
+ * Provides parameters for constructing indexes via {@see IndexCreatorProvider}.
+ * The responsibility for ensuring that the correct parameters for a particular
+ * index type lies with the caller.
+ */
+public interface IndexCreationContext {
+
+ FieldSpec getFieldSpec();
+
+ File getIndexDir();
+
+ boolean isOnHeap();
+
+ int getLengthOfLongestEntry();
+
+ int getMaxNumberOfMultiValueElements();
+
+ int getMaxRowLengthInBytes();
+
+ boolean isSorted();
+
+ int getCardinality();
+
+ int getTotalNumberOfEntries();
+
+ int getTotalDocs();
+
+ boolean hasDictionary();
+
+ BloomFilter forBloomFilter(BloomFilterConfig bloomFilterConfig);
+
+ Forward forForwardIndex(ChunkCompressionType chunkCompressionType,
+ @Nullable Map<String, Map<String, String>> columnProperties);
+
+ Geospatial forGeospatialIndex(H3IndexConfig h3IndexConfig);
+
+ Inverted forInvertedIndex();
+
+ Json forJsonIndex();
+
+ Range forRangeIndex(ColumnMetadata columnMetadata, int rangeIndexVersion);
+
+ Text forTextIndex(FSTType fstType, String[] sortedUniqueElementsArray);
Review comment:
This is confusing. Will this support all 3 flavors of text index ?
- Lucene text
-Lucene FST
- Native FST
##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/IndexCreationContext.java
##########
@@ -0,0 +1,481 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.creator;
+
+import java.io.File;
+import java.util.Map;
+import java.util.Objects;
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.index.creator.H3IndexConfig;
+import org.apache.pinot.spi.config.table.BloomFilterConfig;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+/**
+ * Provides parameters for constructing indexes via {@see IndexCreatorProvider}.
+ * The responsibility for ensuring that the correct parameters for a particular
+ * index type lies with the caller.
+ */
+public interface IndexCreationContext {
+
+ FieldSpec getFieldSpec();
+
+ File getIndexDir();
+
+ boolean isOnHeap();
+
+ int getLengthOfLongestEntry();
+
+ int getMaxNumberOfMultiValueElements();
+
+ int getMaxRowLengthInBytes();
+
+ boolean isSorted();
+
+ int getCardinality();
+
+ int getTotalNumberOfEntries();
+
+ int getTotalDocs();
+
+ boolean hasDictionary();
+
+ BloomFilter forBloomFilter(BloomFilterConfig bloomFilterConfig);
+
+ Forward forForwardIndex(ChunkCompressionType chunkCompressionType,
+ @Nullable Map<String, Map<String, String>> columnProperties);
+
+ Geospatial forGeospatialIndex(H3IndexConfig h3IndexConfig);
+
+ Inverted forInvertedIndex();
+
+ Json forJsonIndex();
+
+ Range forRangeIndex(ColumnMetadata columnMetadata, int rangeIndexVersion);
+
+ Text forTextIndex(FSTType fstType, String[] sortedUniqueElementsArray);
Review comment:
This is confusing. Will this support all 3 flavors of text index ?
- Lucene text
- Lucene FST
- Native FST
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r770079177
##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/InvertedIndexHandler.java
##########
@@ -100,9 +104,8 @@ private void createInvertedIndexForColumn(ColumnMetadata columnMetadata)
// Create new inverted index for the column.
LOGGER.info("Creating new inverted index for segment: {}, column: {}", segmentName, column);
int numDocs = columnMetadata.getTotalDocs();
- try (OffHeapBitmapInvertedIndexCreator creator = new OffHeapBitmapInvertedIndexCreator(_indexDir,
- columnMetadata.getFieldSpec(), columnMetadata.getCardinality(), numDocs,
- columnMetadata.getTotalNumberOfEntries())) {
+ try (DictionaryBasedInvertedIndexCreator creator = _indexCreatorProvider.newInvertedIndexCreator(
Review comment:
the default value is false
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#issuecomment-990031441
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7885](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2b42bc9) into [master](https://codecov.io/gh/apache/pinot/commit/fb12a509056b4e36724126cef9a344a5e1e89bb1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (fb12a50) will **decrease** coverage by `57.00%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7885 +/- ##
=============================================
- Coverage 71.34% 14.33% -57.01%
+ Complexity 4087 80 -4007
=============================================
Files 1587 1547 -40
Lines 82071 80473 -1598
Branches 12267 12076 -191
=============================================
- Hits 58553 11539 -47014
- Misses 19550 68073 +48523
+ Partials 3968 861 -3107
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `?` | |
| unittests2 | `14.33% <0.00%> (-0.11%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/pinot/core/minion/RawIndexConverter.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9taW5pb24vUmF3SW5kZXhDb252ZXJ0ZXIuamF2YQ==) | `0.00% <0.00%> (-56.61%)` | :arrow_down: |
| [...nverttorawindex/ConvertToRawIndexTaskExecutor.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvY29udmVydHRvcmF3aW5kZXgvQ29udmVydFRvUmF3SW5kZXhUYXNrRXhlY3V0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...ment/creator/impl/DefaultIndexCreatorProvider.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9EZWZhdWx0SW5kZXhDcmVhdG9yUHJvdmlkZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...ment/creator/impl/SegmentColumnarIndexCreator.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9TZWdtZW50Q29sdW1uYXJJbmRleENyZWF0b3IuamF2YQ==) | `0.00% <0.00%> (-84.40%)` | :arrow_down: |
| [...ocal/segment/index/loader/IndexHandlerFactory.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9JbmRleEhhbmRsZXJGYWN0b3J5LmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...t/index/loader/bloomfilter/BloomFilterHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9ibG9vbWZpbHRlci9CbG9vbUZpbHRlckhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (-77.59%)` | :arrow_down: |
| [...nt/index/loader/invertedindex/FSTIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0ZTVEluZGV4SGFuZGxlci5qYXZh) | `0.00% <0.00%> (-82.70%)` | :arrow_down: |
| [...ent/index/loader/invertedindex/H3IndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0gzSW5kZXhIYW5kbGVyLmphdmE=) | `0.00% <0.00%> (-72.59%)` | :arrow_down: |
| [...dex/loader/invertedindex/InvertedIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0ludmVydGVkSW5kZXhIYW5kbGVyLmphdmE=) | `0.00% <0.00%> (-93.88%)` | :arrow_down: |
| [...t/index/loader/invertedindex/JsonIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0pzb25JbmRleEhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (-72.89%)` | :arrow_down: |
| ... and [1275 more](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [fb12a50...2b42bc9](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7885: make index creator provision interceptible
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#issuecomment-990031441
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7885](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (874335b) into [master](https://codecov.io/gh/apache/pinot/commit/86278194233781d8001498f0c2be4a8a3299cd9d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8627819) will **decrease** coverage by `1.03%`.
> The diff coverage is `84.46%`.
[](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7885 +/- ##
============================================
- Coverage 71.37% 70.34% -1.04%
- Complexity 4082 4100 +18
============================================
Files 1583 1586 +3
Lines 81868 82000 +132
Branches 12239 12241 +2
============================================
- Hits 58436 57679 -757
- Misses 19470 20366 +896
+ Partials 3962 3955 -7
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `27.73% <0.00%> (-0.08%)` | :arrow_down: |
| unittests1 | `68.57% <84.46%> (+0.10%)` | :arrow_up: |
| unittests2 | `14.44% <0.00%> (-0.06%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/pinot/core/minion/RawIndexConverter.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9taW5pb24vUmF3SW5kZXhDb252ZXJ0ZXIuamF2YQ==) | `0.00% <0.00%> (-56.61%)` | :arrow_down: |
| [...ment/creator/impl/DefaultIndexCreatorProvider.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9EZWZhdWx0SW5kZXhDcmVhdG9yUHJvdmlkZXIuamF2YQ==) | `66.25% <66.25%> (ø)` | |
| [...t/index/loader/invertedindex/JsonIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0pzb25JbmRleEhhbmRsZXIuamF2YQ==) | `73.33% <66.66%> (+0.45%)` | :arrow_up: |
| [...not/segment/spi/creator/IndexCreatorProviders.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2NyZWF0b3IvSW5kZXhDcmVhdG9yUHJvdmlkZXJzLmphdmE=) | `69.23% <69.23%> (ø)` | |
| [.../index/loader/invertedindex/RangeIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L1JhbmdlSW5kZXhIYW5kbGVyLmphdmE=) | `44.44% <85.71%> (+3.44%)` | :arrow_up: |
| [...ment/creator/impl/SegmentColumnarIndexCreator.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9TZWdtZW50Q29sdW1uYXJJbmRleENyZWF0b3IuamF2YQ==) | `87.22% <90.00%> (+2.83%)` | :arrow_up: |
| [...ocal/segment/index/loader/IndexHandlerFactory.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9JbmRleEhhbmRsZXJGYWN0b3J5LmphdmE=) | `100.00% <100.00%> (ø)` | |
| [...t/index/loader/bloomfilter/BloomFilterHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9ibG9vbWZpbHRlci9CbG9vbUZpbHRlckhhbmRsZXIuamF2YQ==) | `78.33% <100.00%> (+0.74%)` | :arrow_up: |
| [...nt/index/loader/invertedindex/FSTIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0ZTVEluZGV4SGFuZGxlci5qYXZh) | `86.79% <100.00%> (+4.10%)` | :arrow_up: |
| [...ent/index/loader/invertedindex/H3IndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0gzSW5kZXhIYW5kbGVyLmphdmE=) | `73.84% <100.00%> (+1.26%)` | :arrow_up: |
| ... and [116 more](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [8627819...874335b](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#issuecomment-990031441
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7885](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (989f68b) into [master](https://codecov.io/gh/apache/pinot/commit/fb12a509056b4e36724126cef9a344a5e1e89bb1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (fb12a50) will **decrease** coverage by `56.99%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7885 +/- ##
=============================================
- Coverage 71.34% 14.34% -57.00%
+ Complexity 4087 80 -4007
=============================================
Files 1587 1545 -42
Lines 82071 80341 -1730
Branches 12267 12066 -201
=============================================
- Hits 58553 11528 -47025
- Misses 19550 67956 +48406
+ Partials 3968 857 -3111
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `?` | |
| unittests2 | `14.34% <0.00%> (-0.10%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/pinot/core/minion/RawIndexConverter.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9taW5pb24vUmF3SW5kZXhDb252ZXJ0ZXIuamF2YQ==) | `0.00% <0.00%> (-56.61%)` | :arrow_down: |
| [...ment/creator/impl/DefaultIndexCreatorProvider.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9EZWZhdWx0SW5kZXhDcmVhdG9yUHJvdmlkZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...ment/creator/impl/SegmentColumnarIndexCreator.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9TZWdtZW50Q29sdW1uYXJJbmRleENyZWF0b3IuamF2YQ==) | `0.00% <0.00%> (-84.40%)` | :arrow_down: |
| [...ocal/segment/index/loader/IndexHandlerFactory.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9JbmRleEhhbmRsZXJGYWN0b3J5LmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...t/index/loader/bloomfilter/BloomFilterHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9ibG9vbWZpbHRlci9CbG9vbUZpbHRlckhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (-77.59%)` | :arrow_down: |
| [...nt/index/loader/invertedindex/FSTIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0ZTVEluZGV4SGFuZGxlci5qYXZh) | `0.00% <0.00%> (-82.70%)` | :arrow_down: |
| [...ent/index/loader/invertedindex/H3IndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0gzSW5kZXhIYW5kbGVyLmphdmE=) | `0.00% <0.00%> (-72.59%)` | :arrow_down: |
| [...dex/loader/invertedindex/InvertedIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0ludmVydGVkSW5kZXhIYW5kbGVyLmphdmE=) | `0.00% <0.00%> (-93.88%)` | :arrow_down: |
| [...t/index/loader/invertedindex/JsonIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0pzb25JbmRleEhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (-72.89%)` | :arrow_down: |
| [.../index/loader/invertedindex/RangeIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L1JhbmdlSW5kZXhIYW5kbGVyLmphdmE=) | `0.00% <0.00%> (-41.00%)` | :arrow_down: |
| ... and [1272 more](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [fb12a50...989f68b](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r770023148
##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/DefaultIndexCreatorProvider.java
##########
@@ -0,0 +1,249 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.Map;
+import java.util.Objects;
+import org.apache.pinot.segment.local.io.writer.impl.BaseChunkSVForwardIndexWriter;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.MultiValueFixedByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.MultiValueUnsortedForwardIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.MultiValueVarByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueFixedByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueSortedForwardIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueUnsortedForwardIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueVarByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.OffHeapBitmapInvertedIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.OnHeapBitmapInvertedIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.geospatial.OffHeapH3IndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.geospatial.OnHeapH3IndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.json.OffHeapJsonIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.json.OnHeapJsonIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.text.LuceneFSTIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.text.LuceneTextIndexCreator;
+import org.apache.pinot.segment.local.utils.nativefst.NativeFSTIndexCreator;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.DictionaryBasedInvertedIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.GeoSpatialIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.JsonIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.TextIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.H3IndexResolution;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.config.table.FieldConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+public final class DefaultIndexCreatorProvider implements IndexCreatorProvider {
+
+ @Override
+ public ForwardIndexCreator newForwardIndexCreator(IndexCreationContext context)
+ throws Exception {
+ if (context.isRawColumn()) {
+ boolean deriveNumDocsPerChunk =
+ shouldDeriveNumDocsPerChunk(context.getFieldSpec().getName(), context.getColumnProperties());
+ int writerVersion = rawIndexWriterVersion(context.getFieldSpec().getName(), context.getColumnProperties());
+ if (context.getFieldSpec().isSingleValueField()) {
+ return getRawIndexCreatorForSVColumn(context.getIndexDir(), context.getChunkCompressionType(),
+ context.getFieldSpec().getName(),
+ context.getFieldSpec().getDataType().getStoredType(), context.getTotalDocs(),
+ context.getLengthOfLongestEntry(), deriveNumDocsPerChunk, writerVersion);
+ } else {
+ return getRawIndexCreatorForMVColumn(context.getIndexDir(), context.getChunkCompressionType(),
+ context.getFieldSpec().getName(), context.getFieldSpec().getDataType().getStoredType(),
+ context.getTotalDocs(),
+ context.getMaxNumberOfMultiValueElements(), deriveNumDocsPerChunk, writerVersion,
+ context.getMaxRowLengthInBytes());
+ }
+ } else {
+ if (context.getFieldSpec().isSingleValueField()) {
+ if (context.isSorted()) {
+ return new SingleValueSortedForwardIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(),
+ context.getDistinctValueCount());
+ } else {
+ return new SingleValueUnsortedForwardIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(),
+ context.getDistinctValueCount(),
+ context.getTotalDocs());
+ }
+ } else {
+ return new MultiValueUnsortedForwardIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(),
+ context.getDistinctValueCount(),
+ context.getTotalDocs(),
+ context.getTotalNumberOfEntries());
+ }
+ }
+ }
+
+ @Override
+ public DictionaryBasedInvertedIndexCreator newInvertedIndexCreator(IndexCreationContext context)
+ throws IOException {
+ if (context.isOnHeap()) {
+ return new OnHeapBitmapInvertedIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(),
+ context.getDistinctValueCount());
+ } else {
+ return new OffHeapBitmapInvertedIndexCreator(context.getIndexDir(), context.getFieldSpec(),
+ context.getDistinctValueCount(), context.getTotalDocs(),
+ context.getTotalNumberOfEntries());
+ }
+ }
+
+ @Override
+ public JsonIndexCreator newJsonIndexCreator(IndexCreationContext context)
+ throws IOException {
+ Preconditions.checkState(context.getFieldSpec().isSingleValueField(),
+ "Json index is currently only supported on single-value columns");
+ Preconditions.checkState(context.getFieldSpec().getDataType().getStoredType() == FieldSpec.DataType.STRING,
+ "Json index is currently only supported on STRING columns");
+ return context.isOnHeap() ? new OnHeapJsonIndexCreator(context.getIndexDir(),
+ context.getFieldSpec().getName())
+ : new OffHeapJsonIndexCreator(context.getIndexDir(), context.getFieldSpec().getName());
+ }
+
+ @Override
+ public TextIndexCreator newTextIndexCreator(IndexCreationContext context)
+ throws IOException {
+ // Initialize text index creator
+ Preconditions.checkState(context.getFieldSpec().getDataType().getStoredType() == FieldSpec.DataType.STRING,
+ "Text index is currently only supported on STRING type columns");
+ return new LuceneTextIndexCreator(context.getFieldSpec().getName(), context.getIndexDir(),
+ true /* commitOnClose */);
+ }
+
+ @Override
+ public TextIndexCreator newFSTIndexCreator(IndexCreationContext context)
+ throws IOException {
+ Preconditions.checkState(context.getFieldSpec().isSingleValueField(),
+ "FST index is currently only supported on single-value columns");
+ Preconditions.checkState(context.getFieldSpec().getDataType().getStoredType() == FieldSpec.DataType.STRING,
+ "FST index is currently only supported on STRING type columns");
+ Preconditions.checkState(!context.isRawColumn(),
+ "FST index is currently only supported on dictionary-encoded columns");
+ String[] sortedValues = (String[]) context.getSortedUniqueElementsArray();
+ if (context.getFstType() == FSTType.NATIVE) {
+ return new NativeFSTIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(), sortedValues);
+ } else {
+ return new LuceneFSTIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(), sortedValues);
+ }
+ }
+
+ @Override
+ public GeoSpatialIndexCreator newGeoSpatialIndexCreator(IndexCreationContext context)
+ throws IOException {
+ Preconditions.checkState(context.getFieldSpec().isSingleValueField(),
+ "H3 index is currently only supported on single-value columns");
+ Preconditions.checkState(context.getFieldSpec().getDataType().getStoredType() == FieldSpec.DataType.BYTES,
+ "H3 index is currently only supported on BYTES columns");
+ H3IndexResolution resolution = Objects.requireNonNull(context.getH3IndexConfig()).getResolution();
+ return context.isOnHeap() ? new OnHeapH3IndexCreator(context.getIndexDir(),
+ context.getFieldSpec().getName(), resolution)
+ : new OffHeapH3IndexCreator(context.getIndexDir(), context.getFieldSpec().getName(), resolution);
+ }
+
+ public static boolean shouldDeriveNumDocsPerChunk(String columnName,
+ Map<String, Map<String, String>> columnProperties) {
+ if (columnProperties != null) {
+ Map<String, String> properties = columnProperties.get(columnName);
+ return properties != null && Boolean.parseBoolean(
+ properties.get(FieldConfig.DERIVE_NUM_DOCS_PER_CHUNK_RAW_INDEX_KEY));
+ }
+ return false;
+ }
+
+ public static int rawIndexWriterVersion(String columnName, Map<String, Map<String, String>> columnProperties) {
Review comment:
This code was copied and pasted from `SegmentColumnarIndexCreator` - it's not a new method name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r768304406
##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/TextIndexCreatorProvider.java
##########
@@ -0,0 +1,47 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.creator;
+
+import java.io.IOException;
+import org.apache.pinot.segment.spi.index.creator.TextIndexCreator;
+
+
+public interface TextIndexCreatorProvider {
+ /**
+ * Creates a {@see TextIndexCreator} from information about index creation.
+ * This allows a plugin to pattern match index creation information to select
+ * an appropriate implementation.
+ * @param context context about the index creation.
+ * @return a {@see ForwardIndexCreator}
+ * @throws IOException whenever something goes wrong matching or constructing the creator
+ */
+ TextIndexCreator newTextIndexCreator(IndexCreationContext context)
Review comment:
All current three impls (Lucene text, lucene fst, native fst) implement TextIndexCreator interface. So, can we try to keep a single `newTextIndexCreator` and let the `IndexCreationContext` dictate which impl to create ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r770883789
##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/IndexCreationContext.java
##########
@@ -0,0 +1,467 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.creator;
+
+import java.io.File;
+import java.util.Map;
+import java.util.Objects;
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.index.creator.H3IndexConfig;
+import org.apache.pinot.spi.config.table.BloomFilterConfig;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+/**
+ * Provides parameters for constructing indexes via {@see IndexCreatorProvider}.
+ * The responsibility for ensuring that the correct parameters for a particular
+ * index type lies with the caller.
+ */
+public interface IndexCreationContext {
Review comment:
They can be added later if they're ever necessary. It's always easier to add things than take them away.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#issuecomment-990031441
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7885](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (152a55d) into [master](https://codecov.io/gh/apache/pinot/commit/fb12a509056b4e36724126cef9a344a5e1e89bb1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (fb12a50) will **decrease** coverage by `6.38%`.
> The diff coverage is `74.53%`.
[](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7885 +/- ##
============================================
- Coverage 71.34% 64.95% -6.39%
- Complexity 4087 4099 +12
============================================
Files 1587 1547 -40
Lines 82071 80421 -1650
Branches 12267 12069 -198
============================================
- Hits 58553 52238 -6315
- Misses 19550 24459 +4909
+ Partials 3968 3724 -244
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `68.33% <75.09%> (+<0.01%)` | :arrow_up: |
| unittests2 | `14.35% <0.00%> (-0.10%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/pinot/core/minion/RawIndexConverter.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9taW5pb24vUmF3SW5kZXhDb252ZXJ0ZXIuamF2YQ==) | `0.00% <0.00%> (-56.61%)` | :arrow_down: |
| [...nverttorawindex/ConvertToRawIndexTaskExecutor.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvY29udmVydHRvcmF3aW5kZXgvQ29udmVydFRvUmF3SW5kZXhUYXNrRXhlY3V0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...not/segment/spi/creator/IndexCreatorProviders.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2NyZWF0b3IvSW5kZXhDcmVhdG9yUHJvdmlkZXJzLmphdmE=) | `52.38% <52.38%> (ø)` | |
| [...t/index/loader/invertedindex/JsonIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0pzb25JbmRleEhhbmRsZXIuamF2YQ==) | `73.33% <66.66%> (+0.45%)` | :arrow_up: |
| [...ment/creator/impl/DefaultIndexCreatorProvider.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9EZWZhdWx0SW5kZXhDcmVhdG9yUHJvdmlkZXIuamF2YQ==) | `67.14% <67.14%> (ø)` | |
| [.../index/loader/invertedindex/RangeIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L1JhbmdlSW5kZXhIYW5kbGVyLmphdmE=) | `44.44% <85.71%> (+3.44%)` | :arrow_up: |
| [...ment/creator/impl/SegmentColumnarIndexCreator.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9TZWdtZW50Q29sdW1uYXJJbmRleENyZWF0b3IuamF2YQ==) | `86.99% <87.50%> (+2.60%)` | :arrow_up: |
| [...ocal/segment/index/loader/IndexHandlerFactory.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9JbmRleEhhbmRsZXJGYWN0b3J5LmphdmE=) | `100.00% <100.00%> (ø)` | |
| [...t/index/loader/bloomfilter/BloomFilterHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9ibG9vbWZpbHRlci9CbG9vbUZpbHRlckhhbmRsZXIuamF2YQ==) | `78.33% <100.00%> (+0.74%)` | :arrow_up: |
| [...nt/index/loader/invertedindex/FSTIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0ZTVEluZGV4SGFuZGxlci5qYXZh) | `86.79% <100.00%> (+4.10%)` | :arrow_up: |
| ... and [388 more](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [fb12a50...152a55d](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#issuecomment-990031441
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7885](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3ad1746) into [master](https://codecov.io/gh/apache/pinot/commit/fb12a509056b4e36724126cef9a344a5e1e89bb1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (fb12a50) will **decrease** coverage by `6.35%`.
> The diff coverage is `80.80%`.
[](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7885 +/- ##
============================================
- Coverage 71.34% 64.98% -6.36%
- Complexity 4087 4105 +18
============================================
Files 1587 1547 -40
Lines 82071 80370 -1701
Branches 12267 12066 -201
============================================
- Hits 58553 52228 -6325
- Misses 19550 24420 +4870
+ Partials 3968 3722 -246
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `68.39% <81.45%> (+0.06%)` | :arrow_up: |
| unittests2 | `14.35% <0.00%> (-0.10%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/pinot/core/minion/RawIndexConverter.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9taW5pb24vUmF3SW5kZXhDb252ZXJ0ZXIuamF2YQ==) | `0.00% <0.00%> (-56.61%)` | :arrow_down: |
| [...nverttorawindex/ConvertToRawIndexTaskExecutor.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvY29udmVydHRvcmF3aW5kZXgvQ29udmVydFRvUmF3SW5kZXhUYXNrRXhlY3V0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...not/segment/spi/creator/IndexCreatorProviders.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2NyZWF0b3IvSW5kZXhDcmVhdG9yUHJvdmlkZXJzLmphdmE=) | `52.38% <52.38%> (ø)` | |
| [...t/index/loader/invertedindex/JsonIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0pzb25JbmRleEhhbmRsZXIuamF2YQ==) | `73.33% <66.66%> (+0.45%)` | :arrow_up: |
| [...ment/creator/impl/DefaultIndexCreatorProvider.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9EZWZhdWx0SW5kZXhDcmVhdG9yUHJvdmlkZXIuamF2YQ==) | `67.14% <67.14%> (ø)` | |
| [.../index/loader/invertedindex/RangeIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L1JhbmdlSW5kZXhIYW5kbGVyLmphdmE=) | `44.44% <85.71%> (+3.44%)` | :arrow_up: |
| [...ment/creator/impl/SegmentColumnarIndexCreator.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9TZWdtZW50Q29sdW1uYXJJbmRleENyZWF0b3IuamF2YQ==) | `86.99% <87.50%> (+2.60%)` | :arrow_up: |
| [...ocal/segment/index/loader/IndexHandlerFactory.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9JbmRleEhhbmRsZXJGYWN0b3J5LmphdmE=) | `100.00% <100.00%> (ø)` | |
| [...t/index/loader/bloomfilter/BloomFilterHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9ibG9vbWZpbHRlci9CbG9vbUZpbHRlckhhbmRsZXIuamF2YQ==) | `78.33% <100.00%> (+0.74%)` | :arrow_up: |
| [...nt/index/loader/invertedindex/FSTIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0ZTVEluZGV4SGFuZGxlci5qYXZh) | `86.79% <100.00%> (+4.10%)` | :arrow_up: |
| ... and [389 more](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [fb12a50...3ad1746](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#issuecomment-990031441
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7885](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b3961f0) into [master](https://codecov.io/gh/apache/pinot/commit/fb12a509056b4e36724126cef9a344a5e1e89bb1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (fb12a50) will **decrease** coverage by `43.73%`.
> The diff coverage is `17.71%`.
> :exclamation: Current head b3961f0 differs from pull request most recent head 152a55d. Consider uploading reports for the commit 152a55d to get more accurate results
[](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7885 +/- ##
=============================================
- Coverage 71.34% 27.60% -43.74%
=============================================
Files 1587 1583 -4
Lines 82071 81944 -127
Branches 12267 12234 -33
=============================================
- Hits 58553 22622 -35931
- Misses 19550 57261 +37711
+ Partials 3968 2061 -1907
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `27.60% <17.71%> (-0.05%)` | :arrow_down: |
| unittests1 | `?` | |
| unittests2 | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../java/org/apache/pinot/common/utils/DataTable.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvRGF0YVRhYmxlLmphdmE=) | `89.47% <ø> (ø)` | |
| [...e/pinot/core/common/datatable/DataTableImplV2.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vZGF0YXRhYmxlL0RhdGFUYWJsZUltcGxWMi5qYXZh) | `0.00% <0.00%> (-77.78%)` | :arrow_down: |
| [...rg/apache/pinot/core/minion/RawIndexConverter.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9taW5pb24vUmF3SW5kZXhDb252ZXJ0ZXIuamF2YQ==) | `0.00% <0.00%> (-56.61%)` | :arrow_down: |
| [...ache/pinot/core/plan/InstanceResponsePlanNode.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0luc3RhbmNlUmVzcG9uc2VQbGFuTm9kZS5qYXZh) | `100.00% <ø> (ø)` | |
| [...nverttorawindex/ConvertToRawIndexTaskExecutor.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvY29udmVydHRvcmF3aW5kZXgvQ29udmVydFRvUmF3SW5kZXhUYXNrRXhlY3V0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...ment/creator/impl/DefaultIndexCreatorProvider.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9EZWZhdWx0SW5kZXhDcmVhdG9yUHJvdmlkZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...ment/creator/impl/SegmentColumnarIndexCreator.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9TZWdtZW50Q29sdW1uYXJJbmRleENyZWF0b3IuamF2YQ==) | `0.00% <0.00%> (-84.40%)` | :arrow_down: |
| [...ocal/segment/index/loader/IndexHandlerFactory.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9JbmRleEhhbmRsZXJGYWN0b3J5LmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...t/index/loader/bloomfilter/BloomFilterHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9ibG9vbWZpbHRlci9CbG9vbUZpbHRlckhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (-77.59%)` | :arrow_down: |
| [...nt/index/loader/invertedindex/FSTIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0ZTVEluZGV4SGFuZGxlci5qYXZh) | `0.00% <0.00%> (-82.70%)` | :arrow_down: |
| ... and [1172 more](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [fb12a50...152a55d](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r768314829
##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/IndexCreatorProviders.java
##########
@@ -0,0 +1,143 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.creator;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.lang.invoke.MethodType;
+import java.util.concurrent.atomic.AtomicReference;
+import org.apache.pinot.segment.spi.index.creator.BloomFilterCreator;
+import org.apache.pinot.segment.spi.index.creator.CombinedInvertedIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.DictionaryBasedInvertedIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.GeoSpatialIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.JsonIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.TextIndexCreator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Plugin registration point to allow index creation logic to be swapped out.
+ * Plugins should not reimplement Pinot's default index creation logic.
+ * Users provide an override to Pinot's index creation logic. This is simplified
+ * by extending {@see IndexCreatorProviders.Default}
+ */
+public final class IndexCreatorProviders {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(IndexCreatorProviders.class);
+
+ static final IndexCreatorProvider DEFAULT = defaultProvider();
+ private static final AtomicReference<IndexCreatorProvider> REGISTRATION = new AtomicReference<>(DEFAULT);
+
+ private IndexCreatorProviders() {
+ }
+
+ /**
+ * The caller provides a decorator to wrap the default provider, which allows plugins to create
+ * a delegation chain.
+ * @param provider index creation override
+ * @return true if this is the first invocation and the provider has not yet been used.
+ */
+ public static boolean registerProvider(IndexCreatorProvider provider) {
+ return REGISTRATION.compareAndSet(DEFAULT, provider);
+ }
+
+ /**
+ * Obtain the registered index creator provider. If the user has provided an override, then it will be used instead.
+ * If the user has not provided an override yet, then this action will prevent them from doing so.
+ * @return the global index provision logic.
+ */
+ public static IndexCreatorProvider getIndexCreatorProvider() {
+ return Holder.PROVIDER;
+ }
+
+ private static final class Holder {
+ public static final IndexCreatorProvider PROVIDER = REGISTRATION.get();
+ }
+
+ private static IndexCreatorProvider defaultProvider() {
+ // use MethodHandle to break circular dependency and keep implementation details encapsulated within
+ // pinot-segment-local
+ String className = "org.apache.pinot.segment.local.segment.creator.impl.DefaultIndexCreatorProvider";
+ try {
+ Class<?> clazz = Class.forName(className, false, IndexCreatorProviders.class.getClassLoader());
+ return (IndexCreatorProvider) MethodHandles.publicLookup()
+ .findConstructor(clazz, MethodType.methodType(void.class)).invoke();
+ } catch (Throwable implausible) {
+ LOGGER.error("could not construct MethodHandle for {}", className,
+ implausible);
+ throw new IllegalStateException(implausible);
+ }
+ }
+
+ /**
+ * Extend this class to override index creation
+ */
+ public static class Default implements IndexCreatorProvider {
Review comment:
IIUC, this is the one that allows to mix ? So for example, someone might want to use existing forward index, dictionary and everything else but might want to plugin a different inverted index ? So, this can be sub-classed and `newInvertedIndexCreator` can be overridden to get custom behavior. The sub-class can then be registered ? Is this understanding correct ?
Also, can we try to add one test (or may not be needed if example usage pattern is going to be covered in docs later on) ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r770367746
##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/IndexCreationContext.java
##########
@@ -0,0 +1,481 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.creator;
+
+import java.io.File;
+import java.util.Map;
+import java.util.Objects;
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.index.creator.H3IndexConfig;
+import org.apache.pinot.spi.config.table.BloomFilterConfig;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+/**
+ * Provides parameters for constructing indexes via {@see IndexCreatorProvider}.
+ * The responsibility for ensuring that the correct parameters for a particular
+ * index type lies with the caller.
+ */
+public interface IndexCreationContext {
+
+ FieldSpec getFieldSpec();
+
+ File getIndexDir();
+
+ boolean isOnHeap();
+
+ int getLengthOfLongestEntry();
+
+ int getMaxNumberOfMultiValueElements();
+
+ int getMaxRowLengthInBytes();
+
+ boolean isSorted();
+
+ int getCardinality();
+
+ int getTotalNumberOfEntries();
+
+ int getTotalDocs();
+
+ boolean hasDictionary();
+
+ BloomFilter forBloomFilter(BloomFilterConfig bloomFilterConfig);
+
+ Forward forForwardIndex(ChunkCompressionType chunkCompressionType,
+ @Nullable Map<String, Map<String, String>> columnProperties);
+
+ Geospatial forGeospatialIndex(H3IndexConfig h3IndexConfig);
+
+ Inverted forInvertedIndex();
+
+ Json forJsonIndex();
+
+ Range forRangeIndex(ColumnMetadata columnMetadata, int rangeIndexVersion);
+
+ Text forTextIndex(FSTType fstType, String[] sortedUniqueElementsArray);
Review comment:
FWIW, the reason for this confusion is I just don't want to override text index construction and am only interested in forward, inverted, and JSON indexes, but the scope of this PR expanded to cover text. I don't fully understand the text indexing features and they're only included at all for completeness.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r768459249
##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/IndexCreationContext.java
##########
@@ -0,0 +1,256 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.creator;
+
+import java.io.File;
+import java.util.Map;
+import java.util.Objects;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.index.creator.H3IndexConfig;
+import org.apache.pinot.spi.config.table.BloomFilterConfig;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+public class IndexCreationContext {
Review comment:
This is not an extension point, and can't be customised. It is just the carrier of parameters from caller to creator. I may make some refinement here to use the type system to ensure e.g. a caller who wants a bloom filter always provides the correct parameters
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#issuecomment-993005556
@richardstartin just seeing this. I will review it today. Looks like the design doc link was intended to be an issue but it is linking to this PR ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#issuecomment-990031441
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7885](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2b42bc9) into [master](https://codecov.io/gh/apache/pinot/commit/fb12a509056b4e36724126cef9a344a5e1e89bb1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (fb12a50) will **decrease** coverage by `6.31%`.
> The diff coverage is `82.15%`.
[](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7885 +/- ##
============================================
- Coverage 71.34% 65.02% -6.32%
- Complexity 4087 4110 +23
============================================
Files 1587 1547 -40
Lines 82071 80473 -1598
Branches 12267 12076 -191
============================================
- Hits 58553 52328 -6225
- Misses 19550 24409 +4859
+ Partials 3968 3736 -232
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `68.43% <82.66%> (+0.10%)` | :arrow_up: |
| unittests2 | `14.33% <0.00%> (-0.11%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/pinot/core/minion/RawIndexConverter.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9taW5pb24vUmF3SW5kZXhDb252ZXJ0ZXIuamF2YQ==) | `0.00% <0.00%> (-56.61%)` | :arrow_down: |
| [...nverttorawindex/ConvertToRawIndexTaskExecutor.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvY29udmVydHRvcmF3aW5kZXgvQ29udmVydFRvUmF3SW5kZXhUYXNrRXhlY3V0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...not/segment/spi/creator/IndexCreatorProviders.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2NyZWF0b3IvSW5kZXhDcmVhdG9yUHJvdmlkZXJzLmphdmE=) | `45.71% <45.71%> (ø)` | |
| [...t/index/loader/invertedindex/JsonIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0pzb25JbmRleEhhbmRsZXIuamF2YQ==) | `73.33% <66.66%> (+0.45%)` | :arrow_up: |
| [...ment/creator/impl/DefaultIndexCreatorProvider.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9EZWZhdWx0SW5kZXhDcmVhdG9yUHJvdmlkZXIuamF2YQ==) | `67.56% <67.56%> (ø)` | |
| [.../index/loader/invertedindex/RangeIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L1JhbmdlSW5kZXhIYW5kbGVyLmphdmE=) | `44.44% <85.71%> (+3.44%)` | :arrow_up: |
| [...ment/creator/impl/SegmentColumnarIndexCreator.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9TZWdtZW50Q29sdW1uYXJJbmRleENyZWF0b3IuamF2YQ==) | `87.26% <89.74%> (+2.87%)` | :arrow_up: |
| [...ocal/segment/index/loader/IndexHandlerFactory.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9JbmRleEhhbmRsZXJGYWN0b3J5LmphdmE=) | `100.00% <100.00%> (ø)` | |
| [...t/index/loader/bloomfilter/BloomFilterHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9ibG9vbWZpbHRlci9CbG9vbUZpbHRlckhhbmRsZXIuamF2YQ==) | `78.33% <100.00%> (+0.74%)` | :arrow_up: |
| [...nt/index/loader/invertedindex/FSTIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0ZTVEluZGV4SGFuZGxlci5qYXZh) | `86.79% <100.00%> (+4.10%)` | :arrow_up: |
| ... and [392 more](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [fb12a50...2b42bc9](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r770838596
##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/IndexCreationContext.java
##########
@@ -0,0 +1,454 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.creator;
+
+import java.io.File;
+import java.util.Map;
+import java.util.Objects;
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.index.creator.H3IndexConfig;
+import org.apache.pinot.spi.config.table.BloomFilterConfig;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+/**
+ * Provides parameters for constructing indexes via {@see IndexCreatorProvider}.
+ * The responsibility for ensuring that the correct parameters for a particular
+ * index type lies with the caller.
+ */
+public interface IndexCreationContext {
+
+ FieldSpec getFieldSpec();
+
+ File getIndexDir();
+
+ boolean isOnHeap();
+
+ int getLengthOfLongestEntry();
+
+ int getMaxNumberOfMultiValueElements();
+
+ int getMaxRowLengthInBytes();
+
+ boolean isSorted();
+
+ int getCardinality();
+
+ int getTotalNumberOfEntries();
+
+ int getTotalDocs();
+
+ boolean hasDictionary();
+
+ final class Builder {
+ private File _indexDir;
+ private int _lengthOfLongestEntry;
+ private int _maxNumberOfMultiValueElements;
+ private int _maxRowLengthInBytes;
+ private boolean _onHeap = false;
+ private FieldSpec _fieldSpec;
+ private boolean _sorted;
+ private int _cardinality;
+ private int _totalNumberOfEntries;
+ private int _totalDocs;
+ private boolean _hasDictionary = true;
+
+ public Builder withColumnIndexCreationInfo(ColumnIndexCreationInfo columnIndexCreationInfo) {
+ return withLengthOfLongestEntry(columnIndexCreationInfo.getLengthOfLongestEntry())
+ .withMaxNumberOfMultiValueElements(columnIndexCreationInfo.getMaxNumberOfMultiValueElements())
+ .withMaxRowLengthInBytes(columnIndexCreationInfo.getMaxRowLengthInBytes());
+ }
+
+ public Builder withIndexDir(File indexDir) {
+ _indexDir = indexDir;
+ return this;
+ }
+
+ public Builder onHeap(boolean onHeap) {
+ _onHeap = onHeap;
+ return this;
+ }
+
+ public Builder withColumnMetadata(ColumnMetadata columnMetadata) {
+ return withFieldSpec(columnMetadata.getFieldSpec())
+ .sorted(columnMetadata.isSorted())
+ .withCardinality(columnMetadata.getCardinality())
+ .withTotalNumberOfEntries(columnMetadata.getTotalNumberOfEntries())
+ .withTotalDocs(columnMetadata.getTotalDocs())
+ .withDictionary(columnMetadata.hasDictionary());
+ }
+
+ public Builder withLengthOfLongestEntry(int lengthOfLongestEntry) {
+ _lengthOfLongestEntry = lengthOfLongestEntry;
+ return this;
+ }
+
+ public Builder withMaxNumberOfMultiValueElements(int maxNumberOfMultiValueElements) {
+ _maxNumberOfMultiValueElements = maxNumberOfMultiValueElements;
+ return this;
+ }
+
+ public Builder withMaxRowLengthInBytes(int maxRowLengthInBytes) {
+ _maxRowLengthInBytes = maxRowLengthInBytes;
+ return this;
+ }
+
+ public Builder withFieldSpec(FieldSpec fieldSpec) {
+ _fieldSpec = fieldSpec;
+ return this;
+ }
+
+ public Builder sorted(boolean sorted) {
+ _sorted = sorted;
+ return this;
+ }
+
+ public Builder withCardinality(int cardinality) {
+ _cardinality = cardinality;
+ return this;
+ }
+
+ public Builder withTotalNumberOfEntries(int totalNumberOfEntries) {
+ _totalNumberOfEntries = totalNumberOfEntries;
+ return this;
+ }
+
+ public Builder withTotalDocs(int totalDocs) {
+ _totalDocs = totalDocs;
+ return this;
+ }
+
+ public Builder withDictionary(boolean hasDictionary) {
+ _hasDictionary = hasDictionary;
+ return this;
+ }
+
+ public Common build() {
+ return new Common(Objects.requireNonNull(_indexDir),
+ _lengthOfLongestEntry, _maxNumberOfMultiValueElements, _maxRowLengthInBytes,
+ _onHeap, Objects.requireNonNull(_fieldSpec),
+ _sorted, _cardinality, _totalNumberOfEntries, _totalDocs, _hasDictionary);
+ }
+ }
+
+ static Builder builder() {
+ return new Builder();
+ }
+
+ final class Common implements IndexCreationContext {
+
+ private final File _indexDir;
+ private final int _lengthOfLongestEntry;
+ private final int _maxNumberOfMultiValueElements;
+ private final int _maxRowLengthInBytes;
+ private final boolean _onHeap;
+ private final FieldSpec _fieldSpec;
+ private final boolean _sorted;
+ private final int _cardinality;
+ private final int _totalNumberOfEntries;
+ private final int _totalDocs;
+ private final boolean _hasDictionary;
+
+ public Common(File indexDir, int lengthOfLongestEntry,
+ int maxNumberOfMultiValueElements, int maxRowLengthInBytes, boolean onHeap,
+ FieldSpec fieldSpec, boolean sorted, int cardinality, int totalNumberOfEntries,
+ int totalDocs, boolean hasDictionary) {
+ _indexDir = indexDir;
+ _lengthOfLongestEntry = lengthOfLongestEntry;
+ _maxNumberOfMultiValueElements = maxNumberOfMultiValueElements;
+ _maxRowLengthInBytes = maxRowLengthInBytes;
+ _onHeap = onHeap;
+ _fieldSpec = fieldSpec;
+ _sorted = sorted;
+ _cardinality = cardinality;
+ _totalNumberOfEntries = totalNumberOfEntries;
+ _totalDocs = totalDocs;
+ _hasDictionary = hasDictionary;
+ }
+
+ public FieldSpec getFieldSpec() {
+ return _fieldSpec;
+ }
+
+ public File getIndexDir() {
+ return _indexDir;
+ }
+
+ public boolean isOnHeap() {
+ return _onHeap;
+ }
+
+ public int getLengthOfLongestEntry() {
+ return _lengthOfLongestEntry;
+ }
+
+ public int getMaxNumberOfMultiValueElements() {
+ return _maxNumberOfMultiValueElements;
+ }
+
+ public int getMaxRowLengthInBytes() {
+ return _maxRowLengthInBytes;
+ }
+
+ public boolean isSorted() {
+ return _sorted;
+ }
+
+ public int getCardinality() {
+ return _cardinality;
+ }
+
+ public int getTotalNumberOfEntries() {
+ return _totalNumberOfEntries;
+ }
+
+ public int getTotalDocs() {
+ return _totalDocs;
+ }
+
+ public boolean hasDictionary() {
+ return _hasDictionary;
+ }
+
+ public BloomFilter forBloomFilter(BloomFilterConfig bloomFilterConfig) {
+ return new BloomFilter(this, bloomFilterConfig);
+ }
+
+ public Forward forForwardIndex(ChunkCompressionType chunkCompressionType,
+ @Nullable Map<String, Map<String, String>> columnProperties) {
+ return new Forward(this, chunkCompressionType, columnProperties);
+ }
+
+ public Text forFSTIndex(FSTType fstType, String[] sortedUniqueElementsArray) {
+ return new Text(this, fstType, sortedUniqueElementsArray);
+ }
+
+ public Geospatial forGeospatialIndex(H3IndexConfig h3IndexConfig) {
+ return new Geospatial(this, h3IndexConfig);
+ }
+
+ public Inverted forInvertedIndex() {
+ return new Inverted(this);
+ }
+
+ public Json forJsonIndex() {
+ return new Json(this);
+ }
+
+ public Range forRangeIndex(ColumnMetadata columnMetadata, int rangeIndexVersion) {
+ return new Range(this, columnMetadata, rangeIndexVersion);
+ }
+
+ public Text forTextIndex(boolean commitOnClose) {
+ return new Text(this, commitOnClose);
+ }
+ }
+
+ class Wrapper implements IndexCreationContext {
+
+ private final IndexCreationContext _delegate;
+
+ Wrapper(IndexCreationContext delegate) {
+ _delegate = delegate;
+ }
+
+ @Override
+ public FieldSpec getFieldSpec() {
+ return _delegate.getFieldSpec();
+ }
+
+ @Override
+ public File getIndexDir() {
+ return _delegate.getIndexDir();
+ }
+
+ @Override
+ public boolean isOnHeap() {
+ return _delegate.isOnHeap();
+ }
+
+ @Override
+ public int getLengthOfLongestEntry() {
+ return _delegate.getLengthOfLongestEntry();
+ }
+
+ @Override
+ public int getMaxNumberOfMultiValueElements() {
+ return _delegate.getMaxNumberOfMultiValueElements();
+ }
+
+ @Override
+ public int getMaxRowLengthInBytes() {
+ return _delegate.getMaxRowLengthInBytes();
+ }
+
+ @Override
+ public boolean isSorted() {
+ return _delegate.isSorted();
+ }
+
+ @Override
+ public int getCardinality() {
+ return _delegate.getCardinality();
+ }
+
+ @Override
+ public int getTotalNumberOfEntries() {
+ return _delegate.getTotalNumberOfEntries();
+ }
+
+ @Override
+ public int getTotalDocs() {
+ return _delegate.getTotalDocs();
+ }
+
+ @Override
+ public boolean hasDictionary() {
+ return _delegate.hasDictionary();
+ }
+ }
+
+ class BloomFilter extends Wrapper {
+
+ private final BloomFilterConfig _bloomFilterConfig;
+
+ public BloomFilter(IndexCreationContext wrapped, BloomFilterConfig bloomFilterConfig) {
+ super(wrapped);
+ _bloomFilterConfig = bloomFilterConfig;
+ }
+
+ @Nullable
+ public BloomFilterConfig getBloomFilterConfig() {
+ return _bloomFilterConfig;
+ }
+ }
+
+ class Forward extends Wrapper {
+
+ private final ChunkCompressionType _chunkCompressionType;
+ private final Map<String, Map<String, String>> _columnProperties;
+
+ Forward(IndexCreationContext delegate,
+ ChunkCompressionType chunkCompressionType,
+ @Nullable Map<String, Map<String, String>> columnProperties) {
+ super(delegate);
+ _chunkCompressionType = chunkCompressionType;
+ _columnProperties = columnProperties;
+ }
+
+ public ChunkCompressionType getChunkCompressionType() {
+ return _chunkCompressionType;
+ }
+
+ @Nullable
+ public Map<String, Map<String, String>> getColumnProperties() {
+ return _columnProperties;
+ }
+ }
+
+ class Geospatial extends Wrapper {
+
+ private final H3IndexConfig _h3IndexConfig;
+
+ Geospatial(IndexCreationContext delegate, H3IndexConfig h3IndexConfig) {
+ super(delegate);
+ _h3IndexConfig = h3IndexConfig;
+ }
+
+ public H3IndexConfig getH3IndexConfig() {
+ return _h3IndexConfig;
+ }
+ }
+
+ class Inverted extends Wrapper {
+
+ Inverted(IndexCreationContext delegate) {
+ super(delegate);
+ }
+ }
+
+ class Json extends Wrapper {
+
+ Json(IndexCreationContext delegate) {
+ super(delegate);
+ }
+ }
+
+ class Range extends Wrapper {
+
+ private final ColumnMetadata _columnMetadata;
+ private final int _rangeIndexVersion;
+
+ Range(IndexCreationContext delegate, ColumnMetadata columnMetadata, int rangeIndexVersion) {
+ super(delegate);
+ _columnMetadata = columnMetadata;
+ _rangeIndexVersion = rangeIndexVersion;
+ }
+
+ public ColumnMetadata getColumnMetadata() {
+ return _columnMetadata;
+ }
+
+ public int getRangeIndexVersion() {
+ return _rangeIndexVersion;
+ }
+ }
+
+ class Text extends Wrapper {
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r770028671
##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/IndexCreationContext.java
##########
@@ -0,0 +1,261 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.creator;
+
+import java.io.File;
+import java.util.Map;
+import java.util.Objects;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.index.creator.H3IndexConfig;
+import org.apache.pinot.spi.config.table.BloomFilterConfig;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+/**
+ * Provides parameters for constructing indexes via {@see IndexCreatorProvider}.
+ * The responsibility for ensuring that the correct parameters for a particular
+ * index type lies with the caller.
+ */
+public class IndexCreationContext {
+
+ public static final class Builder {
+ private File _indexDir;
+ private Map<String, Map<String, String>> _columnProperties;
+ private FSTType _fstType;
+ private H3IndexConfig _h3IndexConfig;
+ private int _lengthOfLongestEntry;
+ private int _maxNumberOfMultiValueElements;
+ private int _maxRowLengthInBytes;
+ private Object _sortedUniqueElementsArray;
+ private boolean _onHeap = false;
+ private ChunkCompressionType _chunkCompressionType;
+ public BloomFilterConfig _bloomFilterConfig;
+ private int _rangeIndexVersion;
+ private ColumnMetadata _columnMetadata;
Review comment:
I will refactor this so there isn't one monolithic context
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r770884366
##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/IndexCreationContext.java
##########
@@ -0,0 +1,467 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.creator;
+
+import java.io.File;
+import java.util.Map;
+import java.util.Objects;
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.index.creator.H3IndexConfig;
+import org.apache.pinot.spi.config.table.BloomFilterConfig;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+/**
+ * Provides parameters for constructing indexes via {@see IndexCreatorProvider}.
+ * The responsibility for ensuring that the correct parameters for a particular
+ * index type lies with the caller.
+ */
+public interface IndexCreationContext {
+
+ FieldSpec getFieldSpec();
+
+ File getIndexDir();
+
+ boolean isOnHeap();
+
+ int getLengthOfLongestEntry();
+
+ int getMaxNumberOfMultiValueElements();
+
+ int getMaxRowLengthInBytes();
+
+ boolean isSorted();
+
+ int getCardinality();
+
+ int getTotalNumberOfEntries();
+
+ int getTotalDocs();
+
+ boolean hasDictionary();
+
+ final class Builder {
+ private File _indexDir;
+ private int _lengthOfLongestEntry;
+ private int _maxNumberOfMultiValueElements;
+ private int _maxRowLengthInBytes;
+ private boolean _onHeap = false;
+ private FieldSpec _fieldSpec;
+ private boolean _sorted;
+ private int _cardinality;
+ private int _totalNumberOfEntries;
+ private int _totalDocs;
+ private boolean _hasDictionary = true;
+
+ public Builder withColumnIndexCreationInfo(ColumnIndexCreationInfo columnIndexCreationInfo) {
+ return withLengthOfLongestEntry(columnIndexCreationInfo.getLengthOfLongestEntry())
+ .withMaxNumberOfMultiValueElements(columnIndexCreationInfo.getMaxNumberOfMultiValueElements())
+ .withMaxRowLengthInBytes(columnIndexCreationInfo.getMaxRowLengthInBytes());
+ }
+
+ public Builder withIndexDir(File indexDir) {
+ _indexDir = indexDir;
+ return this;
+ }
+
+ public Builder onHeap(boolean onHeap) {
+ _onHeap = onHeap;
+ return this;
+ }
+
+ public Builder withColumnMetadata(ColumnMetadata columnMetadata) {
+ return withFieldSpec(columnMetadata.getFieldSpec())
+ .sorted(columnMetadata.isSorted())
+ .withCardinality(columnMetadata.getCardinality())
+ .withTotalNumberOfEntries(columnMetadata.getTotalNumberOfEntries())
+ .withTotalDocs(columnMetadata.getTotalDocs())
+ .withDictionary(columnMetadata.hasDictionary());
+ }
+
+ public Builder withLengthOfLongestEntry(int lengthOfLongestEntry) {
+ _lengthOfLongestEntry = lengthOfLongestEntry;
+ return this;
+ }
+
+ public Builder withMaxNumberOfMultiValueElements(int maxNumberOfMultiValueElements) {
+ _maxNumberOfMultiValueElements = maxNumberOfMultiValueElements;
+ return this;
+ }
+
+ public Builder withMaxRowLengthInBytes(int maxRowLengthInBytes) {
+ _maxRowLengthInBytes = maxRowLengthInBytes;
+ return this;
+ }
+
+ public Builder withFieldSpec(FieldSpec fieldSpec) {
+ _fieldSpec = fieldSpec;
+ return this;
+ }
+
+ public Builder sorted(boolean sorted) {
+ _sorted = sorted;
+ return this;
+ }
+
+ public Builder withCardinality(int cardinality) {
+ _cardinality = cardinality;
+ return this;
+ }
+
+ public Builder withTotalNumberOfEntries(int totalNumberOfEntries) {
+ _totalNumberOfEntries = totalNumberOfEntries;
+ return this;
+ }
+
+ public Builder withTotalDocs(int totalDocs) {
+ _totalDocs = totalDocs;
+ return this;
+ }
+
+ public Builder withDictionary(boolean hasDictionary) {
+ _hasDictionary = hasDictionary;
+ return this;
+ }
+
+ public Common build() {
+ return new Common(Objects.requireNonNull(_indexDir),
+ _lengthOfLongestEntry, _maxNumberOfMultiValueElements, _maxRowLengthInBytes,
+ _onHeap, Objects.requireNonNull(_fieldSpec),
+ _sorted, _cardinality, _totalNumberOfEntries, _totalDocs, _hasDictionary);
+ }
+ }
+
+ static Builder builder() {
+ return new Builder();
+ }
+
+ final class Common implements IndexCreationContext {
+
+ private final File _indexDir;
+ private final int _lengthOfLongestEntry;
+ private final int _maxNumberOfMultiValueElements;
+ private final int _maxRowLengthInBytes;
+ private final boolean _onHeap;
+ private final FieldSpec _fieldSpec;
+ private final boolean _sorted;
+ private final int _cardinality;
+ private final int _totalNumberOfEntries;
+ private final int _totalDocs;
+ private final boolean _hasDictionary;
+
+ public Common(File indexDir, int lengthOfLongestEntry,
+ int maxNumberOfMultiValueElements, int maxRowLengthInBytes, boolean onHeap,
+ FieldSpec fieldSpec, boolean sorted, int cardinality, int totalNumberOfEntries,
+ int totalDocs, boolean hasDictionary) {
+ _indexDir = indexDir;
+ _lengthOfLongestEntry = lengthOfLongestEntry;
+ _maxNumberOfMultiValueElements = maxNumberOfMultiValueElements;
+ _maxRowLengthInBytes = maxRowLengthInBytes;
+ _onHeap = onHeap;
+ _fieldSpec = fieldSpec;
+ _sorted = sorted;
+ _cardinality = cardinality;
+ _totalNumberOfEntries = totalNumberOfEntries;
+ _totalDocs = totalDocs;
+ _hasDictionary = hasDictionary;
+ }
+
+ public FieldSpec getFieldSpec() {
+ return _fieldSpec;
+ }
+
+ public File getIndexDir() {
+ return _indexDir;
+ }
+
+ public boolean isOnHeap() {
+ return _onHeap;
+ }
+
+ public int getLengthOfLongestEntry() {
+ return _lengthOfLongestEntry;
+ }
+
+ public int getMaxNumberOfMultiValueElements() {
+ return _maxNumberOfMultiValueElements;
+ }
+
+ public int getMaxRowLengthInBytes() {
+ return _maxRowLengthInBytes;
+ }
+
+ public boolean isSorted() {
+ return _sorted;
+ }
+
+ public int getCardinality() {
+ return _cardinality;
+ }
+
+ public int getTotalNumberOfEntries() {
+ return _totalNumberOfEntries;
+ }
+
+ public int getTotalDocs() {
+ return _totalDocs;
+ }
+
+ public boolean hasDictionary() {
+ return _hasDictionary;
+ }
+
+ public BloomFilter forBloomFilter(BloomFilterConfig bloomFilterConfig) {
+ return new BloomFilter(this, bloomFilterConfig);
+ }
+
+ public Forward forForwardIndex(ChunkCompressionType chunkCompressionType,
+ @Nullable Map<String, Map<String, String>> columnProperties) {
+ return new Forward(this, chunkCompressionType, columnProperties);
+ }
+
+ public Text forFSTIndex(FSTType fstType, String[] sortedUniqueElementsArray) {
+ return new Text(this, fstType, sortedUniqueElementsArray);
+ }
+
+ public Geospatial forGeospatialIndex(H3IndexConfig h3IndexConfig) {
+ return new Geospatial(this, h3IndexConfig);
+ }
+
+ public Inverted forInvertedIndex() {
+ return new Inverted(this);
+ }
+
+ public Json forJsonIndex() {
+ return new Json(this);
+ }
+
+ public Range forRangeIndex(int rangeIndexVersion, Comparable<?> min, Comparable<?> max) {
+ return new Range(this, rangeIndexVersion, min, max);
+ }
+
+ public Text forTextIndex(boolean commitOnClose) {
+ return new Text(this, commitOnClose);
+ }
+ }
+
+ class Wrapper implements IndexCreationContext {
+
+ private final IndexCreationContext _delegate;
+
+ Wrapper(IndexCreationContext delegate) {
+ _delegate = delegate;
+ }
+
+ @Override
+ public FieldSpec getFieldSpec() {
+ return _delegate.getFieldSpec();
+ }
+
+ @Override
+ public File getIndexDir() {
+ return _delegate.getIndexDir();
+ }
+
+ @Override
+ public boolean isOnHeap() {
+ return _delegate.isOnHeap();
+ }
+
+ @Override
+ public int getLengthOfLongestEntry() {
+ return _delegate.getLengthOfLongestEntry();
+ }
+
+ @Override
+ public int getMaxNumberOfMultiValueElements() {
+ return _delegate.getMaxNumberOfMultiValueElements();
+ }
+
+ @Override
+ public int getMaxRowLengthInBytes() {
+ return _delegate.getMaxRowLengthInBytes();
+ }
+
+ @Override
+ public boolean isSorted() {
+ return _delegate.isSorted();
+ }
+
+ @Override
+ public int getCardinality() {
+ return _delegate.getCardinality();
+ }
+
+ @Override
+ public int getTotalNumberOfEntries() {
+ return _delegate.getTotalNumberOfEntries();
+ }
+
+ @Override
+ public int getTotalDocs() {
+ return _delegate.getTotalDocs();
+ }
+
+ @Override
+ public boolean hasDictionary() {
+ return _delegate.hasDictionary();
+ }
+ }
+
+ class BloomFilter extends Wrapper {
+
+ private final BloomFilterConfig _bloomFilterConfig;
+
+ public BloomFilter(IndexCreationContext wrapped, BloomFilterConfig bloomFilterConfig) {
+ super(wrapped);
+ _bloomFilterConfig = bloomFilterConfig;
+ }
+
+ @Nullable
+ public BloomFilterConfig getBloomFilterConfig() {
+ return _bloomFilterConfig;
+ }
+ }
+
+ class Forward extends Wrapper {
+
+ private final ChunkCompressionType _chunkCompressionType;
+ private final Map<String, Map<String, String>> _columnProperties;
+
+ Forward(IndexCreationContext delegate,
+ ChunkCompressionType chunkCompressionType,
+ @Nullable Map<String, Map<String, String>> columnProperties) {
+ super(delegate);
+ _chunkCompressionType = chunkCompressionType;
+ _columnProperties = columnProperties;
+ }
+
+ public ChunkCompressionType getChunkCompressionType() {
+ return _chunkCompressionType;
+ }
+
+ @Nullable
+ public Map<String, Map<String, String>> getColumnProperties() {
+ return _columnProperties;
+ }
+ }
+
+ class Geospatial extends Wrapper {
+
+ private final H3IndexConfig _h3IndexConfig;
+
+ Geospatial(IndexCreationContext delegate, H3IndexConfig h3IndexConfig) {
+ super(delegate);
+ _h3IndexConfig = h3IndexConfig;
+ }
+
+ public H3IndexConfig getH3IndexConfig() {
+ return _h3IndexConfig;
+ }
+ }
+
+ class Inverted extends Wrapper {
+
+ Inverted(IndexCreationContext delegate) {
+ super(delegate);
+ }
+ }
+
+ class Json extends Wrapper {
+
+ Json(IndexCreationContext delegate) {
+ super(delegate);
+ }
+ }
+
+ class Range extends Wrapper {
+
+ private final Comparable<?> _min;
+ private final Comparable<?> _max;
+ private final int _rangeIndexVersion;
+
+
+ Range(IndexCreationContext delegate, int rangeIndexVersion, Comparable<?> min, Comparable<?> max) {
+ super(delegate);
+ _rangeIndexVersion = rangeIndexVersion;
+ _min = min;
+ _max = max;
+ }
+
+ public Comparable<?> getMin() {
+ return _min;
+ }
+
+ public Comparable<?> getMax() {
+ return _max;
+ }
+
+ public int getRangeIndexVersion() {
+ return _rangeIndexVersion;
+ }
+ }
+
+ class Text extends Wrapper {
+ private final boolean _commitOnClose;
Review comment:
well, it was hardcoded to be always true, with a comment to name it. Now it has a variable to give it a name and declare its purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7885: make index creator provision interceptible
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r766013317
##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/DefaultIndexCreatorProvider.java
##########
@@ -0,0 +1,249 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.Map;
+import java.util.Objects;
+import org.apache.pinot.segment.local.io.writer.impl.BaseChunkSVForwardIndexWriter;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.MultiValueFixedByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.MultiValueUnsortedForwardIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.MultiValueVarByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueFixedByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueSortedForwardIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueUnsortedForwardIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueVarByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.OffHeapBitmapInvertedIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.OnHeapBitmapInvertedIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.geospatial.OffHeapH3IndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.geospatial.OnHeapH3IndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.json.OffHeapJsonIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.json.OnHeapJsonIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.text.LuceneFSTIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.text.LuceneTextIndexCreator;
+import org.apache.pinot.segment.local.utils.nativefst.NativeFSTIndexCreator;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.DictionaryBasedInvertedIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.GeoSpatialIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.JsonIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.TextIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.H3IndexResolution;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.config.table.FieldConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+public final class DefaultIndexCreatorProvider implements IndexCreatorProvider {
+
+ @Override
+ public ForwardIndexCreator newForwardIndexCreator(IndexCreationContext context)
+ throws Exception {
+ if (context.isRawColumn()) {
+ boolean deriveNumDocsPerChunk =
+ shouldDeriveNumDocsPerChunk(context.getFieldSpec().getName(), context.getColumnProperties());
+ int writerVersion = rawIndexWriterVersion(context.getFieldSpec().getName(), context.getColumnProperties());
+ if (context.getFieldSpec().isSingleValueField()) {
+ return getRawIndexCreatorForSVColumn(context.getIndexDir(), context.getChunkCompressionType(),
+ context.getFieldSpec().getName(),
+ context.getFieldSpec().getDataType().getStoredType(), context.getTotalDocs(),
+ context.getLengthOfLongestEntry(), deriveNumDocsPerChunk, writerVersion);
+ } else {
+ return getRawIndexCreatorForMVColumn(context.getIndexDir(), context.getChunkCompressionType(),
+ context.getFieldSpec().getName(), context.getFieldSpec().getDataType().getStoredType(),
+ context.getTotalDocs(),
+ context.getMaxNumberOfMultiValueElements(), deriveNumDocsPerChunk, writerVersion,
+ context.getMaxRowLengthInBytes());
+ }
+ } else {
+ if (context.getFieldSpec().isSingleValueField()) {
+ if (context.isSorted()) {
+ return new SingleValueSortedForwardIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(),
+ context.getDistinctValueCount());
+ } else {
+ return new SingleValueUnsortedForwardIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(),
+ context.getDistinctValueCount(),
+ context.getTotalDocs());
+ }
+ } else {
+ return new MultiValueUnsortedForwardIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(),
+ context.getDistinctValueCount(),
+ context.getTotalDocs(),
+ context.getTotalNumberOfEntries());
+ }
+ }
+ }
+
+ @Override
+ public DictionaryBasedInvertedIndexCreator newInvertedIndexCreator(IndexCreationContext context)
+ throws IOException {
+ if (context.isOnHeap()) {
+ return new OnHeapBitmapInvertedIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(),
+ context.getDistinctValueCount());
+ } else {
+ return new OffHeapBitmapInvertedIndexCreator(context.getIndexDir(), context.getFieldSpec(),
+ context.getDistinctValueCount(), context.getTotalDocs(),
+ context.getTotalNumberOfEntries());
+ }
+ }
+
+ @Override
+ public JsonIndexCreator newJsonIndexCreator(IndexCreationContext context)
+ throws IOException {
+ Preconditions.checkState(context.getFieldSpec().isSingleValueField(),
+ "Json index is currently only supported on single-value columns");
+ Preconditions.checkState(context.getFieldSpec().getDataType().getStoredType() == FieldSpec.DataType.STRING,
+ "Json index is currently only supported on STRING columns");
+ return context.isOnHeap() ? new OnHeapJsonIndexCreator(context.getIndexDir(),
+ context.getFieldSpec().getName())
+ : new OffHeapJsonIndexCreator(context.getIndexDir(), context.getFieldSpec().getName());
+ }
+
+ @Override
+ public TextIndexCreator newTextIndexCreator(IndexCreationContext context)
+ throws IOException {
+ // Initialize text index creator
+ Preconditions.checkState(context.getFieldSpec().getDataType().getStoredType() == FieldSpec.DataType.STRING,
+ "Text index is currently only supported on STRING type columns");
+ return new LuceneTextIndexCreator(context.getFieldSpec().getName(), context.getIndexDir(),
+ true /* commitOnClose */);
+ }
+
+ @Override
+ public TextIndexCreator newFSTIndexCreator(IndexCreationContext context)
+ throws IOException {
+ Preconditions.checkState(context.getFieldSpec().isSingleValueField(),
+ "FST index is currently only supported on single-value columns");
+ Preconditions.checkState(context.getFieldSpec().getDataType().getStoredType() == FieldSpec.DataType.STRING,
+ "FST index is currently only supported on STRING type columns");
+ Preconditions.checkState(!context.isRawColumn(),
+ "FST index is currently only supported on dictionary-encoded columns");
+ String[] sortedValues = (String[]) context.getSortedUniqueElementsArray();
+ if (context.getFstType() == FSTType.NATIVE) {
+ return new NativeFSTIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(), sortedValues);
+ } else {
+ return new LuceneFSTIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(), sortedValues);
+ }
Review comment:
@atris FYI as FST indexing expert this will be called wherever a `TextIndexCreator` would have been constructed verbatim.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7885: make index creator provision interceptible
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r766013018
##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/DefaultIndexCreatorProvider.java
##########
@@ -0,0 +1,249 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.Map;
+import java.util.Objects;
+import org.apache.pinot.segment.local.io.writer.impl.BaseChunkSVForwardIndexWriter;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.MultiValueFixedByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.MultiValueUnsortedForwardIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.MultiValueVarByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueFixedByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueSortedForwardIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueUnsortedForwardIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueVarByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.OffHeapBitmapInvertedIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.OnHeapBitmapInvertedIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.geospatial.OffHeapH3IndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.geospatial.OnHeapH3IndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.json.OffHeapJsonIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.json.OnHeapJsonIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.text.LuceneFSTIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.text.LuceneTextIndexCreator;
+import org.apache.pinot.segment.local.utils.nativefst.NativeFSTIndexCreator;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.DictionaryBasedInvertedIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.GeoSpatialIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.JsonIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.TextIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.H3IndexResolution;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.config.table.FieldConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+public final class DefaultIndexCreatorProvider implements IndexCreatorProvider {
+
+ @Override
+ public ForwardIndexCreator newForwardIndexCreator(IndexCreationContext context)
+ throws Exception {
+ if (context.isRawColumn()) {
+ boolean deriveNumDocsPerChunk =
+ shouldDeriveNumDocsPerChunk(context.getFieldSpec().getName(), context.getColumnProperties());
+ int writerVersion = rawIndexWriterVersion(context.getFieldSpec().getName(), context.getColumnProperties());
+ if (context.getFieldSpec().isSingleValueField()) {
+ return getRawIndexCreatorForSVColumn(context.getIndexDir(), context.getChunkCompressionType(),
+ context.getFieldSpec().getName(),
+ context.getFieldSpec().getDataType().getStoredType(), context.getTotalDocs(),
+ context.getLengthOfLongestEntry(), deriveNumDocsPerChunk, writerVersion);
+ } else {
+ return getRawIndexCreatorForMVColumn(context.getIndexDir(), context.getChunkCompressionType(),
+ context.getFieldSpec().getName(), context.getFieldSpec().getDataType().getStoredType(),
+ context.getTotalDocs(),
+ context.getMaxNumberOfMultiValueElements(), deriveNumDocsPerChunk, writerVersion,
+ context.getMaxRowLengthInBytes());
+ }
+ } else {
+ if (context.getFieldSpec().isSingleValueField()) {
+ if (context.isSorted()) {
+ return new SingleValueSortedForwardIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(),
+ context.getDistinctValueCount());
+ } else {
+ return new SingleValueUnsortedForwardIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(),
+ context.getDistinctValueCount(),
+ context.getTotalDocs());
+ }
+ } else {
+ return new MultiValueUnsortedForwardIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(),
+ context.getDistinctValueCount(),
+ context.getTotalDocs(),
+ context.getTotalNumberOfEntries());
+ }
+ }
+ }
+
+ @Override
+ public DictionaryBasedInvertedIndexCreator newInvertedIndexCreator(IndexCreationContext context)
+ throws IOException {
+ if (context.isOnHeap()) {
+ return new OnHeapBitmapInvertedIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(),
+ context.getDistinctValueCount());
+ } else {
+ return new OffHeapBitmapInvertedIndexCreator(context.getIndexDir(), context.getFieldSpec(),
+ context.getDistinctValueCount(), context.getTotalDocs(),
+ context.getTotalNumberOfEntries());
+ }
+ }
+
+ @Override
+ public JsonIndexCreator newJsonIndexCreator(IndexCreationContext context)
+ throws IOException {
+ Preconditions.checkState(context.getFieldSpec().isSingleValueField(),
+ "Json index is currently only supported on single-value columns");
+ Preconditions.checkState(context.getFieldSpec().getDataType().getStoredType() == FieldSpec.DataType.STRING,
+ "Json index is currently only supported on STRING columns");
+ return context.isOnHeap() ? new OnHeapJsonIndexCreator(context.getIndexDir(),
+ context.getFieldSpec().getName())
+ : new OffHeapJsonIndexCreator(context.getIndexDir(), context.getFieldSpec().getName());
+ }
+
+ @Override
+ public TextIndexCreator newTextIndexCreator(IndexCreationContext context)
+ throws IOException {
+ // Initialize text index creator
+ Preconditions.checkState(context.getFieldSpec().getDataType().getStoredType() == FieldSpec.DataType.STRING,
+ "Text index is currently only supported on STRING type columns");
+ return new LuceneTextIndexCreator(context.getFieldSpec().getName(), context.getIndexDir(),
+ true /* commitOnClose */);
Review comment:
@siddharthteotia FYI as text indexing expert this will be called wherever a `TextIndexCreator` would have been constructed verbatim.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r769145870
##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/TextIndexCreatorProvider.java
##########
@@ -0,0 +1,37 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.creator;
+
+import java.io.IOException;
+import org.apache.pinot.segment.spi.index.creator.TextIndexCreator;
+
+
+public interface TextIndexCreatorProvider {
+
+ /**
+ * Creates a {@see TextIndexCreator} from information about index creation.
+ * This allows a plugin to pattern match index creation information to select
+ * an appropriate implementation.
+ * @param context context about the index creation.
+ * @return a {@see ForwardIndexCreator}
+ * @throws IOException whenever something goes wrong matching or constructing the creator
+ */
+ TextIndexCreator newFSTIndexCreator(IndexCreationContext context)
Review comment:
(major) the `newTextIndexCreator()` method is missing
##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/DefaultIndexCreatorProvider.java
##########
@@ -0,0 +1,249 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.Map;
+import java.util.Objects;
+import org.apache.pinot.segment.local.io.writer.impl.BaseChunkSVForwardIndexWriter;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.MultiValueFixedByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.MultiValueUnsortedForwardIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.MultiValueVarByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueFixedByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueSortedForwardIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueUnsortedForwardIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueVarByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.OffHeapBitmapInvertedIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.OnHeapBitmapInvertedIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.geospatial.OffHeapH3IndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.geospatial.OnHeapH3IndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.json.OffHeapJsonIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.json.OnHeapJsonIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.text.LuceneFSTIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.text.LuceneTextIndexCreator;
+import org.apache.pinot.segment.local.utils.nativefst.NativeFSTIndexCreator;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.DictionaryBasedInvertedIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.GeoSpatialIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.JsonIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.TextIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.H3IndexResolution;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.config.table.FieldConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+public final class DefaultIndexCreatorProvider implements IndexCreatorProvider {
+
+ @Override
+ public ForwardIndexCreator newForwardIndexCreator(IndexCreationContext context)
+ throws Exception {
+ if (context.isRawColumn()) {
+ boolean deriveNumDocsPerChunk =
+ shouldDeriveNumDocsPerChunk(context.getFieldSpec().getName(), context.getColumnProperties());
+ int writerVersion = rawIndexWriterVersion(context.getFieldSpec().getName(), context.getColumnProperties());
+ if (context.getFieldSpec().isSingleValueField()) {
+ return getRawIndexCreatorForSVColumn(context.getIndexDir(), context.getChunkCompressionType(),
+ context.getFieldSpec().getName(),
+ context.getFieldSpec().getDataType().getStoredType(), context.getTotalDocs(),
+ context.getLengthOfLongestEntry(), deriveNumDocsPerChunk, writerVersion);
+ } else {
+ return getRawIndexCreatorForMVColumn(context.getIndexDir(), context.getChunkCompressionType(),
+ context.getFieldSpec().getName(), context.getFieldSpec().getDataType().getStoredType(),
+ context.getTotalDocs(),
+ context.getMaxNumberOfMultiValueElements(), deriveNumDocsPerChunk, writerVersion,
+ context.getMaxRowLengthInBytes());
+ }
+ } else {
+ if (context.getFieldSpec().isSingleValueField()) {
+ if (context.isSorted()) {
+ return new SingleValueSortedForwardIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(),
+ context.getDistinctValueCount());
+ } else {
+ return new SingleValueUnsortedForwardIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(),
+ context.getDistinctValueCount(),
+ context.getTotalDocs());
+ }
+ } else {
+ return new MultiValueUnsortedForwardIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(),
+ context.getDistinctValueCount(),
+ context.getTotalDocs(),
+ context.getTotalNumberOfEntries());
+ }
+ }
+ }
+
+ @Override
+ public DictionaryBasedInvertedIndexCreator newInvertedIndexCreator(IndexCreationContext context)
+ throws IOException {
+ if (context.isOnHeap()) {
+ return new OnHeapBitmapInvertedIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(),
+ context.getDistinctValueCount());
+ } else {
+ return new OffHeapBitmapInvertedIndexCreator(context.getIndexDir(), context.getFieldSpec(),
+ context.getDistinctValueCount(), context.getTotalDocs(),
+ context.getTotalNumberOfEntries());
+ }
+ }
+
+ @Override
+ public JsonIndexCreator newJsonIndexCreator(IndexCreationContext context)
+ throws IOException {
+ Preconditions.checkState(context.getFieldSpec().isSingleValueField(),
+ "Json index is currently only supported on single-value columns");
+ Preconditions.checkState(context.getFieldSpec().getDataType().getStoredType() == FieldSpec.DataType.STRING,
+ "Json index is currently only supported on STRING columns");
+ return context.isOnHeap() ? new OnHeapJsonIndexCreator(context.getIndexDir(),
+ context.getFieldSpec().getName())
+ : new OffHeapJsonIndexCreator(context.getIndexDir(), context.getFieldSpec().getName());
+ }
+
+ @Override
+ public TextIndexCreator newTextIndexCreator(IndexCreationContext context)
+ throws IOException {
+ // Initialize text index creator
+ Preconditions.checkState(context.getFieldSpec().getDataType().getStoredType() == FieldSpec.DataType.STRING,
+ "Text index is currently only supported on STRING type columns");
+ return new LuceneTextIndexCreator(context.getFieldSpec().getName(), context.getIndexDir(),
+ true /* commitOnClose */);
+ }
+
+ @Override
+ public TextIndexCreator newFSTIndexCreator(IndexCreationContext context)
+ throws IOException {
+ Preconditions.checkState(context.getFieldSpec().isSingleValueField(),
+ "FST index is currently only supported on single-value columns");
+ Preconditions.checkState(context.getFieldSpec().getDataType().getStoredType() == FieldSpec.DataType.STRING,
+ "FST index is currently only supported on STRING type columns");
+ Preconditions.checkState(!context.isRawColumn(),
+ "FST index is currently only supported on dictionary-encoded columns");
+ String[] sortedValues = (String[]) context.getSortedUniqueElementsArray();
+ if (context.getFstType() == FSTType.NATIVE) {
+ return new NativeFSTIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(), sortedValues);
+ } else {
+ return new LuceneFSTIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(), sortedValues);
+ }
+ }
+
+ @Override
+ public GeoSpatialIndexCreator newGeoSpatialIndexCreator(IndexCreationContext context)
+ throws IOException {
+ Preconditions.checkState(context.getFieldSpec().isSingleValueField(),
+ "H3 index is currently only supported on single-value columns");
+ Preconditions.checkState(context.getFieldSpec().getDataType().getStoredType() == FieldSpec.DataType.BYTES,
+ "H3 index is currently only supported on BYTES columns");
+ H3IndexResolution resolution = Objects.requireNonNull(context.getH3IndexConfig()).getResolution();
+ return context.isOnHeap() ? new OnHeapH3IndexCreator(context.getIndexDir(),
+ context.getFieldSpec().getName(), resolution)
+ : new OffHeapH3IndexCreator(context.getIndexDir(), context.getFieldSpec().getName(), resolution);
+ }
+
+ public static boolean shouldDeriveNumDocsPerChunk(String columnName,
+ Map<String, Map<String, String>> columnProperties) {
+ if (columnProperties != null) {
+ Map<String, String> properties = columnProperties.get(columnName);
+ return properties != null && Boolean.parseBoolean(
+ properties.get(FieldConfig.DERIVE_NUM_DOCS_PER_CHUNK_RAW_INDEX_KEY));
+ }
+ return false;
+ }
+
+ public static int rawIndexWriterVersion(String columnName, Map<String, Map<String, String>> columnProperties) {
Review comment:
Seems also missed in a branch ^^
##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/IndexCreationContext.java
##########
@@ -0,0 +1,261 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.creator;
+
+import java.io.File;
+import java.util.Map;
+import java.util.Objects;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.index.creator.H3IndexConfig;
+import org.apache.pinot.spi.config.table.BloomFilterConfig;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+/**
+ * Provides parameters for constructing indexes via {@see IndexCreatorProvider}.
+ * The responsibility for ensuring that the correct parameters for a particular
+ * index type lies with the caller.
+ */
+public class IndexCreationContext {
+
+ public static final class Builder {
+ private File _indexDir;
+ private Map<String, Map<String, String>> _columnProperties;
+ private FSTType _fstType;
+ private H3IndexConfig _h3IndexConfig;
+ private int _lengthOfLongestEntry;
+ private int _maxNumberOfMultiValueElements;
+ private int _maxRowLengthInBytes;
+ private Object _sortedUniqueElementsArray;
+ private boolean _onHeap = false;
+ private ChunkCompressionType _chunkCompressionType;
+ public BloomFilterConfig _bloomFilterConfig;
+ private int _rangeIndexVersion;
+ private ColumnMetadata _columnMetadata;
Review comment:
I feel we should not include `ColumnMetadata` within the `IndexCreationContext`. Conceptually column metadata should be available only after the indexes are created. Suggest putting the required properties such as field spec, min/max values as individual fields in the `IndexCreationContext`
##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java
##########
@@ -179,126 +167,71 @@ public void init(SegmentGeneratorConfig segmentCreationSpec, SegmentIndexCreatio
}
String columnName = fieldSpec.getName();
- DataType storedType = fieldSpec.getDataType().getStoredType();
- ColumnIndexCreationInfo indexCreationInfo = indexCreationInfoMap.get(columnName);
- Preconditions.checkNotNull(indexCreationInfo, "Missing index creation info for column: %s", columnName);
- boolean dictEnabledColumn = createDictionaryForColumn(indexCreationInfo, segmentCreationSpec, fieldSpec);
-
+ ColumnIndexCreationInfo columnIndexCreationInfo = indexCreationInfoMap.get(columnName);
+ Preconditions.checkNotNull(columnIndexCreationInfo, "Missing index creation info for column: %s", columnName);
+ boolean dictEnabledColumn = createDictionaryForColumn(columnIndexCreationInfo, segmentCreationSpec, fieldSpec);
+ Preconditions.checkState(dictEnabledColumn || !invertedIndexColumns.contains(columnName),
+ "Cannot create inverted index for raw index column: %s", columnName);
+ ColumnMetadata columnMetadata = ColumnMetadataImpl.builder()
+ .setCardinality(columnIndexCreationInfo.getDistinctValueCount())
+ .setHasDictionary(dictEnabledColumn)
+ .setFieldSpec(fieldSpec)
+ .setTotalDocs(segmentIndexCreationInfo.getTotalDocs())
+ .setTotalNumberOfEntries(columnIndexCreationInfo.getTotalNumberOfEntries())
+ .setSorted(columnIndexCreationInfo.isSorted())
+ .build();
+ IndexCreationContext context = IndexCreationContext.builder()
+ .withIndexDir(_indexDir)
+ .withColumnMetadata(columnMetadata)
+ .withSegmentGeneratorConfig(segmentCreationSpec)
+ .withColumnIndexCreationInfo(columnIndexCreationInfo)
+ .withFSTType(_config.getFSTIndexType())
+ .withH3IndexConfig(h3IndexConfigs.get(columnName))
+ .withCompressionType(dictEnabledColumn ? null : getColumnCompressionType(segmentCreationSpec, fieldSpec))
+ .build();
+ // Initialize forward index creator
+ _forwardIndexCreatorMap.put(columnName, _indexCreatorProvider.newForwardIndexCreator(context));
+
+ // Initialize inverted index creator; skip creating inverted index if sorted
+ if (invertedIndexColumns.contains(columnName) && !columnIndexCreationInfo.isSorted()) {
+ _invertedIndexCreatorMap.put(columnName, _indexCreatorProvider.newInvertedIndexCreator(context));
+ }
if (dictEnabledColumn) {
// Create dictionary-encoded index
-
// Initialize dictionary creator
SegmentDictionaryCreator dictionaryCreator =
- new SegmentDictionaryCreator(indexCreationInfo.getSortedUniqueElementsArray(), fieldSpec, _indexDir,
- indexCreationInfo.isUseVarLengthDictionary());
+ new SegmentDictionaryCreator(columnIndexCreationInfo.getSortedUniqueElementsArray(), fieldSpec, _indexDir,
+ columnIndexCreationInfo.isUseVarLengthDictionary());
_dictionaryCreatorMap.put(columnName, dictionaryCreator);
-
// Create dictionary
try {
dictionaryCreator.build();
} catch (Exception e) {
LOGGER.error("Error building dictionary for field: {}, cardinality: {}, number of bytes per entry: {}",
- fieldSpec.getName(), indexCreationInfo.getDistinctValueCount(), dictionaryCreator.getNumBytesPerEntry());
+ fieldSpec.getName(), columnIndexCreationInfo.getDistinctValueCount(),
+ dictionaryCreator.getNumBytesPerEntry());
throw e;
}
-
- // Initialize forward index creator
- int cardinality = indexCreationInfo.getDistinctValueCount();
- if (fieldSpec.isSingleValueField()) {
- if (indexCreationInfo.isSorted()) {
- _forwardIndexCreatorMap.put(columnName,
- new SingleValueSortedForwardIndexCreator(_indexDir, columnName, cardinality));
- } else {
- _forwardIndexCreatorMap.put(columnName,
- new SingleValueUnsortedForwardIndexCreator(_indexDir, columnName, cardinality, _totalDocs));
- }
- } else {
- _forwardIndexCreatorMap.put(columnName,
- new MultiValueUnsortedForwardIndexCreator(_indexDir, columnName, cardinality, _totalDocs,
- indexCreationInfo.getTotalNumberOfEntries()));
- }
-
- // Initialize inverted index creator; skip creating inverted index if sorted
- if (invertedIndexColumns.contains(columnName) && !indexCreationInfo.isSorted()) {
- if (segmentCreationSpec.isOnHeap()) {
- _invertedIndexCreatorMap.put(columnName,
- new OnHeapBitmapInvertedIndexCreator(_indexDir, columnName, cardinality));
- } else {
- _invertedIndexCreatorMap.put(columnName,
- new OffHeapBitmapInvertedIndexCreator(_indexDir, fieldSpec, cardinality, _totalDocs,
- indexCreationInfo.getTotalNumberOfEntries()));
- }
- }
- } else {
- // Create raw index
- Preconditions.checkState(!invertedIndexColumns.contains(columnName),
- "Cannot create inverted index for raw index column: %s", columnName);
-
- ChunkCompressionType compressionType = getColumnCompressionType(segmentCreationSpec, fieldSpec);
-
- // Initialize forward index creator
- boolean deriveNumDocsPerChunk =
- shouldDeriveNumDocsPerChunk(columnName, segmentCreationSpec.getColumnProperties());
- int writerVersion = rawIndexWriterVersion(columnName, segmentCreationSpec.getColumnProperties());
- if (fieldSpec.isSingleValueField()) {
- _forwardIndexCreatorMap.put(columnName,
- getRawIndexCreatorForSVColumn(_indexDir, compressionType, columnName, storedType, _totalDocs,
- indexCreationInfo.getLengthOfLongestEntry(), deriveNumDocsPerChunk, writerVersion));
- } else {
- _forwardIndexCreatorMap.put(columnName,
- getRawIndexCreatorForMVColumn(_indexDir, compressionType, columnName, storedType, _totalDocs,
- indexCreationInfo.getMaxNumberOfMultiValueElements(), deriveNumDocsPerChunk, writerVersion,
- indexCreationInfo.getMaxRowLengthInBytes()));
- }
}
if (textIndexColumns.contains(columnName)) {
// Initialize text index creator
- Preconditions.checkState(storedType == DataType.STRING,
+ Preconditions.checkState(fieldSpec.getDataType().getStoredType() == FieldSpec.DataType.STRING,
"Text index is currently only supported on STRING type columns");
_textIndexCreatorMap.put(columnName,
new LuceneTextIndexCreator(columnName, _indexDir, true /* commitOnClose */));
Review comment:
Any specific reason why this one is not included in the index creator interface?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#issuecomment-990031441
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7885](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (4d1d254) into [master](https://codecov.io/gh/apache/pinot/commit/fb12a509056b4e36724126cef9a344a5e1e89bb1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (fb12a50) will **decrease** coverage by `2.97%`.
> The diff coverage is `80.46%`.
[](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7885 +/- ##
============================================
- Coverage 71.34% 68.37% -2.98%
+ Complexity 4087 4026 -61
============================================
Files 1587 1197 -390
Lines 82071 59501 -22570
Branches 12267 9175 -3092
============================================
- Hits 58553 40683 -17870
+ Misses 19550 15949 -3601
+ Partials 3968 2869 -1099
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `68.37% <80.46%> (+0.03%)` | :arrow_up: |
| unittests2 | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/pinot/core/minion/RawIndexConverter.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9taW5pb24vUmF3SW5kZXhDb252ZXJ0ZXIuamF2YQ==) | `0.00% <0.00%> (-56.61%)` | :arrow_down: |
| [...not/segment/spi/creator/IndexCreatorProviders.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2NyZWF0b3IvSW5kZXhDcmVhdG9yUHJvdmlkZXJzLmphdmE=) | `40.90% <40.90%> (ø)` | |
| [...ment/creator/impl/DefaultIndexCreatorProvider.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9EZWZhdWx0SW5kZXhDcmVhdG9yUHJvdmlkZXIuamF2YQ==) | `65.27% <65.27%> (ø)` | |
| [...t/index/loader/invertedindex/JsonIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0pzb25JbmRleEhhbmRsZXIuamF2YQ==) | `73.33% <66.66%> (+0.45%)` | :arrow_up: |
| [.../index/loader/invertedindex/RangeIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L1JhbmdlSW5kZXhIYW5kbGVyLmphdmE=) | `44.44% <85.71%> (+3.44%)` | :arrow_up: |
| [...ment/creator/impl/SegmentColumnarIndexCreator.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9TZWdtZW50Q29sdW1uYXJJbmRleENyZWF0b3IuamF2YQ==) | `87.22% <90.00%> (+2.83%)` | :arrow_up: |
| [...ocal/segment/index/loader/IndexHandlerFactory.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9JbmRleEhhbmRsZXJGYWN0b3J5LmphdmE=) | `100.00% <100.00%> (ø)` | |
| [...t/index/loader/bloomfilter/BloomFilterHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9ibG9vbWZpbHRlci9CbG9vbUZpbHRlckhhbmRsZXIuamF2YQ==) | `78.33% <100.00%> (+0.74%)` | :arrow_up: |
| [...nt/index/loader/invertedindex/FSTIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0ZTVEluZGV4SGFuZGxlci5qYXZh) | `86.79% <100.00%> (+4.10%)` | :arrow_up: |
| [...ent/index/loader/invertedindex/H3IndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0gzSW5kZXhIYW5kbGVyLmphdmE=) | `73.84% <100.00%> (+1.26%)` | :arrow_up: |
| ... and [628 more](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [fb12a50...4d1d254](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#issuecomment-990031441
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7885](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a7fc6ea) into [master](https://codecov.io/gh/apache/pinot/commit/fb12a509056b4e36724126cef9a344a5e1e89bb1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (fb12a50) will **decrease** coverage by `56.98%`.
> The diff coverage is `0.00%`.
> :exclamation: Current head a7fc6ea differs from pull request most recent head 3ac3b11. Consider uploading reports for the commit 3ac3b11 to get more accurate results
[](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7885 +/- ##
=============================================
- Coverage 71.34% 14.35% -56.99%
+ Complexity 4087 80 -4007
=============================================
Files 1587 1547 -40
Lines 82071 80374 -1697
Branches 12267 12066 -201
=============================================
- Hits 58553 11541 -47012
- Misses 19550 67980 +48430
+ Partials 3968 853 -3115
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `?` | |
| unittests2 | `14.35% <0.00%> (-0.09%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/pinot/core/minion/RawIndexConverter.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9taW5pb24vUmF3SW5kZXhDb252ZXJ0ZXIuamF2YQ==) | `0.00% <0.00%> (-56.61%)` | :arrow_down: |
| [...nverttorawindex/ConvertToRawIndexTaskExecutor.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvY29udmVydHRvcmF3aW5kZXgvQ29udmVydFRvUmF3SW5kZXhUYXNrRXhlY3V0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...ment/creator/impl/DefaultIndexCreatorProvider.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9EZWZhdWx0SW5kZXhDcmVhdG9yUHJvdmlkZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...ment/creator/impl/SegmentColumnarIndexCreator.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9TZWdtZW50Q29sdW1uYXJJbmRleENyZWF0b3IuamF2YQ==) | `0.00% <0.00%> (-84.40%)` | :arrow_down: |
| [...ocal/segment/index/loader/IndexHandlerFactory.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9JbmRleEhhbmRsZXJGYWN0b3J5LmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...t/index/loader/bloomfilter/BloomFilterHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9ibG9vbWZpbHRlci9CbG9vbUZpbHRlckhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (-77.59%)` | :arrow_down: |
| [...nt/index/loader/invertedindex/FSTIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0ZTVEluZGV4SGFuZGxlci5qYXZh) | `0.00% <0.00%> (-82.70%)` | :arrow_down: |
| [...ent/index/loader/invertedindex/H3IndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0gzSW5kZXhIYW5kbGVyLmphdmE=) | `0.00% <0.00%> (-72.59%)` | :arrow_down: |
| [...dex/loader/invertedindex/InvertedIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0ludmVydGVkSW5kZXhIYW5kbGVyLmphdmE=) | `0.00% <0.00%> (-93.88%)` | :arrow_down: |
| [...t/index/loader/invertedindex/JsonIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0pzb25JbmRleEhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (-72.89%)` | :arrow_down: |
| ... and [1274 more](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [fb12a50...3ac3b11](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#issuecomment-990031441
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7885](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3ac3b11) into [master](https://codecov.io/gh/apache/pinot/commit/fb12a509056b4e36724126cef9a344a5e1e89bb1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (fb12a50) will **decrease** coverage by `1.25%`.
> The diff coverage is `80.00%`.
[](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7885 +/- ##
============================================
- Coverage 71.34% 70.09% -1.26%
- Complexity 4087 4101 +14
============================================
Files 1587 1592 +5
Lines 82071 82241 +170
Branches 12267 12269 +2
============================================
- Hits 58553 57646 -907
- Misses 19550 20645 +1095
+ Partials 3968 3950 -18
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `27.64% <0.00%> (-0.01%)` | :arrow_down: |
| unittests1 | `68.36% <80.64%> (+0.02%)` | :arrow_up: |
| unittests2 | `14.35% <0.00%> (-0.10%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/pinot/core/minion/RawIndexConverter.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9taW5pb24vUmF3SW5kZXhDb252ZXJ0ZXIuamF2YQ==) | `0.00% <0.00%> (-56.61%)` | :arrow_down: |
| [...nverttorawindex/ConvertToRawIndexTaskExecutor.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvY29udmVydHRvcmF3aW5kZXgvQ29udmVydFRvUmF3SW5kZXhUYXNrRXhlY3V0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...not/segment/spi/creator/IndexCreatorProviders.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2NyZWF0b3IvSW5kZXhDcmVhdG9yUHJvdmlkZXJzLmphdmE=) | `42.85% <42.85%> (ø)` | |
| [...t/index/loader/invertedindex/JsonIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0pzb25JbmRleEhhbmRsZXIuamF2YQ==) | `73.33% <66.66%> (+0.45%)` | :arrow_up: |
| [...ment/creator/impl/DefaultIndexCreatorProvider.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9EZWZhdWx0SW5kZXhDcmVhdG9yUHJvdmlkZXIuamF2YQ==) | `67.14% <67.14%> (ø)` | |
| [.../index/loader/invertedindex/RangeIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L1JhbmdlSW5kZXhIYW5kbGVyLmphdmE=) | `44.44% <85.71%> (+3.44%)` | :arrow_up: |
| [...ment/creator/impl/SegmentColumnarIndexCreator.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9TZWdtZW50Q29sdW1uYXJJbmRleENyZWF0b3IuamF2YQ==) | `86.99% <87.50%> (+2.60%)` | :arrow_up: |
| [...ocal/segment/index/loader/IndexHandlerFactory.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9JbmRleEhhbmRsZXJGYWN0b3J5LmphdmE=) | `100.00% <100.00%> (ø)` | |
| [...t/index/loader/bloomfilter/BloomFilterHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9ibG9vbWZpbHRlci9CbG9vbUZpbHRlckhhbmRsZXIuamF2YQ==) | `78.33% <100.00%> (+0.74%)` | :arrow_up: |
| [...nt/index/loader/invertedindex/FSTIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0ZTVEluZGV4SGFuZGxlci5qYXZh) | `86.79% <100.00%> (+4.10%)` | :arrow_up: |
| ... and [129 more](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [fb12a50...3ac3b11](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r768723722
##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/IndexCreatorProviders.java
##########
@@ -0,0 +1,143 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.creator;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.lang.invoke.MethodType;
+import java.util.concurrent.atomic.AtomicReference;
+import org.apache.pinot.segment.spi.index.creator.BloomFilterCreator;
+import org.apache.pinot.segment.spi.index.creator.CombinedInvertedIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.DictionaryBasedInvertedIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.GeoSpatialIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.JsonIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.TextIndexCreator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Plugin registration point to allow index creation logic to be swapped out.
+ * Plugins should not reimplement Pinot's default index creation logic.
+ * Users provide an override to Pinot's index creation logic. This is simplified
+ * by extending {@see IndexCreatorProviders.Default}
+ */
+public final class IndexCreatorProviders {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(IndexCreatorProviders.class);
+
+ static final IndexCreatorProvider DEFAULT = defaultProvider();
+ private static final AtomicReference<IndexCreatorProvider> REGISTRATION = new AtomicReference<>(DEFAULT);
+
+ private IndexCreatorProviders() {
+ }
+
+ /**
+ * The caller provides a decorator to wrap the default provider, which allows plugins to create
+ * a delegation chain.
+ * @param provider index creation override
+ * @return true if this is the first invocation and the provider has not yet been used.
+ */
+ public static boolean registerProvider(IndexCreatorProvider provider) {
+ return REGISTRATION.compareAndSet(DEFAULT, provider);
+ }
+
+ /**
+ * Obtain the registered index creator provider. If the user has provided an override, then it will be used instead.
+ * If the user has not provided an override yet, then this action will prevent them from doing so.
+ * @return the global index provision logic.
+ */
+ public static IndexCreatorProvider getIndexCreatorProvider() {
+ return Holder.PROVIDER;
+ }
+
+ private static final class Holder {
+ public static final IndexCreatorProvider PROVIDER = REGISTRATION.get();
+ }
+
+ private static IndexCreatorProvider defaultProvider() {
+ // use MethodHandle to break circular dependency and keep implementation details encapsulated within
+ // pinot-segment-local
+ String className = "org.apache.pinot.segment.local.segment.creator.impl.DefaultIndexCreatorProvider";
+ try {
+ Class<?> clazz = Class.forName(className, false, IndexCreatorProviders.class.getClassLoader());
+ return (IndexCreatorProvider) MethodHandles.publicLookup()
+ .findConstructor(clazz, MethodType.methodType(void.class)).invoke();
+ } catch (Throwable implausible) {
+ LOGGER.error("could not construct MethodHandle for {}", className,
+ implausible);
+ throw new IllegalStateException(implausible);
+ }
+ }
+
+ /**
+ * Extend this class to override index creation
+ */
+ public static class Default implements IndexCreatorProvider {
Review comment:
Added a test which should make the mechanism (and motivation) fairly clear: https://github.com/apache/pinot/pull/7885/commits/e79a4a78ee464c67b68fcb205e34e207ae98c393
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r768314829
##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/IndexCreatorProviders.java
##########
@@ -0,0 +1,143 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.creator;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.lang.invoke.MethodType;
+import java.util.concurrent.atomic.AtomicReference;
+import org.apache.pinot.segment.spi.index.creator.BloomFilterCreator;
+import org.apache.pinot.segment.spi.index.creator.CombinedInvertedIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.DictionaryBasedInvertedIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.GeoSpatialIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.JsonIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.TextIndexCreator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Plugin registration point to allow index creation logic to be swapped out.
+ * Plugins should not reimplement Pinot's default index creation logic.
+ * Users provide an override to Pinot's index creation logic. This is simplified
+ * by extending {@see IndexCreatorProviders.Default}
+ */
+public final class IndexCreatorProviders {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(IndexCreatorProviders.class);
+
+ static final IndexCreatorProvider DEFAULT = defaultProvider();
+ private static final AtomicReference<IndexCreatorProvider> REGISTRATION = new AtomicReference<>(DEFAULT);
+
+ private IndexCreatorProviders() {
+ }
+
+ /**
+ * The caller provides a decorator to wrap the default provider, which allows plugins to create
+ * a delegation chain.
+ * @param provider index creation override
+ * @return true if this is the first invocation and the provider has not yet been used.
+ */
+ public static boolean registerProvider(IndexCreatorProvider provider) {
+ return REGISTRATION.compareAndSet(DEFAULT, provider);
+ }
+
+ /**
+ * Obtain the registered index creator provider. If the user has provided an override, then it will be used instead.
+ * If the user has not provided an override yet, then this action will prevent them from doing so.
+ * @return the global index provision logic.
+ */
+ public static IndexCreatorProvider getIndexCreatorProvider() {
+ return Holder.PROVIDER;
+ }
+
+ private static final class Holder {
+ public static final IndexCreatorProvider PROVIDER = REGISTRATION.get();
+ }
+
+ private static IndexCreatorProvider defaultProvider() {
+ // use MethodHandle to break circular dependency and keep implementation details encapsulated within
+ // pinot-segment-local
+ String className = "org.apache.pinot.segment.local.segment.creator.impl.DefaultIndexCreatorProvider";
+ try {
+ Class<?> clazz = Class.forName(className, false, IndexCreatorProviders.class.getClassLoader());
+ return (IndexCreatorProvider) MethodHandles.publicLookup()
+ .findConstructor(clazz, MethodType.methodType(void.class)).invoke();
+ } catch (Throwable implausible) {
+ LOGGER.error("could not construct MethodHandle for {}", className,
+ implausible);
+ throw new IllegalStateException(implausible);
+ }
+ }
+
+ /**
+ * Extend this class to override index creation
+ */
+ public static class Default implements IndexCreatorProvider {
Review comment:
IIUC, this is the one that allows to mix ? So for example, someone might want to use existing forward index, dictionary and everything else but might want to plugin a different inverted index ? So, this can be sub-classed and `newInvertedIndexCreator` can be overridden to get custom behavior. The sub-class can then be registered ? Is this understanding correct ?
Also, can we try to add one test for something to be handy for devs. May not be needed if example usage pattern is going to be covered in docs later on?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r768319018
##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/IndexCreationContext.java
##########
@@ -0,0 +1,256 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.creator;
+
+import java.io.File;
+import java.util.Map;
+import java.util.Objects;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.index.creator.H3IndexConfig;
+import org.apache.pinot.spi.config.table.BloomFilterConfig;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+public class IndexCreationContext {
Review comment:
(nit) brief javadoc please ?
Also to understand the end state better, this is not customizable right ? `SegmentColumnarIndexCreator` will pick up the pluggable creator and then build this context from indexCreationInfo / stats and the invoke the creator's APIs ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r768317277
##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/IndexCreatorProviders.java
##########
@@ -0,0 +1,143 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.creator;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.lang.invoke.MethodType;
+import java.util.concurrent.atomic.AtomicReference;
+import org.apache.pinot.segment.spi.index.creator.BloomFilterCreator;
+import org.apache.pinot.segment.spi.index.creator.CombinedInvertedIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.DictionaryBasedInvertedIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.GeoSpatialIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.JsonIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.TextIndexCreator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Plugin registration point to allow index creation logic to be swapped out.
+ * Plugins should not reimplement Pinot's default index creation logic.
+ * Users provide an override to Pinot's index creation logic. This is simplified
+ * by extending {@see IndexCreatorProviders.Default}
+ */
+public final class IndexCreatorProviders {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(IndexCreatorProviders.class);
+
+ static final IndexCreatorProvider DEFAULT = defaultProvider();
+ private static final AtomicReference<IndexCreatorProvider> REGISTRATION = new AtomicReference<>(DEFAULT);
+
+ private IndexCreatorProviders() {
+ }
+
+ /**
+ * The caller provides a decorator to wrap the default provider, which allows plugins to create
+ * a delegation chain.
+ * @param provider index creation override
+ * @return true if this is the first invocation and the provider has not yet been used.
+ */
+ public static boolean registerProvider(IndexCreatorProvider provider) {
+ return REGISTRATION.compareAndSet(DEFAULT, provider);
+ }
+
+ /**
+ * Obtain the registered index creator provider. If the user has provided an override, then it will be used instead.
+ * If the user has not provided an override yet, then this action will prevent them from doing so.
+ * @return the global index provision logic.
+ */
+ public static IndexCreatorProvider getIndexCreatorProvider() {
+ return Holder.PROVIDER;
+ }
+
+ private static final class Holder {
+ public static final IndexCreatorProvider PROVIDER = REGISTRATION.get();
+ }
+
+ private static IndexCreatorProvider defaultProvider() {
+ // use MethodHandle to break circular dependency and keep implementation details encapsulated within
+ // pinot-segment-local
+ String className = "org.apache.pinot.segment.local.segment.creator.impl.DefaultIndexCreatorProvider";
+ try {
+ Class<?> clazz = Class.forName(className, false, IndexCreatorProviders.class.getClassLoader());
+ return (IndexCreatorProvider) MethodHandles.publicLookup()
+ .findConstructor(clazz, MethodType.methodType(void.class)).invoke();
+ } catch (Throwable implausible) {
+ LOGGER.error("could not construct MethodHandle for {}", className,
+ implausible);
+ throw new IllegalStateException(implausible);
+ }
+ }
+
+ /**
+ * Extend this class to override index creation
+ */
+ public static class Default implements IndexCreatorProvider {
Review comment:
Also I wonder why `DefaultIndexCreatorProvider` is final class because in theory user can just sub-class it and register ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r768319018
##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/IndexCreationContext.java
##########
@@ -0,0 +1,256 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.creator;
+
+import java.io.File;
+import java.util.Map;
+import java.util.Objects;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.index.creator.H3IndexConfig;
+import org.apache.pinot.spi.config.table.BloomFilterConfig;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+public class IndexCreationContext {
Review comment:
(nit) brief javadoc please ?
Also to understand the end state better, this is not customizable right ? `SegmentColumnarIndexCreator` will pick up the pluggable creator and then build this context from indexCreationInfo / stats and the invoke the creator's APIs or is there more to it ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#issuecomment-990031441
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7885](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d4a8970) into [master](https://codecov.io/gh/apache/pinot/commit/fb12a509056b4e36724126cef9a344a5e1e89bb1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (fb12a50) will **decrease** coverage by `0.02%`.
> The diff coverage is `81.92%`.
[](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7885 +/- ##
============================================
- Coverage 71.34% 71.31% -0.03%
- Complexity 4087 4102 +15
============================================
Files 1587 1590 +3
Lines 82071 82212 +141
Branches 12267 12269 +2
============================================
+ Hits 58553 58633 +80
- Misses 19550 19612 +62
+ Partials 3968 3967 -1
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `29.14% <0.38%> (-0.13%)` | :arrow_down: |
| integration2 | `27.58% <0.00%> (-0.07%)` | :arrow_down: |
| unittests1 | `68.39% <81.53%> (+0.06%)` | :arrow_up: |
| unittests2 | `14.36% <0.00%> (-0.09%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...not/segment/spi/creator/IndexCreatorProviders.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2NyZWF0b3IvSW5kZXhDcmVhdG9yUHJvdmlkZXJzLmphdmE=) | `40.90% <40.90%> (ø)` | |
| [...ment/creator/impl/DefaultIndexCreatorProvider.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9EZWZhdWx0SW5kZXhDcmVhdG9yUHJvdmlkZXIuamF2YQ==) | `66.25% <66.25%> (ø)` | |
| [...t/index/loader/invertedindex/JsonIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0pzb25JbmRleEhhbmRsZXIuamF2YQ==) | `73.33% <66.66%> (+0.45%)` | :arrow_up: |
| [.../index/loader/invertedindex/RangeIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L1JhbmdlSW5kZXhIYW5kbGVyLmphdmE=) | `44.44% <85.71%> (+3.44%)` | :arrow_up: |
| [...ment/creator/impl/SegmentColumnarIndexCreator.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9TZWdtZW50Q29sdW1uYXJJbmRleENyZWF0b3IuamF2YQ==) | `87.22% <90.00%> (+2.83%)` | :arrow_up: |
| [...rg/apache/pinot/core/minion/RawIndexConverter.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9taW5pb24vUmF3SW5kZXhDb252ZXJ0ZXIuamF2YQ==) | `56.60% <100.00%> (ø)` | |
| [...ocal/segment/index/loader/IndexHandlerFactory.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9JbmRleEhhbmRsZXJGYWN0b3J5LmphdmE=) | `100.00% <100.00%> (ø)` | |
| [...t/index/loader/bloomfilter/BloomFilterHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9ibG9vbWZpbHRlci9CbG9vbUZpbHRlckhhbmRsZXIuamF2YQ==) | `78.33% <100.00%> (+0.74%)` | :arrow_up: |
| [...nt/index/loader/invertedindex/FSTIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0ZTVEluZGV4SGFuZGxlci5qYXZh) | `86.79% <100.00%> (+4.10%)` | :arrow_up: |
| [...ent/index/loader/invertedindex/H3IndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0gzSW5kZXhIYW5kbGVyLmphdmE=) | `73.84% <100.00%> (+1.26%)` | :arrow_up: |
| ... and [24 more](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [fb12a50...d4a8970](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r768459249
##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/IndexCreationContext.java
##########
@@ -0,0 +1,256 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.creator;
+
+import java.io.File;
+import java.util.Map;
+import java.util.Objects;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.index.creator.H3IndexConfig;
+import org.apache.pinot.spi.config.table.BloomFilterConfig;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+public class IndexCreationContext {
Review comment:
This cannot be customised, this is just the carrier of parameters from caller to creator. I may make some refinement here to use the type system to ensure e.g. a caller who wants a bloom filter always provides the correct parameters
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r770166077
##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/IndexCreationContext.java
##########
@@ -0,0 +1,481 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.creator;
+
+import java.io.File;
+import java.util.Map;
+import java.util.Objects;
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.index.creator.H3IndexConfig;
+import org.apache.pinot.spi.config.table.BloomFilterConfig;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+/**
+ * Provides parameters for constructing indexes via {@see IndexCreatorProvider}.
+ * The responsibility for ensuring that the correct parameters for a particular
+ * index type lies with the caller.
+ */
+public interface IndexCreationContext {
+
+ FieldSpec getFieldSpec();
+
+ File getIndexDir();
+
+ boolean isOnHeap();
+
+ int getLengthOfLongestEntry();
+
+ int getMaxNumberOfMultiValueElements();
+
+ int getMaxRowLengthInBytes();
+
+ boolean isSorted();
+
+ int getCardinality();
+
+ int getTotalNumberOfEntries();
+
+ int getTotalDocs();
+
+ boolean hasDictionary();
+
+ BloomFilter forBloomFilter(BloomFilterConfig bloomFilterConfig);
+
+ Forward forForwardIndex(ChunkCompressionType chunkCompressionType,
+ @Nullable Map<String, Map<String, String>> columnProperties);
+
+ Geospatial forGeospatialIndex(H3IndexConfig h3IndexConfig);
+
+ Inverted forInvertedIndex();
+
+ Json forJsonIndex();
+
+ Range forRangeIndex(ColumnMetadata columnMetadata, int rangeIndexVersion);
+
+ Text forTextIndex(FSTType fstType, String[] sortedUniqueElementsArray);
Review comment:
This is confusing. Will this support all current 3 flavors of text index ?
- Lucene text
- Lucene FST
- Native FST
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r770076937
##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/IndexCreationContext.java
##########
@@ -0,0 +1,261 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.creator;
+
+import java.io.File;
+import java.util.Map;
+import java.util.Objects;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.index.creator.H3IndexConfig;
+import org.apache.pinot.spi.config.table.BloomFilterConfig;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+/**
+ * Provides parameters for constructing indexes via {@see IndexCreatorProvider}.
+ * The responsibility for ensuring that the correct parameters for a particular
+ * index type lies with the caller.
+ */
+public class IndexCreationContext {
+
+ public static final class Builder {
+ private File _indexDir;
+ private Map<String, Map<String, String>> _columnProperties;
+ private FSTType _fstType;
+ private H3IndexConfig _h3IndexConfig;
+ private int _lengthOfLongestEntry;
+ private int _maxNumberOfMultiValueElements;
+ private int _maxRowLengthInBytes;
+ private Object _sortedUniqueElementsArray;
+ private boolean _onHeap = false;
+ private ChunkCompressionType _chunkCompressionType;
+ public BloomFilterConfig _bloomFilterConfig;
+ private int _rangeIndexVersion;
+ private ColumnMetadata _columnMetadata;
Review comment:
Please see refactor here: https://github.com/apache/pinot/pull/7885/commits/38b54fccda1893aeeeccc7d0d198f4f48e62465c
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #7885: make index creator provision interceptible
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r766072395
##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/IndexCreatorProvider.java
##########
@@ -0,0 +1,95 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.creator;
+
+import java.io.IOException;
+import org.apache.pinot.segment.spi.index.creator.DictionaryBasedInvertedIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.GeoSpatialIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.JsonIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.TextIndexCreator;
+
+
+/**
+ * Plugin interface to abstract index creation.
+ */
+public interface IndexCreatorProvider {
Review comment:
Let's also add `newRangeIndexCreator()` and `newBloomFilterCreator()` to the provider, even though they might not be used
##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/DefaultIndexCreatorProvider.java
##########
@@ -0,0 +1,249 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.Map;
+import java.util.Objects;
+import org.apache.pinot.segment.local.io.writer.impl.BaseChunkSVForwardIndexWriter;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.MultiValueFixedByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.MultiValueUnsortedForwardIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.MultiValueVarByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueFixedByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueSortedForwardIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueUnsortedForwardIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueVarByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.OffHeapBitmapInvertedIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.OnHeapBitmapInvertedIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.geospatial.OffHeapH3IndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.geospatial.OnHeapH3IndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.json.OffHeapJsonIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.json.OnHeapJsonIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.text.LuceneFSTIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.text.LuceneTextIndexCreator;
+import org.apache.pinot.segment.local.utils.nativefst.NativeFSTIndexCreator;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.DictionaryBasedInvertedIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.GeoSpatialIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.JsonIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.TextIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.H3IndexResolution;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.config.table.FieldConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+public final class DefaultIndexCreatorProvider implements IndexCreatorProvider {
+
+ @Override
+ public ForwardIndexCreator newForwardIndexCreator(IndexCreationContext context)
+ throws Exception {
+ if (context.isRawColumn()) {
+ boolean deriveNumDocsPerChunk =
+ shouldDeriveNumDocsPerChunk(context.getFieldSpec().getName(), context.getColumnProperties());
+ int writerVersion = rawIndexWriterVersion(context.getFieldSpec().getName(), context.getColumnProperties());
+ if (context.getFieldSpec().isSingleValueField()) {
+ return getRawIndexCreatorForSVColumn(context.getIndexDir(), context.getChunkCompressionType(),
+ context.getFieldSpec().getName(),
+ context.getFieldSpec().getDataType().getStoredType(), context.getTotalDocs(),
+ context.getLengthOfLongestEntry(), deriveNumDocsPerChunk, writerVersion);
+ } else {
+ return getRawIndexCreatorForMVColumn(context.getIndexDir(), context.getChunkCompressionType(),
+ context.getFieldSpec().getName(), context.getFieldSpec().getDataType().getStoredType(),
+ context.getTotalDocs(),
+ context.getMaxNumberOfMultiValueElements(), deriveNumDocsPerChunk, writerVersion,
+ context.getMaxRowLengthInBytes());
+ }
+ } else {
+ if (context.getFieldSpec().isSingleValueField()) {
+ if (context.isSorted()) {
+ return new SingleValueSortedForwardIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(),
+ context.getDistinctValueCount());
+ } else {
+ return new SingleValueUnsortedForwardIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(),
+ context.getDistinctValueCount(),
+ context.getTotalDocs());
+ }
+ } else {
+ return new MultiValueUnsortedForwardIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(),
+ context.getDistinctValueCount(),
+ context.getTotalDocs(),
+ context.getTotalNumberOfEntries());
+ }
+ }
+ }
+
+ @Override
+ public DictionaryBasedInvertedIndexCreator newInvertedIndexCreator(IndexCreationContext context)
+ throws IOException {
+ if (context.isOnHeap()) {
+ return new OnHeapBitmapInvertedIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(),
+ context.getDistinctValueCount());
+ } else {
+ return new OffHeapBitmapInvertedIndexCreator(context.getIndexDir(), context.getFieldSpec(),
+ context.getDistinctValueCount(), context.getTotalDocs(),
+ context.getTotalNumberOfEntries());
+ }
+ }
+
+ @Override
+ public JsonIndexCreator newJsonIndexCreator(IndexCreationContext context)
+ throws IOException {
+ Preconditions.checkState(context.getFieldSpec().isSingleValueField(),
+ "Json index is currently only supported on single-value columns");
+ Preconditions.checkState(context.getFieldSpec().getDataType().getStoredType() == FieldSpec.DataType.STRING,
+ "Json index is currently only supported on STRING columns");
+ return context.isOnHeap() ? new OnHeapJsonIndexCreator(context.getIndexDir(),
+ context.getFieldSpec().getName())
+ : new OffHeapJsonIndexCreator(context.getIndexDir(), context.getFieldSpec().getName());
+ }
+
+ @Override
+ public TextIndexCreator newTextIndexCreator(IndexCreationContext context)
+ throws IOException {
+ // Initialize text index creator
+ Preconditions.checkState(context.getFieldSpec().getDataType().getStoredType() == FieldSpec.DataType.STRING,
+ "Text index is currently only supported on STRING type columns");
+ return new LuceneTextIndexCreator(context.getFieldSpec().getName(), context.getIndexDir(),
+ true /* commitOnClose */);
+ }
+
+ @Override
+ public TextIndexCreator newFSTIndexCreator(IndexCreationContext context)
+ throws IOException {
+ Preconditions.checkState(context.getFieldSpec().isSingleValueField(),
+ "FST index is currently only supported on single-value columns");
+ Preconditions.checkState(context.getFieldSpec().getDataType().getStoredType() == FieldSpec.DataType.STRING,
+ "FST index is currently only supported on STRING type columns");
+ Preconditions.checkState(!context.isRawColumn(),
+ "FST index is currently only supported on dictionary-encoded columns");
+ String[] sortedValues = (String[]) context.getSortedUniqueElementsArray();
+ if (context.getFstType() == FSTType.NATIVE) {
+ return new NativeFSTIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(), sortedValues);
+ } else {
+ return new LuceneFSTIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(), sortedValues);
+ }
+ }
+
+ @Override
+ public GeoSpatialIndexCreator newGeoSpatialIndexCreator(IndexCreationContext context)
+ throws IOException {
+ Preconditions.checkState(context.getFieldSpec().isSingleValueField(),
+ "H3 index is currently only supported on single-value columns");
+ Preconditions.checkState(context.getFieldSpec().getDataType().getStoredType() == FieldSpec.DataType.BYTES,
+ "H3 index is currently only supported on BYTES columns");
+ H3IndexResolution resolution = Objects.requireNonNull(context.getH3IndexConfig()).getResolution();
+ return context.isOnHeap() ? new OnHeapH3IndexCreator(context.getIndexDir(),
+ context.getFieldSpec().getName(), resolution)
+ : new OffHeapH3IndexCreator(context.getIndexDir(), context.getFieldSpec().getName(), resolution);
+ }
+
+ public static boolean shouldDeriveNumDocsPerChunk(String columnName,
+ Map<String, Map<String, String>> columnProperties) {
+ if (columnProperties != null) {
+ Map<String, String> properties = columnProperties.get(columnName);
+ return properties != null && Boolean.parseBoolean(
+ properties.get(FieldConfig.DERIVE_NUM_DOCS_PER_CHUNK_RAW_INDEX_KEY));
+ }
+ return false;
+ }
+
+ public static int rawIndexWriterVersion(String columnName, Map<String, Map<String, String>> columnProperties) {
Review comment:
(minor) rename to `getRawIndexWriterVersion`
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/minion/RawIndexConverter.java
##########
@@ -206,7 +206,7 @@ private void convertColumn(FieldSpec fieldSpec)
DataType storedType = dictionary.getValueType();
int numDocs = _originalSegmentMetadata.getTotalDocs();
int lengthOfLongestEntry = _originalSegmentMetadata.getColumnMetadataFor(columnName).getColumnMaxLength();
- try (ForwardIndexCreator rawIndexCreator = SegmentColumnarIndexCreator
+ try (ForwardIndexCreator rawIndexCreator = DefaultIndexCreatorProvider
Review comment:
This should also use the registered index creator provider
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r770853668
##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/IndexCreationContext.java
##########
@@ -0,0 +1,467 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.creator;
+
+import java.io.File;
+import java.util.Map;
+import java.util.Objects;
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.index.creator.H3IndexConfig;
+import org.apache.pinot.spi.config.table.BloomFilterConfig;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+/**
+ * Provides parameters for constructing indexes via {@see IndexCreatorProvider}.
+ * The responsibility for ensuring that the correct parameters for a particular
+ * index type lies with the caller.
+ */
+public interface IndexCreationContext {
+
+ FieldSpec getFieldSpec();
+
+ File getIndexDir();
+
+ boolean isOnHeap();
+
+ int getLengthOfLongestEntry();
+
+ int getMaxNumberOfMultiValueElements();
+
+ int getMaxRowLengthInBytes();
+
+ boolean isSorted();
+
+ int getCardinality();
+
+ int getTotalNumberOfEntries();
+
+ int getTotalDocs();
+
+ boolean hasDictionary();
+
+ final class Builder {
+ private File _indexDir;
+ private int _lengthOfLongestEntry;
+ private int _maxNumberOfMultiValueElements;
+ private int _maxRowLengthInBytes;
+ private boolean _onHeap = false;
+ private FieldSpec _fieldSpec;
+ private boolean _sorted;
+ private int _cardinality;
+ private int _totalNumberOfEntries;
+ private int _totalDocs;
+ private boolean _hasDictionary = true;
+
+ public Builder withColumnIndexCreationInfo(ColumnIndexCreationInfo columnIndexCreationInfo) {
+ return withLengthOfLongestEntry(columnIndexCreationInfo.getLengthOfLongestEntry())
+ .withMaxNumberOfMultiValueElements(columnIndexCreationInfo.getMaxNumberOfMultiValueElements())
+ .withMaxRowLengthInBytes(columnIndexCreationInfo.getMaxRowLengthInBytes());
+ }
+
+ public Builder withIndexDir(File indexDir) {
+ _indexDir = indexDir;
+ return this;
+ }
+
+ public Builder onHeap(boolean onHeap) {
+ _onHeap = onHeap;
+ return this;
+ }
+
+ public Builder withColumnMetadata(ColumnMetadata columnMetadata) {
+ return withFieldSpec(columnMetadata.getFieldSpec())
+ .sorted(columnMetadata.isSorted())
+ .withCardinality(columnMetadata.getCardinality())
+ .withTotalNumberOfEntries(columnMetadata.getTotalNumberOfEntries())
+ .withTotalDocs(columnMetadata.getTotalDocs())
+ .withDictionary(columnMetadata.hasDictionary());
+ }
+
+ public Builder withLengthOfLongestEntry(int lengthOfLongestEntry) {
+ _lengthOfLongestEntry = lengthOfLongestEntry;
+ return this;
+ }
+
+ public Builder withMaxNumberOfMultiValueElements(int maxNumberOfMultiValueElements) {
+ _maxNumberOfMultiValueElements = maxNumberOfMultiValueElements;
+ return this;
+ }
+
+ public Builder withMaxRowLengthInBytes(int maxRowLengthInBytes) {
+ _maxRowLengthInBytes = maxRowLengthInBytes;
+ return this;
+ }
+
+ public Builder withFieldSpec(FieldSpec fieldSpec) {
+ _fieldSpec = fieldSpec;
+ return this;
+ }
+
+ public Builder sorted(boolean sorted) {
+ _sorted = sorted;
+ return this;
+ }
+
+ public Builder withCardinality(int cardinality) {
+ _cardinality = cardinality;
+ return this;
+ }
+
+ public Builder withTotalNumberOfEntries(int totalNumberOfEntries) {
+ _totalNumberOfEntries = totalNumberOfEntries;
+ return this;
+ }
+
+ public Builder withTotalDocs(int totalDocs) {
+ _totalDocs = totalDocs;
+ return this;
+ }
+
+ public Builder withDictionary(boolean hasDictionary) {
+ _hasDictionary = hasDictionary;
+ return this;
+ }
+
+ public Common build() {
+ return new Common(Objects.requireNonNull(_indexDir),
+ _lengthOfLongestEntry, _maxNumberOfMultiValueElements, _maxRowLengthInBytes,
+ _onHeap, Objects.requireNonNull(_fieldSpec),
+ _sorted, _cardinality, _totalNumberOfEntries, _totalDocs, _hasDictionary);
+ }
+ }
+
+ static Builder builder() {
+ return new Builder();
+ }
+
+ final class Common implements IndexCreationContext {
+
+ private final File _indexDir;
+ private final int _lengthOfLongestEntry;
+ private final int _maxNumberOfMultiValueElements;
+ private final int _maxRowLengthInBytes;
+ private final boolean _onHeap;
+ private final FieldSpec _fieldSpec;
+ private final boolean _sorted;
+ private final int _cardinality;
+ private final int _totalNumberOfEntries;
+ private final int _totalDocs;
+ private final boolean _hasDictionary;
+
+ public Common(File indexDir, int lengthOfLongestEntry,
+ int maxNumberOfMultiValueElements, int maxRowLengthInBytes, boolean onHeap,
+ FieldSpec fieldSpec, boolean sorted, int cardinality, int totalNumberOfEntries,
+ int totalDocs, boolean hasDictionary) {
+ _indexDir = indexDir;
+ _lengthOfLongestEntry = lengthOfLongestEntry;
+ _maxNumberOfMultiValueElements = maxNumberOfMultiValueElements;
+ _maxRowLengthInBytes = maxRowLengthInBytes;
+ _onHeap = onHeap;
+ _fieldSpec = fieldSpec;
+ _sorted = sorted;
+ _cardinality = cardinality;
+ _totalNumberOfEntries = totalNumberOfEntries;
+ _totalDocs = totalDocs;
+ _hasDictionary = hasDictionary;
+ }
+
+ public FieldSpec getFieldSpec() {
+ return _fieldSpec;
+ }
+
+ public File getIndexDir() {
+ return _indexDir;
+ }
+
+ public boolean isOnHeap() {
+ return _onHeap;
+ }
+
+ public int getLengthOfLongestEntry() {
+ return _lengthOfLongestEntry;
+ }
+
+ public int getMaxNumberOfMultiValueElements() {
+ return _maxNumberOfMultiValueElements;
+ }
+
+ public int getMaxRowLengthInBytes() {
+ return _maxRowLengthInBytes;
+ }
+
+ public boolean isSorted() {
+ return _sorted;
+ }
+
+ public int getCardinality() {
+ return _cardinality;
+ }
+
+ public int getTotalNumberOfEntries() {
+ return _totalNumberOfEntries;
+ }
+
+ public int getTotalDocs() {
+ return _totalDocs;
+ }
+
+ public boolean hasDictionary() {
+ return _hasDictionary;
+ }
+
+ public BloomFilter forBloomFilter(BloomFilterConfig bloomFilterConfig) {
+ return new BloomFilter(this, bloomFilterConfig);
+ }
+
+ public Forward forForwardIndex(ChunkCompressionType chunkCompressionType,
+ @Nullable Map<String, Map<String, String>> columnProperties) {
+ return new Forward(this, chunkCompressionType, columnProperties);
+ }
+
+ public Text forFSTIndex(FSTType fstType, String[] sortedUniqueElementsArray) {
+ return new Text(this, fstType, sortedUniqueElementsArray);
+ }
+
+ public Geospatial forGeospatialIndex(H3IndexConfig h3IndexConfig) {
+ return new Geospatial(this, h3IndexConfig);
+ }
+
+ public Inverted forInvertedIndex() {
+ return new Inverted(this);
+ }
+
+ public Json forJsonIndex() {
+ return new Json(this);
+ }
+
+ public Range forRangeIndex(int rangeIndexVersion, Comparable<?> min, Comparable<?> max) {
+ return new Range(this, rangeIndexVersion, min, max);
+ }
+
+ public Text forTextIndex(boolean commitOnClose) {
+ return new Text(this, commitOnClose);
+ }
+ }
+
+ class Wrapper implements IndexCreationContext {
+
+ private final IndexCreationContext _delegate;
+
+ Wrapper(IndexCreationContext delegate) {
+ _delegate = delegate;
+ }
+
+ @Override
+ public FieldSpec getFieldSpec() {
+ return _delegate.getFieldSpec();
+ }
+
+ @Override
+ public File getIndexDir() {
+ return _delegate.getIndexDir();
+ }
+
+ @Override
+ public boolean isOnHeap() {
+ return _delegate.isOnHeap();
+ }
+
+ @Override
+ public int getLengthOfLongestEntry() {
+ return _delegate.getLengthOfLongestEntry();
+ }
+
+ @Override
+ public int getMaxNumberOfMultiValueElements() {
+ return _delegate.getMaxNumberOfMultiValueElements();
+ }
+
+ @Override
+ public int getMaxRowLengthInBytes() {
+ return _delegate.getMaxRowLengthInBytes();
+ }
+
+ @Override
+ public boolean isSorted() {
+ return _delegate.isSorted();
+ }
+
+ @Override
+ public int getCardinality() {
+ return _delegate.getCardinality();
+ }
+
+ @Override
+ public int getTotalNumberOfEntries() {
+ return _delegate.getTotalNumberOfEntries();
+ }
+
+ @Override
+ public int getTotalDocs() {
+ return _delegate.getTotalDocs();
+ }
+
+ @Override
+ public boolean hasDictionary() {
+ return _delegate.hasDictionary();
+ }
+ }
+
+ class BloomFilter extends Wrapper {
+
+ private final BloomFilterConfig _bloomFilterConfig;
+
+ public BloomFilter(IndexCreationContext wrapped, BloomFilterConfig bloomFilterConfig) {
+ super(wrapped);
+ _bloomFilterConfig = bloomFilterConfig;
+ }
+
+ @Nullable
+ public BloomFilterConfig getBloomFilterConfig() {
+ return _bloomFilterConfig;
+ }
+ }
+
+ class Forward extends Wrapper {
+
+ private final ChunkCompressionType _chunkCompressionType;
+ private final Map<String, Map<String, String>> _columnProperties;
+
+ Forward(IndexCreationContext delegate,
+ ChunkCompressionType chunkCompressionType,
+ @Nullable Map<String, Map<String, String>> columnProperties) {
+ super(delegate);
+ _chunkCompressionType = chunkCompressionType;
+ _columnProperties = columnProperties;
+ }
+
+ public ChunkCompressionType getChunkCompressionType() {
+ return _chunkCompressionType;
+ }
+
+ @Nullable
+ public Map<String, Map<String, String>> getColumnProperties() {
+ return _columnProperties;
+ }
+ }
+
+ class Geospatial extends Wrapper {
+
+ private final H3IndexConfig _h3IndexConfig;
+
+ Geospatial(IndexCreationContext delegate, H3IndexConfig h3IndexConfig) {
+ super(delegate);
+ _h3IndexConfig = h3IndexConfig;
+ }
+
+ public H3IndexConfig getH3IndexConfig() {
+ return _h3IndexConfig;
+ }
+ }
+
+ class Inverted extends Wrapper {
+
+ Inverted(IndexCreationContext delegate) {
+ super(delegate);
+ }
+ }
+
+ class Json extends Wrapper {
+
+ Json(IndexCreationContext delegate) {
+ super(delegate);
+ }
+ }
+
+ class Range extends Wrapper {
+
+ private final Comparable<?> _min;
+ private final Comparable<?> _max;
+ private final int _rangeIndexVersion;
+
+
+ Range(IndexCreationContext delegate, int rangeIndexVersion, Comparable<?> min, Comparable<?> max) {
+ super(delegate);
+ _rangeIndexVersion = rangeIndexVersion;
+ _min = min;
+ _max = max;
+ }
+
+ public Comparable<?> getMin() {
+ return _min;
+ }
+
+ public Comparable<?> getMax() {
+ return _max;
+ }
+
+ public int getRangeIndexVersion() {
+ return _rangeIndexVersion;
+ }
+ }
+
+ class Text extends Wrapper {
+ private final boolean _commitOnClose;
Review comment:
This is not really required as it only applies to mutable index (for consuming segment)
##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/IndexCreationContext.java
##########
@@ -0,0 +1,467 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.creator;
+
+import java.io.File;
+import java.util.Map;
+import java.util.Objects;
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.index.creator.H3IndexConfig;
+import org.apache.pinot.spi.config.table.BloomFilterConfig;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+/**
+ * Provides parameters for constructing indexes via {@see IndexCreatorProvider}.
+ * The responsibility for ensuring that the correct parameters for a particular
+ * index type lies with the caller.
+ */
+public interface IndexCreationContext {
Review comment:
There are some index independent properties that can be moved to the common: `min`, `max`, `sortedUniqueValues`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang merged pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang merged pull request #7885:
URL: https://github.com/apache/pinot/pull/7885
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r768314829
##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/IndexCreatorProviders.java
##########
@@ -0,0 +1,143 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.creator;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.lang.invoke.MethodType;
+import java.util.concurrent.atomic.AtomicReference;
+import org.apache.pinot.segment.spi.index.creator.BloomFilterCreator;
+import org.apache.pinot.segment.spi.index.creator.CombinedInvertedIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.DictionaryBasedInvertedIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.GeoSpatialIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.JsonIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.TextIndexCreator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Plugin registration point to allow index creation logic to be swapped out.
+ * Plugins should not reimplement Pinot's default index creation logic.
+ * Users provide an override to Pinot's index creation logic. This is simplified
+ * by extending {@see IndexCreatorProviders.Default}
+ */
+public final class IndexCreatorProviders {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(IndexCreatorProviders.class);
+
+ static final IndexCreatorProvider DEFAULT = defaultProvider();
+ private static final AtomicReference<IndexCreatorProvider> REGISTRATION = new AtomicReference<>(DEFAULT);
+
+ private IndexCreatorProviders() {
+ }
+
+ /**
+ * The caller provides a decorator to wrap the default provider, which allows plugins to create
+ * a delegation chain.
+ * @param provider index creation override
+ * @return true if this is the first invocation and the provider has not yet been used.
+ */
+ public static boolean registerProvider(IndexCreatorProvider provider) {
+ return REGISTRATION.compareAndSet(DEFAULT, provider);
+ }
+
+ /**
+ * Obtain the registered index creator provider. If the user has provided an override, then it will be used instead.
+ * If the user has not provided an override yet, then this action will prevent them from doing so.
+ * @return the global index provision logic.
+ */
+ public static IndexCreatorProvider getIndexCreatorProvider() {
+ return Holder.PROVIDER;
+ }
+
+ private static final class Holder {
+ public static final IndexCreatorProvider PROVIDER = REGISTRATION.get();
+ }
+
+ private static IndexCreatorProvider defaultProvider() {
+ // use MethodHandle to break circular dependency and keep implementation details encapsulated within
+ // pinot-segment-local
+ String className = "org.apache.pinot.segment.local.segment.creator.impl.DefaultIndexCreatorProvider";
+ try {
+ Class<?> clazz = Class.forName(className, false, IndexCreatorProviders.class.getClassLoader());
+ return (IndexCreatorProvider) MethodHandles.publicLookup()
+ .findConstructor(clazz, MethodType.methodType(void.class)).invoke();
+ } catch (Throwable implausible) {
+ LOGGER.error("could not construct MethodHandle for {}", className,
+ implausible);
+ throw new IllegalStateException(implausible);
+ }
+ }
+
+ /**
+ * Extend this class to override index creation
+ */
+ public static class Default implements IndexCreatorProvider {
Review comment:
IIUC, this is the one that allows to mix ? So for example, someone might want to use existing forward index, dictionary and everything else but might want to plugin a different inverted index ? So, this can be sub-classed and newInvertedIndexCreator can be overridden to get custom behavior. The sub-class can then be registered ? Is this flow correct ?
Also, can we try to add one test (or may not be needed if example usage pattern is going to be covered in docs later on) ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r768311643
##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/DefaultIndexCreatorProvider.java
##########
@@ -0,0 +1,266 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.Map;
+import java.util.Objects;
+import org.apache.pinot.segment.local.io.writer.impl.BaseChunkSVForwardIndexWriter;
+import org.apache.pinot.segment.local.segment.creator.impl.bloom.OnHeapGuavaBloomFilterCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.MultiValueFixedByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.MultiValueUnsortedForwardIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.MultiValueVarByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueFixedByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueSortedForwardIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueUnsortedForwardIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueVarByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.BitSlicedRangeIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.OffHeapBitmapInvertedIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.OnHeapBitmapInvertedIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.RangeIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.geospatial.OffHeapH3IndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.geospatial.OnHeapH3IndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.json.OffHeapJsonIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.json.OnHeapJsonIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.text.LuceneFSTIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.text.LuceneTextIndexCreator;
+import org.apache.pinot.segment.local.utils.nativefst.NativeFSTIndexCreator;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.BloomFilterCreator;
+import org.apache.pinot.segment.spi.index.creator.CombinedInvertedIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.DictionaryBasedInvertedIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.GeoSpatialIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.JsonIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.TextIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.H3IndexResolution;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.config.table.FieldConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+public final class DefaultIndexCreatorProvider implements IndexCreatorProvider {
Review comment:
(nit) brief javadoc please ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r768456475
##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/IndexCreatorProviders.java
##########
@@ -0,0 +1,143 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.creator;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.lang.invoke.MethodType;
+import java.util.concurrent.atomic.AtomicReference;
+import org.apache.pinot.segment.spi.index.creator.BloomFilterCreator;
+import org.apache.pinot.segment.spi.index.creator.CombinedInvertedIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.DictionaryBasedInvertedIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.GeoSpatialIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.JsonIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.TextIndexCreator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Plugin registration point to allow index creation logic to be swapped out.
+ * Plugins should not reimplement Pinot's default index creation logic.
+ * Users provide an override to Pinot's index creation logic. This is simplified
+ * by extending {@see IndexCreatorProviders.Default}
+ */
+public final class IndexCreatorProviders {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(IndexCreatorProviders.class);
+
+ static final IndexCreatorProvider DEFAULT = defaultProvider();
+ private static final AtomicReference<IndexCreatorProvider> REGISTRATION = new AtomicReference<>(DEFAULT);
+
+ private IndexCreatorProviders() {
+ }
+
+ /**
+ * The caller provides a decorator to wrap the default provider, which allows plugins to create
+ * a delegation chain.
+ * @param provider index creation override
+ * @return true if this is the first invocation and the provider has not yet been used.
+ */
+ public static boolean registerProvider(IndexCreatorProvider provider) {
+ return REGISTRATION.compareAndSet(DEFAULT, provider);
+ }
+
+ /**
+ * Obtain the registered index creator provider. If the user has provided an override, then it will be used instead.
+ * If the user has not provided an override yet, then this action will prevent them from doing so.
+ * @return the global index provision logic.
+ */
+ public static IndexCreatorProvider getIndexCreatorProvider() {
+ return Holder.PROVIDER;
+ }
+
+ private static final class Holder {
+ public static final IndexCreatorProvider PROVIDER = REGISTRATION.get();
+ }
+
+ private static IndexCreatorProvider defaultProvider() {
+ // use MethodHandle to break circular dependency and keep implementation details encapsulated within
+ // pinot-segment-local
+ String className = "org.apache.pinot.segment.local.segment.creator.impl.DefaultIndexCreatorProvider";
+ try {
+ Class<?> clazz = Class.forName(className, false, IndexCreatorProviders.class.getClassLoader());
+ return (IndexCreatorProvider) MethodHandles.publicLookup()
+ .findConstructor(clazz, MethodType.methodType(void.class)).invoke();
+ } catch (Throwable implausible) {
+ LOGGER.error("could not construct MethodHandle for {}", className,
+ implausible);
+ throw new IllegalStateException(implausible);
+ }
+ }
+
+ /**
+ * Extend this class to override index creation
+ */
+ public static class Default implements IndexCreatorProvider {
Review comment:
`DefaultIndexCreatorProvider` is in a core module which an external plugin should not have a dependency on, it has access to all kinds of internal APIs which should not be exposed via SPI. The user of the SPI doesn't need a dependency on core modules, just the SPI, and they just need to extend `Default` without knowing what the defaults are.
For example usage, let's say the user implements a new kind of inverted index only for `INT` columns, for the sake of example, but wants to use the defaults otherwise. They would subclass Default:
```java
public class MyIndexCreatorProvider extends IndexCreatorProviders.Default {
@Override
public DictionaryBasedInvertedIndexCreator newInvertedIndexProvider(IndexCreationContext context) {
if (context.getColumnMetadata().getDataType() == INT) {
return new MyCustomInvertedIndexCreator(context.getColumnMetadata());
}
return super.newInvertedIndexProvider(context)
}
}
...
IndexCreatorProviders.registerProvider(new MyPluginProvider());
```
The user doesn't care about forward indexes so doesn't implement the method. The idea behind an inheritance based API is that new index types can be added and old extensions will be both source- and binary-compatible.
##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/IndexCreatorProviders.java
##########
@@ -0,0 +1,143 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.creator;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.lang.invoke.MethodType;
+import java.util.concurrent.atomic.AtomicReference;
+import org.apache.pinot.segment.spi.index.creator.BloomFilterCreator;
+import org.apache.pinot.segment.spi.index.creator.CombinedInvertedIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.DictionaryBasedInvertedIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.GeoSpatialIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.JsonIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.TextIndexCreator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Plugin registration point to allow index creation logic to be swapped out.
+ * Plugins should not reimplement Pinot's default index creation logic.
+ * Users provide an override to Pinot's index creation logic. This is simplified
+ * by extending {@see IndexCreatorProviders.Default}
+ */
+public final class IndexCreatorProviders {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(IndexCreatorProviders.class);
+
+ static final IndexCreatorProvider DEFAULT = defaultProvider();
+ private static final AtomicReference<IndexCreatorProvider> REGISTRATION = new AtomicReference<>(DEFAULT);
+
+ private IndexCreatorProviders() {
+ }
+
+ /**
+ * The caller provides a decorator to wrap the default provider, which allows plugins to create
+ * a delegation chain.
+ * @param provider index creation override
+ * @return true if this is the first invocation and the provider has not yet been used.
+ */
+ public static boolean registerProvider(IndexCreatorProvider provider) {
+ return REGISTRATION.compareAndSet(DEFAULT, provider);
+ }
+
+ /**
+ * Obtain the registered index creator provider. If the user has provided an override, then it will be used instead.
+ * If the user has not provided an override yet, then this action will prevent them from doing so.
+ * @return the global index provision logic.
+ */
+ public static IndexCreatorProvider getIndexCreatorProvider() {
+ return Holder.PROVIDER;
+ }
+
+ private static final class Holder {
+ public static final IndexCreatorProvider PROVIDER = REGISTRATION.get();
+ }
+
+ private static IndexCreatorProvider defaultProvider() {
+ // use MethodHandle to break circular dependency and keep implementation details encapsulated within
+ // pinot-segment-local
+ String className = "org.apache.pinot.segment.local.segment.creator.impl.DefaultIndexCreatorProvider";
+ try {
+ Class<?> clazz = Class.forName(className, false, IndexCreatorProviders.class.getClassLoader());
+ return (IndexCreatorProvider) MethodHandles.publicLookup()
+ .findConstructor(clazz, MethodType.methodType(void.class)).invoke();
+ } catch (Throwable implausible) {
+ LOGGER.error("could not construct MethodHandle for {}", className,
+ implausible);
+ throw new IllegalStateException(implausible);
+ }
+ }
+
+ /**
+ * Extend this class to override index creation
+ */
+ public static class Default implements IndexCreatorProvider {
Review comment:
`DefaultIndexCreatorProvider` is in a core module which an external plugin should not have a dependency on, it has access to all kinds of internal APIs which should not be exposed via SPI. The user of the SPI doesn't need a dependency on core modules, just the SPI, and they just need to extend `Default` without knowing what the defaults are.
For example usage, let's say the user implements a new kind of inverted index only for `INT` columns, for the sake of example, but wants to use the defaults otherwise. They would subclass Default:
```java
public class MyIndexCreatorProvider extends IndexCreatorProviders.Default {
@Override
public DictionaryBasedInvertedIndexCreator newInvertedIndexProvider(IndexCreationContext context) {
if (context.getColumnMetadata().getDataType() == INT) {
return new MyCustomInvertedIndexCreator(context.getColumnMetadata());
}
return super.newInvertedIndexProvider(context)
}
}
...
IndexCreatorProviders.registerProvider(new MyIndexCreatorProvider());
```
The user doesn't care about forward indexes so doesn't implement the method. The idea behind an inheritance based API is that new index types can be added and old extensions will be both source- and binary-compatible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r768308452
##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/IndexCreationContext.java
##########
@@ -0,0 +1,256 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.creator;
+
+import java.io.File;
+import java.util.Map;
+import java.util.Objects;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.index.creator.H3IndexConfig;
+import org.apache.pinot.spi.config.table.BloomFilterConfig;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+public class IndexCreationContext {
+
+ public static final class Builder {
+ private File _indexDir;
+ private Map<String, Map<String, String>> _columnProperties;
+ private FSTType _fstType;
+ private H3IndexConfig _h3IndexConfig;
+ private int _lengthOfLongestEntry;
+ private int _maxNumberOfMultiValueElements;
+ private int _maxRowLengthInBytes;
+ private Object _sortedUniqueElementsArray;
+ private boolean _onHeap = false;
+ private ChunkCompressionType _chunkCompressionType;
+ public BloomFilterConfig _bloomFilterConfig;
+ private int _rangeIndexVersion;
+ private ColumnMetadata _columnMetadata;
+
+ public Builder withColumnIndexCreationInfo(ColumnIndexCreationInfo columnIndexCreationInfo) {
+ return withLengthOfLongestEntry(columnIndexCreationInfo.getLengthOfLongestEntry())
+ .withMaxNumberOfMultiValueElements(columnIndexCreationInfo.getMaxNumberOfMultiValueElements())
+ .withMaxRowLengthInBytes(columnIndexCreationInfo.getMaxRowLengthInBytes())
+ .withSortedUniqueElementsArray(columnIndexCreationInfo.getSortedUniqueElementsArray());
+ }
+
+ public Builder withIndexDir(File indexDir) {
+ _indexDir = indexDir;
+ return this;
+ }
+
+ public Builder withSegmentGeneratorConfig(@Nonnull SegmentGeneratorConfig segmentGeneratorConfig) {
+ return onHeap(segmentGeneratorConfig.isOnHeap())
+ .withColumnProperties(segmentGeneratorConfig.getColumnProperties());
+ }
+
+ public Builder onHeap(boolean onHeap) {
+ _onHeap = onHeap;
+ return this;
+ }
+
+ public Builder withColumnProperties(Map<String, Map<String, String>> columnProperties) {
+ _columnProperties = columnProperties;
+ return this;
+ }
+
+ public Builder withColumnMetadata(ColumnMetadata columnMetadata) {
+ _columnMetadata = columnMetadata;
+ return this;
+ }
+
+ public Builder withFSTType(FSTType fstType) {
+ _fstType = fstType;
+ return this;
+ }
+
+ public Builder withH3IndexConfig(H3IndexConfig h3IndexConfig) {
+ _h3IndexConfig = h3IndexConfig;
+ return this;
+ }
+
+ public Builder withLengthOfLongestEntry(int lengthOfLongestEntry) {
+ _lengthOfLongestEntry = lengthOfLongestEntry;
+ return this;
+ }
+
+ public Builder withMaxNumberOfMultiValueElements(int maxNumberOfMultiValueElements) {
+ _maxNumberOfMultiValueElements = maxNumberOfMultiValueElements;
+ return this;
+ }
+
+ public Builder withMaxRowLengthInBytes(int maxRowLengthInBytes) {
+ _maxRowLengthInBytes = maxRowLengthInBytes;
+ return this;
+ }
+
+ public Builder withSortedUniqueElementsArray(Object sortedUniqueElementsArray) {
+ _sortedUniqueElementsArray = sortedUniqueElementsArray;
+ return this;
+ }
+
+ public Builder withCompressionType(ChunkCompressionType chunkCompressionType) {
+ _chunkCompressionType = chunkCompressionType;
+ return this;
+ }
+
+ public Builder withBloomFilterConfig(BloomFilterConfig bloomFilterConfig) {
+ _bloomFilterConfig = bloomFilterConfig;
+ return this;
+ }
+
+ public Builder withRangeIndexVersion(int rangeIndexVersion) {
+ _rangeIndexVersion = rangeIndexVersion;
+ return this;
+ }
+
+ public IndexCreationContext build() {
+ return new IndexCreationContext(Objects.requireNonNull(_indexDir),
+ _columnProperties, _h3IndexConfig, _fstType, _lengthOfLongestEntry,
+ _maxNumberOfMultiValueElements, _maxRowLengthInBytes,
+ _sortedUniqueElementsArray, _onHeap, _chunkCompressionType, _bloomFilterConfig,
+ _rangeIndexVersion,
+ Objects.requireNonNull(_columnMetadata));
+ }
+ }
+
+ public static IndexCreationContext.Builder builder() {
+ return new Builder();
+ }
+
+ private final File _indexDir;
+ private final Map<String, Map<String, String>> _columnProperties;
+ private final FSTType _fstType;
+ private final H3IndexConfig _h3IndexConfig;
+ private final int _lengthOfLongestEntry;
+ private final int _maxNumberOfMultiValueElements;
+ private final int _maxRowLengthInBytes;
+ private final Object _sortedUniqueElementsArray;
+ private final boolean _onHeap;
+ private final ChunkCompressionType _chunkCompressionType;
+ public final BloomFilterConfig _bloomFilterConfig;
+ private final int _rangeIndexVersion;
+ private final ColumnMetadata _columnMetadata;
+
+ public IndexCreationContext(File indexDir, @Nullable Map<String, Map<String, String>> columnProperties,
+ @Nullable H3IndexConfig h3IndexConfig, @Nullable FSTType fstType, int lengthOfLongestEntry,
+ int maxNumberOfMultiValueElements, int maxRowLengthInBytes, Object sortedUniqueElementsArray, boolean onHeap,
+ @Nullable ChunkCompressionType chunkCompressionType, @Nullable BloomFilterConfig bloomFilterConfig,
+ int rangeIndexVersion, ColumnMetadata columnMetadata) {
+ _indexDir = indexDir;
+ _columnProperties = columnProperties;
+ _fstType = fstType;
+ _h3IndexConfig = h3IndexConfig;
+ _lengthOfLongestEntry = lengthOfLongestEntry;
+ _maxNumberOfMultiValueElements = maxNumberOfMultiValueElements;
+ _maxRowLengthInBytes = maxRowLengthInBytes;
+ _sortedUniqueElementsArray = sortedUniqueElementsArray;
+ _onHeap = onHeap;
+ _chunkCompressionType = chunkCompressionType;
+ _bloomFilterConfig = bloomFilterConfig;
+ _rangeIndexVersion = rangeIndexVersion;
+ _columnMetadata = columnMetadata;
+ }
+
+ public FieldSpec getFieldSpec() {
+ return _columnMetadata.getFieldSpec();
+ }
+
+ public File getIndexDir() {
+ return _indexDir;
+ }
+
+ @Nullable
+ public Map<String, Map<String, String>> getColumnProperties() {
+ return _columnProperties;
+ }
+
+ public boolean isOnHeap() {
+ return _onHeap;
+ }
+
+ public int getLengthOfLongestEntry() {
+ return _lengthOfLongestEntry;
+ }
+
+ public int getMaxNumberOfMultiValueElements() {
+ return _maxNumberOfMultiValueElements;
+ }
+
+ public int getMaxRowLengthInBytes() {
+ return _maxRowLengthInBytes;
+ }
+
+ public boolean isSorted() {
+ return _columnMetadata.isSorted();
+ }
+
+ public int getCardinality() {
+ return _columnMetadata.getCardinality();
+ }
+
+ public int getTotalNumberOfEntries() {
+ return _columnMetadata.getTotalNumberOfEntries();
+ }
+
+ public Object getSortedUniqueElementsArray() {
+ return _sortedUniqueElementsArray;
+ }
+
+ public int getTotalDocs() {
+ return _columnMetadata.getTotalDocs();
+ }
+
+ public boolean isRawColumn() {
Review comment:
(nit) should we call it isDictionaryBased or hasDictionary() ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
klsince commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r767990031
##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/DefaultIndexCreatorProvider.java
##########
@@ -0,0 +1,274 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.Map;
+import java.util.Objects;
+import org.apache.pinot.segment.local.io.writer.impl.BaseChunkSVForwardIndexWriter;
+import org.apache.pinot.segment.local.segment.creator.impl.bloom.OnHeapGuavaBloomFilterCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.MultiValueFixedByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.MultiValueUnsortedForwardIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.MultiValueVarByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueFixedByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueSortedForwardIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueUnsortedForwardIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueVarByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.BitSlicedRangeIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.OffHeapBitmapInvertedIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.OnHeapBitmapInvertedIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.RangeIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.geospatial.OffHeapH3IndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.geospatial.OnHeapH3IndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.json.OffHeapJsonIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.json.OnHeapJsonIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.text.LuceneFSTIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.text.LuceneTextIndexCreator;
+import org.apache.pinot.segment.local.utils.nativefst.NativeFSTIndexCreator;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.BloomFilterCreator;
+import org.apache.pinot.segment.spi.index.creator.CombinedInvertedIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.DictionaryBasedInvertedIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.GeoSpatialIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.JsonIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.TextIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.H3IndexResolution;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.config.table.FieldConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+public final class DefaultIndexCreatorProvider implements IndexCreatorProvider {
+
+ @Override
+ public ForwardIndexCreator newForwardIndexCreator(IndexCreationContext context)
+ throws Exception {
+ if (context.isRawColumn()) {
+ boolean deriveNumDocsPerChunk =
+ shouldDeriveNumDocsPerChunk(context.getFieldSpec().getName(), context.getColumnProperties());
+ int writerVersion = rawIndexWriterVersion(context.getFieldSpec().getName(), context.getColumnProperties());
+ if (context.getFieldSpec().isSingleValueField()) {
+ return getRawIndexCreatorForSVColumn(context.getIndexDir(), context.getChunkCompressionType(),
+ context.getFieldSpec().getName(),
Review comment:
nit: format? and similar format issue in a few other places.
btw, love the whole refactoring. the logic flow looks a lot concise to me now! 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r768443968
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/minion/RawIndexConverter.java
##########
@@ -206,7 +206,7 @@ private void convertColumn(FieldSpec fieldSpec)
DataType storedType = dictionary.getValueType();
int numDocs = _originalSegmentMetadata.getTotalDocs();
int lengthOfLongestEntry = _originalSegmentMetadata.getColumnMetadataFor(columnName).getColumnMaxLength();
- try (ForwardIndexCreator rawIndexCreator = SegmentColumnarIndexCreator
+ try (ForwardIndexCreator rawIndexCreator = DefaultIndexCreatorProvider
Review comment:
Sorry it was solved but I appear to have backed it out in a rebase at some point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7885: make index creator provision interceptible
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#issuecomment-990031441
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7885](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (4cc35e3) into [master](https://codecov.io/gh/apache/pinot/commit/86278194233781d8001498f0c2be4a8a3299cd9d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8627819) will **increase** coverage by `5.11%`.
> The diff coverage is `81.70%`.
[](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7885 +/- ##
============================================
+ Coverage 65.10% 70.21% +5.11%
- Complexity 4078 4089 +11
============================================
Files 1538 1586 +48
Lines 79998 81949 +1951
Branches 12036 12241 +205
============================================
+ Hits 52085 57544 +5459
+ Misses 24187 20453 -3734
- Partials 3726 3952 +226
```
| Flag | Coverage Δ | |
|---|---|---|
| integration2 | `27.67% <0.00%> (?)` | |
| unittests1 | `68.47% <81.70%> (+0.01%)` | :arrow_up: |
| unittests2 | `14.40% <0.00%> (-0.07%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/pinot/core/minion/RawIndexConverter.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9taW5pb24vUmF3SW5kZXhDb252ZXJ0ZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...not/segment/spi/creator/IndexCreatorProviders.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2NyZWF0b3IvSW5kZXhDcmVhdG9yUHJvdmlkZXJzLmphdmE=) | `69.23% <69.23%> (ø)` | |
| [...ment/creator/impl/DefaultIndexCreatorProvider.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9EZWZhdWx0SW5kZXhDcmVhdG9yUHJvdmlkZXIuamF2YQ==) | `73.17% <73.17%> (ø)` | |
| [...ment/creator/impl/SegmentColumnarIndexCreator.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9TZWdtZW50Q29sdW1uYXJJbmRleENyZWF0b3IuamF2YQ==) | `86.76% <88.88%> (+2.37%)` | :arrow_up: |
| [...inot/segment/spi/creator/IndexCreationContext.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2NyZWF0b3IvSW5kZXhDcmVhdGlvbkNvbnRleHQuamF2YQ==) | `100.00% <100.00%> (ø)` | |
| [...gregation/function/StUnionAggregationFunction.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9hZ2dyZWdhdGlvbi9mdW5jdGlvbi9TdFVuaW9uQWdncmVnYXRpb25GdW5jdGlvbi5qYXZh) | `73.52% <0.00%> (-2.95%)` | :arrow_down: |
| [.../pinot/core/query/scheduler/PriorityScheduler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9zY2hlZHVsZXIvUHJpb3JpdHlTY2hlZHVsZXIuamF2YQ==) | `80.82% <0.00%> (-2.74%)` | :arrow_down: |
| [.../aggregation/function/ModeAggregationFunction.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9hZ2dyZWdhdGlvbi9mdW5jdGlvbi9Nb2RlQWdncmVnYXRpb25GdW5jdGlvbi5qYXZh) | `88.10% <0.00%> (-0.55%)` | :arrow_down: |
| [...va/org/apache/pinot/controller/ControllerConf.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9Db250cm9sbGVyQ29uZi5qYXZh) | `56.50% <0.00%> (ø)` | |
| [...inot/server/api/resources/ServerResourceUtils.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvYXBpL3Jlc291cmNlcy9TZXJ2ZXJSZXNvdXJjZVV0aWxzLmphdmE=) | `45.45% <0.00%> (ø)` | |
| ... and [329 more](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [8627819...4cc35e3](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#issuecomment-990031441
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7885](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (152a55d) into [master](https://codecov.io/gh/apache/pinot/commit/fb12a509056b4e36724126cef9a344a5e1e89bb1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (fb12a50) will **decrease** coverage by `56.98%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7885 +/- ##
=============================================
- Coverage 71.34% 14.35% -56.99%
+ Complexity 4087 80 -4007
=============================================
Files 1587 1547 -40
Lines 82071 80421 -1650
Branches 12267 12069 -198
=============================================
- Hits 58553 11544 -47009
- Misses 19550 68023 +48473
+ Partials 3968 854 -3114
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `?` | |
| unittests2 | `14.35% <0.00%> (-0.10%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/pinot/core/minion/RawIndexConverter.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9taW5pb24vUmF3SW5kZXhDb252ZXJ0ZXIuamF2YQ==) | `0.00% <0.00%> (-56.61%)` | :arrow_down: |
| [...nverttorawindex/ConvertToRawIndexTaskExecutor.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvY29udmVydHRvcmF3aW5kZXgvQ29udmVydFRvUmF3SW5kZXhUYXNrRXhlY3V0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...ment/creator/impl/DefaultIndexCreatorProvider.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9EZWZhdWx0SW5kZXhDcmVhdG9yUHJvdmlkZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...ment/creator/impl/SegmentColumnarIndexCreator.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9TZWdtZW50Q29sdW1uYXJJbmRleENyZWF0b3IuamF2YQ==) | `0.00% <0.00%> (-84.40%)` | :arrow_down: |
| [...ocal/segment/index/loader/IndexHandlerFactory.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9JbmRleEhhbmRsZXJGYWN0b3J5LmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...t/index/loader/bloomfilter/BloomFilterHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9ibG9vbWZpbHRlci9CbG9vbUZpbHRlckhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (-77.59%)` | :arrow_down: |
| [...nt/index/loader/invertedindex/FSTIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0ZTVEluZGV4SGFuZGxlci5qYXZh) | `0.00% <0.00%> (-82.70%)` | :arrow_down: |
| [...ent/index/loader/invertedindex/H3IndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0gzSW5kZXhIYW5kbGVyLmphdmE=) | `0.00% <0.00%> (-72.59%)` | :arrow_down: |
| [...dex/loader/invertedindex/InvertedIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0ludmVydGVkSW5kZXhIYW5kbGVyLmphdmE=) | `0.00% <0.00%> (-93.88%)` | :arrow_down: |
| [...t/index/loader/invertedindex/JsonIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0pzb25JbmRleEhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (-72.89%)` | :arrow_down: |
| ... and [1273 more](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [fb12a50...152a55d](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r770163310
##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java
##########
@@ -179,126 +167,71 @@ public void init(SegmentGeneratorConfig segmentCreationSpec, SegmentIndexCreatio
}
String columnName = fieldSpec.getName();
- DataType storedType = fieldSpec.getDataType().getStoredType();
- ColumnIndexCreationInfo indexCreationInfo = indexCreationInfoMap.get(columnName);
- Preconditions.checkNotNull(indexCreationInfo, "Missing index creation info for column: %s", columnName);
- boolean dictEnabledColumn = createDictionaryForColumn(indexCreationInfo, segmentCreationSpec, fieldSpec);
-
+ ColumnIndexCreationInfo columnIndexCreationInfo = indexCreationInfoMap.get(columnName);
+ Preconditions.checkNotNull(columnIndexCreationInfo, "Missing index creation info for column: %s", columnName);
+ boolean dictEnabledColumn = createDictionaryForColumn(columnIndexCreationInfo, segmentCreationSpec, fieldSpec);
+ Preconditions.checkState(dictEnabledColumn || !invertedIndexColumns.contains(columnName),
+ "Cannot create inverted index for raw index column: %s", columnName);
+ ColumnMetadata columnMetadata = ColumnMetadataImpl.builder()
+ .setCardinality(columnIndexCreationInfo.getDistinctValueCount())
+ .setHasDictionary(dictEnabledColumn)
+ .setFieldSpec(fieldSpec)
+ .setTotalDocs(segmentIndexCreationInfo.getTotalDocs())
+ .setTotalNumberOfEntries(columnIndexCreationInfo.getTotalNumberOfEntries())
+ .setSorted(columnIndexCreationInfo.isSorted())
+ .build();
+ IndexCreationContext context = IndexCreationContext.builder()
+ .withIndexDir(_indexDir)
+ .withColumnMetadata(columnMetadata)
+ .withSegmentGeneratorConfig(segmentCreationSpec)
+ .withColumnIndexCreationInfo(columnIndexCreationInfo)
+ .withFSTType(_config.getFSTIndexType())
+ .withH3IndexConfig(h3IndexConfigs.get(columnName))
+ .withCompressionType(dictEnabledColumn ? null : getColumnCompressionType(segmentCreationSpec, fieldSpec))
+ .build();
+ // Initialize forward index creator
+ _forwardIndexCreatorMap.put(columnName, _indexCreatorProvider.newForwardIndexCreator(context));
+
+ // Initialize inverted index creator; skip creating inverted index if sorted
+ if (invertedIndexColumns.contains(columnName) && !columnIndexCreationInfo.isSorted()) {
+ _invertedIndexCreatorMap.put(columnName, _indexCreatorProvider.newInvertedIndexCreator(context));
+ }
if (dictEnabledColumn) {
// Create dictionary-encoded index
-
// Initialize dictionary creator
SegmentDictionaryCreator dictionaryCreator =
- new SegmentDictionaryCreator(indexCreationInfo.getSortedUniqueElementsArray(), fieldSpec, _indexDir,
- indexCreationInfo.isUseVarLengthDictionary());
+ new SegmentDictionaryCreator(columnIndexCreationInfo.getSortedUniqueElementsArray(), fieldSpec, _indexDir,
+ columnIndexCreationInfo.isUseVarLengthDictionary());
_dictionaryCreatorMap.put(columnName, dictionaryCreator);
-
// Create dictionary
try {
dictionaryCreator.build();
} catch (Exception e) {
LOGGER.error("Error building dictionary for field: {}, cardinality: {}, number of bytes per entry: {}",
- fieldSpec.getName(), indexCreationInfo.getDistinctValueCount(), dictionaryCreator.getNumBytesPerEntry());
+ fieldSpec.getName(), columnIndexCreationInfo.getDistinctValueCount(),
+ dictionaryCreator.getNumBytesPerEntry());
throw e;
}
-
- // Initialize forward index creator
- int cardinality = indexCreationInfo.getDistinctValueCount();
- if (fieldSpec.isSingleValueField()) {
- if (indexCreationInfo.isSorted()) {
- _forwardIndexCreatorMap.put(columnName,
- new SingleValueSortedForwardIndexCreator(_indexDir, columnName, cardinality));
- } else {
- _forwardIndexCreatorMap.put(columnName,
- new SingleValueUnsortedForwardIndexCreator(_indexDir, columnName, cardinality, _totalDocs));
- }
- } else {
- _forwardIndexCreatorMap.put(columnName,
- new MultiValueUnsortedForwardIndexCreator(_indexDir, columnName, cardinality, _totalDocs,
- indexCreationInfo.getTotalNumberOfEntries()));
- }
-
- // Initialize inverted index creator; skip creating inverted index if sorted
- if (invertedIndexColumns.contains(columnName) && !indexCreationInfo.isSorted()) {
- if (segmentCreationSpec.isOnHeap()) {
- _invertedIndexCreatorMap.put(columnName,
- new OnHeapBitmapInvertedIndexCreator(_indexDir, columnName, cardinality));
- } else {
- _invertedIndexCreatorMap.put(columnName,
- new OffHeapBitmapInvertedIndexCreator(_indexDir, fieldSpec, cardinality, _totalDocs,
- indexCreationInfo.getTotalNumberOfEntries()));
- }
- }
- } else {
- // Create raw index
- Preconditions.checkState(!invertedIndexColumns.contains(columnName),
- "Cannot create inverted index for raw index column: %s", columnName);
-
- ChunkCompressionType compressionType = getColumnCompressionType(segmentCreationSpec, fieldSpec);
-
- // Initialize forward index creator
- boolean deriveNumDocsPerChunk =
- shouldDeriveNumDocsPerChunk(columnName, segmentCreationSpec.getColumnProperties());
- int writerVersion = rawIndexWriterVersion(columnName, segmentCreationSpec.getColumnProperties());
- if (fieldSpec.isSingleValueField()) {
- _forwardIndexCreatorMap.put(columnName,
- getRawIndexCreatorForSVColumn(_indexDir, compressionType, columnName, storedType, _totalDocs,
- indexCreationInfo.getLengthOfLongestEntry(), deriveNumDocsPerChunk, writerVersion));
- } else {
- _forwardIndexCreatorMap.put(columnName,
- getRawIndexCreatorForMVColumn(_indexDir, compressionType, columnName, storedType, _totalDocs,
- indexCreationInfo.getMaxNumberOfMultiValueElements(), deriveNumDocsPerChunk, writerVersion,
- indexCreationInfo.getMaxRowLengthInBytes()));
- }
}
if (textIndexColumns.contains(columnName)) {
// Initialize text index creator
- Preconditions.checkState(storedType == DataType.STRING,
+ Preconditions.checkState(fieldSpec.getDataType().getStoredType() == FieldSpec.DataType.STRING,
"Text index is currently only supported on STRING type columns");
_textIndexCreatorMap.put(columnName,
new LuceneTextIndexCreator(columnName, _indexDir, true /* commitOnClose */));
Review comment:
I don't follow this. When I reviewed it last time, TextIndexCreatorProvider supported both lucene text fst - https://github.com/apache/pinot/pull/7885#discussion_r768304406
Why do we have to exclude it ? The same distinction that we have today between lucene text index, lucene fst and native fst should be carried into this interface. When native fst was designed, we had agreed that the other 2 are not going to go away from the code (although users might be provided a migration path if need be). So, since they will continue to exist, TextIndexCreatorProvider should support all flavors imo.
##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java
##########
@@ -179,126 +167,71 @@ public void init(SegmentGeneratorConfig segmentCreationSpec, SegmentIndexCreatio
}
String columnName = fieldSpec.getName();
- DataType storedType = fieldSpec.getDataType().getStoredType();
- ColumnIndexCreationInfo indexCreationInfo = indexCreationInfoMap.get(columnName);
- Preconditions.checkNotNull(indexCreationInfo, "Missing index creation info for column: %s", columnName);
- boolean dictEnabledColumn = createDictionaryForColumn(indexCreationInfo, segmentCreationSpec, fieldSpec);
-
+ ColumnIndexCreationInfo columnIndexCreationInfo = indexCreationInfoMap.get(columnName);
+ Preconditions.checkNotNull(columnIndexCreationInfo, "Missing index creation info for column: %s", columnName);
+ boolean dictEnabledColumn = createDictionaryForColumn(columnIndexCreationInfo, segmentCreationSpec, fieldSpec);
+ Preconditions.checkState(dictEnabledColumn || !invertedIndexColumns.contains(columnName),
+ "Cannot create inverted index for raw index column: %s", columnName);
+ ColumnMetadata columnMetadata = ColumnMetadataImpl.builder()
+ .setCardinality(columnIndexCreationInfo.getDistinctValueCount())
+ .setHasDictionary(dictEnabledColumn)
+ .setFieldSpec(fieldSpec)
+ .setTotalDocs(segmentIndexCreationInfo.getTotalDocs())
+ .setTotalNumberOfEntries(columnIndexCreationInfo.getTotalNumberOfEntries())
+ .setSorted(columnIndexCreationInfo.isSorted())
+ .build();
+ IndexCreationContext context = IndexCreationContext.builder()
+ .withIndexDir(_indexDir)
+ .withColumnMetadata(columnMetadata)
+ .withSegmentGeneratorConfig(segmentCreationSpec)
+ .withColumnIndexCreationInfo(columnIndexCreationInfo)
+ .withFSTType(_config.getFSTIndexType())
+ .withH3IndexConfig(h3IndexConfigs.get(columnName))
+ .withCompressionType(dictEnabledColumn ? null : getColumnCompressionType(segmentCreationSpec, fieldSpec))
+ .build();
+ // Initialize forward index creator
+ _forwardIndexCreatorMap.put(columnName, _indexCreatorProvider.newForwardIndexCreator(context));
+
+ // Initialize inverted index creator; skip creating inverted index if sorted
+ if (invertedIndexColumns.contains(columnName) && !columnIndexCreationInfo.isSorted()) {
+ _invertedIndexCreatorMap.put(columnName, _indexCreatorProvider.newInvertedIndexCreator(context));
+ }
if (dictEnabledColumn) {
// Create dictionary-encoded index
-
// Initialize dictionary creator
SegmentDictionaryCreator dictionaryCreator =
- new SegmentDictionaryCreator(indexCreationInfo.getSortedUniqueElementsArray(), fieldSpec, _indexDir,
- indexCreationInfo.isUseVarLengthDictionary());
+ new SegmentDictionaryCreator(columnIndexCreationInfo.getSortedUniqueElementsArray(), fieldSpec, _indexDir,
+ columnIndexCreationInfo.isUseVarLengthDictionary());
_dictionaryCreatorMap.put(columnName, dictionaryCreator);
-
// Create dictionary
try {
dictionaryCreator.build();
} catch (Exception e) {
LOGGER.error("Error building dictionary for field: {}, cardinality: {}, number of bytes per entry: {}",
- fieldSpec.getName(), indexCreationInfo.getDistinctValueCount(), dictionaryCreator.getNumBytesPerEntry());
+ fieldSpec.getName(), columnIndexCreationInfo.getDistinctValueCount(),
+ dictionaryCreator.getNumBytesPerEntry());
throw e;
}
-
- // Initialize forward index creator
- int cardinality = indexCreationInfo.getDistinctValueCount();
- if (fieldSpec.isSingleValueField()) {
- if (indexCreationInfo.isSorted()) {
- _forwardIndexCreatorMap.put(columnName,
- new SingleValueSortedForwardIndexCreator(_indexDir, columnName, cardinality));
- } else {
- _forwardIndexCreatorMap.put(columnName,
- new SingleValueUnsortedForwardIndexCreator(_indexDir, columnName, cardinality, _totalDocs));
- }
- } else {
- _forwardIndexCreatorMap.put(columnName,
- new MultiValueUnsortedForwardIndexCreator(_indexDir, columnName, cardinality, _totalDocs,
- indexCreationInfo.getTotalNumberOfEntries()));
- }
-
- // Initialize inverted index creator; skip creating inverted index if sorted
- if (invertedIndexColumns.contains(columnName) && !indexCreationInfo.isSorted()) {
- if (segmentCreationSpec.isOnHeap()) {
- _invertedIndexCreatorMap.put(columnName,
- new OnHeapBitmapInvertedIndexCreator(_indexDir, columnName, cardinality));
- } else {
- _invertedIndexCreatorMap.put(columnName,
- new OffHeapBitmapInvertedIndexCreator(_indexDir, fieldSpec, cardinality, _totalDocs,
- indexCreationInfo.getTotalNumberOfEntries()));
- }
- }
- } else {
- // Create raw index
- Preconditions.checkState(!invertedIndexColumns.contains(columnName),
- "Cannot create inverted index for raw index column: %s", columnName);
-
- ChunkCompressionType compressionType = getColumnCompressionType(segmentCreationSpec, fieldSpec);
-
- // Initialize forward index creator
- boolean deriveNumDocsPerChunk =
- shouldDeriveNumDocsPerChunk(columnName, segmentCreationSpec.getColumnProperties());
- int writerVersion = rawIndexWriterVersion(columnName, segmentCreationSpec.getColumnProperties());
- if (fieldSpec.isSingleValueField()) {
- _forwardIndexCreatorMap.put(columnName,
- getRawIndexCreatorForSVColumn(_indexDir, compressionType, columnName, storedType, _totalDocs,
- indexCreationInfo.getLengthOfLongestEntry(), deriveNumDocsPerChunk, writerVersion));
- } else {
- _forwardIndexCreatorMap.put(columnName,
- getRawIndexCreatorForMVColumn(_indexDir, compressionType, columnName, storedType, _totalDocs,
- indexCreationInfo.getMaxNumberOfMultiValueElements(), deriveNumDocsPerChunk, writerVersion,
- indexCreationInfo.getMaxRowLengthInBytes()));
- }
}
if (textIndexColumns.contains(columnName)) {
// Initialize text index creator
- Preconditions.checkState(storedType == DataType.STRING,
+ Preconditions.checkState(fieldSpec.getDataType().getStoredType() == FieldSpec.DataType.STRING,
"Text index is currently only supported on STRING type columns");
_textIndexCreatorMap.put(columnName,
new LuceneTextIndexCreator(columnName, _indexDir, true /* commitOnClose */));
Review comment:
I don't follow this. When I reviewed it last time, TextIndexCreatorProvider supported both lucene text and fst - https://github.com/apache/pinot/pull/7885#discussion_r768304406
Why do we have to exclude it ? The same distinction that we have today between lucene text index, lucene fst and native fst should be carried into this interface. When native fst was designed, we had agreed that the other 2 are not going to go away from the code (although users might be provided a migration path if need be). So, since they will continue to exist, TextIndexCreatorProvider should support all flavors imo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r770353762
##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/IndexCreationContext.java
##########
@@ -0,0 +1,481 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.creator;
+
+import java.io.File;
+import java.util.Map;
+import java.util.Objects;
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.index.creator.H3IndexConfig;
+import org.apache.pinot.spi.config.table.BloomFilterConfig;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+/**
+ * Provides parameters for constructing indexes via {@see IndexCreatorProvider}.
+ * The responsibility for ensuring that the correct parameters for a particular
+ * index type lies with the caller.
+ */
+public interface IndexCreationContext {
+
+ FieldSpec getFieldSpec();
+
+ File getIndexDir();
+
+ boolean isOnHeap();
+
+ int getLengthOfLongestEntry();
+
+ int getMaxNumberOfMultiValueElements();
+
+ int getMaxRowLengthInBytes();
+
+ boolean isSorted();
+
+ int getCardinality();
+
+ int getTotalNumberOfEntries();
+
+ int getTotalDocs();
+
+ boolean hasDictionary();
+
+ BloomFilter forBloomFilter(BloomFilterConfig bloomFilterConfig);
+
+ Forward forForwardIndex(ChunkCompressionType chunkCompressionType,
+ @Nullable Map<String, Map<String, String>> columnProperties);
+
+ Geospatial forGeospatialIndex(H3IndexConfig h3IndexConfig);
+
+ Inverted forInvertedIndex();
+
+ Json forJsonIndex();
+
+ Range forRangeIndex(ColumnMetadata columnMetadata, int rangeIndexVersion);
+
+ Text forTextIndex(FSTType fstType, String[] sortedUniqueElementsArray);
Review comment:
I'll add another method to make the distinction between text and FST
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7885: make index creator provision interceptible
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r766087852
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/minion/RawIndexConverter.java
##########
@@ -206,7 +206,7 @@ private void convertColumn(FieldSpec fieldSpec)
DataType storedType = dictionary.getValueType();
int numDocs = _originalSegmentMetadata.getTotalDocs();
int lengthOfLongestEntry = _originalSegmentMetadata.getColumnMetadataFor(columnName).getColumnMaxLength();
- try (ForwardIndexCreator rawIndexCreator = SegmentColumnarIndexCreator
+ try (ForwardIndexCreator rawIndexCreator = DefaultIndexCreatorProvider
Review comment:
Good catch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
richardstartin commented on pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#issuecomment-995617069
@siddharthteotia there is now only one method for creating text indexes now, if the `FSTType` is null, it will create a text index, otherwise an FST index.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#issuecomment-990031441
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7885](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f861dd6) into [master](https://codecov.io/gh/apache/pinot/commit/fb12a509056b4e36724126cef9a344a5e1e89bb1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (fb12a50) will **decrease** coverage by `43.69%`.
> The diff coverage is `15.66%`.
> :exclamation: Current head f861dd6 differs from pull request most recent head 21140ba. Consider uploading reports for the commit 21140ba to get more accurate results
[](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7885 +/- ##
=============================================
- Coverage 71.34% 27.65% -43.70%
=============================================
Files 1587 1583 -4
Lines 82071 81996 -75
Branches 12267 12241 -26
=============================================
- Hits 58553 22675 -35878
- Misses 19550 57261 +37711
+ Partials 3968 2060 -1908
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `27.65% <15.66%> (+<0.01%)` | :arrow_up: |
| unittests1 | `?` | |
| unittests2 | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../java/org/apache/pinot/common/utils/DataTable.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvRGF0YVRhYmxlLmphdmE=) | `89.47% <ø> (ø)` | |
| [...e/pinot/core/common/datatable/DataTableImplV2.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vZGF0YXRhYmxlL0RhdGFUYWJsZUltcGxWMi5qYXZh) | `0.00% <0.00%> (-77.78%)` | :arrow_down: |
| [...rg/apache/pinot/core/minion/RawIndexConverter.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9taW5pb24vUmF3SW5kZXhDb252ZXJ0ZXIuamF2YQ==) | `0.00% <0.00%> (-56.61%)` | :arrow_down: |
| [...ache/pinot/core/plan/InstanceResponsePlanNode.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0luc3RhbmNlUmVzcG9uc2VQbGFuTm9kZS5qYXZh) | `100.00% <ø> (ø)` | |
| [...nverttorawindex/ConvertToRawIndexTaskExecutor.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvY29udmVydHRvcmF3aW5kZXgvQ29udmVydFRvUmF3SW5kZXhUYXNrRXhlY3V0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...ment/creator/impl/DefaultIndexCreatorProvider.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9EZWZhdWx0SW5kZXhDcmVhdG9yUHJvdmlkZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...ment/creator/impl/SegmentColumnarIndexCreator.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9TZWdtZW50Q29sdW1uYXJJbmRleENyZWF0b3IuamF2YQ==) | `0.00% <0.00%> (-84.40%)` | :arrow_down: |
| [...ocal/segment/index/loader/IndexHandlerFactory.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9JbmRleEhhbmRsZXJGYWN0b3J5LmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...t/index/loader/bloomfilter/BloomFilterHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9ibG9vbWZpbHRlci9CbG9vbUZpbHRlckhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (-77.59%)` | :arrow_down: |
| [...nt/index/loader/invertedindex/FSTIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0ZTVEluZGV4SGFuZGxlci5qYXZh) | `0.00% <0.00%> (-82.70%)` | :arrow_down: |
| ... and [1171 more](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [fb12a50...21140ba](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r770024976
##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/IndexCreationContext.java
##########
@@ -0,0 +1,261 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.creator;
+
+import java.io.File;
+import java.util.Map;
+import java.util.Objects;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.index.creator.H3IndexConfig;
+import org.apache.pinot.spi.config.table.BloomFilterConfig;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+/**
+ * Provides parameters for constructing indexes via {@see IndexCreatorProvider}.
+ * The responsibility for ensuring that the correct parameters for a particular
+ * index type lies with the caller.
+ */
+public class IndexCreationContext {
+
+ public static final class Builder {
+ private File _indexDir;
+ private Map<String, Map<String, String>> _columnProperties;
+ private FSTType _fstType;
+ private H3IndexConfig _h3IndexConfig;
+ private int _lengthOfLongestEntry;
+ private int _maxNumberOfMultiValueElements;
+ private int _maxRowLengthInBytes;
+ private Object _sortedUniqueElementsArray;
+ private boolean _onHeap = false;
+ private ChunkCompressionType _chunkCompressionType;
+ public BloomFilterConfig _bloomFilterConfig;
+ private int _rangeIndexVersion;
+ private ColumnMetadata _columnMetadata;
Review comment:
I thought about it and it needs to have `ColumnMetadata` because the range index needs to know the min and max values, because the index can't be built incrementally. I don't see the harm in bundling parameters from `SegmentColumnarIndexCreator` in `ColumnMetadata` just because min and max won't be known; it would only cause a problem if `SegmentColumnarIndexCreator` tried to create an index it didn't have enough to build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#issuecomment-990031441
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7885](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2b42bc9) into [master](https://codecov.io/gh/apache/pinot/commit/fb12a509056b4e36724126cef9a344a5e1e89bb1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (fb12a50) will **decrease** coverage by `0.85%`.
> The diff coverage is `84.61%`.
[](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7885 +/- ##
============================================
- Coverage 71.34% 70.48% -0.86%
- Complexity 4087 4110 +23
============================================
Files 1587 1592 +5
Lines 82071 82344 +273
Branches 12267 12279 +12
============================================
- Hits 58553 58042 -511
- Misses 19550 20323 +773
- Partials 3968 3979 +11
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `29.14% <2.46%> (-0.13%)` | :arrow_down: |
| integration2 | `?` | |
| unittests1 | `68.43% <82.66%> (+0.10%)` | :arrow_up: |
| unittests2 | `14.33% <0.00%> (-0.11%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...not/segment/spi/creator/IndexCreatorProviders.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2NyZWF0b3IvSW5kZXhDcmVhdG9yUHJvdmlkZXJzLmphdmE=) | `45.71% <45.71%> (ø)` | |
| [...t/index/loader/invertedindex/JsonIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0pzb25JbmRleEhhbmRsZXIuamF2YQ==) | `73.33% <66.66%> (+0.45%)` | :arrow_up: |
| [...ment/creator/impl/DefaultIndexCreatorProvider.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9EZWZhdWx0SW5kZXhDcmVhdG9yUHJvdmlkZXIuamF2YQ==) | `67.56% <67.56%> (ø)` | |
| [.../index/loader/invertedindex/RangeIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L1JhbmdlSW5kZXhIYW5kbGVyLmphdmE=) | `44.44% <85.71%> (+3.44%)` | :arrow_up: |
| [...ment/creator/impl/SegmentColumnarIndexCreator.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9TZWdtZW50Q29sdW1uYXJJbmRleENyZWF0b3IuamF2YQ==) | `87.26% <89.74%> (+2.87%)` | :arrow_up: |
| [...rg/apache/pinot/core/minion/RawIndexConverter.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9taW5pb24vUmF3SW5kZXhDb252ZXJ0ZXIuamF2YQ==) | `57.79% <100.00%> (+1.19%)` | :arrow_up: |
| [...nverttorawindex/ConvertToRawIndexTaskExecutor.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvY29udmVydHRvcmF3aW5kZXgvQ29udmVydFRvUmF3SW5kZXhUYXNrRXhlY3V0b3IuamF2YQ==) | `100.00% <100.00%> (ø)` | |
| [...ocal/segment/index/loader/IndexHandlerFactory.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9JbmRleEhhbmRsZXJGYWN0b3J5LmphdmE=) | `100.00% <100.00%> (ø)` | |
| [...t/index/loader/bloomfilter/BloomFilterHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9ibG9vbWZpbHRlci9CbG9vbUZpbHRlckhhbmRsZXIuamF2YQ==) | `78.33% <100.00%> (+0.74%)` | :arrow_up: |
| [...nt/index/loader/invertedindex/FSTIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0ZTVEluZGV4SGFuZGxlci5qYXZh) | `86.79% <100.00%> (+4.10%)` | :arrow_up: |
| ... and [115 more](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [fb12a50...2b42bc9](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
klsince commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r770044288
##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/InvertedIndexHandler.java
##########
@@ -100,9 +104,8 @@ private void createInvertedIndexForColumn(ColumnMetadata columnMetadata)
// Create new inverted index for the column.
LOGGER.info("Creating new inverted index for segment: {}, column: {}", segmentName, column);
int numDocs = columnMetadata.getTotalDocs();
- try (OffHeapBitmapInvertedIndexCreator creator = new OffHeapBitmapInvertedIndexCreator(_indexDir,
- columnMetadata.getFieldSpec(), columnMetadata.getCardinality(), numDocs,
- columnMetadata.getTotalNumberOfEntries())) {
+ try (DictionaryBasedInvertedIndexCreator creator = _indexCreatorProvider.newInvertedIndexCreator(
Review comment:
nit: set context to use offheap version explicitly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#issuecomment-990031441
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7885](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (218b5a2) into [master](https://codecov.io/gh/apache/pinot/commit/fb12a509056b4e36724126cef9a344a5e1e89bb1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (fb12a50) will **decrease** coverage by `43.71%`.
> The diff coverage is `15.66%`.
> :exclamation: Current head 218b5a2 differs from pull request most recent head 2b42bc9. Consider uploading reports for the commit 2b42bc9 to get more accurate results
[](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7885 +/- ##
=============================================
- Coverage 71.34% 27.63% -43.72%
=============================================
Files 1587 1583 -4
Lines 82071 81996 -75
Branches 12267 12241 -26
=============================================
- Hits 58553 22657 -35896
- Misses 19550 57276 +37726
+ Partials 3968 2063 -1905
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `27.63% <15.66%> (-0.03%)` | :arrow_down: |
| unittests1 | `?` | |
| unittests2 | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../java/org/apache/pinot/common/utils/DataTable.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvRGF0YVRhYmxlLmphdmE=) | `89.47% <ø> (ø)` | |
| [...e/pinot/core/common/datatable/DataTableImplV2.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vZGF0YXRhYmxlL0RhdGFUYWJsZUltcGxWMi5qYXZh) | `0.00% <0.00%> (-77.78%)` | :arrow_down: |
| [...rg/apache/pinot/core/minion/RawIndexConverter.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9taW5pb24vUmF3SW5kZXhDb252ZXJ0ZXIuamF2YQ==) | `0.00% <0.00%> (-56.61%)` | :arrow_down: |
| [...ache/pinot/core/plan/InstanceResponsePlanNode.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0luc3RhbmNlUmVzcG9uc2VQbGFuTm9kZS5qYXZh) | `100.00% <ø> (ø)` | |
| [...nverttorawindex/ConvertToRawIndexTaskExecutor.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvY29udmVydHRvcmF3aW5kZXgvQ29udmVydFRvUmF3SW5kZXhUYXNrRXhlY3V0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...ment/creator/impl/DefaultIndexCreatorProvider.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9EZWZhdWx0SW5kZXhDcmVhdG9yUHJvdmlkZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...ment/creator/impl/SegmentColumnarIndexCreator.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9TZWdtZW50Q29sdW1uYXJJbmRleENyZWF0b3IuamF2YQ==) | `0.00% <0.00%> (-84.40%)` | :arrow_down: |
| [...ocal/segment/index/loader/IndexHandlerFactory.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9JbmRleEhhbmRsZXJGYWN0b3J5LmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...t/index/loader/bloomfilter/BloomFilterHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9ibG9vbWZpbHRlci9CbG9vbUZpbHRlckhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (-77.59%)` | :arrow_down: |
| [...nt/index/loader/invertedindex/FSTIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0ZTVEluZGV4SGFuZGxlci5qYXZh) | `0.00% <0.00%> (-82.70%)` | :arrow_down: |
| ... and [1171 more](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [fb12a50...2b42bc9](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r770772471
##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/IndexCreationContext.java
##########
@@ -0,0 +1,454 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.creator;
+
+import java.io.File;
+import java.util.Map;
+import java.util.Objects;
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.index.creator.H3IndexConfig;
+import org.apache.pinot.spi.config.table.BloomFilterConfig;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+/**
+ * Provides parameters for constructing indexes via {@see IndexCreatorProvider}.
+ * The responsibility for ensuring that the correct parameters for a particular
+ * index type lies with the caller.
+ */
+public interface IndexCreationContext {
+
+ FieldSpec getFieldSpec();
+
+ File getIndexDir();
+
+ boolean isOnHeap();
+
+ int getLengthOfLongestEntry();
+
+ int getMaxNumberOfMultiValueElements();
+
+ int getMaxRowLengthInBytes();
+
+ boolean isSorted();
+
+ int getCardinality();
+
+ int getTotalNumberOfEntries();
+
+ int getTotalDocs();
+
+ boolean hasDictionary();
+
+ final class Builder {
+ private File _indexDir;
+ private int _lengthOfLongestEntry;
+ private int _maxNumberOfMultiValueElements;
+ private int _maxRowLengthInBytes;
+ private boolean _onHeap = false;
+ private FieldSpec _fieldSpec;
+ private boolean _sorted;
+ private int _cardinality;
+ private int _totalNumberOfEntries;
+ private int _totalDocs;
+ private boolean _hasDictionary = true;
+
+ public Builder withColumnIndexCreationInfo(ColumnIndexCreationInfo columnIndexCreationInfo) {
+ return withLengthOfLongestEntry(columnIndexCreationInfo.getLengthOfLongestEntry())
+ .withMaxNumberOfMultiValueElements(columnIndexCreationInfo.getMaxNumberOfMultiValueElements())
+ .withMaxRowLengthInBytes(columnIndexCreationInfo.getMaxRowLengthInBytes());
+ }
+
+ public Builder withIndexDir(File indexDir) {
+ _indexDir = indexDir;
+ return this;
+ }
+
+ public Builder onHeap(boolean onHeap) {
+ _onHeap = onHeap;
+ return this;
+ }
+
+ public Builder withColumnMetadata(ColumnMetadata columnMetadata) {
+ return withFieldSpec(columnMetadata.getFieldSpec())
+ .sorted(columnMetadata.isSorted())
+ .withCardinality(columnMetadata.getCardinality())
+ .withTotalNumberOfEntries(columnMetadata.getTotalNumberOfEntries())
+ .withTotalDocs(columnMetadata.getTotalDocs())
+ .withDictionary(columnMetadata.hasDictionary());
+ }
+
+ public Builder withLengthOfLongestEntry(int lengthOfLongestEntry) {
+ _lengthOfLongestEntry = lengthOfLongestEntry;
+ return this;
+ }
+
+ public Builder withMaxNumberOfMultiValueElements(int maxNumberOfMultiValueElements) {
+ _maxNumberOfMultiValueElements = maxNumberOfMultiValueElements;
+ return this;
+ }
+
+ public Builder withMaxRowLengthInBytes(int maxRowLengthInBytes) {
+ _maxRowLengthInBytes = maxRowLengthInBytes;
+ return this;
+ }
+
+ public Builder withFieldSpec(FieldSpec fieldSpec) {
+ _fieldSpec = fieldSpec;
+ return this;
+ }
+
+ public Builder sorted(boolean sorted) {
+ _sorted = sorted;
+ return this;
+ }
+
+ public Builder withCardinality(int cardinality) {
+ _cardinality = cardinality;
+ return this;
+ }
+
+ public Builder withTotalNumberOfEntries(int totalNumberOfEntries) {
+ _totalNumberOfEntries = totalNumberOfEntries;
+ return this;
+ }
+
+ public Builder withTotalDocs(int totalDocs) {
+ _totalDocs = totalDocs;
+ return this;
+ }
+
+ public Builder withDictionary(boolean hasDictionary) {
+ _hasDictionary = hasDictionary;
+ return this;
+ }
+
+ public Common build() {
+ return new Common(Objects.requireNonNull(_indexDir),
+ _lengthOfLongestEntry, _maxNumberOfMultiValueElements, _maxRowLengthInBytes,
+ _onHeap, Objects.requireNonNull(_fieldSpec),
+ _sorted, _cardinality, _totalNumberOfEntries, _totalDocs, _hasDictionary);
+ }
+ }
+
+ static Builder builder() {
+ return new Builder();
+ }
+
+ final class Common implements IndexCreationContext {
+
+ private final File _indexDir;
+ private final int _lengthOfLongestEntry;
+ private final int _maxNumberOfMultiValueElements;
+ private final int _maxRowLengthInBytes;
+ private final boolean _onHeap;
+ private final FieldSpec _fieldSpec;
+ private final boolean _sorted;
+ private final int _cardinality;
+ private final int _totalNumberOfEntries;
+ private final int _totalDocs;
+ private final boolean _hasDictionary;
+
+ public Common(File indexDir, int lengthOfLongestEntry,
+ int maxNumberOfMultiValueElements, int maxRowLengthInBytes, boolean onHeap,
+ FieldSpec fieldSpec, boolean sorted, int cardinality, int totalNumberOfEntries,
+ int totalDocs, boolean hasDictionary) {
+ _indexDir = indexDir;
+ _lengthOfLongestEntry = lengthOfLongestEntry;
+ _maxNumberOfMultiValueElements = maxNumberOfMultiValueElements;
+ _maxRowLengthInBytes = maxRowLengthInBytes;
+ _onHeap = onHeap;
+ _fieldSpec = fieldSpec;
+ _sorted = sorted;
+ _cardinality = cardinality;
+ _totalNumberOfEntries = totalNumberOfEntries;
+ _totalDocs = totalDocs;
+ _hasDictionary = hasDictionary;
+ }
+
+ public FieldSpec getFieldSpec() {
+ return _fieldSpec;
+ }
+
+ public File getIndexDir() {
+ return _indexDir;
+ }
+
+ public boolean isOnHeap() {
+ return _onHeap;
+ }
+
+ public int getLengthOfLongestEntry() {
+ return _lengthOfLongestEntry;
+ }
+
+ public int getMaxNumberOfMultiValueElements() {
+ return _maxNumberOfMultiValueElements;
+ }
+
+ public int getMaxRowLengthInBytes() {
+ return _maxRowLengthInBytes;
+ }
+
+ public boolean isSorted() {
+ return _sorted;
+ }
+
+ public int getCardinality() {
+ return _cardinality;
+ }
+
+ public int getTotalNumberOfEntries() {
+ return _totalNumberOfEntries;
+ }
+
+ public int getTotalDocs() {
+ return _totalDocs;
+ }
+
+ public boolean hasDictionary() {
+ return _hasDictionary;
+ }
+
+ public BloomFilter forBloomFilter(BloomFilterConfig bloomFilterConfig) {
+ return new BloomFilter(this, bloomFilterConfig);
+ }
+
+ public Forward forForwardIndex(ChunkCompressionType chunkCompressionType,
+ @Nullable Map<String, Map<String, String>> columnProperties) {
+ return new Forward(this, chunkCompressionType, columnProperties);
+ }
+
+ public Text forFSTIndex(FSTType fstType, String[] sortedUniqueElementsArray) {
+ return new Text(this, fstType, sortedUniqueElementsArray);
+ }
+
+ public Geospatial forGeospatialIndex(H3IndexConfig h3IndexConfig) {
+ return new Geospatial(this, h3IndexConfig);
+ }
+
+ public Inverted forInvertedIndex() {
+ return new Inverted(this);
+ }
+
+ public Json forJsonIndex() {
+ return new Json(this);
+ }
+
+ public Range forRangeIndex(ColumnMetadata columnMetadata, int rangeIndexVersion) {
+ return new Range(this, columnMetadata, rangeIndexVersion);
+ }
+
+ public Text forTextIndex(boolean commitOnClose) {
+ return new Text(this, commitOnClose);
+ }
+ }
+
+ class Wrapper implements IndexCreationContext {
+
+ private final IndexCreationContext _delegate;
+
+ Wrapper(IndexCreationContext delegate) {
+ _delegate = delegate;
+ }
+
+ @Override
+ public FieldSpec getFieldSpec() {
+ return _delegate.getFieldSpec();
+ }
+
+ @Override
+ public File getIndexDir() {
+ return _delegate.getIndexDir();
+ }
+
+ @Override
+ public boolean isOnHeap() {
+ return _delegate.isOnHeap();
+ }
+
+ @Override
+ public int getLengthOfLongestEntry() {
+ return _delegate.getLengthOfLongestEntry();
+ }
+
+ @Override
+ public int getMaxNumberOfMultiValueElements() {
+ return _delegate.getMaxNumberOfMultiValueElements();
+ }
+
+ @Override
+ public int getMaxRowLengthInBytes() {
+ return _delegate.getMaxRowLengthInBytes();
+ }
+
+ @Override
+ public boolean isSorted() {
+ return _delegate.isSorted();
+ }
+
+ @Override
+ public int getCardinality() {
+ return _delegate.getCardinality();
+ }
+
+ @Override
+ public int getTotalNumberOfEntries() {
+ return _delegate.getTotalNumberOfEntries();
+ }
+
+ @Override
+ public int getTotalDocs() {
+ return _delegate.getTotalDocs();
+ }
+
+ @Override
+ public boolean hasDictionary() {
+ return _delegate.hasDictionary();
+ }
+ }
+
+ class BloomFilter extends Wrapper {
+
+ private final BloomFilterConfig _bloomFilterConfig;
+
+ public BloomFilter(IndexCreationContext wrapped, BloomFilterConfig bloomFilterConfig) {
+ super(wrapped);
+ _bloomFilterConfig = bloomFilterConfig;
+ }
+
+ @Nullable
+ public BloomFilterConfig getBloomFilterConfig() {
+ return _bloomFilterConfig;
+ }
+ }
+
+ class Forward extends Wrapper {
+
+ private final ChunkCompressionType _chunkCompressionType;
+ private final Map<String, Map<String, String>> _columnProperties;
+
+ Forward(IndexCreationContext delegate,
+ ChunkCompressionType chunkCompressionType,
+ @Nullable Map<String, Map<String, String>> columnProperties) {
+ super(delegate);
+ _chunkCompressionType = chunkCompressionType;
+ _columnProperties = columnProperties;
+ }
+
+ public ChunkCompressionType getChunkCompressionType() {
+ return _chunkCompressionType;
+ }
+
+ @Nullable
+ public Map<String, Map<String, String>> getColumnProperties() {
+ return _columnProperties;
+ }
+ }
+
+ class Geospatial extends Wrapper {
+
+ private final H3IndexConfig _h3IndexConfig;
+
+ Geospatial(IndexCreationContext delegate, H3IndexConfig h3IndexConfig) {
+ super(delegate);
+ _h3IndexConfig = h3IndexConfig;
+ }
+
+ public H3IndexConfig getH3IndexConfig() {
+ return _h3IndexConfig;
+ }
+ }
+
+ class Inverted extends Wrapper {
+
+ Inverted(IndexCreationContext delegate) {
+ super(delegate);
+ }
+ }
+
+ class Json extends Wrapper {
+
+ Json(IndexCreationContext delegate) {
+ super(delegate);
+ }
+ }
+
+ class Range extends Wrapper {
+
+ private final ColumnMetadata _columnMetadata;
+ private final int _rangeIndexVersion;
+
+ Range(IndexCreationContext delegate, ColumnMetadata columnMetadata, int rangeIndexVersion) {
+ super(delegate);
+ _columnMetadata = columnMetadata;
+ _rangeIndexVersion = rangeIndexVersion;
+ }
+
+ public ColumnMetadata getColumnMetadata() {
+ return _columnMetadata;
+ }
+
+ public int getRangeIndexVersion() {
+ return _rangeIndexVersion;
+ }
+ }
+
+ class Text extends Wrapper {
Review comment:
(nit) would be good to add a short note on 2 constructors highlighting the first one is for Lucene text and other one is for 2 flavors of FST.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] amrishlal edited a comment on pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
amrishlal edited a comment on pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#issuecomment-995038011
If needed in future, would it be possible to extend these interfaces to allow for creation of multiple indices on a column or would it lead to breaking existing index plugins? For example, I could create an H3 index and a quad-tree index on a geospatial column, or a flat index and a tree index on a JSON column and then depending upon the type of query being evaluated, pick one out of the two available indices to evaluate the query.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] amrishlal commented on pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
amrishlal commented on pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#issuecomment-995038011
If needed in future, would it be possible to extend these interfaces to allow for creation of multiple indices on a column? For example, I could create an H3 index and a quad-tree index on a geospatial column, or a flat index and a tree index on a JSON column and then depending upon the type of query being evaluated, pick one out of the two available indices to evaluate the query.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r769706242
##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java
##########
@@ -179,126 +167,71 @@ public void init(SegmentGeneratorConfig segmentCreationSpec, SegmentIndexCreatio
}
String columnName = fieldSpec.getName();
- DataType storedType = fieldSpec.getDataType().getStoredType();
- ColumnIndexCreationInfo indexCreationInfo = indexCreationInfoMap.get(columnName);
- Preconditions.checkNotNull(indexCreationInfo, "Missing index creation info for column: %s", columnName);
- boolean dictEnabledColumn = createDictionaryForColumn(indexCreationInfo, segmentCreationSpec, fieldSpec);
-
+ ColumnIndexCreationInfo columnIndexCreationInfo = indexCreationInfoMap.get(columnName);
+ Preconditions.checkNotNull(columnIndexCreationInfo, "Missing index creation info for column: %s", columnName);
+ boolean dictEnabledColumn = createDictionaryForColumn(columnIndexCreationInfo, segmentCreationSpec, fieldSpec);
+ Preconditions.checkState(dictEnabledColumn || !invertedIndexColumns.contains(columnName),
+ "Cannot create inverted index for raw index column: %s", columnName);
+ ColumnMetadata columnMetadata = ColumnMetadataImpl.builder()
+ .setCardinality(columnIndexCreationInfo.getDistinctValueCount())
+ .setHasDictionary(dictEnabledColumn)
+ .setFieldSpec(fieldSpec)
+ .setTotalDocs(segmentIndexCreationInfo.getTotalDocs())
+ .setTotalNumberOfEntries(columnIndexCreationInfo.getTotalNumberOfEntries())
+ .setSorted(columnIndexCreationInfo.isSorted())
+ .build();
+ IndexCreationContext context = IndexCreationContext.builder()
+ .withIndexDir(_indexDir)
+ .withColumnMetadata(columnMetadata)
+ .withSegmentGeneratorConfig(segmentCreationSpec)
+ .withColumnIndexCreationInfo(columnIndexCreationInfo)
+ .withFSTType(_config.getFSTIndexType())
+ .withH3IndexConfig(h3IndexConfigs.get(columnName))
+ .withCompressionType(dictEnabledColumn ? null : getColumnCompressionType(segmentCreationSpec, fieldSpec))
+ .build();
+ // Initialize forward index creator
+ _forwardIndexCreatorMap.put(columnName, _indexCreatorProvider.newForwardIndexCreator(context));
+
+ // Initialize inverted index creator; skip creating inverted index if sorted
+ if (invertedIndexColumns.contains(columnName) && !columnIndexCreationInfo.isSorted()) {
+ _invertedIndexCreatorMap.put(columnName, _indexCreatorProvider.newInvertedIndexCreator(context));
+ }
if (dictEnabledColumn) {
// Create dictionary-encoded index
-
// Initialize dictionary creator
SegmentDictionaryCreator dictionaryCreator =
- new SegmentDictionaryCreator(indexCreationInfo.getSortedUniqueElementsArray(), fieldSpec, _indexDir,
- indexCreationInfo.isUseVarLengthDictionary());
+ new SegmentDictionaryCreator(columnIndexCreationInfo.getSortedUniqueElementsArray(), fieldSpec, _indexDir,
+ columnIndexCreationInfo.isUseVarLengthDictionary());
_dictionaryCreatorMap.put(columnName, dictionaryCreator);
-
// Create dictionary
try {
dictionaryCreator.build();
} catch (Exception e) {
LOGGER.error("Error building dictionary for field: {}, cardinality: {}, number of bytes per entry: {}",
- fieldSpec.getName(), indexCreationInfo.getDistinctValueCount(), dictionaryCreator.getNumBytesPerEntry());
+ fieldSpec.getName(), columnIndexCreationInfo.getDistinctValueCount(),
+ dictionaryCreator.getNumBytesPerEntry());
throw e;
}
-
- // Initialize forward index creator
- int cardinality = indexCreationInfo.getDistinctValueCount();
- if (fieldSpec.isSingleValueField()) {
- if (indexCreationInfo.isSorted()) {
- _forwardIndexCreatorMap.put(columnName,
- new SingleValueSortedForwardIndexCreator(_indexDir, columnName, cardinality));
- } else {
- _forwardIndexCreatorMap.put(columnName,
- new SingleValueUnsortedForwardIndexCreator(_indexDir, columnName, cardinality, _totalDocs));
- }
- } else {
- _forwardIndexCreatorMap.put(columnName,
- new MultiValueUnsortedForwardIndexCreator(_indexDir, columnName, cardinality, _totalDocs,
- indexCreationInfo.getTotalNumberOfEntries()));
- }
-
- // Initialize inverted index creator; skip creating inverted index if sorted
- if (invertedIndexColumns.contains(columnName) && !indexCreationInfo.isSorted()) {
- if (segmentCreationSpec.isOnHeap()) {
- _invertedIndexCreatorMap.put(columnName,
- new OnHeapBitmapInvertedIndexCreator(_indexDir, columnName, cardinality));
- } else {
- _invertedIndexCreatorMap.put(columnName,
- new OffHeapBitmapInvertedIndexCreator(_indexDir, fieldSpec, cardinality, _totalDocs,
- indexCreationInfo.getTotalNumberOfEntries()));
- }
- }
- } else {
- // Create raw index
- Preconditions.checkState(!invertedIndexColumns.contains(columnName),
- "Cannot create inverted index for raw index column: %s", columnName);
-
- ChunkCompressionType compressionType = getColumnCompressionType(segmentCreationSpec, fieldSpec);
-
- // Initialize forward index creator
- boolean deriveNumDocsPerChunk =
- shouldDeriveNumDocsPerChunk(columnName, segmentCreationSpec.getColumnProperties());
- int writerVersion = rawIndexWriterVersion(columnName, segmentCreationSpec.getColumnProperties());
- if (fieldSpec.isSingleValueField()) {
- _forwardIndexCreatorMap.put(columnName,
- getRawIndexCreatorForSVColumn(_indexDir, compressionType, columnName, storedType, _totalDocs,
- indexCreationInfo.getLengthOfLongestEntry(), deriveNumDocsPerChunk, writerVersion));
- } else {
- _forwardIndexCreatorMap.put(columnName,
- getRawIndexCreatorForMVColumn(_indexDir, compressionType, columnName, storedType, _totalDocs,
- indexCreationInfo.getMaxNumberOfMultiValueElements(), deriveNumDocsPerChunk, writerVersion,
- indexCreationInfo.getMaxRowLengthInBytes()));
- }
}
if (textIndexColumns.contains(columnName)) {
// Initialize text index creator
- Preconditions.checkState(storedType == DataType.STRING,
+ Preconditions.checkState(fieldSpec.getDataType().getStoredType() == FieldSpec.DataType.STRING,
"Text index is currently only supported on STRING type columns");
_textIndexCreatorMap.put(columnName,
new LuceneTextIndexCreator(columnName, _indexDir, true /* commitOnClose */));
Review comment:
The distinction between text indexes and FST indexes is currently muddy and I don't want to enshrine it in an interface. Therefore, the existing text index is completely excluded from the override mechanism.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#issuecomment-990031441
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7885](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9dfc034) into [master](https://codecov.io/gh/apache/pinot/commit/fb12a509056b4e36724126cef9a344a5e1e89bb1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (fb12a50) will **decrease** coverage by `1.22%`.
> The diff coverage is `81.20%`.
[](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7885 +/- ##
============================================
- Coverage 71.34% 70.11% -1.23%
- Complexity 4087 4103 +16
============================================
Files 1587 1592 +5
Lines 82071 82321 +250
Branches 12267 12272 +5
============================================
- Hits 58553 57723 -830
- Misses 19550 20645 +1095
+ Partials 3968 3953 -15
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `27.63% <0.00%> (-0.02%)` | :arrow_down: |
| unittests1 | `68.39% <81.75%> (+0.05%)` | :arrow_up: |
| unittests2 | `14.34% <0.00%> (-0.11%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/pinot/core/minion/RawIndexConverter.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9taW5pb24vUmF3SW5kZXhDb252ZXJ0ZXIuamF2YQ==) | `0.00% <0.00%> (-56.61%)` | :arrow_down: |
| [...nverttorawindex/ConvertToRawIndexTaskExecutor.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvY29udmVydHRvcmF3aW5kZXgvQ29udmVydFRvUmF3SW5kZXhUYXNrRXhlY3V0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...not/segment/spi/creator/IndexCreatorProviders.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2NyZWF0b3IvSW5kZXhDcmVhdG9yUHJvdmlkZXJzLmphdmE=) | `52.38% <52.38%> (ø)` | |
| [...t/index/loader/invertedindex/JsonIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0pzb25JbmRleEhhbmRsZXIuamF2YQ==) | `73.33% <66.66%> (+0.45%)` | :arrow_up: |
| [...ment/creator/impl/DefaultIndexCreatorProvider.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9EZWZhdWx0SW5kZXhDcmVhdG9yUHJvdmlkZXIuamF2YQ==) | `67.14% <67.14%> (ø)` | |
| [.../index/loader/invertedindex/RangeIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L1JhbmdlSW5kZXhIYW5kbGVyLmphdmE=) | `44.44% <85.71%> (+3.44%)` | :arrow_up: |
| [...ment/creator/impl/SegmentColumnarIndexCreator.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9TZWdtZW50Q29sdW1uYXJJbmRleENyZWF0b3IuamF2YQ==) | `87.02% <87.17%> (+2.63%)` | :arrow_up: |
| [...inot/segment/spi/creator/IndexCreationContext.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2NyZWF0b3IvSW5kZXhDcmVhdGlvbkNvbnRleHQuamF2YQ==) | `94.11% <94.11%> (ø)` | |
| [...ocal/segment/index/loader/IndexHandlerFactory.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9JbmRleEhhbmRsZXJGYWN0b3J5LmphdmE=) | `100.00% <100.00%> (ø)` | |
| [...t/index/loader/bloomfilter/BloomFilterHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9ibG9vbWZpbHRlci9CbG9vbUZpbHRlckhhbmRsZXIuamF2YQ==) | `78.33% <100.00%> (+0.74%)` | :arrow_up: |
| ... and [133 more](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [fb12a50...9dfc034](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #7885: make index creator provision interceptible
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#issuecomment-990031441
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7885](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (4cc35e3) into [master](https://codecov.io/gh/apache/pinot/commit/86278194233781d8001498f0c2be4a8a3299cd9d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8627819) will **decrease** coverage by `50.69%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7885 +/- ##
=============================================
- Coverage 65.10% 14.40% -50.70%
+ Complexity 4078 80 -3998
=============================================
Files 1538 1541 +3
Lines 79998 80079 +81
Branches 12036 12038 +2
=============================================
- Hits 52085 11538 -40547
- Misses 24187 67686 +43499
+ Partials 3726 855 -2871
```
| Flag | Coverage Δ | |
|---|---|---|
| unittests1 | `?` | |
| unittests2 | `14.40% <0.00%> (-0.07%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/pinot/core/minion/RawIndexConverter.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9taW5pb24vUmF3SW5kZXhDb252ZXJ0ZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...ment/creator/impl/DefaultIndexCreatorProvider.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9EZWZhdWx0SW5kZXhDcmVhdG9yUHJvdmlkZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...ment/creator/impl/SegmentColumnarIndexCreator.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9TZWdtZW50Q29sdW1uYXJJbmRleENyZWF0b3IuamF2YQ==) | `0.00% <0.00%> (-84.40%)` | :arrow_down: |
| [...inot/segment/spi/creator/IndexCreationContext.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2NyZWF0b3IvSW5kZXhDcmVhdGlvbkNvbnRleHQuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...not/segment/spi/creator/IndexCreatorProviders.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2NyZWF0b3IvSW5kZXhDcmVhdG9yUHJvdmlkZXJzLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...ain/java/org/apache/pinot/core/data/table/Key.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL3RhYmxlL0tleS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../java/org/apache/pinot/spi/utils/BooleanUtils.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQm9vbGVhblV0aWxzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../java/org/apache/pinot/core/data/table/Record.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL3RhYmxlL1JlY29yZC5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../java/org/apache/pinot/core/util/GroupByUtils.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS91dGlsL0dyb3VwQnlVdGlscy5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...ava/org/apache/pinot/spi/config/table/FSTType.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvY29uZmlnL3RhYmxlL0ZTVFR5cGUuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [1061 more](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [8627819...4cc35e3](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r768307803
##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/DefaultIndexCreatorProvider.java
##########
@@ -0,0 +1,266 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.Map;
+import java.util.Objects;
+import org.apache.pinot.segment.local.io.writer.impl.BaseChunkSVForwardIndexWriter;
+import org.apache.pinot.segment.local.segment.creator.impl.bloom.OnHeapGuavaBloomFilterCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.MultiValueFixedByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.MultiValueUnsortedForwardIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.MultiValueVarByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueFixedByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueSortedForwardIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueUnsortedForwardIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueVarByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.BitSlicedRangeIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.OffHeapBitmapInvertedIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.OnHeapBitmapInvertedIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.RangeIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.geospatial.OffHeapH3IndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.geospatial.OnHeapH3IndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.json.OffHeapJsonIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.json.OnHeapJsonIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.text.LuceneFSTIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.text.LuceneTextIndexCreator;
+import org.apache.pinot.segment.local.utils.nativefst.NativeFSTIndexCreator;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.BloomFilterCreator;
+import org.apache.pinot.segment.spi.index.creator.CombinedInvertedIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.DictionaryBasedInvertedIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.GeoSpatialIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.JsonIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.TextIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.H3IndexResolution;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.config.table.FieldConfig;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+public final class DefaultIndexCreatorProvider implements IndexCreatorProvider {
+
+ @Override
+ public ForwardIndexCreator newForwardIndexCreator(IndexCreationContext context)
+ throws Exception {
+ if (context.isRawColumn()) {
+ boolean deriveNumDocsPerChunk =
+ shouldDeriveNumDocsPerChunk(context.getFieldSpec().getName(), context.getColumnProperties());
+ int writerVersion = rawIndexWriterVersion(context.getFieldSpec().getName(), context.getColumnProperties());
+ if (context.getFieldSpec().isSingleValueField()) {
+ return getRawIndexCreatorForSVColumn(context.getIndexDir(), context.getChunkCompressionType(),
+ context.getFieldSpec().getName(), context.getFieldSpec().getDataType().getStoredType(),
+ context.getTotalDocs(), context.getLengthOfLongestEntry(), deriveNumDocsPerChunk, writerVersion);
+ } else {
+ return getRawIndexCreatorForMVColumn(context.getIndexDir(), context.getChunkCompressionType(),
+ context.getFieldSpec().getName(), context.getFieldSpec().getDataType().getStoredType(),
+ context.getTotalDocs(), context.getMaxNumberOfMultiValueElements(), deriveNumDocsPerChunk, writerVersion,
+ context.getMaxRowLengthInBytes());
+ }
+ } else {
+ if (context.getFieldSpec().isSingleValueField()) {
+ if (context.isSorted()) {
+ return new SingleValueSortedForwardIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(),
+ context.getCardinality());
+ } else {
+ return new SingleValueUnsortedForwardIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(),
+ context.getCardinality(), context.getTotalDocs());
+ }
+ } else {
+ return new MultiValueUnsortedForwardIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(),
+ context.getCardinality(), context.getTotalDocs(), context.getTotalNumberOfEntries());
+ }
+ }
+ }
+
+ @Override
+ public DictionaryBasedInvertedIndexCreator newInvertedIndexCreator(IndexCreationContext context)
+ throws IOException {
+ if (context.isOnHeap()) {
+ return new OnHeapBitmapInvertedIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(),
+ context.getCardinality());
+ } else {
+ return new OffHeapBitmapInvertedIndexCreator(context.getIndexDir(), context.getFieldSpec(),
+ context.getCardinality(), context.getTotalDocs(), context.getTotalNumberOfEntries());
+ }
+ }
+
+ @Override
+ public JsonIndexCreator newJsonIndexCreator(IndexCreationContext context)
+ throws IOException {
+ Preconditions.checkState(context.getFieldSpec().isSingleValueField(),
+ "Json index is currently only supported on single-value columns");
+ Preconditions.checkState(context.getFieldSpec().getDataType().getStoredType() == FieldSpec.DataType.STRING,
+ "Json index is currently only supported on STRING columns");
+ return context.isOnHeap() ? new OnHeapJsonIndexCreator(context.getIndexDir(), context.getFieldSpec().getName())
+ : new OffHeapJsonIndexCreator(context.getIndexDir(), context.getFieldSpec().getName());
+ }
+
+ @Override
+ public TextIndexCreator newTextIndexCreator(IndexCreationContext context)
+ throws IOException {
+ // Initialize text index creator
+ Preconditions.checkState(context.getFieldSpec().getDataType().getStoredType() == FieldSpec.DataType.STRING,
+ "Text index is currently only supported on STRING type columns");
+ return new LuceneTextIndexCreator(context.getFieldSpec().getName(), context.getIndexDir(),
+ true /* commitOnClose */);
+ }
+
+ @Override
+ public TextIndexCreator newFSTIndexCreator(IndexCreationContext context)
+ throws IOException {
+ Preconditions.checkState(context.getFieldSpec().isSingleValueField(),
+ "FST index is currently only supported on single-value columns");
+ Preconditions.checkState(context.getFieldSpec().getDataType().getStoredType() == FieldSpec.DataType.STRING,
+ "FST index is currently only supported on STRING type columns");
+ Preconditions.checkState(!context.isRawColumn(),
+ "FST index is currently only supported on dictionary-encoded columns");
+ String[] sortedValues = (String[]) context.getSortedUniqueElementsArray();
+ if (context.getFstType() == FSTType.NATIVE) {
+ return new NativeFSTIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(), sortedValues);
+ } else {
+ return new LuceneFSTIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(), sortedValues);
+ }
+ }
+
+ @Override
+ public GeoSpatialIndexCreator newGeoSpatialIndexCreator(IndexCreationContext context)
+ throws IOException {
+ Preconditions.checkState(context.getFieldSpec().isSingleValueField(),
+ "H3 index is currently only supported on single-value columns");
+ Preconditions.checkState(context.getFieldSpec().getDataType().getStoredType() == FieldSpec.DataType.BYTES,
+ "H3 index is currently only supported on BYTES columns");
+ H3IndexResolution resolution = Objects.requireNonNull(context.getH3IndexConfig()).getResolution();
+ return context.isOnHeap() ? new OnHeapH3IndexCreator(context.getIndexDir(), context.getFieldSpec().getName(),
+ resolution) : new OffHeapH3IndexCreator(context.getIndexDir(), context.getFieldSpec().getName(), resolution);
+ }
+
+ public static boolean shouldDeriveNumDocsPerChunk(String columnName,
Review comment:
Are we planning to refactor the code in SegmentColumnarIndexCreator to avoid duplicate code and let it reuse these static methods for `shouldDeriveNumDocsPerChunk` and `rawIndexWriterVersion` ? Similarly `getRawIndexCreatorForSVColumn` can also be written just once ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#discussion_r768625404
##########
File path: pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/TextIndexCreatorProvider.java
##########
@@ -0,0 +1,47 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.spi.creator;
+
+import java.io.IOException;
+import org.apache.pinot.segment.spi.index.creator.TextIndexCreator;
+
+
+public interface TextIndexCreatorProvider {
+ /**
+ * Creates a {@see TextIndexCreator} from information about index creation.
+ * This allows a plugin to pattern match index creation information to select
+ * an appropriate implementation.
+ * @param context context about the index creation.
+ * @return a {@see ForwardIndexCreator}
+ * @throws IOException whenever something goes wrong matching or constructing the creator
+ */
+ TextIndexCreator newTextIndexCreator(IndexCreationContext context)
Review comment:
Not really, not without adding more information to `IndexCreatorContext` because of the way it has been purposefully kept separate, which has rippled into the configuration layer. What I think would make sense is not to allow overriding of the Lucene based text index at all (it's also awkward to do for the reader), and leave it completely untouched by this change. At some point in the future we can make a decision about whether and how to migrate the first incarnation of text index to the FST indexes, but leave this distinction out of this SPI for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7885: make index creator provision pluggable
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7885:
URL: https://github.com/apache/pinot/pull/7885#issuecomment-990031441
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7885](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e79a4a7) into [master](https://codecov.io/gh/apache/pinot/commit/fb12a509056b4e36724126cef9a344a5e1e89bb1?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (fb12a50) will **decrease** coverage by `56.99%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7885 +/- ##
=============================================
- Coverage 71.34% 14.34% -57.00%
+ Complexity 4087 80 -4007
=============================================
Files 1587 1547 -40
Lines 82071 80370 -1701
Branches 12267 12066 -201
=============================================
- Hits 58553 11531 -47022
- Misses 19550 67984 +48434
+ Partials 3968 855 -3113
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `?` | |
| unittests2 | `14.34% <0.00%> (-0.11%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...rg/apache/pinot/core/minion/RawIndexConverter.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9taW5pb24vUmF3SW5kZXhDb252ZXJ0ZXIuamF2YQ==) | `0.00% <0.00%> (-56.61%)` | :arrow_down: |
| [...nverttorawindex/ConvertToRawIndexTaskExecutor.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvY29udmVydHRvcmF3aW5kZXgvQ29udmVydFRvUmF3SW5kZXhUYXNrRXhlY3V0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...ment/creator/impl/DefaultIndexCreatorProvider.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9EZWZhdWx0SW5kZXhDcmVhdG9yUHJvdmlkZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...ment/creator/impl/SegmentColumnarIndexCreator.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9TZWdtZW50Q29sdW1uYXJJbmRleENyZWF0b3IuamF2YQ==) | `0.00% <0.00%> (-84.40%)` | :arrow_down: |
| [...ocal/segment/index/loader/IndexHandlerFactory.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9JbmRleEhhbmRsZXJGYWN0b3J5LmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...t/index/loader/bloomfilter/BloomFilterHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9ibG9vbWZpbHRlci9CbG9vbUZpbHRlckhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (-77.59%)` | :arrow_down: |
| [...nt/index/loader/invertedindex/FSTIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0ZTVEluZGV4SGFuZGxlci5qYXZh) | `0.00% <0.00%> (-82.70%)` | :arrow_down: |
| [...ent/index/loader/invertedindex/H3IndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0gzSW5kZXhIYW5kbGVyLmphdmE=) | `0.00% <0.00%> (-72.59%)` | :arrow_down: |
| [...dex/loader/invertedindex/InvertedIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0ludmVydGVkSW5kZXhIYW5kbGVyLmphdmE=) | `0.00% <0.00%> (-93.88%)` | :arrow_down: |
| [...t/index/loader/invertedindex/JsonIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0pzb25JbmRleEhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (-72.89%)` | :arrow_down: |
| ... and [1273 more](https://codecov.io/gh/apache/pinot/pull/7885/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [fb12a50...e79a4a7](https://codecov.io/gh/apache/pinot/pull/7885?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org