You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/08/11 18:04:48 UTC

[GitHub] [pinot] klsince opened a new pull request #7286: Cleanup indices in place while reloading segment - interface changes

klsince opened a new pull request #7286:
URL: https://github.com/apache/pinot/pull/7286


   ## Description
   This RP starts to implement a feature to clean up obsolete indices during segment reload. As the first RP, 
   1. added a new method in SegmentDirectory iface, retainIndices()
   2. added a helper method getIndicesFromTableConfig() as input param for retainIndices()
   3. re-arranged the existing codes in SegmentPreprocessor to be more readable
   
   Use this one to get feedbacks on where change is to be added. Implementation comes in next RP. But still, lemme know if this should be split to smaller RPs for easy review.  
   
   FYI, this one is different from  https://github.com/apache/pinot/pull/7249 (which forces to download segment to overwrite local one for cleanup). This one does cleanup inside local segment folder, e.g. it removes obsolete indices from the columns.psf index file for v3 segment format. 
   
   ## Upgrade Notes
   Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion)
   * [ ] Yes (Please label as **<code>backward-incompat</code>**, and complete the section below on Release Notes)
   
   Does this PR fix a zero-downtime upgrade introduced earlier?
   * [ ] Yes (Please label this as **<code>backward-incompat</code>**, and complete the section below on Release Notes)
   
   Does this PR otherwise need attention when creating release notes? Things to consider:
   - New configuration options
   - Deprecation of configurations
   - Signature changes to public methods/interfaces
   - New plugins added or old plugins removed
   * [ ] Yes (Please label this PR as **<code>release-notes</code>** and complete the section on Release Notes)
   ## Release Notes
   <!-- If you have tagged this as either backward-incompat or release-notes,
   you MUST add text here that you would like to see appear in release notes of the
   next release. -->
   
   <!-- If you have a series of commits adding or enabling a feature, then
   add this section only in final commit that marks the feature completed.
   Refer to earlier release notes to see examples of text.
   -->
   ## Documentation
   <!-- If you have introduced a new feature or configuration, please add it to the documentation as well.
   See https://docs.pinot.apache.org/developers/developers-and-contributors/update-document
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter edited a comment on pull request #7286: Remove obsolete indices during segment reloading for V3 format (wrap up)

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7286:
URL: https://github.com/apache/pinot/pull/7286#issuecomment-897062223


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7286](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7253003) into [master](https://codecov.io/gh/apache/pinot/commit/c62a1fb328f1b151fa4a7a4a6ef17875662d6059?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (c62a1fb) will **decrease** coverage by `6.69%`.
   > The diff coverage is `85.81%`.
   
   > :exclamation: Current head 7253003 differs from pull request most recent head bec81d9. Consider uploading reports for the commit bec81d9 to get more accurate results
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7286/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #7286      +/-   ##
   ============================================
   - Coverage     71.65%   64.96%   -6.70%     
   - Complexity     3311     3346      +35     
   ============================================
     Files          1515     1470      -45     
     Lines         75006    73067    -1939     
     Branches      10910    10708     -202     
   ============================================
   - Hits          53749    47467    -6282     
   - Misses        17628    22223    +4595     
   + Partials       3629     3377     -252     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `?` | |
   | unittests1 | `69.68% <85.81%> (+0.16%)` | :arrow_up: |
   | unittests2 | `14.50% <2.07%> (+0.02%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...rg/apache/pinot/core/minion/RawIndexConverter.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9taW5pb24vUmF3SW5kZXhDb252ZXJ0ZXIuamF2YQ==) | `0.00% <0.00%> (-56.61%)` | :arrow_down: |
   | [...not/common/metadata/segment/SegmentZKMetadata.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0YWRhdGEvc2VnbWVudC9TZWdtZW50WktNZXRhZGF0YS5qYXZh) | `48.52% <28.57%> (-24.21%)` | :arrow_down: |
   | [...ava/org/apache/pinot/client/ConnectionFactory.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Nvbm5lY3Rpb25GYWN0b3J5LmphdmE=) | `66.66% <50.00%> (+8.33%)` | :arrow_up: |
   | [...ache/pinot/core/data/table/SimpleIndexedTable.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL3RhYmxlL1NpbXBsZUluZGV4ZWRUYWJsZS5qYXZh) | `92.30% <66.66%> (-2.52%)` | :arrow_down: |
   | [...org/apache/pinot/core/data/table/IndexedTable.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL3RhYmxlL0luZGV4ZWRUYWJsZS5qYXZh) | `79.48% <70.00%> (-9.99%)` | :arrow_down: |
   | [.../pinot/core/data/table/ConcurrentIndexedTable.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL3RhYmxlL0NvbmN1cnJlbnRJbmRleGVkVGFibGUuamF2YQ==) | `81.81% <80.00%> (-7.74%)` | :arrow_down: |
   | [...t/index/loader/bloomfilter/BloomFilterHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9ibG9vbWZpbHRlci9CbG9vbUZpbHRlckhhbmRsZXIuamF2YQ==) | `77.58% <84.21%> (+59.72%)` | :arrow_up: |
   | [...t/index/loader/invertedindex/JsonIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0pzb25JbmRleEhhbmRsZXIuamF2YQ==) | `74.60% <85.00%> (+39.60%)` | :arrow_up: |
   | [...ent/index/loader/invertedindex/H3IndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0gzSW5kZXhIYW5kbGVyLmphdmE=) | `72.58% <85.71%> (+4.83%)` | :arrow_up: |
   | [...x/loader/defaultcolumn/V3DefaultColumnHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9kZWZhdWx0Y29sdW1uL1YzRGVmYXVsdENvbHVtbkhhbmRsZXIuamF2YQ==) | `90.90% <86.66%> (+36.36%)` | :arrow_up: |
   | ... and [396 more](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [c62a1fb...bec81d9](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter edited a comment on pull request #7286: Remove obsolete indices during segment reloading for V3 format (wrap up)

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7286:
URL: https://github.com/apache/pinot/pull/7286#issuecomment-897062223


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7286](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (263882a) into [master](https://codecov.io/gh/apache/pinot/commit/c62a1fb328f1b151fa4a7a4a6ef17875662d6059?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (c62a1fb) will **decrease** coverage by `39.47%`.
   > The diff coverage is `22.69%`.
   
   > :exclamation: Current head 263882a differs from pull request most recent head 4ee9b17. Consider uploading reports for the commit 4ee9b17 to get more accurate results
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7286/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@              Coverage Diff              @@
   ##             master    #7286       +/-   ##
   =============================================
   - Coverage     71.65%   32.18%   -39.48%     
   + Complexity     3311       93     -3218     
   =============================================
     Files          1515     1508        -7     
     Lines         75006    74659      -347     
     Branches      10910    10876       -34     
   =============================================
   - Hits          53749    24026    -29723     
   - Misses        17628    48564    +30936     
   + Partials       3629     2069     -1560     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `30.34% <21.16%> (-0.07%)` | :arrow_down: |
   | integration2 | `29.10% <22.39%> (-0.01%)` | :arrow_down: |
   | unittests1 | `?` | |
   | unittests2 | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [.../main/java/org/apache/pinot/client/Connection.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Nvbm5lY3Rpb24uamF2YQ==) | `21.73% <0.00%> (-20.82%)` | :arrow_down: |
   | [.../org/apache/pinot/client/SimpleBrokerSelector.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L1NpbXBsZUJyb2tlclNlbGVjdG9yLmphdmE=) | `0.00% <0.00%> (-83.34%)` | :arrow_down: |
   | [...common/exception/SchemaAlreadyExistsException.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZXhjZXB0aW9uL1NjaGVtYUFscmVhZHlFeGlzdHNFeGNlcHRpb24uamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...ller/api/resources/PinotSchemaRestletResource.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9hcGkvcmVzb3VyY2VzL1Bpbm90U2NoZW1hUmVzdGxldFJlc291cmNlLmphdmE=) | `23.93% <0.00%> (-14.81%)` | :arrow_down: |
   | [...re/data/table/UnboundedConcurrentIndexedTable.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL3RhYmxlL1VuYm91bmRlZENvbmN1cnJlbnRJbmRleGVkVGFibGUuamF2YQ==) | `0.00% <0.00%> (-22.73%)` | :arrow_down: |
   | [...perator/combine/GroupByOrderByCombineOperator.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9jb21iaW5lL0dyb3VwQnlPcmRlckJ5Q29tYmluZU9wZXJhdG9yLmphdmE=) | `60.49% <0.00%> (-22.23%)` | :arrow_down: |
   | [...indexsegment/immutable/ImmutableSegmentLoader.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pbmRleHNlZ21lbnQvaW1tdXRhYmxlL0ltbXV0YWJsZVNlZ21lbnRMb2FkZXIuamF2YQ==) | `0.00% <0.00%> (-91.18%)` | :arrow_down: |
   | [...ocal/segment/index/loader/IndexHandlerFactory.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9JbmRleEhhbmRsZXJGYWN0b3J5LmphdmE=) | `0.00% <0.00%> (ø)` | |
   | [...ocal/segment/index/loader/SegmentPreProcessor.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9TZWdtZW50UHJlUHJvY2Vzc29yLmphdmE=) | `0.00% <0.00%> (-68.00%)` | :arrow_down: |
   | [...t/index/loader/bloomfilter/BloomFilterHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9ibG9vbWZpbHRlci9CbG9vbUZpbHRlckhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (-17.86%)` | :arrow_down: |
   | ... and [1016 more](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [c62a1fb...4ee9b17](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] klsince commented on a change in pull request #7286: Remove obsolete indices during segment reloading for V3 format (wrap up)

Posted by GitBox <gi...@apache.org>.
klsince commented on a change in pull request #7286:
URL: https://github.com/apache/pinot/pull/7286#discussion_r696832920



##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/SegmentPreProcessor.java
##########
@@ -112,31 +111,23 @@ public void process()
       rangeIndexHandler.createRangeIndices();
 
       // Create text indices according to the index config.
-      Set<String> textIndexColumns = _indexLoadingConfig.getTextIndexColumns();
-      if (!textIndexColumns.isEmpty()) {
-        TextIndexHandler textIndexHandler =
-            new TextIndexHandler(_indexDir, _segmentMetadata, textIndexColumns, segmentWriter);
-        textIndexHandler.createTextIndexesOnSegmentLoad();
-      }
+      TextIndexHandler textIndexHandler =
+          new TextIndexHandler(_indexDir, _segmentMetadata, _indexLoadingConfig, segmentWriter);
+      textIndexHandler.createTextIndexesOnSegmentLoad();

Review comment:
       actually, those createXXX() method right now indeed just adds indices, as the removal happens in the constructor. but think I can move removal logic into the updateXXX() method.

##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/bloomfilter/BloomFilterHandler.java
##########
@@ -67,12 +67,20 @@ public BloomFilterHandler(File indexDir, SegmentMetadataImpl segmentMetadata, In
     _segmentVersion = segmentMetadata.getVersion();
     _bloomFilterConfigs = indexLoadingConfig.getBloomFilterConfigs();
 
-    for (String column : _bloomFilterConfigs.keySet()) {
+    Set<String> columnsInCfg = _bloomFilterConfigs.keySet();
+    for (String column : columnsInCfg) {
       ColumnMetadata columnMetadata = segmentMetadata.getColumnMetadataFor(column);
       if (columnMetadata != null) {
         _bloomFilterColumns.add(columnMetadata);
       }
     }
+    // Remove indices not set in table config any more
+    Set<String> localColumns = _segmentWriter.toSegmentDirectory().getColumnsWithIndex(ColumnIndexType.BLOOM_FILTER);
+    for (String column : localColumns) {
+      if (!columnsInCfg.contains(column)) {
+        _segmentWriter.removeIndex(column, ColumnIndexType.BLOOM_FILTER);
+      }
+    }

Review comment:
       cool!




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter edited a comment on pull request #7286: Cleanup indices in place while reloading segment - interface changes

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7286:
URL: https://github.com/apache/pinot/pull/7286#issuecomment-897062223


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7286](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1a3a03f) into [master](https://codecov.io/gh/apache/pinot/commit/6fb51d2889e5b269e52f483f3cac89f5562a3a62?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6fb51d2) will **decrease** coverage by `0.05%`.
   > The diff coverage is `64.42%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7286/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #7286      +/-   ##
   ============================================
   - Coverage     71.33%   71.27%   -0.06%     
   + Complexity     3363     3297      -66     
   ============================================
     Files          1505     1505              
     Lines         74224    74274      +50     
     Branches      10733    10740       +7     
   ============================================
   - Hits          52950    52942       -8     
   - Misses        17678    17730      +52     
   - Partials       3596     3602       +6     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `30.06% <0.00%> (+0.20%)` | :arrow_up: |
   | integration2 | `29.38% <0.00%> (+0.14%)` | :arrow_up: |
   | unittests1 | `68.99% <64.42%> (-0.04%)` | :arrow_down: |
   | unittests2 | `14.59% <0.00%> (-0.02%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...t/local/segment/store/SegmentLocalFSDirectory.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L3N0b3JlL1NlZ21lbnRMb2NhbEZTRGlyZWN0b3J5LmphdmE=) | `63.75% <0.00%> (-5.09%)` | :arrow_down: |
   | [.../pinot/segment/spi/store/ColumnIndexDirectory.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL3N0b3JlL0NvbHVtbkluZGV4RGlyZWN0b3J5LmphdmE=) | `40.00% <0.00%> (-26.67%)` | :arrow_down: |
   | [...ache/pinot/segment/spi/store/SegmentDirectory.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL3N0b3JlL1NlZ21lbnREaXJlY3RvcnkuamF2YQ==) | `83.33% <ø> (ø)` | |
   | [...ocal/segment/index/loader/SegmentPreProcessor.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9TZWdtZW50UHJlUHJvY2Vzc29yLmphdmE=) | `74.28% <69.62%> (+7.14%)` | :arrow_up: |
   | [...indexsegment/immutable/ImmutableSegmentLoader.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pbmRleHNlZ21lbnQvaW1tdXRhYmxlL0ltbXV0YWJsZVNlZ21lbnRMb2FkZXIuamF2YQ==) | `91.17% <100.00%> (+0.26%)` | :arrow_up: |
   | [...a/manager/realtime/RealtimeSegmentDataManager.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVTZWdtZW50RGF0YU1hbmFnZXIuamF2YQ==) | `50.00% <0.00%> (-25.00%)` | :arrow_down: |
   | [...ntroller/helix/core/minion/TaskMetricsEmitter.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL21pbmlvbi9UYXNrTWV0cmljc0VtaXR0ZXIuamF2YQ==) | `72.72% <0.00%> (-18.19%)` | :arrow_down: |
   | [...nction/DistinctCountBitmapAggregationFunction.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9hZ2dyZWdhdGlvbi9mdW5jdGlvbi9EaXN0aW5jdENvdW50Qml0bWFwQWdncmVnYXRpb25GdW5jdGlvbi5qYXZh) | `41.75% <0.00%> (-7.74%)` | :arrow_down: |
   | [...r/helix/SegmentOnlineOfflineStateModelFactory.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9TZWdtZW50T25saW5lT2ZmbGluZVN0YXRlTW9kZWxGYWN0b3J5LmphdmE=) | `58.87% <0.00%> (-6.55%)` | :arrow_down: |
   | [...core/query/executor/ServerQueryExecutorV1Impl.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9leGVjdXRvci9TZXJ2ZXJRdWVyeUV4ZWN1dG9yVjFJbXBsLmphdmE=) | `81.42% <0.00%> (-4.92%)` | :arrow_down: |
   | ... and [23 more](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [d40e2d3...1a3a03f](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter edited a comment on pull request #7286: Remove obsolete indices during segment reloading for V3 format (wrap up)

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7286:
URL: https://github.com/apache/pinot/pull/7286#issuecomment-897062223


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7286](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (61a3dc2) into [master](https://codecov.io/gh/apache/pinot/commit/bb34406aaa205f5c85b88c928d477fd267eda1b4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bb34406) will **increase** coverage by `5.02%`.
   > The diff coverage is `71.18%`.
   
   > :exclamation: Current head 61a3dc2 differs from pull request most recent head 9171b8b. Consider uploading reports for the commit 9171b8b to get more accurate results
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7286/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #7286      +/-   ##
   ============================================
   + Coverage     64.65%   69.68%   +5.02%     
   + Complexity     3303     3254      -49     
   ============================================
     Files          1496     1114     -382     
     Lines         74289    52747   -21542     
     Branches      10795     7921    -2874     
   ============================================
   - Hits          48033    36755   -11278     
   + Misses        22826    13379    -9447     
   + Partials       3430     2613     -817     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `?` | |
   | unittests1 | `69.68% <71.18%> (+0.21%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [.../pinot/common/messages/RunPeriodicTaskMessage.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWVzc2FnZXMvUnVuUGVyaW9kaWNUYXNrTWVzc2FnZS5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...pinot/core/periodictask/PeriodicTaskScheduler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wZXJpb2RpY3Rhc2svUGVyaW9kaWNUYXNrU2NoZWR1bGVyLmphdmE=) | `61.19% <31.03%> (-23.02%)` | :arrow_down: |
   | [...t/index/loader/invertedindex/JsonIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0pzb25JbmRleEhhbmRsZXIuamF2YQ==) | `75.00% <72.72%> (+40.00%)` | :arrow_up: |
   | [...ache/pinot/core/periodictask/BasePeriodicTask.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wZXJpb2RpY3Rhc2svQmFzZVBlcmlvZGljVGFzay5qYXZh) | `65.62% <74.19%> (+15.62%)` | :arrow_up: |
   | [...x/loader/defaultcolumn/V3DefaultColumnHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9kZWZhdWx0Y29sdW1uL1YzRGVmYXVsdENvbHVtbkhhbmRsZXIuamF2YQ==) | `90.90% <86.66%> (+36.36%)` | :arrow_up: |
   | [...t/index/loader/bloomfilter/BloomFilterHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9ibG9vbWZpbHRlci9CbG9vbUZpbHRlckhhbmRsZXIuamF2YQ==) | `74.60% <88.88%> (+56.74%)` | :arrow_up: |
   | [...ex/loader/invertedindex/LuceneFSTIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0x1Y2VuZUZTVEluZGV4SGFuZGxlci5qYXZh) | `85.18% <88.88%> (-2.05%)` | :arrow_down: |
   | [.../index/loader/invertedindex/RangeIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L1JhbmdlSW5kZXhIYW5kbGVyLmphdmE=) | `41.58% <88.88%> (+32.00%)` | :arrow_up: |
   | [...t/index/loader/invertedindex/TextIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L1RleHRJbmRleEhhbmRsZXIuamF2YQ==) | `89.83% <88.88%> (-2.48%)` | :arrow_down: |
   | [...ent/index/loader/invertedindex/H3IndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0gzSW5kZXhIYW5kbGVyLmphdmE=) | `71.42% <90.00%> (+3.68%)` | :arrow_up: |
   | ... and [596 more](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [bb34406...9171b8b](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter edited a comment on pull request #7286: Remove obsolete indices during segment reloading for V3 format (wrap up)

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7286:
URL: https://github.com/apache/pinot/pull/7286#issuecomment-897062223


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7286](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (4ee9b17) into [master](https://codecov.io/gh/apache/pinot/commit/c62a1fb328f1b151fa4a7a4a6ef17875662d6059?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (c62a1fb) will **decrease** coverage by `1.97%`.
   > The diff coverage is `90.67%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7286/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #7286      +/-   ##
   ============================================
   - Coverage     71.65%   69.68%   -1.98%     
   + Complexity     3311     3256      -55     
   ============================================
     Files          1515     1123     -392     
     Lines         75006    52916   -22090     
     Branches      10910     7963    -2947     
   ============================================
   - Hits          53749    36874   -16875     
   + Misses        17628    13424    -4204     
   + Partials       3629     2618    -1011     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `?` | |
   | unittests1 | `69.68% <90.67%> (+0.16%)` | :arrow_up: |
   | unittests2 | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...rg/apache/pinot/core/minion/RawIndexConverter.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9taW5pb24vUmF3SW5kZXhDb252ZXJ0ZXIuamF2YQ==) | `0.00% <0.00%> (-56.61%)` | :arrow_down: |
   | [...t/index/loader/bloomfilter/BloomFilterHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9ibG9vbWZpbHRlci9CbG9vbUZpbHRlckhhbmRsZXIuamF2YQ==) | `77.58% <84.21%> (+59.72%)` | :arrow_up: |
   | [...t/index/loader/invertedindex/JsonIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0pzb25JbmRleEhhbmRsZXIuamF2YQ==) | `74.60% <85.00%> (+39.60%)` | :arrow_up: |
   | [...ent/index/loader/invertedindex/H3IndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0gzSW5kZXhIYW5kbGVyLmphdmE=) | `72.58% <85.71%> (+4.83%)` | :arrow_up: |
   | [...x/loader/defaultcolumn/V3DefaultColumnHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9kZWZhdWx0Y29sdW1uL1YzRGVmYXVsdENvbHVtbkhhbmRsZXIuamF2YQ==) | `90.90% <86.66%> (+36.36%)` | :arrow_up: |
   | [.../index/loader/invertedindex/RangeIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L1JhbmdlSW5kZXhIYW5kbGVyLmphdmE=) | `41.66% <88.88%> (+32.09%)` | :arrow_up: |
   | [...indexsegment/immutable/ImmutableSegmentLoader.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pbmRleHNlZ21lbnQvaW1tdXRhYmxlL0ltbXV0YWJsZVNlZ21lbnRMb2FkZXIuamF2YQ==) | `90.66% <90.90%> (-0.51%)` | :arrow_down: |
   | [...dex/loader/invertedindex/InvertedIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0ludmVydGVkSW5kZXhIYW5kbGVyLmphdmE=) | `93.87% <94.73%> (+0.39%)` | :arrow_up: |
   | [...t/index/loader/invertedindex/TextIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L1RleHRJbmRleEhhbmRsZXIuamF2YQ==) | `90.74% <94.73%> (-1.57%)` | :arrow_down: |
   | [...ocal/segment/index/loader/IndexHandlerFactory.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9JbmRleEhhbmRsZXJGYWN0b3J5LmphdmE=) | `100.00% <100.00%> (ø)` | |
   | ... and [633 more](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [c62a1fb...4ee9b17](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter edited a comment on pull request #7286: Remove obsolete indices during segment reloading for V3 format (wrap up)

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7286:
URL: https://github.com/apache/pinot/pull/7286#issuecomment-897062223


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7286](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6ac5dc3) into [master](https://codecov.io/gh/apache/pinot/commit/bb34406aaa205f5c85b88c928d477fd267eda1b4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bb34406) will **increase** coverage by `4.99%`.
   > The diff coverage is `90.38%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7286/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #7286      +/-   ##
   ============================================
   + Coverage     64.65%   69.65%   +4.99%     
   + Complexity     3303     3248      -55     
   ============================================
     Files          1496     1113     -383     
     Lines         74289    52705   -21584     
     Branches      10795     7918    -2877     
   ============================================
   - Hits          48033    36712   -11321     
   + Misses        22826    13383    -9443     
   + Partials       3430     2610     -820     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `?` | |
   | unittests1 | `69.65% <90.38%> (+0.18%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...t/index/loader/invertedindex/JsonIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0pzb25JbmRleEhhbmRsZXIuamF2YQ==) | `75.00% <72.72%> (+40.00%)` | :arrow_up: |
   | [...x/loader/defaultcolumn/V3DefaultColumnHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9kZWZhdWx0Y29sdW1uL1YzRGVmYXVsdENvbHVtbkhhbmRsZXIuamF2YQ==) | `90.90% <86.66%> (+36.36%)` | :arrow_up: |
   | [...t/index/loader/bloomfilter/BloomFilterHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9ibG9vbWZpbHRlci9CbG9vbUZpbHRlckhhbmRsZXIuamF2YQ==) | `74.60% <88.88%> (+56.74%)` | :arrow_up: |
   | [...ex/loader/invertedindex/LuceneFSTIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0x1Y2VuZUZTVEluZGV4SGFuZGxlci5qYXZh) | `85.18% <88.88%> (-2.05%)` | :arrow_down: |
   | [.../index/loader/invertedindex/RangeIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L1JhbmdlSW5kZXhIYW5kbGVyLmphdmE=) | `41.58% <88.88%> (+32.00%)` | :arrow_up: |
   | [...t/index/loader/invertedindex/TextIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L1RleHRJbmRleEhhbmRsZXIuamF2YQ==) | `89.83% <88.88%> (-2.48%)` | :arrow_down: |
   | [...ent/index/loader/invertedindex/H3IndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0gzSW5kZXhIYW5kbGVyLmphdmE=) | `71.42% <90.00%> (+3.68%)` | :arrow_up: |
   | [...indexsegment/immutable/ImmutableSegmentLoader.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pbmRleHNlZ21lbnQvaW1tdXRhYmxlL0ltbXV0YWJsZVNlZ21lbnRMb2FkZXIuamF2YQ==) | `91.30% <100.00%> (+0.12%)` | :arrow_up: |
   | [...ocal/segment/index/loader/SegmentPreProcessor.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9TZWdtZW50UHJlUHJvY2Vzc29yLmphdmE=) | `67.14% <100.00%> (-0.86%)` | :arrow_down: |
   | [...loader/defaultcolumn/BaseDefaultColumnHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9kZWZhdWx0Y29sdW1uL0Jhc2VEZWZhdWx0Q29sdW1uSGFuZGxlci5qYXZh) | `67.60% <100.00%> (+1.55%)` | :arrow_up: |
   | ... and [613 more](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [bb34406...6ac5dc3](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] klsince commented on a change in pull request #7286: Remove obsolete indices during segment reloading for V3 format (wrap up)

Posted by GitBox <gi...@apache.org>.
klsince commented on a change in pull request #7286:
URL: https://github.com/apache/pinot/pull/7286#discussion_r698837174



##########
File path: pinot-segment-local/src/test/resources/data/newColumnsSchema3.json
##########
@@ -34,10 +34,6 @@
       "name": "newBooleanSVDimension",
       "defaultNullValue": "false"
     },
-    {

Review comment:
       to test the logic to remove auto-gen'ed default column. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #7286: Remove obsolete indices during segment reloading for V3 format (wrap up)

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #7286:
URL: https://github.com/apache/pinot/pull/7286#discussion_r695092974



##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/store/SingleFileIndexDirectory.java
##########
@@ -385,10 +385,12 @@ public void removeIndex(String columnName, ColumnIndexType indexType) {
 
   @Override
   public Set<String> getColumnsWithIndex(ColumnIndexType type) {
+    // TEXT_INDEX is not tracked via _columnEntries, so need to call

Review comment:
       Specialize the handling for `TEXT` and use the current logic for other index types for efficiency

##########
File path: pinot-segment-local/src/test/resources/data/log4j2.xml
##########
@@ -29,7 +29,7 @@
   </Appenders>
   <Loggers>
     <AsyncRoot level="warn" additivity="false">
-      <AppenderRef ref="console"/>
+      <AppenderRef ref="console" />

Review comment:
       revert

##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/immutable/ImmutableSegmentLoader.java
##########
@@ -57,25 +57,40 @@ private ImmutableSegmentLoader() {
   private static final Logger LOGGER = LoggerFactory.getLogger(ImmutableSegmentLoader.class);
 
   /**
-   * For tests only.
+   * load with empty schema and IndexLoadingConfig is used to get a handler of the segment for
+   * its metadata and any other existing info inside the segment folder. This method is used by
+   * MultiTreeBuilder (with empty schema and IndexLoadingConfig to avoid recursion) and tools.
+   * No need to cleanup indices while calling this load method.
    */
   public static ImmutableSegment load(File indexDir, ReadMode readMode)
       throws Exception {
     IndexLoadingConfig defaultIndexLoadingConfig = new IndexLoadingConfig();
     defaultIndexLoadingConfig.setReadMode(readMode);
-    return load(indexDir, defaultIndexLoadingConfig, null);
+    return load(indexDir, defaultIndexLoadingConfig, null, false);
   }
 
   /**
-   * For tests only.
+   * load with empty schema but a specified IndexLoadingConfig (mostly empty) is also used
+   * to get a handler of the current segment, e.g. used in RawIndexConverter and test cases.
+   * No need to cleanup indices while calling this load method.
    */
   public static ImmutableSegment load(File indexDir, IndexLoadingConfig indexLoadingConfig)
       throws Exception {
-    return load(indexDir, indexLoadingConfig, null);
+    return load(indexDir, indexLoadingConfig, null, false);
   }
 
+  /**
+   * load with specified schema and IndexLoadingConfig and clean up obsolete indices
+   * according to the IndexLoadingConfig.
+   */
   public static ImmutableSegment load(File indexDir, IndexLoadingConfig indexLoadingConfig, @Nullable Schema schema)
       throws Exception {
+    return load(indexDir, indexLoadingConfig, schema, true);
+  }
+
+  private static ImmutableSegment load(File indexDir, IndexLoadingConfig indexLoadingConfig, @Nullable Schema schema,
+      boolean cleanupIndices)

Review comment:
       I think we should make segment pre-processing optional instead of making index cleanup optional. When we do the segment pre-processing, we want to make the segment align with the index loading config and schema.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter edited a comment on pull request #7286: Remove obsolete indices during segment reloading for V3 format (wrap up)

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7286:
URL: https://github.com/apache/pinot/pull/7286#issuecomment-897062223


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7286](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (64cf422) into [master](https://codecov.io/gh/apache/pinot/commit/c62a1fb328f1b151fa4a7a4a6ef17875662d6059?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (c62a1fb) will **decrease** coverage by `2.00%`.
   > The diff coverage is `85.71%`.
   
   > :exclamation: Current head 64cf422 differs from pull request most recent head 4d9cc57. Consider uploading reports for the commit 4d9cc57 to get more accurate results
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7286/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #7286      +/-   ##
   ============================================
   - Coverage     71.65%   69.65%   -2.01%     
   + Complexity     3311     3253      -58     
   ============================================
     Files          1515     1122     -393     
     Lines         75006    52943   -22063     
     Branches      10910     7972    -2938     
   ============================================
   - Hits          53749    36878   -16871     
   + Misses        17628    13440    -4188     
   + Partials       3629     2625    -1004     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `?` | |
   | unittests1 | `69.65% <85.71%> (+0.13%)` | :arrow_up: |
   | unittests2 | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...rg/apache/pinot/core/minion/RawIndexConverter.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9taW5pb24vUmF3SW5kZXhDb252ZXJ0ZXIuamF2YQ==) | `0.00% <0.00%> (-56.61%)` | :arrow_down: |
   | [...t/index/loader/bloomfilter/BloomFilterHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9ibG9vbWZpbHRlci9CbG9vbUZpbHRlckhhbmRsZXIuamF2YQ==) | `72.41% <76.47%> (+54.55%)` | :arrow_up: |
   | [...t/index/loader/invertedindex/JsonIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0pzb25JbmRleEhhbmRsZXIuamF2YQ==) | `69.84% <77.77%> (+34.84%)` | :arrow_up: |
   | [...ent/index/loader/invertedindex/H3IndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0gzSW5kZXhIYW5kbGVyLmphdmE=) | `67.74% <78.94%> (ø)` | |
   | [.../index/loader/invertedindex/RangeIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L1JhbmdlSW5kZXhIYW5kbGVyLmphdmE=) | `38.54% <81.25%> (+28.96%)` | :arrow_up: |
   | [...x/loader/defaultcolumn/V3DefaultColumnHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9kZWZhdWx0Y29sdW1uL1YzRGVmYXVsdENvbHVtbkhhbmRsZXIuamF2YQ==) | `90.90% <85.71%> (+36.36%)` | :arrow_up: |
   | [...dex/loader/invertedindex/InvertedIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0ludmVydGVkSW5kZXhIYW5kbGVyLmphdmE=) | `87.75% <88.23%> (-5.73%)` | :arrow_down: |
   | [...t/index/loader/invertedindex/TextIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L1RleHRJbmRleEhhbmRsZXIuamF2YQ==) | `85.18% <88.23%> (-7.13%)` | :arrow_down: |
   | [...indexsegment/immutable/ImmutableSegmentLoader.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pbmRleHNlZ21lbnQvaW1tdXRhYmxlL0ltbXV0YWJsZVNlZ21lbnRMb2FkZXIuamF2YQ==) | `90.66% <90.90%> (-0.51%)` | :arrow_down: |
   | [...ex/loader/invertedindex/LuceneFSTIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0x1Y2VuZUZTVEluZGV4SGFuZGxlci5qYXZh) | `79.59% <94.11%> (-7.65%)` | :arrow_down: |
   | ... and [629 more](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [c62a1fb...4d9cc57](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter edited a comment on pull request #7286: Remove obsolete indices during segment reloading for V3 format (wrap up)

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7286:
URL: https://github.com/apache/pinot/pull/7286#issuecomment-897062223


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7286](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (44fcfbe) into [master](https://codecov.io/gh/apache/pinot/commit/2d1f7dc9060d45ac9f4ff0621da62790ea9b8e64?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2d1f7dc) will **decrease** coverage by `42.60%`.
   > The diff coverage is `0.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7286/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@              Coverage Diff              @@
   ##             master    #7286       +/-   ##
   =============================================
   - Coverage     71.65%   29.04%   -42.61%     
   + Complexity     3305       84     -3221     
   =============================================
     Files          1515     1506        -9     
     Lines         74992    74690      -302     
     Branches      10903    10880       -23     
   =============================================
   - Hits          53736    21697    -32039     
   - Misses        17629    51026    +33397     
   + Partials       3627     1967     -1660     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `29.04% <0.00%> (-0.04%)` | :arrow_down: |
   | unittests1 | `?` | |
   | unittests2 | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...indexsegment/immutable/ImmutableSegmentLoader.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pbmRleHNlZ21lbnQvaW1tdXRhYmxlL0ltbXV0YWJsZVNlZ21lbnRMb2FkZXIuamF2YQ==) | `0.00% <0.00%> (-91.18%)` | :arrow_down: |
   | [...ocal/segment/index/loader/SegmentPreProcessor.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9TZWdtZW50UHJlUHJvY2Vzc29yLmphdmE=) | `0.00% <0.00%> (-68.00%)` | :arrow_down: |
   | [...t/index/loader/bloomfilter/BloomFilterHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9ibG9vbWZpbHRlci9CbG9vbUZpbHRlckhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (-17.86%)` | :arrow_down: |
   | [...loader/defaultcolumn/BaseDefaultColumnHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9kZWZhdWx0Y29sdW1uL0Jhc2VEZWZhdWx0Q29sdW1uSGFuZGxlci5qYXZh) | `0.00% <0.00%> (-66.05%)` | :arrow_down: |
   | [...x/loader/defaultcolumn/V1DefaultColumnHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9kZWZhdWx0Y29sdW1uL1YxRGVmYXVsdENvbHVtbkhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (-66.67%)` | :arrow_down: |
   | [...x/loader/defaultcolumn/V3DefaultColumnHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9kZWZhdWx0Y29sdW1uL1YzRGVmYXVsdENvbHVtbkhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (-54.55%)` | :arrow_down: |
   | [...ent/index/loader/invertedindex/H3IndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0gzSW5kZXhIYW5kbGVyLmphdmE=) | `0.00% <0.00%> (-67.75%)` | :arrow_down: |
   | [...dex/loader/invertedindex/InvertedIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0ludmVydGVkSW5kZXhIYW5kbGVyLmphdmE=) | `0.00% <0.00%> (-93.48%)` | :arrow_down: |
   | [...t/index/loader/invertedindex/JsonIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0pzb25JbmRleEhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (-35.00%)` | :arrow_down: |
   | [...ex/loader/invertedindex/LuceneFSTIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0x1Y2VuZUZTVEluZGV4SGFuZGxlci5qYXZh) | `0.00% <0.00%> (-87.24%)` | :arrow_down: |
   | ... and [1064 more](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [2d1f7dc...44fcfbe](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang merged pull request #7286: Remove obsolete indices during segment reloading for V3 format (wrap up)

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang merged pull request #7286:
URL: https://github.com/apache/pinot/pull/7286


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter edited a comment on pull request #7286: Remove obsolete indices during segment reloading for V3 format (wrap up)

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7286:
URL: https://github.com/apache/pinot/pull/7286#issuecomment-897062223


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7286](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (61a3dc2) into [master](https://codecov.io/gh/apache/pinot/commit/bb34406aaa205f5c85b88c928d477fd267eda1b4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bb34406) will **increase** coverage by `5.87%`.
   > The diff coverage is `54.57%`.
   
   > :exclamation: Current head 61a3dc2 differs from pull request most recent head 9171b8b. Consider uploading reports for the commit 9171b8b to get more accurate results
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7286/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #7286      +/-   ##
   ============================================
   + Coverage     64.65%   70.52%   +5.87%     
   - Complexity     3303     3430     +127     
   ============================================
     Files          1496     1508      +12     
     Lines         74289    74799     +510     
     Branches      10795    10864      +69     
   ============================================
   + Hits          48033    52754    +4721     
   + Misses        22826    18416    -4410     
   - Partials       3430     3629     +199     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `29.13% <16.27%> (-0.06%)` | :arrow_down: |
   | unittests1 | `69.68% <71.18%> (+0.21%)` | :arrow_up: |
   | unittests2 | `14.47% <8.63%> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...t/broker/api/resources/PinotBrokerHealthCheck.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYXBpL3Jlc291cmNlcy9QaW5vdEJyb2tlckhlYWx0aENoZWNrLmphdmE=) | `0.00% <0.00%> (ø)` | |
   | [...org/apache/pinot/broker/queryquota/HitCounter.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcXVlcnlxdW90YS9IaXRDb3VudGVyLmphdmE=) | `100.00% <ø> (+31.03%)` | :arrow_up: |
   | [...che/pinot/broker/queryquota/MaxHitRateTracker.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcXVlcnlxdW90YS9NYXhIaXRSYXRlVHJhY2tlci5qYXZh) | `95.00% <ø> (+58.63%)` | :arrow_up: |
   | [...roker/requesthandler/BaseBrokerRequestHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcmVxdWVzdGhhbmRsZXIvQmFzZUJyb2tlclJlcXVlc3RIYW5kbGVyLmphdmE=) | `70.99% <ø> (+4.72%)` | :arrow_up: |
   | [.../routing/segmentpruner/PartitionSegmentPruner.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9zZWdtZW50cHJ1bmVyL1BhcnRpdGlvblNlZ21lbnRQcnVuZXIuamF2YQ==) | `61.24% <0.00%> (+13.58%)` | :arrow_up: |
   | [...roker/routing/segmentpruner/TimeSegmentPruner.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9zZWdtZW50cHJ1bmVyL1RpbWVTZWdtZW50UHJ1bmVyLmphdmE=) | `80.95% <0.00%> (+80.95%)` | :arrow_up: |
   | [.../pinot/common/messages/RunPeriodicTaskMessage.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWVzc2FnZXMvUnVuUGVyaW9kaWNUYXNrTWVzc2FnZS5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...ache/pinot/controller/api/resources/Constants.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9hcGkvcmVzb3VyY2VzL0NvbnN0YW50cy5qYXZh) | `23.07% <ø> (ø)` | |
   | [...es/PinotControllerPeriodicTaskRestletResource.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9hcGkvcmVzb3VyY2VzL1Bpbm90Q29udHJvbGxlclBlcmlvZGljVGFza1Jlc3RsZXRSZXNvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [.../helix/core/minion/MinionInstancesCleanupTask.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL21pbmlvbi9NaW5pb25JbnN0YW5jZXNDbGVhbnVwVGFzay5qYXZh) | `63.63% <ø> (ø)` | |
   | ... and [320 more](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [bb34406...9171b8b](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter edited a comment on pull request #7286: Cleanup indices in place while reloading segment - interface changes

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7286:
URL: https://github.com/apache/pinot/pull/7286#issuecomment-897062223


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7286](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1a3a03f) into [master](https://codecov.io/gh/apache/pinot/commit/6fb51d2889e5b269e52f483f3cac89f5562a3a62?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6fb51d2) will **decrease** coverage by `6.78%`.
   > The diff coverage is `64.42%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7286/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #7286      +/-   ##
   ============================================
   - Coverage     71.33%   64.55%   -6.79%     
   + Complexity     3363     3297      -66     
   ============================================
     Files          1505     1459      -46     
     Lines         74224    72364    -1860     
     Branches      10733    10536     -197     
   ============================================
   - Hits          52950    46713    -6237     
   - Misses        17678    22314    +4636     
   + Partials       3596     3337     -259     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `?` | |
   | unittests1 | `68.99% <64.42%> (-0.04%)` | :arrow_down: |
   | unittests2 | `14.59% <0.00%> (-0.02%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...t/local/segment/store/SegmentLocalFSDirectory.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L3N0b3JlL1NlZ21lbnRMb2NhbEZTRGlyZWN0b3J5LmphdmE=) | `63.75% <0.00%> (-5.09%)` | :arrow_down: |
   | [.../pinot/segment/spi/store/ColumnIndexDirectory.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL3N0b3JlL0NvbHVtbkluZGV4RGlyZWN0b3J5LmphdmE=) | `40.00% <0.00%> (-26.67%)` | :arrow_down: |
   | [...ache/pinot/segment/spi/store/SegmentDirectory.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL3N0b3JlL1NlZ21lbnREaXJlY3RvcnkuamF2YQ==) | `83.33% <ø> (ø)` | |
   | [...ocal/segment/index/loader/SegmentPreProcessor.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9TZWdtZW50UHJlUHJvY2Vzc29yLmphdmE=) | `74.28% <69.62%> (+7.14%)` | :arrow_up: |
   | [...indexsegment/immutable/ImmutableSegmentLoader.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pbmRleHNlZ21lbnQvaW1tdXRhYmxlL0ltbXV0YWJsZVNlZ21lbnRMb2FkZXIuamF2YQ==) | `91.17% <100.00%> (+0.26%)` | :arrow_up: |
   | [...a/org/apache/pinot/common/metrics/MinionMeter.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25NZXRlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...g/apache/pinot/common/metrics/ControllerMeter.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Db250cm9sbGVyTWV0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../apache/pinot/common/metrics/BrokerQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Ccm9rZXJRdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../apache/pinot/common/metrics/MinionQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25RdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...pache/pinot/common/utils/grpc/GrpcQueryClient.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvZ3JwYy9HcnBjUXVlcnlDbGllbnQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [353 more](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [d40e2d3...1a3a03f](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter edited a comment on pull request #7286: Remove obsolete indices during segment reloading for V3 format (wrap up)

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7286:
URL: https://github.com/apache/pinot/pull/7286#issuecomment-897062223


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7286](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (4ad428a) into [master](https://codecov.io/gh/apache/pinot/commit/bb34406aaa205f5c85b88c928d477fd267eda1b4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bb34406) will **decrease** coverage by `34.24%`.
   > The diff coverage is `0.00%`.
   
   > :exclamation: Current head 4ad428a differs from pull request most recent head 9dc4307. Consider uploading reports for the commit 9dc4307 to get more accurate results
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7286/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@              Coverage Diff              @@
   ##             master    #7286       +/-   ##
   =============================================
   - Coverage     64.65%   30.40%   -34.25%     
   + Complexity     3303       87     -3216     
   =============================================
     Files          1496     1496               
     Lines         74289    74336       +47     
     Branches      10795    10813       +18     
   =============================================
   - Hits          48033    22605    -25428     
   - Misses        22826    49728    +26902     
   + Partials       3430     2003     -1427     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `30.40% <0.00%> (-0.03%)` | :arrow_down: |
   | integration2 | `?` | |
   | unittests1 | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...indexsegment/immutable/ImmutableSegmentLoader.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pbmRleHNlZ21lbnQvaW1tdXRhYmxlL0ltbXV0YWJsZVNlZ21lbnRMb2FkZXIuamF2YQ==) | `0.00% <0.00%> (-91.18%)` | :arrow_down: |
   | [...ocal/segment/index/loader/SegmentPreProcessor.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9TZWdtZW50UHJlUHJvY2Vzc29yLmphdmE=) | `0.00% <0.00%> (-68.00%)` | :arrow_down: |
   | [...t/index/loader/bloomfilter/BloomFilterHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9ibG9vbWZpbHRlci9CbG9vbUZpbHRlckhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (-17.86%)` | :arrow_down: |
   | [...loader/defaultcolumn/BaseDefaultColumnHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9kZWZhdWx0Y29sdW1uL0Jhc2VEZWZhdWx0Q29sdW1uSGFuZGxlci5qYXZh) | `0.00% <0.00%> (-66.05%)` | :arrow_down: |
   | [...x/loader/defaultcolumn/V1DefaultColumnHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9kZWZhdWx0Y29sdW1uL1YxRGVmYXVsdENvbHVtbkhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (-66.67%)` | :arrow_down: |
   | [...x/loader/defaultcolumn/V3DefaultColumnHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9kZWZhdWx0Y29sdW1uL1YzRGVmYXVsdENvbHVtbkhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (-54.55%)` | :arrow_down: |
   | [...ent/index/loader/invertedindex/H3IndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0gzSW5kZXhIYW5kbGVyLmphdmE=) | `0.00% <0.00%> (-67.75%)` | :arrow_down: |
   | [...dex/loader/invertedindex/InvertedIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0ludmVydGVkSW5kZXhIYW5kbGVyLmphdmE=) | `0.00% <0.00%> (-93.48%)` | :arrow_down: |
   | [...t/index/loader/invertedindex/JsonIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0pzb25JbmRleEhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (-35.00%)` | :arrow_down: |
   | [...ex/loader/invertedindex/LuceneFSTIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0x1Y2VuZUZTVEluZGV4SGFuZGxlci5qYXZh) | `0.00% <0.00%> (-87.24%)` | :arrow_down: |
   | ... and [882 more](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [bb34406...9dc4307](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter edited a comment on pull request #7286: Remove obsolete indices during segment reloading for V3 format (wrap up)

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7286:
URL: https://github.com/apache/pinot/pull/7286#issuecomment-897062223


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7286](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (61a3dc2) into [master](https://codecov.io/gh/apache/pinot/commit/bb34406aaa205f5c85b88c928d477fd267eda1b4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bb34406) will **decrease** coverage by `2.37%`.
   > The diff coverage is `51.86%`.
   
   > :exclamation: Current head 61a3dc2 differs from pull request most recent head 9171b8b. Consider uploading reports for the commit 9171b8b to get more accurate results
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7286/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #7286      +/-   ##
   ============================================
   - Coverage     64.65%   62.27%   -2.38%     
   - Complexity     3303     3338      +35     
   ============================================
     Files          1496     1499       +3     
     Lines         74289    74451     +162     
     Branches      10795    10826      +31     
   ============================================
   - Hits          48033    46368    -1665     
   - Misses        22826    24723    +1897     
   + Partials       3430     3360      -70     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `29.13% <16.27%> (-0.06%)` | :arrow_down: |
   | unittests1 | `69.68% <71.18%> (+0.21%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...t/broker/api/resources/PinotBrokerHealthCheck.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYXBpL3Jlc291cmNlcy9QaW5vdEJyb2tlckhlYWx0aENoZWNrLmphdmE=) | `0.00% <0.00%> (ø)` | |
   | [...org/apache/pinot/broker/queryquota/HitCounter.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcXVlcnlxdW90YS9IaXRDb3VudGVyLmphdmE=) | `67.85% <ø> (-1.11%)` | :arrow_down: |
   | [...che/pinot/broker/queryquota/MaxHitRateTracker.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcXVlcnlxdW90YS9NYXhIaXRSYXRlVHJhY2tlci5qYXZh) | `30.00% <ø> (-6.37%)` | :arrow_down: |
   | [...roker/requesthandler/BaseBrokerRequestHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcmVxdWVzdGhhbmRsZXIvQmFzZUJyb2tlclJlcXVlc3RIYW5kbGVyLmphdmE=) | `65.75% <ø> (-0.51%)` | :arrow_down: |
   | [.../routing/segmentpruner/PartitionSegmentPruner.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9zZWdtZW50cHJ1bmVyL1BhcnRpdGlvblNlZ21lbnRQcnVuZXIuamF2YQ==) | `0.00% <0.00%> (-47.66%)` | :arrow_down: |
   | [...roker/routing/segmentpruner/TimeSegmentPruner.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9zZWdtZW50cHJ1bmVyL1RpbWVTZWdtZW50UHJ1bmVyLmphdmE=) | `0.00% <0.00%> (ø)` | |
   | [...r/routing/segmentpruner/interval/IntervalTree.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9zZWdtZW50cHJ1bmVyL2ludGVydmFsL0ludGVydmFsVHJlZS5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [.../pinot/common/messages/RunPeriodicTaskMessage.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWVzc2FnZXMvUnVuUGVyaW9kaWNUYXNrTWVzc2FnZS5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...ache/pinot/controller/api/resources/Constants.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9hcGkvcmVzb3VyY2VzL0NvbnN0YW50cy5qYXZh) | `23.07% <ø> (ø)` | |
   | [...es/PinotControllerPeriodicTaskRestletResource.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9hcGkvcmVzb3VyY2VzL1Bpbm90Q29udHJvbGxlclBlcmlvZGljVGFza1Jlc3RsZXRSZXNvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
   | ... and [170 more](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [bb34406...9171b8b](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter edited a comment on pull request #7286: Remove obsolete indices during segment reloading for V3 format (wrap up)

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7286:
URL: https://github.com/apache/pinot/pull/7286#issuecomment-897062223






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] klsince commented on a change in pull request #7286: Remove obsolete indices during segment reloading for V3 format (wrap up)

Posted by GitBox <gi...@apache.org>.
klsince commented on a change in pull request #7286:
URL: https://github.com/apache/pinot/pull/7286#discussion_r698836449



##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/IndexHandlerFactory.java
##########
@@ -0,0 +1,63 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import java.io.File;
+import org.apache.pinot.segment.local.segment.index.loader.bloomfilter.BloomFilterHandler;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.H3IndexHandler;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.InvertedIndexHandler;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.JsonIndexHandler;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.LuceneFSTIndexHandler;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.RangeIndexHandler;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.TextIndexHandler;
+import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+
+
+public class IndexHandlerFactory {
+  private IndexHandlerFactory() {
+  }
+
+  private static final IndexHandler DEFAULT_HANDLER = () -> {
+    // noop by default.
+  };
+
+  public static IndexHandler newHandler(ColumnIndexType type, File indexDir, SegmentMetadataImpl segmentMetadata,

Review comment:
       fine to me if it's consistent to existing convention. I usually call it newXX() if it should return a newly created object; getXX() to return object that can be reused.

##########
File path: pinot-segment-local/src/test/resources/data/newColumnsSchema3.json
##########
@@ -34,10 +34,6 @@
       "name": "newBooleanSVDimension",
       "defaultNullValue": "false"
     },
-    {

Review comment:
       to test the logic to remove auto-gen'ed default column. 

##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/InvertedIndexHandler.java
##########
@@ -84,7 +92,7 @@ private void createInvertedIndexForColumn(ColumnMetadata columnMetadata)
       if (_segmentWriter.hasIndexFor(column, ColumnIndexType.INVERTED_INDEX)) {

Review comment:
       yeah, seems able to. will update those places. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter edited a comment on pull request #7286: Remove obsolete indices during segment reloading for V3 format (wrap up)

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7286:
URL: https://github.com/apache/pinot/pull/7286#issuecomment-897062223


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7286](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9dc4307) into [master](https://codecov.io/gh/apache/pinot/commit/bb34406aaa205f5c85b88c928d477fd267eda1b4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bb34406) will **increase** coverage by `5.02%`.
   > The diff coverage is `90.38%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7286/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #7286      +/-   ##
   ============================================
   + Coverage     64.65%   69.68%   +5.02%     
   + Complexity     3303     3254      -49     
   ============================================
     Files          1496     1113     -383     
     Lines         74289    52705   -21584     
     Branches      10795     7918    -2877     
   ============================================
   - Hits          48033    36726   -11307     
   + Misses        22826    13368    -9458     
   + Partials       3430     2611     -819     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `?` | |
   | unittests1 | `69.68% <90.38%> (+0.21%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...t/index/loader/invertedindex/JsonIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0pzb25JbmRleEhhbmRsZXIuamF2YQ==) | `75.00% <72.72%> (+40.00%)` | :arrow_up: |
   | [...x/loader/defaultcolumn/V3DefaultColumnHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9kZWZhdWx0Y29sdW1uL1YzRGVmYXVsdENvbHVtbkhhbmRsZXIuamF2YQ==) | `90.90% <86.66%> (+36.36%)` | :arrow_up: |
   | [...t/index/loader/bloomfilter/BloomFilterHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9ibG9vbWZpbHRlci9CbG9vbUZpbHRlckhhbmRsZXIuamF2YQ==) | `74.60% <88.88%> (+56.74%)` | :arrow_up: |
   | [...ex/loader/invertedindex/LuceneFSTIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0x1Y2VuZUZTVEluZGV4SGFuZGxlci5qYXZh) | `85.18% <88.88%> (-2.05%)` | :arrow_down: |
   | [.../index/loader/invertedindex/RangeIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L1JhbmdlSW5kZXhIYW5kbGVyLmphdmE=) | `41.58% <88.88%> (+32.00%)` | :arrow_up: |
   | [...t/index/loader/invertedindex/TextIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L1RleHRJbmRleEhhbmRsZXIuamF2YQ==) | `89.83% <88.88%> (-2.48%)` | :arrow_down: |
   | [...ent/index/loader/invertedindex/H3IndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0gzSW5kZXhIYW5kbGVyLmphdmE=) | `71.42% <90.00%> (+3.68%)` | :arrow_up: |
   | [...indexsegment/immutable/ImmutableSegmentLoader.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pbmRleHNlZ21lbnQvaW1tdXRhYmxlL0ltbXV0YWJsZVNlZ21lbnRMb2FkZXIuamF2YQ==) | `91.30% <100.00%> (+0.12%)` | :arrow_up: |
   | [...ocal/segment/index/loader/SegmentPreProcessor.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9TZWdtZW50UHJlUHJvY2Vzc29yLmphdmE=) | `67.14% <100.00%> (-0.86%)` | :arrow_down: |
   | [...loader/defaultcolumn/BaseDefaultColumnHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9kZWZhdWx0Y29sdW1uL0Jhc2VEZWZhdWx0Q29sdW1uSGFuZGxlci5qYXZh) | `67.60% <100.00%> (+1.55%)` | :arrow_up: |
   | ... and [611 more](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [bb34406...9dc4307](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] klsince commented on a change in pull request #7286: Remove obsolete indices during segment reloading for V3 format (wrap up)

Posted by GitBox <gi...@apache.org>.
klsince commented on a change in pull request #7286:
URL: https://github.com/apache/pinot/pull/7286#discussion_r696881763



##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/bloomfilter/BloomFilterHandler.java
##########
@@ -67,12 +67,20 @@ public BloomFilterHandler(File indexDir, SegmentMetadataImpl segmentMetadata, In
     _segmentVersion = segmentMetadata.getVersion();
     _bloomFilterConfigs = indexLoadingConfig.getBloomFilterConfigs();
 
-    for (String column : _bloomFilterConfigs.keySet()) {
+    Set<String> columnsInCfg = _bloomFilterConfigs.keySet();
+    for (String column : columnsInCfg) {
       ColumnMetadata columnMetadata = segmentMetadata.getColumnMetadataFor(column);
       if (columnMetadata != null) {
         _bloomFilterColumns.add(columnMetadata);
       }
     }
+    // Remove indices not set in table config any more
+    Set<String> localColumns = _segmentWriter.toSegmentDirectory().getColumnsWithIndex(ColumnIndexType.BLOOM_FILTER);
+    for (String column : localColumns) {
+      if (!columnsInCfg.contains(column)) {
+        _segmentWriter.removeIndex(column, ColumnIndexType.BLOOM_FILTER);
+      }
+    }

Review comment:
       in fact, the createXX() method already checks the existence so the index won't be created twice on disk. I personally feel the new way is a bit less readable, mainly due to making a copy of Set and use Set.remove(), but lemme know your thoughts. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter edited a comment on pull request #7286: Remove obsolete indices during segment reloading for V3 format (wrap up)

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7286:
URL: https://github.com/apache/pinot/pull/7286#issuecomment-897062223


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7286](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7253003) into [master](https://codecov.io/gh/apache/pinot/commit/c62a1fb328f1b151fa4a7a4a6ef17875662d6059?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (c62a1fb) will **decrease** coverage by `1.09%`.
   > The diff coverage is `87.54%`.
   
   > :exclamation: Current head 7253003 differs from pull request most recent head bec81d9. Consider uploading reports for the commit bec81d9 to get more accurate results
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7286/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #7286      +/-   ##
   ============================================
   - Coverage     71.65%   70.56%   -1.10%     
   - Complexity     3311     3346      +35     
   ============================================
     Files          1515     1516       +1     
     Lines         75006    74992      -14     
     Branches      10910    10914       +4     
   ============================================
   - Hits          53749    52919     -830     
   - Misses        17628    18448     +820     
   + Partials       3629     3625       -4     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `29.08% <19.37%> (-0.03%)` | :arrow_down: |
   | unittests1 | `69.68% <85.81%> (+0.16%)` | :arrow_up: |
   | unittests2 | `14.50% <2.07%> (+0.02%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...rg/apache/pinot/core/minion/RawIndexConverter.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9taW5pb24vUmF3SW5kZXhDb252ZXJ0ZXIuamF2YQ==) | `0.00% <0.00%> (-56.61%)` | :arrow_down: |
   | [...not/common/metadata/segment/SegmentZKMetadata.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0YWRhdGEvc2VnbWVudC9TZWdtZW50WktNZXRhZGF0YS5qYXZh) | `73.96% <57.14%> (+1.23%)` | :arrow_up: |
   | [...ache/pinot/core/data/table/SimpleIndexedTable.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL3RhYmxlL1NpbXBsZUluZGV4ZWRUYWJsZS5qYXZh) | `92.30% <66.66%> (-2.52%)` | :arrow_down: |
   | [...org/apache/pinot/core/data/table/IndexedTable.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL3RhYmxlL0luZGV4ZWRUYWJsZS5qYXZh) | `79.48% <70.00%> (-9.99%)` | :arrow_down: |
   | [.../pinot/core/data/table/ConcurrentIndexedTable.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL3RhYmxlL0NvbmN1cnJlbnRJbmRleGVkVGFibGUuamF2YQ==) | `81.81% <80.00%> (-7.74%)` | :arrow_down: |
   | [...t/index/loader/bloomfilter/BloomFilterHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9ibG9vbWZpbHRlci9CbG9vbUZpbHRlckhhbmRsZXIuamF2YQ==) | `77.58% <84.21%> (+59.72%)` | :arrow_up: |
   | [...t/index/loader/invertedindex/JsonIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0pzb25JbmRleEhhbmRsZXIuamF2YQ==) | `74.60% <85.00%> (+39.60%)` | :arrow_up: |
   | [...ent/index/loader/invertedindex/H3IndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0gzSW5kZXhIYW5kbGVyLmphdmE=) | `72.58% <85.71%> (+4.83%)` | :arrow_up: |
   | [...x/loader/defaultcolumn/V3DefaultColumnHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9kZWZhdWx0Y29sdW1uL1YzRGVmYXVsdENvbHVtbkhhbmRsZXIuamF2YQ==) | `90.90% <86.66%> (+36.36%)` | :arrow_up: |
   | [.../index/loader/invertedindex/RangeIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L1JhbmdlSW5kZXhIYW5kbGVyLmphdmE=) | `41.66% <88.88%> (+32.09%)` | :arrow_up: |
   | ... and [141 more](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [c62a1fb...bec81d9](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter edited a comment on pull request #7286: Remove obsolete indices during segment reloading for V3 format (wrap up)

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7286:
URL: https://github.com/apache/pinot/pull/7286#issuecomment-897062223


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7286](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (4ad428a) into [master](https://codecov.io/gh/apache/pinot/commit/bb34406aaa205f5c85b88c928d477fd267eda1b4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bb34406) will **decrease** coverage by `32.35%`.
   > The diff coverage is `0.00%`.
   
   > :exclamation: Current head 4ad428a differs from pull request most recent head 9dc4307. Consider uploading reports for the commit 9dc4307 to get more accurate results
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7286/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@              Coverage Diff              @@
   ##             master    #7286       +/-   ##
   =============================================
   - Coverage     64.65%   32.30%   -32.36%     
   + Complexity     3303       92     -3211     
   =============================================
     Files          1496     1496               
     Lines         74289    74336       +47     
     Branches      10795    10813       +18     
   =============================================
   - Hits          48033    24012    -24021     
   - Misses        22826    48262    +25436     
   + Partials       3430     2062     -1368     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `30.40% <0.00%> (-0.03%)` | :arrow_down: |
   | integration2 | `29.25% <0.00%> (+0.07%)` | :arrow_up: |
   | unittests1 | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...indexsegment/immutable/ImmutableSegmentLoader.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pbmRleHNlZ21lbnQvaW1tdXRhYmxlL0ltbXV0YWJsZVNlZ21lbnRMb2FkZXIuamF2YQ==) | `0.00% <0.00%> (-91.18%)` | :arrow_down: |
   | [...ocal/segment/index/loader/SegmentPreProcessor.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9TZWdtZW50UHJlUHJvY2Vzc29yLmphdmE=) | `0.00% <0.00%> (-68.00%)` | :arrow_down: |
   | [...t/index/loader/bloomfilter/BloomFilterHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9ibG9vbWZpbHRlci9CbG9vbUZpbHRlckhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (-17.86%)` | :arrow_down: |
   | [...loader/defaultcolumn/BaseDefaultColumnHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9kZWZhdWx0Y29sdW1uL0Jhc2VEZWZhdWx0Q29sdW1uSGFuZGxlci5qYXZh) | `0.00% <0.00%> (-66.05%)` | :arrow_down: |
   | [...x/loader/defaultcolumn/V1DefaultColumnHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9kZWZhdWx0Y29sdW1uL1YxRGVmYXVsdENvbHVtbkhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (-66.67%)` | :arrow_down: |
   | [...x/loader/defaultcolumn/V3DefaultColumnHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9kZWZhdWx0Y29sdW1uL1YzRGVmYXVsdENvbHVtbkhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (-54.55%)` | :arrow_down: |
   | [...ent/index/loader/invertedindex/H3IndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0gzSW5kZXhIYW5kbGVyLmphdmE=) | `0.00% <0.00%> (-67.75%)` | :arrow_down: |
   | [...dex/loader/invertedindex/InvertedIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0ludmVydGVkSW5kZXhIYW5kbGVyLmphdmE=) | `0.00% <0.00%> (-93.48%)` | :arrow_down: |
   | [...t/index/loader/invertedindex/JsonIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0pzb25JbmRleEhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (-35.00%)` | :arrow_down: |
   | [...ex/loader/invertedindex/LuceneFSTIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0x1Y2VuZUZTVEluZGV4SGFuZGxlci5qYXZh) | `0.00% <0.00%> (-87.24%)` | :arrow_down: |
   | ... and [813 more](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [bb34406...9dc4307](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter edited a comment on pull request #7286: Remove obsolete indices during segment reloading for V3 format (wrap up)

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7286:
URL: https://github.com/apache/pinot/pull/7286#issuecomment-897062223


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7286](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (44fcfbe) into [master](https://codecov.io/gh/apache/pinot/commit/2d1f7dc9060d45ac9f4ff0621da62790ea9b8e64?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2d1f7dc) will **decrease** coverage by `39.55%`.
   > The diff coverage is `0.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7286/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@              Coverage Diff              @@
   ##             master    #7286       +/-   ##
   =============================================
   - Coverage     71.65%   32.10%   -39.56%     
   + Complexity     3305       92     -3213     
   =============================================
     Files          1515     1506        -9     
     Lines         74992    74690      -302     
     Branches      10903    10880       -23     
   =============================================
   - Hits          53736    23978    -29758     
   - Misses        17629    48646    +31017     
   + Partials       3627     2066     -1561     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `30.27% <0.00%> (-0.10%)` | :arrow_down: |
   | integration2 | `29.04% <0.00%> (-0.04%)` | :arrow_down: |
   | unittests1 | `?` | |
   | unittests2 | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...indexsegment/immutable/ImmutableSegmentLoader.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pbmRleHNlZ21lbnQvaW1tdXRhYmxlL0ltbXV0YWJsZVNlZ21lbnRMb2FkZXIuamF2YQ==) | `0.00% <0.00%> (-91.18%)` | :arrow_down: |
   | [...ocal/segment/index/loader/SegmentPreProcessor.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9TZWdtZW50UHJlUHJvY2Vzc29yLmphdmE=) | `0.00% <0.00%> (-68.00%)` | :arrow_down: |
   | [...t/index/loader/bloomfilter/BloomFilterHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9ibG9vbWZpbHRlci9CbG9vbUZpbHRlckhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (-17.86%)` | :arrow_down: |
   | [...loader/defaultcolumn/BaseDefaultColumnHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9kZWZhdWx0Y29sdW1uL0Jhc2VEZWZhdWx0Q29sdW1uSGFuZGxlci5qYXZh) | `0.00% <0.00%> (-66.05%)` | :arrow_down: |
   | [...x/loader/defaultcolumn/V1DefaultColumnHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9kZWZhdWx0Y29sdW1uL1YxRGVmYXVsdENvbHVtbkhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (-66.67%)` | :arrow_down: |
   | [...x/loader/defaultcolumn/V3DefaultColumnHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9kZWZhdWx0Y29sdW1uL1YzRGVmYXVsdENvbHVtbkhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (-54.55%)` | :arrow_down: |
   | [...ent/index/loader/invertedindex/H3IndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0gzSW5kZXhIYW5kbGVyLmphdmE=) | `0.00% <0.00%> (-67.75%)` | :arrow_down: |
   | [...dex/loader/invertedindex/InvertedIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0ludmVydGVkSW5kZXhIYW5kbGVyLmphdmE=) | `0.00% <0.00%> (-93.48%)` | :arrow_down: |
   | [...t/index/loader/invertedindex/JsonIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0pzb25JbmRleEhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (-35.00%)` | :arrow_down: |
   | [...ex/loader/invertedindex/LuceneFSTIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0x1Y2VuZUZTVEluZGV4SGFuZGxlci5qYXZh) | `0.00% <0.00%> (-87.24%)` | :arrow_down: |
   | ... and [999 more](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [2d1f7dc...44fcfbe](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter edited a comment on pull request #7286: Remove obsolete indices during segment reloading for V3 format (wrap up)

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7286:
URL: https://github.com/apache/pinot/pull/7286#issuecomment-897062223


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7286](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (604e6ab) into [master](https://codecov.io/gh/apache/pinot/commit/bb34406aaa205f5c85b88c928d477fd267eda1b4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bb34406) will **increase** coverage by `5.08%`.
   > The diff coverage is `90.38%`.
   
   > :exclamation: Current head 604e6ab differs from pull request most recent head 726f88f. Consider uploading reports for the commit 726f88f to get more accurate results
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7286/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #7286      +/-   ##
   ============================================
   + Coverage     64.65%   69.74%   +5.08%     
   + Complexity     3303     3258      -45     
   ============================================
     Files          1496     1113     -383     
     Lines         74289    52705   -21584     
     Branches      10795     7918    -2877     
   ============================================
   - Hits          48033    36758   -11275     
   + Misses        22826    13341    -9485     
   + Partials       3430     2606     -824     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `?` | |
   | unittests1 | `69.74% <90.38%> (+0.27%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...t/index/loader/invertedindex/JsonIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0pzb25JbmRleEhhbmRsZXIuamF2YQ==) | `75.00% <72.72%> (+40.00%)` | :arrow_up: |
   | [...x/loader/defaultcolumn/V3DefaultColumnHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9kZWZhdWx0Y29sdW1uL1YzRGVmYXVsdENvbHVtbkhhbmRsZXIuamF2YQ==) | `90.90% <86.66%> (+36.36%)` | :arrow_up: |
   | [...t/index/loader/bloomfilter/BloomFilterHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9ibG9vbWZpbHRlci9CbG9vbUZpbHRlckhhbmRsZXIuamF2YQ==) | `74.60% <88.88%> (+56.74%)` | :arrow_up: |
   | [...ex/loader/invertedindex/LuceneFSTIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0x1Y2VuZUZTVEluZGV4SGFuZGxlci5qYXZh) | `85.18% <88.88%> (-2.05%)` | :arrow_down: |
   | [.../index/loader/invertedindex/RangeIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L1JhbmdlSW5kZXhIYW5kbGVyLmphdmE=) | `41.58% <88.88%> (+32.00%)` | :arrow_up: |
   | [...t/index/loader/invertedindex/TextIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L1RleHRJbmRleEhhbmRsZXIuamF2YQ==) | `89.83% <88.88%> (-2.48%)` | :arrow_down: |
   | [...ent/index/loader/invertedindex/H3IndexHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0gzSW5kZXhIYW5kbGVyLmphdmE=) | `71.42% <90.00%> (+3.68%)` | :arrow_up: |
   | [...indexsegment/immutable/ImmutableSegmentLoader.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pbmRleHNlZ21lbnQvaW1tdXRhYmxlL0ltbXV0YWJsZVNlZ21lbnRMb2FkZXIuamF2YQ==) | `91.30% <100.00%> (+0.12%)` | :arrow_up: |
   | [...ocal/segment/index/loader/SegmentPreProcessor.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9TZWdtZW50UHJlUHJvY2Vzc29yLmphdmE=) | `67.14% <100.00%> (-0.86%)` | :arrow_down: |
   | [...loader/defaultcolumn/BaseDefaultColumnHandler.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9kZWZhdWx0Y29sdW1uL0Jhc2VEZWZhdWx0Q29sdW1uSGFuZGxlci5qYXZh) | `67.60% <100.00%> (+1.55%)` | :arrow_up: |
   | ... and [613 more](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [bb34406...726f88f](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang merged pull request #7286: Remove obsolete indices during segment reloading for V3 format (wrap up)

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang merged pull request #7286:
URL: https://github.com/apache/pinot/pull/7286


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter edited a comment on pull request #7286: Remove obsolete indices during segment reloading for V3 format (wrap up)

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7286:
URL: https://github.com/apache/pinot/pull/7286#issuecomment-897062223






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] klsince commented on a change in pull request #7286: Remove obsolete indices during segment reloading for V3 format (wrap up)

Posted by GitBox <gi...@apache.org>.
klsince commented on a change in pull request #7286:
URL: https://github.com/apache/pinot/pull/7286#discussion_r695259138



##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/immutable/ImmutableSegmentLoader.java
##########
@@ -57,25 +57,40 @@ private ImmutableSegmentLoader() {
   private static final Logger LOGGER = LoggerFactory.getLogger(ImmutableSegmentLoader.class);
 
   /**
-   * For tests only.
+   * load with empty schema and IndexLoadingConfig is used to get a handler of the segment for
+   * its metadata and any other existing info inside the segment folder. This method is used by
+   * MultiTreeBuilder (with empty schema and IndexLoadingConfig to avoid recursion) and tools.
+   * No need to cleanup indices while calling this load method.
    */
   public static ImmutableSegment load(File indexDir, ReadMode readMode)
       throws Exception {
     IndexLoadingConfig defaultIndexLoadingConfig = new IndexLoadingConfig();
     defaultIndexLoadingConfig.setReadMode(readMode);
-    return load(indexDir, defaultIndexLoadingConfig, null);
+    return load(indexDir, defaultIndexLoadingConfig, null, false);
   }
 
   /**
-   * For tests only.
+   * load with empty schema but a specified IndexLoadingConfig (mostly empty) is also used
+   * to get a handler of the current segment, e.g. used in RawIndexConverter and test cases.
+   * No need to cleanup indices while calling this load method.
    */
   public static ImmutableSegment load(File indexDir, IndexLoadingConfig indexLoadingConfig)
       throws Exception {
-    return load(indexDir, indexLoadingConfig, null);
+    return load(indexDir, indexLoadingConfig, null, false);
   }
 
+  /**
+   * load with specified schema and IndexLoadingConfig and clean up obsolete indices
+   * according to the IndexLoadingConfig.
+   */
   public static ImmutableSegment load(File indexDir, IndexLoadingConfig indexLoadingConfig, @Nullable Schema schema)
       throws Exception {
+    return load(indexDir, indexLoadingConfig, schema, true);
+  }
+
+  private static ImmutableSegment load(File indexDir, IndexLoadingConfig indexLoadingConfig, @Nullable Schema schema,
+      boolean cleanupIndices)

Review comment:
       sgtm




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] klsince commented on a change in pull request #7286: Remove obsolete indices during segment reloading for V3 format (wrap up)

Posted by GitBox <gi...@apache.org>.
klsince commented on a change in pull request #7286:
URL: https://github.com/apache/pinot/pull/7286#discussion_r698836449



##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/IndexHandlerFactory.java
##########
@@ -0,0 +1,63 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import java.io.File;
+import org.apache.pinot.segment.local.segment.index.loader.bloomfilter.BloomFilterHandler;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.H3IndexHandler;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.InvertedIndexHandler;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.JsonIndexHandler;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.LuceneFSTIndexHandler;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.RangeIndexHandler;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.TextIndexHandler;
+import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+
+
+public class IndexHandlerFactory {
+  private IndexHandlerFactory() {
+  }
+
+  private static final IndexHandler DEFAULT_HANDLER = () -> {
+    // noop by default.
+  };
+
+  public static IndexHandler newHandler(ColumnIndexType type, File indexDir, SegmentMetadataImpl segmentMetadata,

Review comment:
       fine to me if it's consistent to existing convention. I usually call it newXX() if it should return a newly created object; getXX() to return object that can be reused.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #7286: Remove obsolete indices during segment reloading for V3 format (wrap up)

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #7286:
URL: https://github.com/apache/pinot/pull/7286#discussion_r698817955



##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/defaultcolumn/V3DefaultColumnHandler.java
##########
@@ -47,39 +45,34 @@ protected boolean updateDefaultColumn(String column, DefaultColumnAction action)
       throws Exception {
     LOGGER.info("Starting default column action: {} on column: {}", action, column);
 
-    // For V3 segment format, only support ADD action
-    // For UPDATE and REMOVE action, throw exception to drop and re-download the segment
-    if (action.isUpdateAction()) {
-      throw new V3UpdateIndexException(
-          "Default value indices for column: " + column + " cannot be updated for V3 format segment.");
+    // For UPDATE and REMOVE action, delete existing dictionary and forward index, and remove column metadata
+    if (action.isUpdateAction() || action.isRemoveAction()) {
+      removeColumnIndices(column);
     }
-
+    // For REMOVE action, no more to do. For ADD and UPDATE action, need to create

Review comment:
       (nit) Move this into the previous if branch

##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/InvertedIndexHandler.java
##########
@@ -84,7 +92,7 @@ private void createInvertedIndexForColumn(ColumnMetadata columnMetadata)
       if (_segmentWriter.hasIndexFor(column, ColumnIndexType.INVERTED_INDEX)) {

Review comment:
       Remove this check? Same for other handlers

##########
File path: pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/OfflineClusterIntegrationTest.java
##########
@@ -327,7 +326,7 @@ public void testUploadSameSegments()
   }
 
   @Test
-  public void testInvertedIndexTriggering()
+  public void testInvertedIndexTriggeringCleanIndicesOnReloading()

Review comment:
       (nit) Suggest still keeping one test, and add/remove indices twice since the logic is very similar

##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/IndexHandlerFactory.java
##########
@@ -0,0 +1,63 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import java.io.File;
+import org.apache.pinot.segment.local.segment.index.loader.bloomfilter.BloomFilterHandler;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.H3IndexHandler;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.InvertedIndexHandler;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.JsonIndexHandler;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.LuceneFSTIndexHandler;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.RangeIndexHandler;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.TextIndexHandler;
+import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+
+
+public class IndexHandlerFactory {
+  private IndexHandlerFactory() {
+  }
+
+  private static final IndexHandler DEFAULT_HANDLER = () -> {
+    // noop by default.
+  };
+
+  public static IndexHandler newHandler(ColumnIndexType type, File indexDir, SegmentMetadataImpl segmentMetadata,

Review comment:
       We usually name the method `getIndexHandler()` for factory.

##########
File path: pinot-segment-local/src/test/resources/data/newColumnsSchema3.json
##########
@@ -34,10 +34,6 @@
       "name": "newBooleanSVDimension",
       "defaultNullValue": "false"
     },
-    {

Review comment:
       Why removing this field?

##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/IndexHandler.java
##########
@@ -0,0 +1,27 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+public interface IndexHandler {
+  /**
+   * Add new indices and remove obsolete indices.

Review comment:
       (nit)
   ```suggestion
      * Adds new indices and removes obsolete indices.
   ```

##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/JsonIndexHandler.java
##########
@@ -88,7 +95,7 @@ private void createJsonIndexForColumn(ColumnMetadata columnMetadata)
       if (_segmentWriter.hasIndexFor(columnName, ColumnIndexType.JSON_INDEX)) {

Review comment:
       We should be able to remove this check?

##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/immutable/ImmutableSegmentLoader.java
##########
@@ -57,55 +57,58 @@ private ImmutableSegmentLoader() {
   private static final Logger LOGGER = LoggerFactory.getLogger(ImmutableSegmentLoader.class);
 
   /**
-   * For tests only.
+   * Loads the segment with empty schema and IndexLoadingConfig. This method is used to
+   * access the segment without modifying it, i.e. in read-only mode.
    */
   public static ImmutableSegment load(File indexDir, ReadMode readMode)
       throws Exception {
     IndexLoadingConfig defaultIndexLoadingConfig = new IndexLoadingConfig();
     defaultIndexLoadingConfig.setReadMode(readMode);
-    return load(indexDir, defaultIndexLoadingConfig, null);
+    return load(indexDir, defaultIndexLoadingConfig, null, false);
   }
 
   /**
-   * For tests only.
+   * Loads the segment with empty schema but a specified IndexLoadingConfig (mostly empty).

Review comment:
       Remove the `mostly empty`? We don't have this assumption for this method

##########
File path: pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/OfflineClusterIntegrationTest.java
##########
@@ -35,7 +35,6 @@
 import java.util.List;

Review comment:
       We should be able to remove the segment version override in this test and test the v3 default column removal

##########
File path: pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/OfflineClusterIntegrationTest.java
##########
@@ -362,56 +361,99 @@ public void testInvertedIndexTriggering()
     long tableSizeWithNewIndex = getTableSize(offlineTableName);
     assertTrue(tableSizeWithNewIndex > tableSizeWithDefaultIndex);
 
-    // Update table config to remove all inverted index.
+    // Update table config to remove the new inverted index, and
+    // reload table to clean the new inverted indices physically.
     tableConfig = getOfflineTableConfig();
-    tableConfig.getIndexingConfig().setInvertedIndexColumns(Collections.emptyList());
+    tableConfig.getIndexingConfig().setInvertedIndexColumns(getInvertedIndexColumns());
     updateTableConfig(tableConfig);
-
-    // Reload table just disables those indices, not clean them physically.
     reloadOfflineTable(offlineTableName);
     TestUtils.waitForCondition(aVoid -> {
       try {
         JsonNode queryResponse2 = postQuery(TEST_UPDATED_INVERTED_INDEX_QUERY);
-        // Total docs should not change during reload
+        // Total docs should not change during reload, but num entries scanned
+        // gets back to total number of documents as the index is removed.
         assertEquals(queryResponse2.get("totalDocs").asLong(), numTotalDocs);
         return queryResponse2.get("numEntriesScannedInFilter").asLong() == numTotalDocs;
       } catch (Exception e) {
         throw new RuntimeException(e);
       }
-    }, 600_000L, "Failed to disable indices");
-    long tableSizeAfterReload = getTableSize(offlineTableName);
-    assertEquals(tableSizeAfterReload, tableSizeWithNewIndex);
+    }, 600_000L, "Failed to cleanup obsolete index");
+    assertEquals(getTableSize(offlineTableName), tableSizeWithDefaultIndex);
+  }
+
+  @Test
+  public void testInvertedIndexTriggeringCleanIndicesWithForceDownload()
+      throws Exception {
+    String offlineTableName = TableNameBuilder.OFFLINE.tableNameWithType(getTableName());

Review comment:
       Let's extract the index addition part of these 2 tests into a common helper method because they are the same

##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/IndexHandler.java
##########
@@ -0,0 +1,27 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+public interface IndexHandler {

Review comment:
       Add some javadoc

##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/IndexHandlerFactory.java
##########
@@ -0,0 +1,63 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import java.io.File;
+import org.apache.pinot.segment.local.segment.index.loader.bloomfilter.BloomFilterHandler;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.H3IndexHandler;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.InvertedIndexHandler;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.JsonIndexHandler;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.LuceneFSTIndexHandler;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.RangeIndexHandler;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.TextIndexHandler;
+import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+
+
+public class IndexHandlerFactory {
+  private IndexHandlerFactory() {
+  }
+
+  private static final IndexHandler DEFAULT_HANDLER = () -> {

Review comment:
       Rename it to `NO_OP_HANDLER`?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] klsince commented on a change in pull request #7286: Remove obsolete indices during segment reloading for V3 format (wrap up)

Posted by GitBox <gi...@apache.org>.
klsince commented on a change in pull request #7286:
URL: https://github.com/apache/pinot/pull/7286#discussion_r696833731



##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/JsonIndexHandler.java
##########
@@ -104,8 +112,9 @@ private void createJsonIndexForColumn(ColumnMetadata columnMetadata)
 
     // Create new json index for the column.
     LOGGER.info("Creating new json index for segment: {}, column: {}", _segmentName, columnName);
-    Preconditions.checkState(columnMetadata.isSingleValue() && columnMetadata.getDataType() == DataType.STRING,
-        "Json index can only be applied to single-value STRING columns");
+    Preconditions.checkState(columnMetadata.isSingleValue() && (columnMetadata.getDataType() == DataType.STRING

Review comment:
       thinking to keep this one here as UT case is easier to add to cover this fix in this PR.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] klsince commented on a change in pull request #7286: Remove obsolete indices during segment reloading for V3 format (wrap up)

Posted by GitBox <gi...@apache.org>.
klsince commented on a change in pull request #7286:
URL: https://github.com/apache/pinot/pull/7286#discussion_r696831890



##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/immutable/ImmutableSegmentLoader.java
##########
@@ -164,6 +168,29 @@ public static ImmutableSegment load(File indexDir, IndexLoadingConfig indexLoadi
     return segment;
   }
 
+  private static void tryToConvertSegment(File indexDir, IndexLoadingConfig indexLoadingConfig,

Review comment:
       will do convertSegmentFormat for short.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter edited a comment on pull request #7286: Cleanup indices in place while reloading segment - interface changes

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7286:
URL: https://github.com/apache/pinot/pull/7286#issuecomment-897062223


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7286](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1a3a03f) into [master](https://codecov.io/gh/apache/pinot/commit/6fb51d2889e5b269e52f483f3cac89f5562a3a62?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6fb51d2) will **decrease** coverage by `0.95%`.
   > The diff coverage is `64.42%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7286/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #7286      +/-   ##
   ============================================
   - Coverage     71.33%   70.38%   -0.96%     
   + Complexity     3363     3297      -66     
   ============================================
     Files          1505     1505              
     Lines         74224    74274      +50     
     Branches      10733    10740       +7     
   ============================================
   - Hits          52950    52279     -671     
   - Misses        17678    18388     +710     
   - Partials       3596     3607      +11     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `30.06% <0.00%> (+0.20%)` | :arrow_up: |
   | integration2 | `?` | |
   | unittests1 | `68.99% <64.42%> (-0.04%)` | :arrow_down: |
   | unittests2 | `14.59% <0.00%> (-0.02%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...t/local/segment/store/SegmentLocalFSDirectory.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L3N0b3JlL1NlZ21lbnRMb2NhbEZTRGlyZWN0b3J5LmphdmE=) | `63.75% <0.00%> (-5.09%)` | :arrow_down: |
   | [.../pinot/segment/spi/store/ColumnIndexDirectory.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL3N0b3JlL0NvbHVtbkluZGV4RGlyZWN0b3J5LmphdmE=) | `40.00% <0.00%> (-26.67%)` | :arrow_down: |
   | [...ache/pinot/segment/spi/store/SegmentDirectory.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL3N0b3JlL1NlZ21lbnREaXJlY3RvcnkuamF2YQ==) | `83.33% <ø> (ø)` | |
   | [...ocal/segment/index/loader/SegmentPreProcessor.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9TZWdtZW50UHJlUHJvY2Vzc29yLmphdmE=) | `74.28% <69.62%> (+7.14%)` | :arrow_up: |
   | [...indexsegment/immutable/ImmutableSegmentLoader.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pbmRleHNlZ21lbnQvaW1tdXRhYmxlL0ltbXV0YWJsZVNlZ21lbnRMb2FkZXIuamF2YQ==) | `91.17% <100.00%> (+0.26%)` | :arrow_up: |
   | [...pache/pinot/common/utils/grpc/GrpcQueryClient.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvZ3JwYy9HcnBjUXVlcnlDbGllbnQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...ore/operator/streaming/StreamingResponseUtils.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9zdHJlYW1pbmcvU3RyZWFtaW5nUmVzcG9uc2VVdGlscy5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...ager/realtime/PeerSchemeSplitSegmentCommitter.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUGVlclNjaGVtZVNwbGl0U2VnbWVudENvbW1pdHRlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...ator/streaming/StreamingSelectionOnlyOperator.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9zdHJlYW1pbmcvU3RyZWFtaW5nU2VsZWN0aW9uT25seU9wZXJhdG9yLmphdmE=) | `0.00% <0.00%> (-92.31%)` | :arrow_down: |
   | [...he/pinot/core/plan/StreamingSelectionPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL1N0cmVhbWluZ1NlbGVjdGlvblBsYW5Ob2RlLmphdmE=) | `0.00% <0.00%> (-90.00%)` | :arrow_down: |
   | ... and [92 more](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [d40e2d3...1a3a03f](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] klsince commented on a change in pull request #7286: Remove obsolete indices during segment reloading for V3 format (wrap up)

Posted by GitBox <gi...@apache.org>.
klsince commented on a change in pull request #7286:
URL: https://github.com/apache/pinot/pull/7286#discussion_r698867283



##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/InvertedIndexHandler.java
##########
@@ -84,7 +92,7 @@ private void createInvertedIndexForColumn(ColumnMetadata columnMetadata)
       if (_segmentWriter.hasIndexFor(column, ColumnIndexType.INVERTED_INDEX)) {

Review comment:
       yeah, seems able to. will update those places. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter commented on pull request #7286: Cleanup indices in place while reloading segment - interface changes

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #7286:
URL: https://github.com/apache/pinot/pull/7286#issuecomment-897062223


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7286](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1a3a03f) into [master](https://codecov.io/gh/apache/pinot/commit/6fb51d2889e5b269e52f483f3cac89f5562a3a62?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6fb51d2) will **decrease** coverage by `2.34%`.
   > The diff coverage is `64.42%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7286/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #7286      +/-   ##
   ============================================
   - Coverage     71.33%   68.99%   -2.35%     
   + Complexity     3363     3205     -158     
   ============================================
     Files          1505     1111     -394     
     Lines         74224    52396   -21828     
     Branches      10733     7819    -2914     
   ============================================
   - Hits          52950    36149   -16801     
   + Misses        17678    13674    -4004     
   + Partials       3596     2573    -1023     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `?` | |
   | unittests1 | `68.99% <64.42%> (-0.04%)` | :arrow_down: |
   | unittests2 | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...t/local/segment/store/SegmentLocalFSDirectory.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L3N0b3JlL1NlZ21lbnRMb2NhbEZTRGlyZWN0b3J5LmphdmE=) | `63.75% <0.00%> (-5.09%)` | :arrow_down: |
   | [.../pinot/segment/spi/store/ColumnIndexDirectory.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL3N0b3JlL0NvbHVtbkluZGV4RGlyZWN0b3J5LmphdmE=) | `40.00% <0.00%> (-26.67%)` | :arrow_down: |
   | [...ache/pinot/segment/spi/store/SegmentDirectory.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL3N0b3JlL1NlZ21lbnREaXJlY3RvcnkuamF2YQ==) | `83.33% <ø> (ø)` | |
   | [...ocal/segment/index/loader/SegmentPreProcessor.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9TZWdtZW50UHJlUHJvY2Vzc29yLmphdmE=) | `74.28% <69.62%> (+7.14%)` | :arrow_up: |
   | [...indexsegment/immutable/ImmutableSegmentLoader.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pbmRleHNlZ21lbnQvaW1tdXRhYmxlL0ltbXV0YWJsZVNlZ21lbnRMb2FkZXIuamF2YQ==) | `91.17% <100.00%> (+0.26%)` | :arrow_up: |
   | [...a/org/apache/pinot/common/metrics/MinionMeter.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25NZXRlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...g/apache/pinot/common/metrics/ControllerMeter.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Db250cm9sbGVyTWV0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../apache/pinot/common/metrics/BrokerQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Ccm9rZXJRdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../apache/pinot/common/metrics/MinionQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25RdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...pache/pinot/common/utils/grpc/GrpcQueryClient.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvZ3JwYy9HcnBjUXVlcnlDbGllbnQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [596 more](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [d40e2d3...1a3a03f](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #7286: Remove obsolete indices during segment reloading for V3 format (wrap up)

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #7286:
URL: https://github.com/apache/pinot/pull/7286#discussion_r698817955



##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/defaultcolumn/V3DefaultColumnHandler.java
##########
@@ -47,39 +45,34 @@ protected boolean updateDefaultColumn(String column, DefaultColumnAction action)
       throws Exception {
     LOGGER.info("Starting default column action: {} on column: {}", action, column);
 
-    // For V3 segment format, only support ADD action
-    // For UPDATE and REMOVE action, throw exception to drop and re-download the segment
-    if (action.isUpdateAction()) {
-      throw new V3UpdateIndexException(
-          "Default value indices for column: " + column + " cannot be updated for V3 format segment.");
+    // For UPDATE and REMOVE action, delete existing dictionary and forward index, and remove column metadata
+    if (action.isUpdateAction() || action.isRemoveAction()) {
+      removeColumnIndices(column);
     }
-
+    // For REMOVE action, no more to do. For ADD and UPDATE action, need to create

Review comment:
       (nit) Move this into the previous if branch

##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/InvertedIndexHandler.java
##########
@@ -84,7 +92,7 @@ private void createInvertedIndexForColumn(ColumnMetadata columnMetadata)
       if (_segmentWriter.hasIndexFor(column, ColumnIndexType.INVERTED_INDEX)) {

Review comment:
       Remove this check? Same for other handlers

##########
File path: pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/OfflineClusterIntegrationTest.java
##########
@@ -327,7 +326,7 @@ public void testUploadSameSegments()
   }
 
   @Test
-  public void testInvertedIndexTriggering()
+  public void testInvertedIndexTriggeringCleanIndicesOnReloading()

Review comment:
       (nit) Suggest still keeping one test, and add/remove indices twice since the logic is very similar

##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/IndexHandlerFactory.java
##########
@@ -0,0 +1,63 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import java.io.File;
+import org.apache.pinot.segment.local.segment.index.loader.bloomfilter.BloomFilterHandler;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.H3IndexHandler;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.InvertedIndexHandler;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.JsonIndexHandler;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.LuceneFSTIndexHandler;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.RangeIndexHandler;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.TextIndexHandler;
+import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+
+
+public class IndexHandlerFactory {
+  private IndexHandlerFactory() {
+  }
+
+  private static final IndexHandler DEFAULT_HANDLER = () -> {
+    // noop by default.
+  };
+
+  public static IndexHandler newHandler(ColumnIndexType type, File indexDir, SegmentMetadataImpl segmentMetadata,

Review comment:
       We usually name the method `getIndexHandler()` for factory.

##########
File path: pinot-segment-local/src/test/resources/data/newColumnsSchema3.json
##########
@@ -34,10 +34,6 @@
       "name": "newBooleanSVDimension",
       "defaultNullValue": "false"
     },
-    {

Review comment:
       Why removing this field?

##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/IndexHandler.java
##########
@@ -0,0 +1,27 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+public interface IndexHandler {
+  /**
+   * Add new indices and remove obsolete indices.

Review comment:
       (nit)
   ```suggestion
      * Adds new indices and removes obsolete indices.
   ```

##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/JsonIndexHandler.java
##########
@@ -88,7 +95,7 @@ private void createJsonIndexForColumn(ColumnMetadata columnMetadata)
       if (_segmentWriter.hasIndexFor(columnName, ColumnIndexType.JSON_INDEX)) {

Review comment:
       We should be able to remove this check?

##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/immutable/ImmutableSegmentLoader.java
##########
@@ -57,55 +57,58 @@ private ImmutableSegmentLoader() {
   private static final Logger LOGGER = LoggerFactory.getLogger(ImmutableSegmentLoader.class);
 
   /**
-   * For tests only.
+   * Loads the segment with empty schema and IndexLoadingConfig. This method is used to
+   * access the segment without modifying it, i.e. in read-only mode.
    */
   public static ImmutableSegment load(File indexDir, ReadMode readMode)
       throws Exception {
     IndexLoadingConfig defaultIndexLoadingConfig = new IndexLoadingConfig();
     defaultIndexLoadingConfig.setReadMode(readMode);
-    return load(indexDir, defaultIndexLoadingConfig, null);
+    return load(indexDir, defaultIndexLoadingConfig, null, false);
   }
 
   /**
-   * For tests only.
+   * Loads the segment with empty schema but a specified IndexLoadingConfig (mostly empty).

Review comment:
       Remove the `mostly empty`? We don't have this assumption for this method

##########
File path: pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/OfflineClusterIntegrationTest.java
##########
@@ -35,7 +35,6 @@
 import java.util.List;

Review comment:
       We should be able to remove the segment version override in this test and test the v3 default column removal

##########
File path: pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/OfflineClusterIntegrationTest.java
##########
@@ -362,56 +361,99 @@ public void testInvertedIndexTriggering()
     long tableSizeWithNewIndex = getTableSize(offlineTableName);
     assertTrue(tableSizeWithNewIndex > tableSizeWithDefaultIndex);
 
-    // Update table config to remove all inverted index.
+    // Update table config to remove the new inverted index, and
+    // reload table to clean the new inverted indices physically.
     tableConfig = getOfflineTableConfig();
-    tableConfig.getIndexingConfig().setInvertedIndexColumns(Collections.emptyList());
+    tableConfig.getIndexingConfig().setInvertedIndexColumns(getInvertedIndexColumns());
     updateTableConfig(tableConfig);
-
-    // Reload table just disables those indices, not clean them physically.
     reloadOfflineTable(offlineTableName);
     TestUtils.waitForCondition(aVoid -> {
       try {
         JsonNode queryResponse2 = postQuery(TEST_UPDATED_INVERTED_INDEX_QUERY);
-        // Total docs should not change during reload
+        // Total docs should not change during reload, but num entries scanned
+        // gets back to total number of documents as the index is removed.
         assertEquals(queryResponse2.get("totalDocs").asLong(), numTotalDocs);
         return queryResponse2.get("numEntriesScannedInFilter").asLong() == numTotalDocs;
       } catch (Exception e) {
         throw new RuntimeException(e);
       }
-    }, 600_000L, "Failed to disable indices");
-    long tableSizeAfterReload = getTableSize(offlineTableName);
-    assertEquals(tableSizeAfterReload, tableSizeWithNewIndex);
+    }, 600_000L, "Failed to cleanup obsolete index");
+    assertEquals(getTableSize(offlineTableName), tableSizeWithDefaultIndex);
+  }
+
+  @Test
+  public void testInvertedIndexTriggeringCleanIndicesWithForceDownload()
+      throws Exception {
+    String offlineTableName = TableNameBuilder.OFFLINE.tableNameWithType(getTableName());

Review comment:
       Let's extract the index addition part of these 2 tests into a common helper method because they are the same

##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/IndexHandler.java
##########
@@ -0,0 +1,27 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+public interface IndexHandler {

Review comment:
       Add some javadoc

##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/IndexHandlerFactory.java
##########
@@ -0,0 +1,63 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import java.io.File;
+import org.apache.pinot.segment.local.segment.index.loader.bloomfilter.BloomFilterHandler;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.H3IndexHandler;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.InvertedIndexHandler;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.JsonIndexHandler;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.LuceneFSTIndexHandler;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.RangeIndexHandler;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.TextIndexHandler;
+import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+
+
+public class IndexHandlerFactory {
+  private IndexHandlerFactory() {
+  }
+
+  private static final IndexHandler DEFAULT_HANDLER = () -> {

Review comment:
       Rename it to `NO_OP_HANDLER`?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] klsince commented on a change in pull request #7286: Remove obsolete indices during segment reloading for V3 format (wrap up)

Posted by GitBox <gi...@apache.org>.
klsince commented on a change in pull request #7286:
URL: https://github.com/apache/pinot/pull/7286#discussion_r695249747



##########
File path: pinot-segment-local/src/test/resources/data/log4j2.xml
##########
@@ -29,7 +29,7 @@
   </Appenders>
   <Loggers>
     <AsyncRoot level="warn" additivity="false">
-      <AppenderRef ref="console"/>
+      <AppenderRef ref="console" />

Review comment:
       interesting. why this was changed 😂  reverting...




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter edited a comment on pull request #7286: Remove obsolete indices during segment reloading for V3 format (wrap up)

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7286:
URL: https://github.com/apache/pinot/pull/7286#issuecomment-897062223


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7286](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (263882a) into [master](https://codecov.io/gh/apache/pinot/commit/c62a1fb328f1b151fa4a7a4a6ef17875662d6059?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (c62a1fb) will **decrease** coverage by `42.55%`.
   > The diff coverage is `22.39%`.
   
   > :exclamation: Current head 263882a differs from pull request most recent head 4ee9b17. Consider uploading reports for the commit 4ee9b17 to get more accurate results
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7286/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@              Coverage Diff              @@
   ##             master    #7286       +/-   ##
   =============================================
   - Coverage     71.65%   29.10%   -42.56%     
   + Complexity     3311       85     -3226     
   =============================================
     Files          1515     1508        -7     
     Lines         75006    74659      -347     
     Branches      10910    10876       -34     
   =============================================
   - Hits          53749    21731    -32018     
   - Misses        17628    50956    +33328     
   + Partials       3629     1972     -1657     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `29.10% <22.39%> (-0.01%)` | :arrow_down: |
   | unittests1 | `?` | |
   | unittests2 | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [.../main/java/org/apache/pinot/client/Connection.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Nvbm5lY3Rpb24uamF2YQ==) | `21.73% <0.00%> (-20.82%)` | :arrow_down: |
   | [.../org/apache/pinot/client/SimpleBrokerSelector.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L1NpbXBsZUJyb2tlclNlbGVjdG9yLmphdmE=) | `0.00% <0.00%> (-83.34%)` | :arrow_down: |
   | [...common/exception/SchemaAlreadyExistsException.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZXhjZXB0aW9uL1NjaGVtYUFscmVhZHlFeGlzdHNFeGNlcHRpb24uamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...ller/api/resources/PinotSchemaRestletResource.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9hcGkvcmVzb3VyY2VzL1Bpbm90U2NoZW1hUmVzdGxldFJlc291cmNlLmphdmE=) | `23.93% <0.00%> (-14.81%)` | :arrow_down: |
   | [...re/data/table/UnboundedConcurrentIndexedTable.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL3RhYmxlL1VuYm91bmRlZENvbmN1cnJlbnRJbmRleGVkVGFibGUuamF2YQ==) | `0.00% <0.00%> (-22.73%)` | :arrow_down: |
   | [...rg/apache/pinot/core/minion/RawIndexConverter.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9taW5pb24vUmF3SW5kZXhDb252ZXJ0ZXIuamF2YQ==) | `0.00% <0.00%> (-56.61%)` | :arrow_down: |
   | [...perator/combine/GroupByOrderByCombineOperator.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9jb21iaW5lL0dyb3VwQnlPcmRlckJ5Q29tYmluZU9wZXJhdG9yLmphdmE=) | `60.49% <0.00%> (-22.23%)` | :arrow_down: |
   | [...indexsegment/immutable/ImmutableSegmentLoader.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pbmRleHNlZ21lbnQvaW1tdXRhYmxlL0ltbXV0YWJsZVNlZ21lbnRMb2FkZXIuamF2YQ==) | `0.00% <0.00%> (-91.18%)` | :arrow_down: |
   | [...ocal/segment/index/loader/IndexHandlerFactory.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9JbmRleEhhbmRsZXJGYWN0b3J5LmphdmE=) | `0.00% <0.00%> (ø)` | |
   | [...ocal/segment/index/loader/SegmentPreProcessor.java](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9TZWdtZW50UHJlUHJvY2Vzc29yLmphdmE=) | `0.00% <0.00%> (-68.00%)` | :arrow_down: |
   | ... and [1078 more](https://codecov.io/gh/apache/pinot/pull/7286/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [c62a1fb...4ee9b17](https://codecov.io/gh/apache/pinot/pull/7286?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] klsince commented on a change in pull request #7286: Remove obsolete indices during segment reloading for V3 format (wrap up)

Posted by GitBox <gi...@apache.org>.
klsince commented on a change in pull request #7286:
URL: https://github.com/apache/pinot/pull/7286#discussion_r696832920



##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/SegmentPreProcessor.java
##########
@@ -112,31 +111,23 @@ public void process()
       rangeIndexHandler.createRangeIndices();
 
       // Create text indices according to the index config.
-      Set<String> textIndexColumns = _indexLoadingConfig.getTextIndexColumns();
-      if (!textIndexColumns.isEmpty()) {
-        TextIndexHandler textIndexHandler =
-            new TextIndexHandler(_indexDir, _segmentMetadata, textIndexColumns, segmentWriter);
-        textIndexHandler.createTextIndexesOnSegmentLoad();
-      }
+      TextIndexHandler textIndexHandler =
+          new TextIndexHandler(_indexDir, _segmentMetadata, _indexLoadingConfig, segmentWriter);
+      textIndexHandler.createTextIndexesOnSegmentLoad();

Review comment:
       updateIndices() sgtm




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #7286: Remove obsolete indices during segment reloading for V3 format (wrap up)

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #7286:
URL: https://github.com/apache/pinot/pull/7286#discussion_r696206001



##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/immutable/ImmutableSegmentLoader.java
##########
@@ -57,55 +57,59 @@ private ImmutableSegmentLoader() {
   private static final Logger LOGGER = LoggerFactory.getLogger(ImmutableSegmentLoader.class);
 
   /**
-   * For tests only.
+   * load with empty schema and IndexLoadingConfig is used to get a handler of the segment for

Review comment:
       (nit) Capitalize it, same for other javadocs
   ```suggestion
      * Loads the segment with...
   ```

##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/immutable/ImmutableSegmentLoader.java
##########
@@ -164,6 +168,29 @@ public static ImmutableSegment load(File indexDir, IndexLoadingConfig indexLoadi
     return segment;
   }
 
+  private static void tryToConvertSegment(File indexDir, IndexLoadingConfig indexLoadingConfig,
+      SegmentMetadataImpl localSegmentMetadata)
+      throws Exception {
+    SegmentVersion segmentVersionToLoad = indexLoadingConfig.getSegmentVersion();
+    if (segmentVersionToLoad == null || SegmentDirectoryPaths.segmentDirectoryFor(indexDir, segmentVersionToLoad)
+        .isDirectory()) {
+      return;
+    }
+    SegmentVersion segmentVersionOnDisk = localSegmentMetadata.getVersion();
+    if (segmentVersionOnDisk == segmentVersionToLoad) {
+      return;
+    }
+    String segmentName = indexDir.getName();
+    LOGGER.info("Segment: {} needs to be converted from version: {} to {}", segmentName, segmentVersionOnDisk,
+        segmentVersionToLoad);
+    SegmentFormatConverter converter =
+        SegmentFormatConverterFactory.getConverter(segmentVersionOnDisk, segmentVersionToLoad);
+    LOGGER.info("Using converter: {} to up-convert segment: {}", converter.getClass().getName(), segmentName);
+    converter.convert(indexDir);
+    LOGGER.info("Successfully up-converted segment: {} from version: {} to {}", segmentName, segmentVersionOnDisk,
+        segmentVersionToLoad);
+  }
+
   private static void preprocessSegment(File indexDir, IndexLoadingConfig indexLoadingConfig, Schema schema)

Review comment:
       Annotate schema as nullable

##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/immutable/ImmutableSegmentLoader.java
##########
@@ -57,55 +57,59 @@ private ImmutableSegmentLoader() {
   private static final Logger LOGGER = LoggerFactory.getLogger(ImmutableSegmentLoader.class);
 
   /**
-   * For tests only.
+   * load with empty schema and IndexLoadingConfig is used to get a handler of the segment for

Review comment:
       This method should be called when we just want to load the segment without modifying it. Let's make it clear in the javadoc.

##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/immutable/ImmutableSegmentLoader.java
##########
@@ -164,6 +168,29 @@ public static ImmutableSegment load(File indexDir, IndexLoadingConfig indexLoadi
     return segment;
   }
 
+  private static void tryToConvertSegment(File indexDir, IndexLoadingConfig indexLoadingConfig,

Review comment:
       Rename it to `convertSegmentFormatIfNeeded`

##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/bloomfilter/BloomFilterHandler.java
##########
@@ -67,12 +67,20 @@ public BloomFilterHandler(File indexDir, SegmentMetadataImpl segmentMetadata, In
     _segmentVersion = segmentMetadata.getVersion();
     _bloomFilterConfigs = indexLoadingConfig.getBloomFilterConfigs();
 
-    for (String column : _bloomFilterConfigs.keySet()) {
+    Set<String> columnsInCfg = _bloomFilterConfigs.keySet();
+    for (String column : columnsInCfg) {
       ColumnMetadata columnMetadata = segmentMetadata.getColumnMetadataFor(column);
       if (columnMetadata != null) {
         _bloomFilterColumns.add(columnMetadata);
       }
     }
+    // Remove indices not set in table config any more
+    Set<String> localColumns = _segmentWriter.toSegmentDirectory().getColumnsWithIndex(ColumnIndexType.BLOOM_FILTER);
+    for (String column : localColumns) {
+      if (!columnsInCfg.contains(column)) {
+        _segmentWriter.removeIndex(column, ColumnIndexType.BLOOM_FILTER);
+      }
+    }

Review comment:
       Exclude local columns from the columns to generate. Same for other handlers
   ```suggestion
       Set<String> existingColumns = _segmentWriter.toSegmentDirectory().getColumnsWithIndex(ColumnIndexType.BLOOM_FILTER);
       Set<String> columnsToAdd = new HashSet<>(_bloomFilterConfigs.keySet());
       for (String column : existingColumns) {
         if (!columnsToAdd.remove(column)) {
           _segmentWriter.removeIndex(column, ColumnIndexType.BLOOM_FILTER);
         }
       }
       for (String column : columnsToAdd) {
         ColumnMetadata columnMetadata = segmentMetadata.getColumnMetadataFor(column);
         if (columnMetadata != null) {
           _bloomFilterColumns.add(columnMetadata);
         }
       }
   ```

##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/store/SingleFileIndexDirectory.java
##########
@@ -386,6 +386,15 @@ public void removeIndex(String columnName, ColumnIndexType indexType) {
   @Override
   public Set<String> getColumnsWithIndex(ColumnIndexType type) {
     Set<String> columns = new HashSet<>();
+    // TEXT_INDEX is not tracked via _columnEntries, so handled separately.

Review comment:
       Move to the bug fix PR

##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/SegmentPreProcessor.java
##########
@@ -112,31 +111,23 @@ public void process()
       rangeIndexHandler.createRangeIndices();
 
       // Create text indices according to the index config.
-      Set<String> textIndexColumns = _indexLoadingConfig.getTextIndexColumns();
-      if (!textIndexColumns.isEmpty()) {
-        TextIndexHandler textIndexHandler =
-            new TextIndexHandler(_indexDir, _segmentMetadata, textIndexColumns, segmentWriter);
-        textIndexHandler.createTextIndexesOnSegmentLoad();
-      }
+      TextIndexHandler textIndexHandler =
+          new TextIndexHandler(_indexDir, _segmentMetadata, _indexLoadingConfig, segmentWriter);
+      textIndexHandler.createTextIndexesOnSegmentLoad();

Review comment:
       We should probably rename the method because it also removes the indices. Probably `updateIndices()`? Same for other handlers

##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/defaultcolumn/V3DefaultColumnHandler.java
##########
@@ -47,39 +45,33 @@ protected boolean updateDefaultColumn(String column, DefaultColumnAction action)
       throws Exception {
     LOGGER.info("Starting default column action: {} on column: {}", action, column);
 
-    // For V3 segment format, only support ADD action
-    // For UPDATE and REMOVE action, throw exception to drop and re-download the segment
-    if (action.isUpdateAction()) {
-      throw new V3UpdateIndexException(
-          "Default value indices for column: " + column + " cannot be updated for V3 format segment.");
+    // For UPDATE and REMOVE action, delete existing dictionary and forward index, and remove column metadata
+    if (action.isUpdateAction() || action.isRemoveAction()) {
+      removeColumnIndices(column);
     }
-
-    if (action.isRemoveAction()) {
-      throw new V3RemoveIndexException(
-          "Default value indices for column: " + column + " cannot be removed for V3 format segment.");
+    if (!action.isAddAction() && !action.isUpdateAction()) {

Review comment:
       ```suggestion
       if (action.isRemoveAction()) {
   ```

##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/store/FilePerIndexDirectory.java
##########
@@ -108,10 +108,13 @@ public void removeIndex(String columnName, ColumnIndexType indexType) {
 
   @Override
   public Set<String> getColumnsWithIndex(ColumnIndexType type) {
+    // _indexBuffers is just a cache of index files, thus not reliable as

Review comment:
       Shall we make these bug fixes in a separate PR for easier review? We can merge the bug fix first

##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/invertedindex/JsonIndexHandler.java
##########
@@ -104,8 +112,9 @@ private void createJsonIndexForColumn(ColumnMetadata columnMetadata)
 
     // Create new json index for the column.
     LOGGER.info("Creating new json index for segment: {}, column: {}", _segmentName, columnName);
-    Preconditions.checkState(columnMetadata.isSingleValue() && columnMetadata.getDataType() == DataType.STRING,
-        "Json index can only be applied to single-value STRING columns");
+    Preconditions.checkState(columnMetadata.isSingleValue() && (columnMetadata.getDataType() == DataType.STRING

Review comment:
       Move to a separate bug fix PR




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org