You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/08/18 17:06:25 UTC
[GitHub] [pinot] klsince opened a new pull request #7319: Move offline seg download code to OfflineTableManager
klsince opened a new pull request #7319:
URL: https://github.com/apache/pinot/pull/7319
## Description
This PR is the first one towards implementing the feature to force download segment for Realtime table during table reloading. This feature has been done for Offline table in https://github.com/apache/pinot/pull/7249.
This PR deletes SegmentFetcherAndLoader; and move its functionality to OfflineTableManager, preparing to combine the download logic with RealtimeTableManager. Besides, adding UTs.
Next PR will combine the codes for Offline and Realtime tables, and enable force download for Realtime table.
## Upgrade Notes
Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion)
* [ ] Yes (Please label as **<code>backward-incompat</code>**, and complete the section below on Release Notes)
Does this PR fix a zero-downtime upgrade introduced earlier?
* [ ] Yes (Please label this as **<code>backward-incompat</code>**, and complete the section below on Release Notes)
Does this PR otherwise need attention when creating release notes? Things to consider:
- New configuration options
- Deprecation of configurations
- Signature changes to public methods/interfaces
- New plugins added or old plugins removed
* [ ] Yes (Please label this PR as **<code>release-notes</code>** and complete the section on Release Notes)
## Release Notes
<!-- If you have tagged this as either backward-incompat or release-notes,
you MUST add text here that you would like to see appear in release notes of the
next release. -->
<!-- If you have a series of commits adding or enabling a feature, then
add this section only in final commit that marks the feature completed.
Refer to earlier release notes to see examples of text.
-->
## Documentation
<!-- If you have introduced a new feature or configuration, please add it to the documentation as well.
See https://docs.pinot.apache.org/developers/developers-and-contributors/update-document
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7319: Move offline seg download code to OfflineTableManager
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#issuecomment-901313573
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7319](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e479f0e) into [master](https://codecov.io/gh/apache/pinot/commit/12e5bcd37350a3e85a9e9630a7f6de05d91b5a24?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (12e5bcd) will **decrease** coverage by `1.13%`.
> The diff coverage is `68.06%`.
[](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7319 +/- ##
============================================
- Coverage 71.32% 70.19% -1.14%
+ Complexity 3367 3281 -86
============================================
Files 1503 1502 -1
Lines 74030 73849 -181
Branches 10718 10692 -26
============================================
- Hits 52803 51839 -964
- Misses 17622 18418 +796
+ Partials 3605 3592 -13
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `29.32% <63.02%> (-0.01%)` | :arrow_down: |
| unittests1 | `69.06% <44.30%> (+0.04%)` | :arrow_up: |
| unittests2 | `14.58% <0.00%> (-0.07%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ata/manager/realtime/RealtimeTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVUYWJsZURhdGFNYW5hZ2VyLmphdmE=) | `66.53% <ø> (-0.41%)` | :arrow_down: |
| [.../pinot/core/data/manager/BaseTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvQmFzZVRhYmxlRGF0YU1hbmFnZXIuamF2YQ==) | `86.51% <50.00%> (-0.84%)` | :arrow_down: |
| [.../data/manager/offline/OfflineTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvb2ZmbGluZS9PZmZsaW5lVGFibGVEYXRhTWFuYWdlci5qYXZh) | `57.14% <53.24%> (-42.86%)` | :arrow_down: |
| [...er/starter/helix/SegmentMessageHandlerFactory.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9TZWdtZW50TWVzc2FnZUhhbmRsZXJGYWN0b3J5LmphdmE=) | `63.63% <75.00%> (-14.57%)` | :arrow_down: |
| [.../pinot/server/starter/helix/BaseServerStarter.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9CYXNlU2VydmVyU3RhcnRlci5qYXZh) | `49.70% <100.00%> (-5.06%)` | :arrow_down: |
| [...server/starter/helix/HelixInstanceDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9IZWxpeEluc3RhbmNlRGF0YU1hbmFnZXIuamF2YQ==) | `79.09% <100.00%> (-3.49%)` | :arrow_down: |
| [...r/helix/SegmentOnlineOfflineStateModelFactory.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9TZWdtZW50T25saW5lT2ZmbGluZVN0YXRlTW9kZWxGYWN0b3J5LmphdmE=) | `61.11% <100.00%> (+2.23%)` | :arrow_up: |
| [...pinot/minion/exception/TaskCancelledException.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vZXhjZXB0aW9uL1Rhc2tDYW5jZWxsZWRFeGNlcHRpb24uamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...ore/startree/executor/StarTreeGroupByExecutor.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zdGFydHJlZS9leGVjdXRvci9TdGFyVHJlZUdyb3VwQnlFeGVjdXRvci5qYXZh) | `0.00% <0.00%> (-86.67%)` | :arrow_down: |
| [.../transform/function/MapValueTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vTWFwVmFsdWVUcmFuc2Zvcm1GdW5jdGlvbi5qYXZh) | `0.00% <0.00%> (-85.30%)` | :arrow_down: |
| ... and [297 more](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [12e5bcd...e479f0e](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7319: Move segment download and segment reload into TableManager
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#issuecomment-901313573
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7319](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6ce60b3) into [master](https://codecov.io/gh/apache/pinot/commit/ad2e0632fa8d8ab84316d25ce2cd4c91c18b11bf?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ad2e063) will **decrease** coverage by `34.08%`.
> The diff coverage is `66.44%`.
[](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7319 +/- ##
=============================================
- Coverage 71.90% 37.81% -34.09%
+ Complexity 3429 92 -3337
=============================================
Files 1517 1516 -1
Lines 75118 75109 -9
Branches 10950 10945 -5
=============================================
- Hits 54013 28406 -25607
- Misses 17470 44492 +27022
+ Partials 3635 2211 -1424
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `30.40% <66.44%> (+0.01%)` | :arrow_up: |
| integration2 | `?` | |
| unittests1 | `?` | |
| unittests2 | `14.48% <0.00%> (-0.04%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...va/org/apache/pinot/spi/utils/CommonConstants.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQ29tbW9uQ29uc3RhbnRzLmphdmE=) | `0.00% <0.00%> (-26.32%)` | :arrow_down: |
| [...ata/manager/realtime/RealtimeTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVUYWJsZURhdGFNYW5hZ2VyLmphdmE=) | `41.86% <33.33%> (-25.62%)` | :arrow_down: |
| [.../pinot/core/data/manager/BaseTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvQmFzZVRhYmxlRGF0YU1hbmFnZXIuamF2YQ==) | `68.58% <55.76%> (-18.77%)` | :arrow_down: |
| [.../pinot/server/starter/helix/BaseServerStarter.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9CYXNlU2VydmVyU3RhcnRlci5qYXZh) | `51.74% <100.00%> (-3.02%)` | :arrow_down: |
| [...server/starter/helix/HelixInstanceDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9IZWxpeEluc3RhbmNlRGF0YU1hbmFnZXIuamF2YQ==) | `84.51% <100.00%> (+4.51%)` | :arrow_up: |
| [...er/starter/helix/SegmentMessageHandlerFactory.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9TZWdtZW50TWVzc2FnZUhhbmRsZXJGYWN0b3J5LmphdmE=) | `72.60% <100.00%> (+0.80%)` | :arrow_up: |
| [...r/helix/SegmentOnlineOfflineStateModelFactory.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9TZWdtZW50T25saW5lT2ZmbGluZVN0YXRlTW9kZWxGYWN0b3J5LmphdmE=) | `58.49% <100.00%> (-0.39%)` | :arrow_down: |
| [...c/main/java/org/apache/pinot/common/tier/Tier.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdGllci9UaWVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../java/org/apache/pinot/spi/utils/BooleanUtils.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQm9vbGVhblV0aWxzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...ava/org/apache/pinot/spi/data/MetricFieldSpec.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9NZXRyaWNGaWVsZFNwZWMuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [877 more](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [ad2e063...6ce60b3](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7319: Move segment download and segment reload into TableManager
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#issuecomment-901313573
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7319](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3d7a45e) into [master](https://codecov.io/gh/apache/pinot/commit/ad2e0632fa8d8ab84316d25ce2cd4c91c18b11bf?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ad2e063) will **decrease** coverage by `2.14%`.
> The diff coverage is `78.89%`.
> :exclamation: Current head 3d7a45e differs from pull request most recent head 6ce60b3. Consider uploading reports for the commit 6ce60b3 to get more accurate results
[](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7319 +/- ##
============================================
- Coverage 71.90% 69.76% -2.15%
+ Complexity 3429 3248 -181
============================================
Files 1517 1123 -394
Lines 75118 53133 -21985
Branches 10950 8008 -2942
============================================
- Hits 54013 37067 -16946
+ Misses 17470 13435 -4035
+ Partials 3635 2631 -1004
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `69.76% <78.89%> (+0.04%)` | :arrow_up: |
| unittests2 | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../pinot/core/data/manager/BaseTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvQmFzZVRhYmxlRGF0YU1hbmFnZXIuamF2YQ==) | `82.19% <77.88%> (-5.16%)` | :arrow_down: |
| [...ata/manager/realtime/RealtimeTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVUYWJsZURhdGFNYW5hZ2VyLmphdmE=) | `11.78% <100.00%> (-55.71%)` | :arrow_down: |
| [...va/org/apache/pinot/spi/utils/CommonConstants.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQ29tbW9uQ29uc3RhbnRzLmphdmE=) | `26.31% <100.00%> (ø)` | |
| [...a/org/apache/pinot/common/metrics/MinionMeter.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25NZXRlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...g/apache/pinot/common/metrics/ControllerMeter.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Db250cm9sbGVyTWV0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/BrokerQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Ccm9rZXJRdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/MinionQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25RdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...he/pinot/common/messages/SegmentReloadMessage.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWVzc2FnZXMvU2VnbWVudFJlbG9hZE1lc3NhZ2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pinot/core/data/manager/realtime/TimerService.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvVGltZXJTZXJ2aWNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...t/core/startree/plan/StarTreeDocIdSetPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zdGFydHJlZS9wbGFuL1N0YXJUcmVlRG9jSWRTZXRQbGFuTm9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [593 more](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [ad2e063...6ce60b3](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7319: Move offline seg download code to OfflineTableManager
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#issuecomment-901313573
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7319](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (dd46cf1) into [master](https://codecov.io/gh/apache/pinot/commit/12e5bcd37350a3e85a9e9630a7f6de05d91b5a24?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (12e5bcd) will **decrease** coverage by `38.78%`.
> The diff coverage is `45.01%`.
> :exclamation: Current head dd46cf1 differs from pull request most recent head 15eb4e8. Consider uploading reports for the commit 15eb4e8 to get more accurate results
[](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7319 +/- ##
=============================================
- Coverage 71.32% 32.54% -38.79%
+ Complexity 3367 91 -3276
=============================================
Files 1503 1493 -10
Lines 74030 73510 -520
Branches 10718 10656 -62
=============================================
- Hits 52803 23924 -28879
- Misses 17622 47534 +29912
+ Partials 3605 2052 -1553
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `30.89% <44.23%> (+0.81%)` | :arrow_up: |
| integration2 | `29.32% <34.17%> (-0.01%)` | :arrow_down: |
| unittests1 | `?` | |
| unittests2 | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../pinot/common/function/DateTimePatternHandler.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vRGF0ZVRpbWVQYXR0ZXJuSGFuZGxlci5qYXZh) | `60.00% <ø> (-23.34%)` | :arrow_down: |
| [...rg/apache/pinot/common/function/DateTimeUtils.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vRGF0ZVRpbWVVdGlscy5qYXZh) | `0.00% <0.00%> (-80.00%)` | :arrow_down: |
| [...ot/common/function/FunctionDefinitionRegistry.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vRnVuY3Rpb25EZWZpbml0aW9uUmVnaXN0cnkuamF2YQ==) | `50.00% <ø> (-38.89%)` | :arrow_down: |
| [.../org/apache/pinot/common/function/TimeZoneKey.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vVGltZVpvbmVLZXkuamF2YQ==) | `0.00% <0.00%> (-38.94%)` | :arrow_down: |
| [...n/function/scalar/DataTypeConversionFunctions.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vc2NhbGFyL0RhdGFUeXBlQ29udmVyc2lvbkZ1bmN0aW9ucy5qYXZh) | `0.00% <ø> (ø)` | |
| [...inot/common/function/scalar/DateTimeFunctions.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vc2NhbGFyL0RhdGVUaW1lRnVuY3Rpb25zLmphdmE=) | `6.94% <0.00%> (-91.67%)` | :arrow_down: |
| [...he/pinot/common/function/scalar/HashFunctions.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vc2NhbGFyL0hhc2hGdW5jdGlvbnMuamF2YQ==) | `0.00% <ø> (-100.00%)` | :arrow_down: |
| [.../org/apache/pinot/common/lineage/LineageEntry.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbGluZWFnZS9MaW5lYWdlRW50cnkuamF2YQ==) | `58.82% <0.00%> (-5.89%)` | :arrow_down: |
| [...not/common/lineage/SegmentLineageAccessHelper.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbGluZWFnZS9TZWdtZW50TGluZWFnZUFjY2Vzc0hlbHBlci5qYXZh) | `100.00% <ø> (+53.33%)` | :arrow_up: |
| [...ot/common/messages/RoutingTableRebuildMessage.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWVzc2FnZXMvUm91dGluZ1RhYmxlUmVidWlsZE1lc3NhZ2UuamF2YQ==) | `81.81% <ø> (+81.81%)` | :arrow_up: |
| ... and [1163 more](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [12e5bcd...15eb4e8](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7319: Move offline seg download code to OfflineTableManager
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#issuecomment-901313573
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7319](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (91e3244) into [master](https://codecov.io/gh/apache/pinot/commit/2d1f7dc9060d45ac9f4ff0621da62790ea9b8e64?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2d1f7dc) will **decrease** coverage by `34.26%`.
> The diff coverage is `62.18%`.
> :exclamation: Current head 91e3244 differs from pull request most recent head e8f676d. Consider uploading reports for the commit e8f676d to get more accurate results
[](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7319 +/- ##
=============================================
- Coverage 71.65% 37.38% -34.27%
+ Complexity 3305 176 -3129
=============================================
Files 1515 1515
Lines 74992 74978 -14
Branches 10903 10898 -5
=============================================
- Hits 53736 28032 -25704
- Misses 17629 44717 +27088
+ Partials 3627 2229 -1398
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `28.95% <62.18%> (-0.13%)` | :arrow_down: |
| unittests1 | `?` | |
| unittests2 | `14.43% <0.00%> (-0.02%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ata/manager/realtime/RealtimeTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVUYWJsZURhdGFNYW5hZ2VyLmphdmE=) | `65.44% <0.00%> (-2.31%)` | :arrow_down: |
| [...t/segment/local/data/manager/TableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9kYXRhL21hbmFnZXIvVGFibGVEYXRhTWFuYWdlci5qYXZh) | `0.00% <0.00%> (ø)` | |
| [.../pinot/core/data/manager/BaseTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvQmFzZVRhYmxlRGF0YU1hbmFnZXIuamF2YQ==) | `65.85% <45.45%> (-21.51%)` | :arrow_down: |
| [...er/starter/helix/SegmentMessageHandlerFactory.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9TZWdtZW50TWVzc2FnZUhhbmRsZXJGYWN0b3J5LmphdmE=) | `63.63% <75.00%> (-14.57%)` | :arrow_down: |
| [.../pinot/server/starter/helix/BaseServerStarter.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9CYXNlU2VydmVyU3RhcnRlci5qYXZh) | `49.70% <100.00%> (-5.06%)` | :arrow_down: |
| [...server/starter/helix/HelixInstanceDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9IZWxpeEluc3RhbmNlRGF0YU1hbmFnZXIuamF2YQ==) | `79.77% <100.00%> (-2.81%)` | :arrow_down: |
| [...r/helix/SegmentOnlineOfflineStateModelFactory.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9TZWdtZW50T25saW5lT2ZmbGluZVN0YXRlTW9kZWxGYWN0b3J5LmphdmE=) | `58.49% <100.00%> (-0.39%)` | :arrow_down: |
| [...c/main/java/org/apache/pinot/common/tier/Tier.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdGllci9UaWVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../java/org/apache/pinot/spi/utils/BooleanUtils.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQm9vbGVhblV0aWxzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...ava/org/apache/pinot/spi/data/MetricFieldSpec.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9NZXRyaWNGaWVsZFNwZWMuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [885 more](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [2d1f7dc...e8f676d](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #7319: Move offline seg download code to OfflineTableManager
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#issuecomment-901313573
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7319](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e479f0e) into [master](https://codecov.io/gh/apache/pinot/commit/12e5bcd37350a3e85a9e9630a7f6de05d91b5a24?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (12e5bcd) will **decrease** coverage by `6.81%`.
> The diff coverage is `44.30%`.
[](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7319 +/- ##
============================================
- Coverage 71.32% 64.50% -6.82%
+ Complexity 3367 3281 -86
============================================
Files 1503 1457 -46
Lines 74030 72034 -1996
Branches 10718 10500 -218
============================================
- Hits 52803 46469 -6334
- Misses 17622 22211 +4589
+ Partials 3605 3354 -251
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `69.06% <44.30%> (+0.04%)` | :arrow_up: |
| unittests2 | `14.58% <0.00%> (-0.07%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ata/manager/realtime/RealtimeTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVUYWJsZURhdGFNYW5hZ2VyLmphdmE=) | `10.61% <ø> (-56.33%)` | :arrow_down: |
| [.../data/manager/offline/OfflineTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvb2ZmbGluZS9PZmZsaW5lVGFibGVEYXRhTWFuYWdlci5qYXZh) | `48.80% <44.15%> (-51.20%)` | :arrow_down: |
| [.../pinot/core/data/manager/BaseTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvQmFzZVRhYmxlRGF0YU1hbmFnZXIuamF2YQ==) | `86.51% <50.00%> (-0.84%)` | :arrow_down: |
| [...a/org/apache/pinot/common/metrics/MinionMeter.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25NZXRlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...g/apache/pinot/common/metrics/ControllerMeter.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Db250cm9sbGVyTWV0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/BrokerQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Ccm9rZXJRdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/MinionQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25RdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pache/pinot/common/utils/grpc/GrpcQueryClient.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvZ3JwYy9HcnBjUXVlcnlDbGllbnQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...he/pinot/common/messages/SegmentReloadMessage.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWVzc2FnZXMvU2VnbWVudFJlbG9hZE1lc3NhZ2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pinot/minion/exception/TaskCancelledException.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vZXhjZXB0aW9uL1Rhc2tDYW5jZWxsZWRFeGNlcHRpb24uamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [509 more](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [12e5bcd...e479f0e](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on a change in pull request #7319: Move offline seg download code to OfflineTableManager
Posted by GitBox <gi...@apache.org>.
klsince commented on a change in pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#discussion_r704593911
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
##########
@@ -256,4 +267,147 @@ public void addSegmentError(String segmentName, SegmentErrorInfo segmentErrorInf
.collect(Collectors.toMap(map -> map.getKey().getSecond(), Map.Entry::getValue));
}
}
+
+ @Override
+ public void addOrReplaceSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentMetadata localMetadata, SegmentZKMetadata zkMetadata, boolean forceDownload)
+ throws Exception {
+ if (!forceDownload && !isNewSegment(zkMetadata, localMetadata)) {
+ LOGGER.info("Segment: {} of table: {} has crc: {} same as before, already loaded, do nothing", segmentName,
+ _tableNameWithType, localMetadata.getCrc());
+ return;
Review comment:
will add a comment to highlight this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7319: Move offline seg download code to OfflineTableManager
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#issuecomment-901313573
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7319](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (46fd9b5) into [master](https://codecov.io/gh/apache/pinot/commit/bb34406aaa205f5c85b88c928d477fd267eda1b4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bb34406) will **increase** coverage by `0.22%`.
> The diff coverage is `65.65%`.
> :exclamation: Current head 46fd9b5 differs from pull request most recent head 331392f. Consider uploading reports for the commit 331392f to get more accurate results
[](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7319 +/- ##
============================================
+ Coverage 64.65% 64.87% +0.22%
+ Complexity 3303 3302 -1
============================================
Files 1496 1460 -36
Lines 74289 72788 -1501
Branches 10795 10633 -162
============================================
- Hits 48033 47224 -809
+ Misses 22826 22193 -633
+ Partials 3430 3371 -59
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `69.50% <72.15%> (+0.03%)` | :arrow_up: |
| unittests2 | `14.52% <8.08%> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...t/broker/api/resources/PinotBrokerHealthCheck.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYXBpL3Jlc291cmNlcy9QaW5vdEJyb2tlckhlYWx0aENoZWNrLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...pinot/broker/api/resources/PinotClientRequest.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYXBpL3Jlc291cmNlcy9QaW5vdENsaWVudFJlcXVlc3QuamF2YQ==) | `0.00% <0.00%> (-35.39%)` | :arrow_down: |
| [...org/apache/pinot/broker/queryquota/HitCounter.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcXVlcnlxdW90YS9IaXRDb3VudGVyLmphdmE=) | `100.00% <ø> (+31.03%)` | :arrow_up: |
| [...che/pinot/broker/queryquota/MaxHitRateTracker.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcXVlcnlxdW90YS9NYXhIaXRSYXRlVHJhY2tlci5qYXZh) | `95.00% <ø> (+58.63%)` | :arrow_up: |
| [...roker/requesthandler/BaseBrokerRequestHandler.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcmVxdWVzdGhhbmRsZXIvQmFzZUJyb2tlclJlcXVlc3RIYW5kbGVyLmphdmE=) | `17.33% <ø> (-48.93%)` | :arrow_down: |
| [.../routing/segmentpruner/PartitionSegmentPruner.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9zZWdtZW50cHJ1bmVyL1BhcnRpdGlvblNlZ21lbnRQcnVuZXIuamF2YQ==) | `61.24% <0.00%> (+13.58%)` | :arrow_up: |
| [...roker/routing/segmentpruner/TimeSegmentPruner.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9zZWdtZW50cHJ1bmVyL1RpbWVTZWdtZW50UHJ1bmVyLmphdmE=) | `80.95% <0.00%> (+80.95%)` | :arrow_up: |
| [...ata/manager/realtime/RealtimeTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVUYWJsZURhdGFNYW5hZ2VyLmphdmE=) | `10.56% <0.00%> (-56.37%)` | :arrow_down: |
| [...t/segment/local/data/manager/TableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9kYXRhL21hbmFnZXIvVGFibGVEYXRhTWFuYWdlci5qYXZh) | `0.00% <0.00%> (ø)` | |
| [.../pinot/core/data/manager/BaseTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvQmFzZVRhYmxlRGF0YU1hbmFnZXIuamF2YQ==) | `81.09% <74.02%> (-6.26%)` | :arrow_down: |
| ... and [493 more](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [bb34406...331392f](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on a change in pull request #7319: Move offline seg download code to OfflineTableManager
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on a change in pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#discussion_r704643135
##########
File path: pinot-core/src/test/java/org/apache/pinot/core/data/manager/BaseTableDataManagerAcquireSegmentTest.java
##########
@@ -240,7 +239,7 @@ public void testReplace()
runStorageServer(numQueryThreads, runTimeSec, tableDataManager); // replaces segments while online
-// System.out.println("Nops = " + _numQueries + ",nDrops=" + _nDestroys + ",nCreates=" + _allSegments.size());
+ // System.out.println("Nops = " + _numQueries + ",nDrops=" + _nDestroys + ",nCreates=" + _allSegments.size());
Review comment:
Instead, print out the random seed once when the test starts. That way, if the test fails, we will be able to reproduce the failure by manually setting the seed to the value printed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on a change in pull request #7319: Move segment download and segment reload into TableManager
Posted by GitBox <gi...@apache.org>.
klsince commented on a change in pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#discussion_r708500264
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
##########
@@ -256,4 +270,219 @@ public void addSegmentError(String segmentName, SegmentErrorInfo segmentErrorInf
.collect(Collectors.toMap(map -> map.getKey().getSecond(), Map.Entry::getValue));
}
}
+
+ @Override
+ public void reloadSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentZKMetadata zkMetadata, SegmentMetadata localMetadata, Schema schema, boolean forceDownload)
+ throws Exception {
+ File indexDir = localMetadata.getIndexDir();
+ Preconditions.checkState(indexDir.isDirectory(), "Index directory: %s is not a directory", indexDir);
+
+ File parentFile = indexDir.getParentFile();
+ File segmentBackupDir =
+ new File(parentFile, indexDir.getName() + CommonConstants.Segment.SEGMENT_BACKUP_DIR_SUFFIX);
+
+ try {
+ // First rename index directory to segment backup directory so that original segment have all file descriptors
+ // point to the segment backup directory to ensure original segment serves queries properly
+
+ // Rename index directory to segment backup directory (atomic)
+ Preconditions.checkState(indexDir.renameTo(segmentBackupDir),
+ "Failed to rename index directory: %s to segment backup directory: %s", indexDir, segmentBackupDir);
+
+ // Download from remote or copy from local backup directory into index directory,
+ // and then continue to load the segment from index directory.
+ boolean shouldDownload = forceDownload || !hasSameCRC(zkMetadata, localMetadata);
+ if (shouldDownload && allowDownload(segmentName, zkMetadata)) {
+ if (forceDownload) {
+ LOGGER.info("Segment: {} of table: {} is forced to download", segmentName, _tableNameWithType);
+ } else {
+ LOGGER.info("Download segment:{} of table: {} as local crc:{} mismatches remote crc: {}", segmentName,
+ _tableNameWithType, localMetadata.getCrc(), zkMetadata.getCrc());
+ }
+ indexDir = downloadSegment(segmentName, zkMetadata);
+ } else {
+ LOGGER.info("Reload the local copy of segment: {} of table: {}", segmentName, _tableNameWithType);
+ FileUtils.copyDirectory(segmentBackupDir, indexDir);
+ }
+
+ // Load from index directory and replace the old segment in memory.
+ addSegment(ImmutableSegmentLoader.load(indexDir, indexLoadingConfig, schema));
+
+ // Rename segment backup directory to segment temporary directory (atomic)
+ // The reason to first rename then delete is that, renaming is an atomic operation, but deleting is not. When we
+ // rename the segment backup directory to segment temporary directory, we know the reload already succeeded, so
+ // that we can safely delete the segment temporary directory
+ File segmentTempDir = new File(parentFile, indexDir.getName() + CommonConstants.Segment.SEGMENT_TEMP_DIR_SUFFIX);
+ Preconditions.checkState(segmentBackupDir.renameTo(segmentTempDir),
+ "Failed to rename segment backup directory: %s to segment temporary directory: %s", segmentBackupDir,
+ segmentTempDir);
+ FileUtils.deleteDirectory(segmentTempDir);
+ } catch (Exception reloadFailureException) {
+ try {
+ LoaderUtils.reloadFailureRecovery(indexDir);
+ } catch (Exception recoveryFailureException) {
+ LOGGER.error("Failed to recover after reload failure", recoveryFailureException);
+ reloadFailureException.addSuppressed(recoveryFailureException);
+ }
+ throw reloadFailureException;
+ }
+ }
+
+ @Override
+ public void addOrReplaceSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentZKMetadata zkMetadata, @Nullable SegmentMetadata localMetadata)
+ throws Exception {
+ if (!isNewSegment(zkMetadata, localMetadata)) {
+ LOGGER.info("Segment: {} of table: {} has crc: {} same as before, already loaded, do nothing", segmentName,
+ _tableNameWithType, localMetadata.getCrc());
+ return;
+ }
+
+ // Try to recover if no local metadata is provided.
+ if (localMetadata == null) {
+ LOGGER.info("Segment: {} of table: {} is not loaded, checking disk", segmentName, _tableNameWithType);
+ localMetadata = recoverSegmentQuietly(segmentName);
+ if (!isNewSegment(zkMetadata, localMetadata)) {
+ LOGGER.info("Segment: {} of table {} has crc: {} same as before, loading", segmentName, _tableNameWithType,
+ localMetadata.getCrc());
+ if (loadSegmentQuietly(segmentName, indexLoadingConfig)) {
+ return;
+ }
+ localMetadata = null;
+ }
+ }
+
+ Preconditions.checkState(allowDownload(segmentName, zkMetadata), "Segment: %s of table: %s does not allow download",
+ segmentName, _tableNameWithType);
+
+ // Download segment and replace the local one, either due to failure to recover local segment,
+ // or the segment data is updated and has new CRC now.
+ if (localMetadata == null) {
+ LOGGER.info("Download segment: {} of table: {} as no good one exists locally", segmentName, _tableNameWithType);
+ } else {
+ LOGGER.info("Download segment: {} of table: {} as local crc: {} mismatches remote crc: {}.", segmentName,
+ _tableNameWithType, localMetadata.getCrc(), zkMetadata.getCrc());
+ }
+ File indexDir = downloadSegment(segmentName, zkMetadata);
+ addSegment(indexDir, indexLoadingConfig);
+ LOGGER.info("Downloaded and replaced segment: {} of table: {} with local crc now becomes: {} and remote crc: {}",
+ segmentName, _tableNameWithType, new SegmentMetadataImpl(indexDir).getCrc(), zkMetadata.getCrc());
+ }
+
+ protected boolean allowDownload(String segmentName, SegmentZKMetadata zkMetadata) {
+ return true;
+ }
+
+ protected File downloadSegment(String segmentName, SegmentZKMetadata zkMetadata)
+ throws Exception {
+ // TODO: may support download from peer servers for RealTime table.
+ return downloadSegmentFromDeepStore(segmentName, zkMetadata);
+ }
+
+ /**
+ * Server restart during segment reload might leave segment directory in inconsistent state, like the index
+ * directory might not exist but segment backup directory existed. This method tries to recover from reload
+ * failure before checking the existence of the index directory and loading segment metadata from it.
+ */
+ private SegmentMetadata recoverSegmentQuietly(String segmentName) {
+ File indexDir = getSegmentDataDir(segmentName);
+ try {
+ LoaderUtils.reloadFailureRecovery(indexDir);
+ if (!indexDir.exists()) {
+ LOGGER.info("Segment: {} of table: {} is not found on disk", segmentName, _tableNameWithType);
+ return null;
+ }
+ SegmentMetadataImpl localMetadata = new SegmentMetadataImpl(indexDir);
+ LOGGER.info("Recovered segment: {} of table: {} with crc: {} from disk", segmentName, _tableNameWithType,
Review comment:
ack. changing to `Segment... is ready for loading`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on a change in pull request #7319: Move segment download and segment reload into TableManager
Posted by GitBox <gi...@apache.org>.
klsince commented on a change in pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#discussion_r708510718
##########
File path: pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixInstanceDataManager.java
##########
@@ -259,53 +255,50 @@ private void reloadSegment(String tableNameWithType, SegmentMetadata segmentMeta
return;
}
- Preconditions.checkState(indexDir.isDirectory(), "Index directory: %s is not a directory", indexDir);
- File parentFile = indexDir.getParentFile();
- File segmentBackupDir =
- new File(parentFile, indexDir.getName() + CommonConstants.Segment.SEGMENT_BACKUP_DIR_SUFFIX);
+ SegmentZKMetadata zkMetadata =
+ ZKMetadataProvider.getSegmentZKMetadata(_propertyStore, tableNameWithType, segmentName);
+ Preconditions.checkNotNull(zkMetadata);
// This method might modify the file on disk. Use segment lock to prevent race condition
Lock segmentLock = SegmentLocks.getSegmentLock(tableNameWithType, segmentName);
try {
segmentLock.lock();
- // First rename index directory to segment backup directory so that original segment have all file descriptors
- // point to the segment backup directory to ensure original segment serves queries properly
-
- // Rename index directory to segment backup directory (atomic)
- Preconditions.checkState(indexDir.renameTo(segmentBackupDir),
- "Failed to rename index directory: %s to segment backup directory: %s", indexDir, segmentBackupDir);
+ // Reloads an existing segment, and the local segment metadata is existing as asserted above.
+ tableDataManager
+ .reloadSegment(segmentName, new IndexLoadingConfig(_instanceDataManagerConfig, tableConfig), zkMetadata,
+ segmentMetadata, schema, forceDownload);
+ LOGGER.info("Reloaded segment: {} of table: {}", segmentName, tableNameWithType);
+ } finally {
+ segmentLock.unlock();
+ }
+ }
- // Copy from segment backup directory back to index directory
- FileUtils.copyDirectory(segmentBackupDir, indexDir);
+ @Override
+ public void addOrReplaceSegment(String tableNameWithType, String segmentName)
+ throws Exception {
+ LOGGER.info("Adding or replacing segment: {} for table: {}", segmentName, tableNameWithType);
- // Load from index directory
- ImmutableSegment immutableSegment = ImmutableSegmentLoader
- .load(indexDir, new IndexLoadingConfig(_instanceDataManagerConfig, tableConfig), schema);
+ // Get updated table config and segment metadata from Zookeeper.
+ TableConfig tableConfig = ZKMetadataProvider.getTableConfig(_propertyStore, tableNameWithType);
Review comment:
I kinda prefer to get tableConfig here in HelixInstanceDataMgr, e.g. `reloadAllSegments()` method gets tableConfig outside the for-loop, inside which it calls `reloadSegment(...tablcConfig)` for each segment. If getting tableConfig in those segment level method in tablemgr, it'd incur a zk call for each segment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on a change in pull request #7319: Move offline seg download code to OfflineTableManager
Posted by GitBox <gi...@apache.org>.
klsince commented on a change in pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#discussion_r704588292
##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/data/manager/TableDataManager.java
##########
@@ -146,4 +156,12 @@ void addSegment(String segmentName, TableConfig tableConfig, IndexLoadingConfig
* @return List of {@link SegmentErrorInfo}
*/
Map<String, SegmentErrorInfo> getSegmentErrors();
+
+ /**
+ * Get the dir for the given table segment, when TableDataManager object is not
+ * yet initialized for the given table.
+ */
+ static File getSegmentDataDir(String instanceDataDir, String tableNameWithType, String segmentName) {
Review comment:
I was thinking to encapsulate the details of assembling the path to segment under table mgr iface. not feel strong about this though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on pull request #7319: Move offline seg download code to OfflineTableManager
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#issuecomment-901316800
Can you please link the issue to the PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #7319: Move offline seg download code to OfflineTableManager
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#discussion_r692458669
##########
File path: pinot-server/src/main/java/org/apache/pinot/server/starter/helix/SegmentOnlineOfflineStateModelFactory.java
##########
@@ -208,7 +205,9 @@ public void onBecomeDroppedFromOffline(Message message, NotificationContext cont
try {
segmentLock.lock();
- final File segmentDir = new File(_fetcherAndLoader.getSegmentLocalDirectory(tableNameWithType, segmentName));
+ TableDataManager tableDataManager = _instanceDataManager.getTableDataManager(tableNameWithType);
+ Preconditions.checkNotNull(tableDataManager);
Review comment:
TableDataManager might not exist if there is no segment loaded for this table yet. Can we read the segment data dir from the `_instanceDataManager`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on a change in pull request #7319: Move offline seg download code to OfflineTableManager
Posted by GitBox <gi...@apache.org>.
klsince commented on a change in pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#discussion_r704813575
##########
File path: pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixInstanceDataManager.java
##########
@@ -311,6 +310,47 @@ private void reloadSegment(String tableNameWithType, SegmentMetadata segmentMeta
}
}
+ @Override
+ public void addOrReplaceSegment(String tableNameWithType, String segmentName, boolean forceDownload)
+ throws Exception {
+ LOGGER.info("Adding or replacing segment: {} for table: {} downloadIsForced: {}", segmentName, tableNameWithType,
+ forceDownload);
+
+ // Get updated table config and segment metadata from Zookeeper.
+ TableConfig tableConfig = ZKMetadataProvider.getTableConfig(_propertyStore, tableNameWithType);
+ Preconditions.checkNotNull(tableConfig);
+ SegmentZKMetadata zkMetadata =
+ ZKMetadataProvider.getSegmentZKMetadata(_propertyStore, tableNameWithType, segmentName);
+ Preconditions.checkNotNull(zkMetadata);
+
+ // This method might modify the file on disk. Use segment lock to prevent race condition
+ Lock segmentLock = SegmentLocks.getSegmentLock(tableNameWithType, segmentName);
+ try {
+ // Lock the segment to get its metadata, so that no other threads are modifying
+ // the disk files of this segment.
+ segmentLock.lock();
+ SegmentMetadata localMetadata = getSegmentMetadata(tableNameWithType, segmentName);
+
+ _tableDataManagerMap.computeIfAbsent(tableNameWithType, k -> createTableDataManager(k, tableConfig))
+ .addOrReplaceSegment(segmentName, new IndexLoadingConfig(_instanceDataManagerConfig, tableConfig),
+ localMetadata, zkMetadata, forceDownload);
+ LOGGER.info("Added or replaced segment: {} of table: {}", segmentName, tableNameWithType);
+ } finally {
+ segmentLock.unlock();
+ }
+ }
+
+ @Override
+ public void addOrReplaceAllSegments(String tableNameWithType, boolean forceDownload)
+ throws Exception {
+ LOGGER.info("Adding or replacing all segments in table: {} downloadIsForced: {}", tableNameWithType, forceDownload);
+ List<SegmentMetadata> segMds = getAllSegmentsMetadata(tableNameWithType);
+ for (SegmentMetadata segMd : segMds) {
+ addOrReplaceSegment(tableNameWithType, segMd.getName(), forceDownload);
Review comment:
after refactoring reloadSegment with forceDownload flag, there is no need for this method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on a change in pull request #7319: Move offline seg download code to OfflineTableManager
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on a change in pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#discussion_r704639430
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
##########
@@ -256,4 +267,147 @@ public void addSegmentError(String segmentName, SegmentErrorInfo segmentErrorInf
.collect(Collectors.toMap(map -> map.getKey().getSecond(), Map.Entry::getValue));
}
}
+
+ @Override
+ public void addOrReplaceSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentMetadata localMetadata, SegmentZKMetadata zkMetadata, boolean forceDownload)
+ throws Exception {
+ if (!forceDownload && !isNewSegment(zkMetadata, localMetadata)) {
Review comment:
Can we code this as `if (!forceDownload) {....} else {....}`. Or, better yet, `if (forceDownload) { ....} else {...}`
It will be much more readable (and better testable) instead of multiple `&&` etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #7319: Move segment download and segment reload into TableManager
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#discussion_r707781438
##########
File path: pinot-core/src/test/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManagerTest.java
##########
@@ -0,0 +1,56 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.data.manager.realtime;
+
+import org.apache.pinot.common.metadata.segment.SegmentZKMetadata;
+import org.apache.pinot.common.utils.HLCSegmentName;
+import org.apache.pinot.common.utils.LLCSegmentName;
+import org.testng.annotations.Test;
+
+import static org.apache.pinot.spi.utils.CommonConstants.Segment.Realtime.Status;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertTrue;
+
+
+public class RealtimeTableDataManagerTest {
+ @Test
+ public void testAllowDownload() {
+ RealtimeTableDataManager mgr = new RealtimeTableDataManager(null);
+
+ final String groupId = "myTable_REALTIME_1234567_0";
Review comment:
(code format) we don't usually put final in local variable
##########
File path: pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixInstanceDataManager.java
##########
@@ -259,53 +255,50 @@ private void reloadSegment(String tableNameWithType, SegmentMetadata segmentMeta
return;
}
- Preconditions.checkState(indexDir.isDirectory(), "Index directory: %s is not a directory", indexDir);
- File parentFile = indexDir.getParentFile();
- File segmentBackupDir =
- new File(parentFile, indexDir.getName() + CommonConstants.Segment.SEGMENT_BACKUP_DIR_SUFFIX);
+ SegmentZKMetadata zkMetadata =
+ ZKMetadataProvider.getSegmentZKMetadata(_propertyStore, tableNameWithType, segmentName);
+ Preconditions.checkNotNull(zkMetadata);
// This method might modify the file on disk. Use segment lock to prevent race condition
Lock segmentLock = SegmentLocks.getSegmentLock(tableNameWithType, segmentName);
try {
segmentLock.lock();
- // First rename index directory to segment backup directory so that original segment have all file descriptors
- // point to the segment backup directory to ensure original segment serves queries properly
-
- // Rename index directory to segment backup directory (atomic)
- Preconditions.checkState(indexDir.renameTo(segmentBackupDir),
- "Failed to rename index directory: %s to segment backup directory: %s", indexDir, segmentBackupDir);
+ // Reloads an existing segment, and the local segment metadata is existing as asserted above.
+ tableDataManager
+ .reloadSegment(segmentName, new IndexLoadingConfig(_instanceDataManagerConfig, tableConfig), zkMetadata,
+ segmentMetadata, schema, forceDownload);
+ LOGGER.info("Reloaded segment: {} of table: {}", segmentName, tableNameWithType);
+ } finally {
+ segmentLock.unlock();
+ }
+ }
- // Copy from segment backup directory back to index directory
- FileUtils.copyDirectory(segmentBackupDir, indexDir);
+ @Override
+ public void addOrReplaceSegment(String tableNameWithType, String segmentName)
+ throws Exception {
+ LOGGER.info("Adding or replacing segment: {} for table: {}", segmentName, tableNameWithType);
- // Load from index directory
- ImmutableSegment immutableSegment = ImmutableSegmentLoader
- .load(indexDir, new IndexLoadingConfig(_instanceDataManagerConfig, tableConfig), schema);
+ // Get updated table config and segment metadata from Zookeeper.
+ TableConfig tableConfig = ZKMetadataProvider.getTableConfig(_propertyStore, tableNameWithType);
Review comment:
Ideally, these should also be pushed down to the table data manager. But seems we need the `_instanceDataManagerConfig` to create the `IndexLoadingConfig`, and pushing the logic down requires more refactoring.
Maybe adding a TODO for future references?
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
##########
@@ -256,4 +270,219 @@ public void addSegmentError(String segmentName, SegmentErrorInfo segmentErrorInf
.collect(Collectors.toMap(map -> map.getKey().getSecond(), Map.Entry::getValue));
}
}
+
+ @Override
+ public void reloadSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentZKMetadata zkMetadata, SegmentMetadata localMetadata, Schema schema, boolean forceDownload)
+ throws Exception {
+ File indexDir = localMetadata.getIndexDir();
+ Preconditions.checkState(indexDir.isDirectory(), "Index directory: %s is not a directory", indexDir);
+
+ File parentFile = indexDir.getParentFile();
+ File segmentBackupDir =
+ new File(parentFile, indexDir.getName() + CommonConstants.Segment.SEGMENT_BACKUP_DIR_SUFFIX);
+
+ try {
+ // First rename index directory to segment backup directory so that original segment have all file descriptors
+ // point to the segment backup directory to ensure original segment serves queries properly
+
+ // Rename index directory to segment backup directory (atomic)
+ Preconditions.checkState(indexDir.renameTo(segmentBackupDir),
+ "Failed to rename index directory: %s to segment backup directory: %s", indexDir, segmentBackupDir);
+
+ // Download from remote or copy from local backup directory into index directory,
+ // and then continue to load the segment from index directory.
+ boolean shouldDownload = forceDownload || !hasSameCRC(zkMetadata, localMetadata);
+ if (shouldDownload && allowDownload(segmentName, zkMetadata)) {
+ if (forceDownload) {
+ LOGGER.info("Segment: {} of table: {} is forced to download", segmentName, _tableNameWithType);
+ } else {
+ LOGGER.info("Download segment:{} of table: {} as local crc:{} mismatches remote crc: {}", segmentName,
+ _tableNameWithType, localMetadata.getCrc(), zkMetadata.getCrc());
+ }
+ indexDir = downloadSegment(segmentName, zkMetadata);
+ } else {
+ LOGGER.info("Reload the local copy of segment: {} of table: {}", segmentName, _tableNameWithType);
+ FileUtils.copyDirectory(segmentBackupDir, indexDir);
+ }
+
+ // Load from index directory and replace the old segment in memory.
+ addSegment(ImmutableSegmentLoader.load(indexDir, indexLoadingConfig, schema));
+
+ // Rename segment backup directory to segment temporary directory (atomic)
+ // The reason to first rename then delete is that, renaming is an atomic operation, but deleting is not. When we
+ // rename the segment backup directory to segment temporary directory, we know the reload already succeeded, so
+ // that we can safely delete the segment temporary directory
+ File segmentTempDir = new File(parentFile, indexDir.getName() + CommonConstants.Segment.SEGMENT_TEMP_DIR_SUFFIX);
+ Preconditions.checkState(segmentBackupDir.renameTo(segmentTempDir),
+ "Failed to rename segment backup directory: %s to segment temporary directory: %s", segmentBackupDir,
+ segmentTempDir);
+ FileUtils.deleteDirectory(segmentTempDir);
+ } catch (Exception reloadFailureException) {
+ try {
+ LoaderUtils.reloadFailureRecovery(indexDir);
+ } catch (Exception recoveryFailureException) {
+ LOGGER.error("Failed to recover after reload failure", recoveryFailureException);
+ reloadFailureException.addSuppressed(recoveryFailureException);
+ }
+ throw reloadFailureException;
+ }
+ }
+
+ @Override
+ public void addOrReplaceSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentZKMetadata zkMetadata, @Nullable SegmentMetadata localMetadata)
+ throws Exception {
+ if (!isNewSegment(zkMetadata, localMetadata)) {
+ LOGGER.info("Segment: {} of table: {} has crc: {} same as before, already loaded, do nothing", segmentName,
+ _tableNameWithType, localMetadata.getCrc());
+ return;
+ }
+
+ // Try to recover if no local metadata is provided.
+ if (localMetadata == null) {
+ LOGGER.info("Segment: {} of table: {} is not loaded, checking disk", segmentName, _tableNameWithType);
+ localMetadata = recoverSegmentQuietly(segmentName);
+ if (!isNewSegment(zkMetadata, localMetadata)) {
+ LOGGER.info("Segment: {} of table {} has crc: {} same as before, loading", segmentName, _tableNameWithType,
+ localMetadata.getCrc());
+ if (loadSegmentQuietly(segmentName, indexLoadingConfig)) {
+ return;
+ }
+ localMetadata = null;
+ }
+ }
+
+ Preconditions.checkState(allowDownload(segmentName, zkMetadata), "Segment: %s of table: %s does not allow download",
+ segmentName, _tableNameWithType);
+
+ // Download segment and replace the local one, either due to failure to recover local segment,
+ // or the segment data is updated and has new CRC now.
+ if (localMetadata == null) {
+ LOGGER.info("Download segment: {} of table: {} as no good one exists locally", segmentName, _tableNameWithType);
+ } else {
+ LOGGER.info("Download segment: {} of table: {} as local crc: {} mismatches remote crc: {}.", segmentName,
+ _tableNameWithType, localMetadata.getCrc(), zkMetadata.getCrc());
+ }
+ File indexDir = downloadSegment(segmentName, zkMetadata);
+ addSegment(indexDir, indexLoadingConfig);
+ LOGGER.info("Downloaded and replaced segment: {} of table: {} with local crc now becomes: {} and remote crc: {}",
+ segmentName, _tableNameWithType, new SegmentMetadataImpl(indexDir).getCrc(), zkMetadata.getCrc());
+ }
+
+ protected boolean allowDownload(String segmentName, SegmentZKMetadata zkMetadata) {
+ return true;
+ }
+
+ protected File downloadSegment(String segmentName, SegmentZKMetadata zkMetadata)
+ throws Exception {
+ // TODO: may support download from peer servers for RealTime table.
+ return downloadSegmentFromDeepStore(segmentName, zkMetadata);
+ }
+
+ /**
+ * Server restart during segment reload might leave segment directory in inconsistent state, like the index
+ * directory might not exist but segment backup directory existed. This method tries to recover from reload
+ * failure before checking the existence of the index directory and loading segment metadata from it.
+ */
+ private SegmentMetadata recoverSegmentQuietly(String segmentName) {
+ File indexDir = getSegmentDataDir(segmentName);
+ try {
+ LoaderUtils.reloadFailureRecovery(indexDir);
+ if (!indexDir.exists()) {
+ LOGGER.info("Segment: {} of table: {} is not found on disk", segmentName, _tableNameWithType);
+ return null;
+ }
+ SegmentMetadataImpl localMetadata = new SegmentMetadataImpl(indexDir);
+ LOGGER.info("Recovered segment: {} of table: {} with crc: {} from disk", segmentName, _tableNameWithType,
+ localMetadata.getCrc());
+ return localMetadata;
+ } catch (Exception e) {
+ LOGGER.error("Failed to recover segment: {} of table: {} from disk", segmentName, _tableNameWithType, e);
+ FileUtils.deleteQuietly(indexDir);
+ return null;
+ }
+ }
+
+ private boolean loadSegmentQuietly(String segmentName, IndexLoadingConfig indexLoadingConfig) {
+ File indexDir = getSegmentDataDir(segmentName);
+ try {
+ addSegment(indexDir, indexLoadingConfig);
+ LOGGER.info("Loaded segment: {} of table: {} from disk", segmentName, _tableNameWithType);
+ return true;
+ } catch (Exception e) {
+ FileUtils.deleteQuietly(indexDir);
+ LOGGER.error("Failed to load segment: {} of table: {} from disk", segmentName, _tableNameWithType, e);
+ return false;
+ }
+ }
+
+ private File downloadSegmentFromDeepStore(String segmentName, SegmentZKMetadata zkMetadata)
+ throws Exception {
+ File tempRootDir = getSegmentDataDir("tmp-" + segmentName + "-" + UUID.randomUUID());
+ FileUtils.forceMkdir(tempRootDir);
+ try {
+ File tarFile = downloadAndDecrypt(segmentName, zkMetadata, tempRootDir);
+ return untarAndMoveSegment(segmentName, tarFile, tempRootDir);
+ } finally {
+ FileUtils.deleteQuietly(tempRootDir);
+ }
+ }
+
+ @VisibleForTesting
+ File downloadAndDecrypt(String segmentName, SegmentZKMetadata zkMetadata, File tempRootDir)
+ throws Exception {
+ File tarFile = new File(tempRootDir, segmentName + TarGzCompressionUtils.TAR_GZ_FILE_EXTENSION);
+ String uri = zkMetadata.getDownloadUrl();
+ try {
+ SegmentFetcherFactory.fetchAndDecryptSegmentToLocal(uri, tarFile, zkMetadata.getCrypterName());
+ LOGGER.info("Downloaded tarred segment: {} for table: {} from: {} to: {}, file length: {}", segmentName,
+ _tableNameWithType, uri, tarFile, tarFile.length());
+ return tarFile;
+ } catch (AttemptsExceededException e) {
+ LOGGER.error("Attempts exceeded when downloading segment: {} for table: {} from: {} to: {}", segmentName,
+ _tableNameWithType, uri, tarFile);
+ _serverMetrics.addMeteredTableValue(_tableNameWithType, ServerMeter.SEGMENT_DOWNLOAD_FAILURES, 1L);
+ Utils.rethrowException(e);
+ }
+ return null;
+ }
+
+ @VisibleForTesting
+ File untarAndMoveSegment(String segmentName, File tarFile, File tempRootDir) {
+ File untarDir = new File(tempRootDir, segmentName);
+ try {
+ // If an exception is thrown when untarring, it means the tar file is broken
+ // or not found after the retry. Thus, there's no need to retry again.
+ File untaredSegDir = TarGzCompressionUtils.untar(tarFile, untarDir).get(0);
+ LOGGER.info("Uncompressed tar file: {} into target dir: {}", tarFile, untarDir);
+ // Replace the existing index directory.
+ File indexDir = getSegmentDataDir(segmentName);
+ FileUtils.deleteDirectory(indexDir);
+ FileUtils.moveDirectory(untaredSegDir, indexDir);
+ LOGGER.info("Successfully downloaded segment: {} of table: {} to index dir: {}", segmentName, _tableNameWithType,
+ indexDir);
+ return indexDir;
+ } catch (Exception e) {
+ LOGGER.error("Failed to untar segment: {} of table: {} from: {} to: {}", segmentName, _tableNameWithType, tarFile,
+ untarDir);
+ _serverMetrics.addMeteredTableValue(_tableNameWithType, ServerMeter.UNTAR_FAILURES, 1L);
+ Utils.rethrowException(e);
Review comment:
Also add `throws Exception` in the method signature
```suggestion
throw e;
```
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
##########
@@ -256,4 +270,219 @@ public void addSegmentError(String segmentName, SegmentErrorInfo segmentErrorInf
.collect(Collectors.toMap(map -> map.getKey().getSecond(), Map.Entry::getValue));
}
}
+
+ @Override
+ public void reloadSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentZKMetadata zkMetadata, SegmentMetadata localMetadata, Schema schema, boolean forceDownload)
+ throws Exception {
+ File indexDir = localMetadata.getIndexDir();
+ Preconditions.checkState(indexDir.isDirectory(), "Index directory: %s is not a directory", indexDir);
+
+ File parentFile = indexDir.getParentFile();
+ File segmentBackupDir =
+ new File(parentFile, indexDir.getName() + CommonConstants.Segment.SEGMENT_BACKUP_DIR_SUFFIX);
+
+ try {
+ // First rename index directory to segment backup directory so that original segment have all file descriptors
+ // point to the segment backup directory to ensure original segment serves queries properly
+
+ // Rename index directory to segment backup directory (atomic)
+ Preconditions.checkState(indexDir.renameTo(segmentBackupDir),
+ "Failed to rename index directory: %s to segment backup directory: %s", indexDir, segmentBackupDir);
+
+ // Download from remote or copy from local backup directory into index directory,
+ // and then continue to load the segment from index directory.
+ boolean shouldDownload = forceDownload || !hasSameCRC(zkMetadata, localMetadata);
+ if (shouldDownload && allowDownload(segmentName, zkMetadata)) {
+ if (forceDownload) {
+ LOGGER.info("Segment: {} of table: {} is forced to download", segmentName, _tableNameWithType);
+ } else {
+ LOGGER.info("Download segment:{} of table: {} as local crc:{} mismatches remote crc: {}", segmentName,
+ _tableNameWithType, localMetadata.getCrc(), zkMetadata.getCrc());
+ }
+ indexDir = downloadSegment(segmentName, zkMetadata);
+ } else {
+ LOGGER.info("Reload the local copy of segment: {} of table: {}", segmentName, _tableNameWithType);
+ FileUtils.copyDirectory(segmentBackupDir, indexDir);
+ }
+
+ // Load from index directory and replace the old segment in memory.
+ addSegment(ImmutableSegmentLoader.load(indexDir, indexLoadingConfig, schema));
+
+ // Rename segment backup directory to segment temporary directory (atomic)
+ // The reason to first rename then delete is that, renaming is an atomic operation, but deleting is not. When we
+ // rename the segment backup directory to segment temporary directory, we know the reload already succeeded, so
+ // that we can safely delete the segment temporary directory
+ File segmentTempDir = new File(parentFile, indexDir.getName() + CommonConstants.Segment.SEGMENT_TEMP_DIR_SUFFIX);
+ Preconditions.checkState(segmentBackupDir.renameTo(segmentTempDir),
+ "Failed to rename segment backup directory: %s to segment temporary directory: %s", segmentBackupDir,
+ segmentTempDir);
+ FileUtils.deleteDirectory(segmentTempDir);
+ } catch (Exception reloadFailureException) {
+ try {
+ LoaderUtils.reloadFailureRecovery(indexDir);
+ } catch (Exception recoveryFailureException) {
+ LOGGER.error("Failed to recover after reload failure", recoveryFailureException);
+ reloadFailureException.addSuppressed(recoveryFailureException);
+ }
+ throw reloadFailureException;
+ }
+ }
+
+ @Override
+ public void addOrReplaceSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentZKMetadata zkMetadata, @Nullable SegmentMetadata localMetadata)
+ throws Exception {
+ if (!isNewSegment(zkMetadata, localMetadata)) {
+ LOGGER.info("Segment: {} of table: {} has crc: {} same as before, already loaded, do nothing", segmentName,
+ _tableNameWithType, localMetadata.getCrc());
+ return;
+ }
+
+ // Try to recover if no local metadata is provided.
+ if (localMetadata == null) {
+ LOGGER.info("Segment: {} of table: {} is not loaded, checking disk", segmentName, _tableNameWithType);
+ localMetadata = recoverSegmentQuietly(segmentName);
+ if (!isNewSegment(zkMetadata, localMetadata)) {
+ LOGGER.info("Segment: {} of table {} has crc: {} same as before, loading", segmentName, _tableNameWithType,
+ localMetadata.getCrc());
+ if (loadSegmentQuietly(segmentName, indexLoadingConfig)) {
+ return;
+ }
+ localMetadata = null;
+ }
+ }
+
+ Preconditions.checkState(allowDownload(segmentName, zkMetadata), "Segment: %s of table: %s does not allow download",
+ segmentName, _tableNameWithType);
+
+ // Download segment and replace the local one, either due to failure to recover local segment,
+ // or the segment data is updated and has new CRC now.
+ if (localMetadata == null) {
+ LOGGER.info("Download segment: {} of table: {} as no good one exists locally", segmentName, _tableNameWithType);
+ } else {
+ LOGGER.info("Download segment: {} of table: {} as local crc: {} mismatches remote crc: {}.", segmentName,
+ _tableNameWithType, localMetadata.getCrc(), zkMetadata.getCrc());
+ }
+ File indexDir = downloadSegment(segmentName, zkMetadata);
+ addSegment(indexDir, indexLoadingConfig);
+ LOGGER.info("Downloaded and replaced segment: {} of table: {} with local crc now becomes: {} and remote crc: {}",
Review comment:
Avoid creating another copy of segment metadata here (`new SegmentMetadataImpl(indexDir).getCrc()`). Local crc should always match remote crc here.
Also, the segment might not be replaced, but newly added
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
##########
@@ -256,4 +270,219 @@ public void addSegmentError(String segmentName, SegmentErrorInfo segmentErrorInf
.collect(Collectors.toMap(map -> map.getKey().getSecond(), Map.Entry::getValue));
}
}
+
+ @Override
+ public void reloadSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentZKMetadata zkMetadata, SegmentMetadata localMetadata, Schema schema, boolean forceDownload)
+ throws Exception {
+ File indexDir = localMetadata.getIndexDir();
+ Preconditions.checkState(indexDir.isDirectory(), "Index directory: %s is not a directory", indexDir);
+
+ File parentFile = indexDir.getParentFile();
+ File segmentBackupDir =
+ new File(parentFile, indexDir.getName() + CommonConstants.Segment.SEGMENT_BACKUP_DIR_SUFFIX);
+
+ try {
+ // First rename index directory to segment backup directory so that original segment have all file descriptors
+ // point to the segment backup directory to ensure original segment serves queries properly
+
+ // Rename index directory to segment backup directory (atomic)
+ Preconditions.checkState(indexDir.renameTo(segmentBackupDir),
+ "Failed to rename index directory: %s to segment backup directory: %s", indexDir, segmentBackupDir);
+
+ // Download from remote or copy from local backup directory into index directory,
+ // and then continue to load the segment from index directory.
+ boolean shouldDownload = forceDownload || !hasSameCRC(zkMetadata, localMetadata);
+ if (shouldDownload && allowDownload(segmentName, zkMetadata)) {
+ if (forceDownload) {
+ LOGGER.info("Segment: {} of table: {} is forced to download", segmentName, _tableNameWithType);
+ } else {
+ LOGGER.info("Download segment:{} of table: {} as local crc:{} mismatches remote crc: {}", segmentName,
+ _tableNameWithType, localMetadata.getCrc(), zkMetadata.getCrc());
+ }
+ indexDir = downloadSegment(segmentName, zkMetadata);
+ } else {
+ LOGGER.info("Reload the local copy of segment: {} of table: {}", segmentName, _tableNameWithType);
+ FileUtils.copyDirectory(segmentBackupDir, indexDir);
+ }
+
+ // Load from index directory and replace the old segment in memory.
+ addSegment(ImmutableSegmentLoader.load(indexDir, indexLoadingConfig, schema));
+
+ // Rename segment backup directory to segment temporary directory (atomic)
+ // The reason to first rename then delete is that, renaming is an atomic operation, but deleting is not. When we
+ // rename the segment backup directory to segment temporary directory, we know the reload already succeeded, so
+ // that we can safely delete the segment temporary directory
+ File segmentTempDir = new File(parentFile, indexDir.getName() + CommonConstants.Segment.SEGMENT_TEMP_DIR_SUFFIX);
+ Preconditions.checkState(segmentBackupDir.renameTo(segmentTempDir),
+ "Failed to rename segment backup directory: %s to segment temporary directory: %s", segmentBackupDir,
+ segmentTempDir);
+ FileUtils.deleteDirectory(segmentTempDir);
+ } catch (Exception reloadFailureException) {
+ try {
+ LoaderUtils.reloadFailureRecovery(indexDir);
+ } catch (Exception recoveryFailureException) {
+ LOGGER.error("Failed to recover after reload failure", recoveryFailureException);
+ reloadFailureException.addSuppressed(recoveryFailureException);
+ }
+ throw reloadFailureException;
+ }
+ }
+
+ @Override
+ public void addOrReplaceSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentZKMetadata zkMetadata, @Nullable SegmentMetadata localMetadata)
+ throws Exception {
+ if (!isNewSegment(zkMetadata, localMetadata)) {
+ LOGGER.info("Segment: {} of table: {} has crc: {} same as before, already loaded, do nothing", segmentName,
+ _tableNameWithType, localMetadata.getCrc());
+ return;
+ }
+
+ // Try to recover if no local metadata is provided.
+ if (localMetadata == null) {
+ LOGGER.info("Segment: {} of table: {} is not loaded, checking disk", segmentName, _tableNameWithType);
+ localMetadata = recoverSegmentQuietly(segmentName);
+ if (!isNewSegment(zkMetadata, localMetadata)) {
+ LOGGER.info("Segment: {} of table {} has crc: {} same as before, loading", segmentName, _tableNameWithType,
+ localMetadata.getCrc());
+ if (loadSegmentQuietly(segmentName, indexLoadingConfig)) {
+ return;
+ }
+ localMetadata = null;
Review comment:
Add some comments on why we set the local metadata to `null` here
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
##########
@@ -256,4 +270,219 @@ public void addSegmentError(String segmentName, SegmentErrorInfo segmentErrorInf
.collect(Collectors.toMap(map -> map.getKey().getSecond(), Map.Entry::getValue));
}
}
+
+ @Override
+ public void reloadSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentZKMetadata zkMetadata, SegmentMetadata localMetadata, Schema schema, boolean forceDownload)
+ throws Exception {
+ File indexDir = localMetadata.getIndexDir();
+ Preconditions.checkState(indexDir.isDirectory(), "Index directory: %s is not a directory", indexDir);
+
+ File parentFile = indexDir.getParentFile();
+ File segmentBackupDir =
+ new File(parentFile, indexDir.getName() + CommonConstants.Segment.SEGMENT_BACKUP_DIR_SUFFIX);
+
+ try {
+ // First rename index directory to segment backup directory so that original segment have all file descriptors
+ // point to the segment backup directory to ensure original segment serves queries properly
+
+ // Rename index directory to segment backup directory (atomic)
+ Preconditions.checkState(indexDir.renameTo(segmentBackupDir),
+ "Failed to rename index directory: %s to segment backup directory: %s", indexDir, segmentBackupDir);
+
+ // Download from remote or copy from local backup directory into index directory,
+ // and then continue to load the segment from index directory.
+ boolean shouldDownload = forceDownload || !hasSameCRC(zkMetadata, localMetadata);
+ if (shouldDownload && allowDownload(segmentName, zkMetadata)) {
+ if (forceDownload) {
+ LOGGER.info("Segment: {} of table: {} is forced to download", segmentName, _tableNameWithType);
+ } else {
+ LOGGER.info("Download segment:{} of table: {} as local crc:{} mismatches remote crc: {}", segmentName,
+ _tableNameWithType, localMetadata.getCrc(), zkMetadata.getCrc());
+ }
+ indexDir = downloadSegment(segmentName, zkMetadata);
+ } else {
+ LOGGER.info("Reload the local copy of segment: {} of table: {}", segmentName, _tableNameWithType);
+ FileUtils.copyDirectory(segmentBackupDir, indexDir);
+ }
+
+ // Load from index directory and replace the old segment in memory.
+ addSegment(ImmutableSegmentLoader.load(indexDir, indexLoadingConfig, schema));
+
+ // Rename segment backup directory to segment temporary directory (atomic)
+ // The reason to first rename then delete is that, renaming is an atomic operation, but deleting is not. When we
+ // rename the segment backup directory to segment temporary directory, we know the reload already succeeded, so
+ // that we can safely delete the segment temporary directory
+ File segmentTempDir = new File(parentFile, indexDir.getName() + CommonConstants.Segment.SEGMENT_TEMP_DIR_SUFFIX);
+ Preconditions.checkState(segmentBackupDir.renameTo(segmentTempDir),
+ "Failed to rename segment backup directory: %s to segment temporary directory: %s", segmentBackupDir,
+ segmentTempDir);
+ FileUtils.deleteDirectory(segmentTempDir);
+ } catch (Exception reloadFailureException) {
+ try {
+ LoaderUtils.reloadFailureRecovery(indexDir);
+ } catch (Exception recoveryFailureException) {
+ LOGGER.error("Failed to recover after reload failure", recoveryFailureException);
+ reloadFailureException.addSuppressed(recoveryFailureException);
+ }
+ throw reloadFailureException;
+ }
+ }
+
+ @Override
+ public void addOrReplaceSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentZKMetadata zkMetadata, @Nullable SegmentMetadata localMetadata)
+ throws Exception {
+ if (!isNewSegment(zkMetadata, localMetadata)) {
+ LOGGER.info("Segment: {} of table: {} has crc: {} same as before, already loaded, do nothing", segmentName,
+ _tableNameWithType, localMetadata.getCrc());
+ return;
+ }
+
+ // Try to recover if no local metadata is provided.
+ if (localMetadata == null) {
+ LOGGER.info("Segment: {} of table: {} is not loaded, checking disk", segmentName, _tableNameWithType);
+ localMetadata = recoverSegmentQuietly(segmentName);
+ if (!isNewSegment(zkMetadata, localMetadata)) {
+ LOGGER.info("Segment: {} of table {} has crc: {} same as before, loading", segmentName, _tableNameWithType,
+ localMetadata.getCrc());
+ if (loadSegmentQuietly(segmentName, indexLoadingConfig)) {
+ return;
+ }
+ localMetadata = null;
+ }
+ }
+
+ Preconditions.checkState(allowDownload(segmentName, zkMetadata), "Segment: %s of table: %s does not allow download",
+ segmentName, _tableNameWithType);
+
+ // Download segment and replace the local one, either due to failure to recover local segment,
+ // or the segment data is updated and has new CRC now.
+ if (localMetadata == null) {
+ LOGGER.info("Download segment: {} of table: {} as no good one exists locally", segmentName, _tableNameWithType);
+ } else {
+ LOGGER.info("Download segment: {} of table: {} as local crc: {} mismatches remote crc: {}.", segmentName,
+ _tableNameWithType, localMetadata.getCrc(), zkMetadata.getCrc());
+ }
+ File indexDir = downloadSegment(segmentName, zkMetadata);
+ addSegment(indexDir, indexLoadingConfig);
+ LOGGER.info("Downloaded and replaced segment: {} of table: {} with local crc now becomes: {} and remote crc: {}",
+ segmentName, _tableNameWithType, new SegmentMetadataImpl(indexDir).getCrc(), zkMetadata.getCrc());
+ }
+
+ protected boolean allowDownload(String segmentName, SegmentZKMetadata zkMetadata) {
+ return true;
+ }
+
+ protected File downloadSegment(String segmentName, SegmentZKMetadata zkMetadata)
+ throws Exception {
+ // TODO: may support download from peer servers for RealTime table.
+ return downloadSegmentFromDeepStore(segmentName, zkMetadata);
+ }
+
+ /**
+ * Server restart during segment reload might leave segment directory in inconsistent state, like the index
+ * directory might not exist but segment backup directory existed. This method tries to recover from reload
+ * failure before checking the existence of the index directory and loading segment metadata from it.
+ */
+ private SegmentMetadata recoverSegmentQuietly(String segmentName) {
+ File indexDir = getSegmentDataDir(segmentName);
+ try {
+ LoaderUtils.reloadFailureRecovery(indexDir);
+ if (!indexDir.exists()) {
+ LOGGER.info("Segment: {} of table: {} is not found on disk", segmentName, _tableNameWithType);
+ return null;
+ }
+ SegmentMetadataImpl localMetadata = new SegmentMetadataImpl(indexDir);
+ LOGGER.info("Recovered segment: {} of table: {} with crc: {} from disk", segmentName, _tableNameWithType,
+ localMetadata.getCrc());
+ return localMetadata;
+ } catch (Exception e) {
+ LOGGER.error("Failed to recover segment: {} of table: {} from disk", segmentName, _tableNameWithType, e);
+ FileUtils.deleteQuietly(indexDir);
+ return null;
+ }
+ }
+
+ private boolean loadSegmentQuietly(String segmentName, IndexLoadingConfig indexLoadingConfig) {
+ File indexDir = getSegmentDataDir(segmentName);
+ try {
+ addSegment(indexDir, indexLoadingConfig);
+ LOGGER.info("Loaded segment: {} of table: {} from disk", segmentName, _tableNameWithType);
+ return true;
+ } catch (Exception e) {
+ FileUtils.deleteQuietly(indexDir);
+ LOGGER.error("Failed to load segment: {} of table: {} from disk", segmentName, _tableNameWithType, e);
+ return false;
+ }
+ }
+
+ private File downloadSegmentFromDeepStore(String segmentName, SegmentZKMetadata zkMetadata)
+ throws Exception {
+ File tempRootDir = getSegmentDataDir("tmp-" + segmentName + "-" + UUID.randomUUID());
+ FileUtils.forceMkdir(tempRootDir);
+ try {
+ File tarFile = downloadAndDecrypt(segmentName, zkMetadata, tempRootDir);
+ return untarAndMoveSegment(segmentName, tarFile, tempRootDir);
+ } finally {
+ FileUtils.deleteQuietly(tempRootDir);
+ }
+ }
+
+ @VisibleForTesting
+ File downloadAndDecrypt(String segmentName, SegmentZKMetadata zkMetadata, File tempRootDir)
+ throws Exception {
+ File tarFile = new File(tempRootDir, segmentName + TarGzCompressionUtils.TAR_GZ_FILE_EXTENSION);
+ String uri = zkMetadata.getDownloadUrl();
+ try {
+ SegmentFetcherFactory.fetchAndDecryptSegmentToLocal(uri, tarFile, zkMetadata.getCrypterName());
+ LOGGER.info("Downloaded tarred segment: {} for table: {} from: {} to: {}, file length: {}", segmentName,
+ _tableNameWithType, uri, tarFile, tarFile.length());
+ return tarFile;
+ } catch (AttemptsExceededException e) {
+ LOGGER.error("Attempts exceeded when downloading segment: {} for table: {} from: {} to: {}", segmentName,
+ _tableNameWithType, uri, tarFile);
+ _serverMetrics.addMeteredTableValue(_tableNameWithType, ServerMeter.SEGMENT_DOWNLOAD_FAILURES, 1L);
+ Utils.rethrowException(e);
Review comment:
```suggestion
throw e;
```
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
##########
@@ -256,4 +270,219 @@ public void addSegmentError(String segmentName, SegmentErrorInfo segmentErrorInf
.collect(Collectors.toMap(map -> map.getKey().getSecond(), Map.Entry::getValue));
}
}
+
+ @Override
+ public void reloadSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentZKMetadata zkMetadata, SegmentMetadata localMetadata, Schema schema, boolean forceDownload)
Review comment:
`schema` is nullable
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
##########
@@ -256,4 +270,219 @@ public void addSegmentError(String segmentName, SegmentErrorInfo segmentErrorInf
.collect(Collectors.toMap(map -> map.getKey().getSecond(), Map.Entry::getValue));
}
}
+
+ @Override
+ public void reloadSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentZKMetadata zkMetadata, SegmentMetadata localMetadata, Schema schema, boolean forceDownload)
+ throws Exception {
+ File indexDir = localMetadata.getIndexDir();
+ Preconditions.checkState(indexDir.isDirectory(), "Index directory: %s is not a directory", indexDir);
+
+ File parentFile = indexDir.getParentFile();
+ File segmentBackupDir =
+ new File(parentFile, indexDir.getName() + CommonConstants.Segment.SEGMENT_BACKUP_DIR_SUFFIX);
+
+ try {
+ // First rename index directory to segment backup directory so that original segment have all file descriptors
+ // point to the segment backup directory to ensure original segment serves queries properly
+
+ // Rename index directory to segment backup directory (atomic)
+ Preconditions.checkState(indexDir.renameTo(segmentBackupDir),
+ "Failed to rename index directory: %s to segment backup directory: %s", indexDir, segmentBackupDir);
+
+ // Download from remote or copy from local backup directory into index directory,
+ // and then continue to load the segment from index directory.
+ boolean shouldDownload = forceDownload || !hasSameCRC(zkMetadata, localMetadata);
+ if (shouldDownload && allowDownload(segmentName, zkMetadata)) {
+ if (forceDownload) {
+ LOGGER.info("Segment: {} of table: {} is forced to download", segmentName, _tableNameWithType);
+ } else {
+ LOGGER.info("Download segment:{} of table: {} as local crc:{} mismatches remote crc: {}", segmentName,
Review comment:
(nit)
```suggestion
LOGGER.info("Download segment: {} of table: {} as local crc: {} mismatches remote crc: {}", segmentName,
```
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
##########
@@ -256,4 +270,219 @@ public void addSegmentError(String segmentName, SegmentErrorInfo segmentErrorInf
.collect(Collectors.toMap(map -> map.getKey().getSecond(), Map.Entry::getValue));
}
}
+
+ @Override
+ public void reloadSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentZKMetadata zkMetadata, SegmentMetadata localMetadata, Schema schema, boolean forceDownload)
+ throws Exception {
+ File indexDir = localMetadata.getIndexDir();
+ Preconditions.checkState(indexDir.isDirectory(), "Index directory: %s is not a directory", indexDir);
+
+ File parentFile = indexDir.getParentFile();
+ File segmentBackupDir =
+ new File(parentFile, indexDir.getName() + CommonConstants.Segment.SEGMENT_BACKUP_DIR_SUFFIX);
+
+ try {
+ // First rename index directory to segment backup directory so that original segment have all file descriptors
+ // point to the segment backup directory to ensure original segment serves queries properly
+
+ // Rename index directory to segment backup directory (atomic)
+ Preconditions.checkState(indexDir.renameTo(segmentBackupDir),
+ "Failed to rename index directory: %s to segment backup directory: %s", indexDir, segmentBackupDir);
+
+ // Download from remote or copy from local backup directory into index directory,
+ // and then continue to load the segment from index directory.
+ boolean shouldDownload = forceDownload || !hasSameCRC(zkMetadata, localMetadata);
+ if (shouldDownload && allowDownload(segmentName, zkMetadata)) {
+ if (forceDownload) {
+ LOGGER.info("Segment: {} of table: {} is forced to download", segmentName, _tableNameWithType);
+ } else {
+ LOGGER.info("Download segment:{} of table: {} as local crc:{} mismatches remote crc: {}", segmentName,
+ _tableNameWithType, localMetadata.getCrc(), zkMetadata.getCrc());
+ }
+ indexDir = downloadSegment(segmentName, zkMetadata);
+ } else {
+ LOGGER.info("Reload the local copy of segment: {} of table: {}", segmentName, _tableNameWithType);
+ FileUtils.copyDirectory(segmentBackupDir, indexDir);
+ }
+
+ // Load from index directory and replace the old segment in memory.
+ addSegment(ImmutableSegmentLoader.load(indexDir, indexLoadingConfig, schema));
+
+ // Rename segment backup directory to segment temporary directory (atomic)
+ // The reason to first rename then delete is that, renaming is an atomic operation, but deleting is not. When we
+ // rename the segment backup directory to segment temporary directory, we know the reload already succeeded, so
+ // that we can safely delete the segment temporary directory
+ File segmentTempDir = new File(parentFile, indexDir.getName() + CommonConstants.Segment.SEGMENT_TEMP_DIR_SUFFIX);
+ Preconditions.checkState(segmentBackupDir.renameTo(segmentTempDir),
+ "Failed to rename segment backup directory: %s to segment temporary directory: %s", segmentBackupDir,
+ segmentTempDir);
+ FileUtils.deleteDirectory(segmentTempDir);
+ } catch (Exception reloadFailureException) {
+ try {
+ LoaderUtils.reloadFailureRecovery(indexDir);
+ } catch (Exception recoveryFailureException) {
+ LOGGER.error("Failed to recover after reload failure", recoveryFailureException);
+ reloadFailureException.addSuppressed(recoveryFailureException);
+ }
+ throw reloadFailureException;
+ }
+ }
+
+ @Override
+ public void addOrReplaceSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentZKMetadata zkMetadata, @Nullable SegmentMetadata localMetadata)
+ throws Exception {
+ if (!isNewSegment(zkMetadata, localMetadata)) {
+ LOGGER.info("Segment: {} of table: {} has crc: {} same as before, already loaded, do nothing", segmentName,
+ _tableNameWithType, localMetadata.getCrc());
+ return;
+ }
+
+ // Try to recover if no local metadata is provided.
+ if (localMetadata == null) {
+ LOGGER.info("Segment: {} of table: {} is not loaded, checking disk", segmentName, _tableNameWithType);
+ localMetadata = recoverSegmentQuietly(segmentName);
+ if (!isNewSegment(zkMetadata, localMetadata)) {
+ LOGGER.info("Segment: {} of table {} has crc: {} same as before, loading", segmentName, _tableNameWithType,
+ localMetadata.getCrc());
+ if (loadSegmentQuietly(segmentName, indexLoadingConfig)) {
+ return;
+ }
+ localMetadata = null;
+ }
+ }
+
+ Preconditions.checkState(allowDownload(segmentName, zkMetadata), "Segment: %s of table: %s does not allow download",
+ segmentName, _tableNameWithType);
+
+ // Download segment and replace the local one, either due to failure to recover local segment,
+ // or the segment data is updated and has new CRC now.
+ if (localMetadata == null) {
+ LOGGER.info("Download segment: {} of table: {} as no good one exists locally", segmentName, _tableNameWithType);
+ } else {
+ LOGGER.info("Download segment: {} of table: {} as local crc: {} mismatches remote crc: {}.", segmentName,
+ _tableNameWithType, localMetadata.getCrc(), zkMetadata.getCrc());
+ }
+ File indexDir = downloadSegment(segmentName, zkMetadata);
+ addSegment(indexDir, indexLoadingConfig);
+ LOGGER.info("Downloaded and replaced segment: {} of table: {} with local crc now becomes: {} and remote crc: {}",
+ segmentName, _tableNameWithType, new SegmentMetadataImpl(indexDir).getCrc(), zkMetadata.getCrc());
+ }
+
+ protected boolean allowDownload(String segmentName, SegmentZKMetadata zkMetadata) {
+ return true;
+ }
+
+ protected File downloadSegment(String segmentName, SegmentZKMetadata zkMetadata)
+ throws Exception {
+ // TODO: may support download from peer servers for RealTime table.
+ return downloadSegmentFromDeepStore(segmentName, zkMetadata);
+ }
+
+ /**
+ * Server restart during segment reload might leave segment directory in inconsistent state, like the index
+ * directory might not exist but segment backup directory existed. This method tries to recover from reload
+ * failure before checking the existence of the index directory and loading segment metadata from it.
+ */
+ private SegmentMetadata recoverSegmentQuietly(String segmentName) {
+ File indexDir = getSegmentDataDir(segmentName);
+ try {
+ LoaderUtils.reloadFailureRecovery(indexDir);
+ if (!indexDir.exists()) {
+ LOGGER.info("Segment: {} of table: {} is not found on disk", segmentName, _tableNameWithType);
+ return null;
+ }
+ SegmentMetadataImpl localMetadata = new SegmentMetadataImpl(indexDir);
+ LOGGER.info("Recovered segment: {} of table: {} with crc: {} from disk", segmentName, _tableNameWithType,
Review comment:
This info is misleading. The segment might not be recovered, but simply not loaded into memory yet.
##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/data/manager/TableDataManager.java
##########
@@ -78,6 +81,27 @@ void addSegment(File indexDir, IndexLoadingConfig indexLoadingConfig)
void addSegment(String segmentName, TableConfig tableConfig, IndexLoadingConfig indexLoadingConfig)
throws Exception;
+ /**
+ * Reloads an existing immutable segment for the table, which can be an OFFLINE or REALTIME table.
+ * A new segment may be downloaded if the local one has a different CRC; or can be forced to download
+ * if forceDownload flag is true. This operation is conducted within a failure handling framework
+ * and made transparent to ongoing queries, because the segment is in online serving state.
+ */
+ void reloadSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentZKMetadata zkMetadata, SegmentMetadata localMetadata, Schema schema, boolean forceDownload)
Review comment:
`schema` is nullable?
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
##########
@@ -256,4 +270,219 @@ public void addSegmentError(String segmentName, SegmentErrorInfo segmentErrorInf
.collect(Collectors.toMap(map -> map.getKey().getSecond(), Map.Entry::getValue));
}
}
+
+ @Override
+ public void reloadSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentZKMetadata zkMetadata, SegmentMetadata localMetadata, Schema schema, boolean forceDownload)
+ throws Exception {
+ File indexDir = localMetadata.getIndexDir();
+ Preconditions.checkState(indexDir.isDirectory(), "Index directory: %s is not a directory", indexDir);
+
+ File parentFile = indexDir.getParentFile();
+ File segmentBackupDir =
+ new File(parentFile, indexDir.getName() + CommonConstants.Segment.SEGMENT_BACKUP_DIR_SUFFIX);
+
+ try {
+ // First rename index directory to segment backup directory so that original segment have all file descriptors
+ // point to the segment backup directory to ensure original segment serves queries properly
+
+ // Rename index directory to segment backup directory (atomic)
+ Preconditions.checkState(indexDir.renameTo(segmentBackupDir),
+ "Failed to rename index directory: %s to segment backup directory: %s", indexDir, segmentBackupDir);
+
+ // Download from remote or copy from local backup directory into index directory,
+ // and then continue to load the segment from index directory.
+ boolean shouldDownload = forceDownload || !hasSameCRC(zkMetadata, localMetadata);
+ if (shouldDownload && allowDownload(segmentName, zkMetadata)) {
+ if (forceDownload) {
+ LOGGER.info("Segment: {} of table: {} is forced to download", segmentName, _tableNameWithType);
+ } else {
+ LOGGER.info("Download segment:{} of table: {} as local crc:{} mismatches remote crc: {}", segmentName,
+ _tableNameWithType, localMetadata.getCrc(), zkMetadata.getCrc());
+ }
+ indexDir = downloadSegment(segmentName, zkMetadata);
+ } else {
+ LOGGER.info("Reload the local copy of segment: {} of table: {}", segmentName, _tableNameWithType);
+ FileUtils.copyDirectory(segmentBackupDir, indexDir);
+ }
+
+ // Load from index directory and replace the old segment in memory.
+ addSegment(ImmutableSegmentLoader.load(indexDir, indexLoadingConfig, schema));
+
+ // Rename segment backup directory to segment temporary directory (atomic)
+ // The reason to first rename then delete is that, renaming is an atomic operation, but deleting is not. When we
+ // rename the segment backup directory to segment temporary directory, we know the reload already succeeded, so
+ // that we can safely delete the segment temporary directory
+ File segmentTempDir = new File(parentFile, indexDir.getName() + CommonConstants.Segment.SEGMENT_TEMP_DIR_SUFFIX);
+ Preconditions.checkState(segmentBackupDir.renameTo(segmentTempDir),
+ "Failed to rename segment backup directory: %s to segment temporary directory: %s", segmentBackupDir,
+ segmentTempDir);
+ FileUtils.deleteDirectory(segmentTempDir);
+ } catch (Exception reloadFailureException) {
+ try {
+ LoaderUtils.reloadFailureRecovery(indexDir);
+ } catch (Exception recoveryFailureException) {
+ LOGGER.error("Failed to recover after reload failure", recoveryFailureException);
+ reloadFailureException.addSuppressed(recoveryFailureException);
+ }
+ throw reloadFailureException;
+ }
+ }
+
+ @Override
+ public void addOrReplaceSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentZKMetadata zkMetadata, @Nullable SegmentMetadata localMetadata)
+ throws Exception {
+ if (!isNewSegment(zkMetadata, localMetadata)) {
+ LOGGER.info("Segment: {} of table: {} has crc: {} same as before, already loaded, do nothing", segmentName,
+ _tableNameWithType, localMetadata.getCrc());
+ return;
+ }
+
+ // Try to recover if no local metadata is provided.
+ if (localMetadata == null) {
+ LOGGER.info("Segment: {} of table: {} is not loaded, checking disk", segmentName, _tableNameWithType);
+ localMetadata = recoverSegmentQuietly(segmentName);
+ if (!isNewSegment(zkMetadata, localMetadata)) {
+ LOGGER.info("Segment: {} of table {} has crc: {} same as before, loading", segmentName, _tableNameWithType,
+ localMetadata.getCrc());
+ if (loadSegmentQuietly(segmentName, indexLoadingConfig)) {
+ return;
+ }
+ localMetadata = null;
+ }
+ }
+
+ Preconditions.checkState(allowDownload(segmentName, zkMetadata), "Segment: %s of table: %s does not allow download",
+ segmentName, _tableNameWithType);
+
+ // Download segment and replace the local one, either due to failure to recover local segment,
+ // or the segment data is updated and has new CRC now.
+ if (localMetadata == null) {
+ LOGGER.info("Download segment: {} of table: {} as no good one exists locally", segmentName, _tableNameWithType);
+ } else {
+ LOGGER.info("Download segment: {} of table: {} as local crc: {} mismatches remote crc: {}.", segmentName,
+ _tableNameWithType, localMetadata.getCrc(), zkMetadata.getCrc());
+ }
+ File indexDir = downloadSegment(segmentName, zkMetadata);
+ addSegment(indexDir, indexLoadingConfig);
+ LOGGER.info("Downloaded and replaced segment: {} of table: {} with local crc now becomes: {} and remote crc: {}",
+ segmentName, _tableNameWithType, new SegmentMetadataImpl(indexDir).getCrc(), zkMetadata.getCrc());
+ }
+
+ protected boolean allowDownload(String segmentName, SegmentZKMetadata zkMetadata) {
+ return true;
+ }
+
+ protected File downloadSegment(String segmentName, SegmentZKMetadata zkMetadata)
+ throws Exception {
+ // TODO: may support download from peer servers for RealTime table.
+ return downloadSegmentFromDeepStore(segmentName, zkMetadata);
+ }
+
+ /**
+ * Server restart during segment reload might leave segment directory in inconsistent state, like the index
+ * directory might not exist but segment backup directory existed. This method tries to recover from reload
+ * failure before checking the existence of the index directory and loading segment metadata from it.
+ */
+ private SegmentMetadata recoverSegmentQuietly(String segmentName) {
+ File indexDir = getSegmentDataDir(segmentName);
+ try {
+ LoaderUtils.reloadFailureRecovery(indexDir);
+ if (!indexDir.exists()) {
+ LOGGER.info("Segment: {} of table: {} is not found on disk", segmentName, _tableNameWithType);
+ return null;
+ }
+ SegmentMetadataImpl localMetadata = new SegmentMetadataImpl(indexDir);
+ LOGGER.info("Recovered segment: {} of table: {} with crc: {} from disk", segmentName, _tableNameWithType,
+ localMetadata.getCrc());
+ return localMetadata;
+ } catch (Exception e) {
+ LOGGER.error("Failed to recover segment: {} of table: {} from disk", segmentName, _tableNameWithType, e);
+ FileUtils.deleteQuietly(indexDir);
+ return null;
+ }
+ }
+
+ private boolean loadSegmentQuietly(String segmentName, IndexLoadingConfig indexLoadingConfig) {
+ File indexDir = getSegmentDataDir(segmentName);
+ try {
+ addSegment(indexDir, indexLoadingConfig);
+ LOGGER.info("Loaded segment: {} of table: {} from disk", segmentName, _tableNameWithType);
+ return true;
+ } catch (Exception e) {
+ FileUtils.deleteQuietly(indexDir);
+ LOGGER.error("Failed to load segment: {} of table: {} from disk", segmentName, _tableNameWithType, e);
+ return false;
+ }
+ }
+
+ private File downloadSegmentFromDeepStore(String segmentName, SegmentZKMetadata zkMetadata)
+ throws Exception {
+ File tempRootDir = getSegmentDataDir("tmp-" + segmentName + "-" + UUID.randomUUID());
+ FileUtils.forceMkdir(tempRootDir);
+ try {
+ File tarFile = downloadAndDecrypt(segmentName, zkMetadata, tempRootDir);
+ return untarAndMoveSegment(segmentName, tarFile, tempRootDir);
+ } finally {
+ FileUtils.deleteQuietly(tempRootDir);
+ }
+ }
+
+ @VisibleForTesting
+ File downloadAndDecrypt(String segmentName, SegmentZKMetadata zkMetadata, File tempRootDir)
+ throws Exception {
+ File tarFile = new File(tempRootDir, segmentName + TarGzCompressionUtils.TAR_GZ_FILE_EXTENSION);
+ String uri = zkMetadata.getDownloadUrl();
+ try {
+ SegmentFetcherFactory.fetchAndDecryptSegmentToLocal(uri, tarFile, zkMetadata.getCrypterName());
+ LOGGER.info("Downloaded tarred segment: {} for table: {} from: {} to: {}, file length: {}", segmentName,
+ _tableNameWithType, uri, tarFile, tarFile.length());
+ return tarFile;
+ } catch (AttemptsExceededException e) {
+ LOGGER.error("Attempts exceeded when downloading segment: {} for table: {} from: {} to: {}", segmentName,
+ _tableNameWithType, uri, tarFile);
+ _serverMetrics.addMeteredTableValue(_tableNameWithType, ServerMeter.SEGMENT_DOWNLOAD_FAILURES, 1L);
+ Utils.rethrowException(e);
+ }
+ return null;
+ }
+
+ @VisibleForTesting
+ File untarAndMoveSegment(String segmentName, File tarFile, File tempRootDir) {
+ File untarDir = new File(tempRootDir, segmentName);
+ try {
+ // If an exception is thrown when untarring, it means the tar file is broken
+ // or not found after the retry. Thus, there's no need to retry again.
+ File untaredSegDir = TarGzCompressionUtils.untar(tarFile, untarDir).get(0);
+ LOGGER.info("Uncompressed tar file: {} into target dir: {}", tarFile, untarDir);
+ // Replace the existing index directory.
+ File indexDir = getSegmentDataDir(segmentName);
+ FileUtils.deleteDirectory(indexDir);
+ FileUtils.moveDirectory(untaredSegDir, indexDir);
+ LOGGER.info("Successfully downloaded segment: {} of table: {} to index dir: {}", segmentName, _tableNameWithType,
+ indexDir);
+ return indexDir;
+ } catch (Exception e) {
+ LOGGER.error("Failed to untar segment: {} of table: {} from: {} to: {}", segmentName, _tableNameWithType, tarFile,
+ untarDir);
+ _serverMetrics.addMeteredTableValue(_tableNameWithType, ServerMeter.UNTAR_FAILURES, 1L);
+ Utils.rethrowException(e);
+ }
+ return null;
+ }
+
+ @VisibleForTesting
+ File getSegmentDataDir(String segmentName) {
+ return new File(_indexDir, segmentName);
+ }
+
+ @VisibleForTesting
+ static boolean isNewSegment(SegmentZKMetadata zkMetadata, SegmentMetadata localMetadata) {
Review comment:
Annotate `localMetadata` as nullable
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
##########
@@ -256,4 +270,219 @@ public void addSegmentError(String segmentName, SegmentErrorInfo segmentErrorInf
.collect(Collectors.toMap(map -> map.getKey().getSecond(), Map.Entry::getValue));
}
}
+
+ @Override
+ public void reloadSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentZKMetadata zkMetadata, SegmentMetadata localMetadata, Schema schema, boolean forceDownload)
+ throws Exception {
+ File indexDir = localMetadata.getIndexDir();
+ Preconditions.checkState(indexDir.isDirectory(), "Index directory: %s is not a directory", indexDir);
+
+ File parentFile = indexDir.getParentFile();
+ File segmentBackupDir =
+ new File(parentFile, indexDir.getName() + CommonConstants.Segment.SEGMENT_BACKUP_DIR_SUFFIX);
+
+ try {
+ // First rename index directory to segment backup directory so that original segment have all file descriptors
+ // point to the segment backup directory to ensure original segment serves queries properly
+
+ // Rename index directory to segment backup directory (atomic)
+ Preconditions.checkState(indexDir.renameTo(segmentBackupDir),
+ "Failed to rename index directory: %s to segment backup directory: %s", indexDir, segmentBackupDir);
+
+ // Download from remote or copy from local backup directory into index directory,
+ // and then continue to load the segment from index directory.
+ boolean shouldDownload = forceDownload || !hasSameCRC(zkMetadata, localMetadata);
+ if (shouldDownload && allowDownload(segmentName, zkMetadata)) {
+ if (forceDownload) {
+ LOGGER.info("Segment: {} of table: {} is forced to download", segmentName, _tableNameWithType);
+ } else {
+ LOGGER.info("Download segment:{} of table: {} as local crc:{} mismatches remote crc: {}", segmentName,
+ _tableNameWithType, localMetadata.getCrc(), zkMetadata.getCrc());
+ }
+ indexDir = downloadSegment(segmentName, zkMetadata);
+ } else {
+ LOGGER.info("Reload the local copy of segment: {} of table: {}", segmentName, _tableNameWithType);
+ FileUtils.copyDirectory(segmentBackupDir, indexDir);
+ }
+
+ // Load from index directory and replace the old segment in memory.
+ addSegment(ImmutableSegmentLoader.load(indexDir, indexLoadingConfig, schema));
+
+ // Rename segment backup directory to segment temporary directory (atomic)
+ // The reason to first rename then delete is that, renaming is an atomic operation, but deleting is not. When we
+ // rename the segment backup directory to segment temporary directory, we know the reload already succeeded, so
+ // that we can safely delete the segment temporary directory
+ File segmentTempDir = new File(parentFile, indexDir.getName() + CommonConstants.Segment.SEGMENT_TEMP_DIR_SUFFIX);
+ Preconditions.checkState(segmentBackupDir.renameTo(segmentTempDir),
+ "Failed to rename segment backup directory: %s to segment temporary directory: %s", segmentBackupDir,
+ segmentTempDir);
+ FileUtils.deleteDirectory(segmentTempDir);
+ } catch (Exception reloadFailureException) {
+ try {
+ LoaderUtils.reloadFailureRecovery(indexDir);
+ } catch (Exception recoveryFailureException) {
+ LOGGER.error("Failed to recover after reload failure", recoveryFailureException);
+ reloadFailureException.addSuppressed(recoveryFailureException);
+ }
+ throw reloadFailureException;
+ }
+ }
+
+ @Override
+ public void addOrReplaceSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentZKMetadata zkMetadata, @Nullable SegmentMetadata localMetadata)
+ throws Exception {
+ if (!isNewSegment(zkMetadata, localMetadata)) {
+ LOGGER.info("Segment: {} of table: {} has crc: {} same as before, already loaded, do nothing", segmentName,
+ _tableNameWithType, localMetadata.getCrc());
+ return;
+ }
+
+ // Try to recover if no local metadata is provided.
+ if (localMetadata == null) {
+ LOGGER.info("Segment: {} of table: {} is not loaded, checking disk", segmentName, _tableNameWithType);
+ localMetadata = recoverSegmentQuietly(segmentName);
+ if (!isNewSegment(zkMetadata, localMetadata)) {
+ LOGGER.info("Segment: {} of table {} has crc: {} same as before, loading", segmentName, _tableNameWithType,
+ localMetadata.getCrc());
+ if (loadSegmentQuietly(segmentName, indexLoadingConfig)) {
+ return;
+ }
+ localMetadata = null;
+ }
+ }
+
+ Preconditions.checkState(allowDownload(segmentName, zkMetadata), "Segment: %s of table: %s does not allow download",
+ segmentName, _tableNameWithType);
+
+ // Download segment and replace the local one, either due to failure to recover local segment,
+ // or the segment data is updated and has new CRC now.
+ if (localMetadata == null) {
+ LOGGER.info("Download segment: {} of table: {} as no good one exists locally", segmentName, _tableNameWithType);
+ } else {
+ LOGGER.info("Download segment: {} of table: {} as local crc: {} mismatches remote crc: {}.", segmentName,
+ _tableNameWithType, localMetadata.getCrc(), zkMetadata.getCrc());
+ }
+ File indexDir = downloadSegment(segmentName, zkMetadata);
+ addSegment(indexDir, indexLoadingConfig);
+ LOGGER.info("Downloaded and replaced segment: {} of table: {} with local crc now becomes: {} and remote crc: {}",
+ segmentName, _tableNameWithType, new SegmentMetadataImpl(indexDir).getCrc(), zkMetadata.getCrc());
+ }
+
+ protected boolean allowDownload(String segmentName, SegmentZKMetadata zkMetadata) {
+ return true;
+ }
+
+ protected File downloadSegment(String segmentName, SegmentZKMetadata zkMetadata)
+ throws Exception {
+ // TODO: may support download from peer servers for RealTime table.
+ return downloadSegmentFromDeepStore(segmentName, zkMetadata);
+ }
+
+ /**
+ * Server restart during segment reload might leave segment directory in inconsistent state, like the index
+ * directory might not exist but segment backup directory existed. This method tries to recover from reload
+ * failure before checking the existence of the index directory and loading segment metadata from it.
+ */
+ private SegmentMetadata recoverSegmentQuietly(String segmentName) {
+ File indexDir = getSegmentDataDir(segmentName);
+ try {
+ LoaderUtils.reloadFailureRecovery(indexDir);
+ if (!indexDir.exists()) {
+ LOGGER.info("Segment: {} of table: {} is not found on disk", segmentName, _tableNameWithType);
+ return null;
+ }
+ SegmentMetadataImpl localMetadata = new SegmentMetadataImpl(indexDir);
+ LOGGER.info("Recovered segment: {} of table: {} with crc: {} from disk", segmentName, _tableNameWithType,
+ localMetadata.getCrc());
+ return localMetadata;
+ } catch (Exception e) {
+ LOGGER.error("Failed to recover segment: {} of table: {} from disk", segmentName, _tableNameWithType, e);
+ FileUtils.deleteQuietly(indexDir);
+ return null;
+ }
+ }
+
+ private boolean loadSegmentQuietly(String segmentName, IndexLoadingConfig indexLoadingConfig) {
+ File indexDir = getSegmentDataDir(segmentName);
+ try {
+ addSegment(indexDir, indexLoadingConfig);
+ LOGGER.info("Loaded segment: {} of table: {} from disk", segmentName, _tableNameWithType);
+ return true;
+ } catch (Exception e) {
+ FileUtils.deleteQuietly(indexDir);
+ LOGGER.error("Failed to load segment: {} of table: {} from disk", segmentName, _tableNameWithType, e);
+ return false;
+ }
+ }
+
+ private File downloadSegmentFromDeepStore(String segmentName, SegmentZKMetadata zkMetadata)
+ throws Exception {
+ File tempRootDir = getSegmentDataDir("tmp-" + segmentName + "-" + UUID.randomUUID());
+ FileUtils.forceMkdir(tempRootDir);
+ try {
+ File tarFile = downloadAndDecrypt(segmentName, zkMetadata, tempRootDir);
+ return untarAndMoveSegment(segmentName, tarFile, tempRootDir);
+ } finally {
+ FileUtils.deleteQuietly(tempRootDir);
+ }
+ }
+
+ @VisibleForTesting
+ File downloadAndDecrypt(String segmentName, SegmentZKMetadata zkMetadata, File tempRootDir)
+ throws Exception {
+ File tarFile = new File(tempRootDir, segmentName + TarGzCompressionUtils.TAR_GZ_FILE_EXTENSION);
+ String uri = zkMetadata.getDownloadUrl();
+ try {
+ SegmentFetcherFactory.fetchAndDecryptSegmentToLocal(uri, tarFile, zkMetadata.getCrypterName());
+ LOGGER.info("Downloaded tarred segment: {} for table: {} from: {} to: {}, file length: {}", segmentName,
+ _tableNameWithType, uri, tarFile, tarFile.length());
+ return tarFile;
+ } catch (AttemptsExceededException e) {
+ LOGGER.error("Attempts exceeded when downloading segment: {} for table: {} from: {} to: {}", segmentName,
+ _tableNameWithType, uri, tarFile);
+ _serverMetrics.addMeteredTableValue(_tableNameWithType, ServerMeter.SEGMENT_DOWNLOAD_FAILURES, 1L);
+ Utils.rethrowException(e);
+ }
+ return null;
Review comment:
Remove this line
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7319: Move segment download and segment reload into TableManager
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#issuecomment-901313573
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7319](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2b4488d) into [master](https://codecov.io/gh/apache/pinot/commit/0e66d688c6f2c9bc41507d35450bfaaf8c12f7de?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0e66d68) will **increase** coverage by `0.00%`.
> The diff coverage is `86.57%`.
[](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7319 +/- ##
=========================================
Coverage 70.70% 70.70%
+ Complexity 3352 3343 -9
=========================================
Files 1517 1516 -1
Lines 75105 75096 -9
Branches 10940 10935 -5
=========================================
- Hits 53100 53094 -6
- Misses 18381 18384 +3
+ Partials 3624 3618 -6
```
| Flag | Coverage Δ | |
|---|---|---|
| integration2 | `29.17% <63.08%> (+0.08%)` | :arrow_up: |
| unittests1 | `69.78% <78.89%> (-0.03%)` | :arrow_down: |
| unittests2 | `14.50% <0.00%> (-0.04%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...er/starter/helix/SegmentMessageHandlerFactory.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9TZWdtZW50TWVzc2FnZUhhbmRsZXJGYWN0b3J5LmphdmE=) | `57.53% <75.00%> (-0.16%)` | :arrow_down: |
| [.../pinot/core/data/manager/BaseTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvQmFzZVRhYmxlRGF0YU1hbmFnZXIuamF2YQ==) | `84.29% <81.73%> (-3.07%)` | :arrow_down: |
| [...ata/manager/realtime/RealtimeTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVUYWJsZURhdGFNYW5hZ2VyLmphdmE=) | `66.66% <100.00%> (-0.42%)` | :arrow_down: |
| [.../pinot/server/starter/helix/BaseServerStarter.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9CYXNlU2VydmVyU3RhcnRlci5qYXZh) | `49.70% <100.00%> (+1.19%)` | :arrow_up: |
| [...server/starter/helix/HelixInstanceDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9IZWxpeEluc3RhbmNlRGF0YU1hbmFnZXIuamF2YQ==) | `83.87% <100.00%> (+5.16%)` | :arrow_up: |
| [...r/helix/SegmentOnlineOfflineStateModelFactory.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9TZWdtZW50T25saW5lT2ZmbGluZVN0YXRlTW9kZWxGYWN0b3J5LmphdmE=) | `58.49% <100.00%> (-0.39%)` | :arrow_down: |
| [...va/org/apache/pinot/spi/utils/CommonConstants.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQ29tbW9uQ29uc3RhbnRzLmphdmE=) | `26.31% <100.00%> (ø)` | |
| [...core/startree/operator/StarTreeFilterOperator.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zdGFydHJlZS9vcGVyYXRvci9TdGFyVHJlZUZpbHRlck9wZXJhdG9yLmphdmE=) | `68.79% <0.00%> (-15.61%)` | :arrow_down: |
| [...nt/local/startree/v2/store/StarTreeDataSource.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zdGFydHJlZS92Mi9zdG9yZS9TdGFyVHJlZURhdGFTb3VyY2UuamF2YQ==) | `40.00% <0.00%> (-13.34%)` | :arrow_down: |
| [...ot/segment/local/startree/OffHeapStarTreeNode.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zdGFydHJlZS9PZmZIZWFwU3RhclRyZWVOb2RlLmphdmE=) | `72.22% <0.00%> (-5.56%)` | :arrow_down: |
| ... and [19 more](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [0e66d68...2b4488d](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on a change in pull request #7319: Move offline seg download code to OfflineTableManager
Posted by GitBox <gi...@apache.org>.
klsince commented on a change in pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#discussion_r704717050
##########
File path: pinot-core/src/test/java/org/apache/pinot/core/data/manager/BaseTableDataManagerAcquireSegmentTest.java
##########
@@ -240,7 +239,7 @@ public void testReplace()
runStorageServer(numQueryThreads, runTimeSec, tableDataManager); // replaces segments while online
-// System.out.println("Nops = " + _numQueries + ",nDrops=" + _nDestroys + ",nCreates=" + _allSegments.size());
+ // System.out.println("Nops = " + _numQueries + ",nDrops=" + _nDestroys + ",nCreates=" + _allSegments.size());
Review comment:
@mcvsubbu thanks for reviewing. about this suggestion, I didn't find a good way to get seed. Instead, it's simpler to set a seed and print it out for record. btw, I see a local random object is used in testReplace() method, but a constant RANDOM object is used in other methods. Is this intended?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7319: Move offline seg download code to OfflineTableManager
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#issuecomment-901313573
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7319](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2b00688) into [master](https://codecov.io/gh/apache/pinot/commit/bb34406aaa205f5c85b88c928d477fd267eda1b4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bb34406) will **increase** coverage by `4.90%`.
> The diff coverage is `71.25%`.
> :exclamation: Current head 2b00688 differs from pull request most recent head a1f78a6. Consider uploading reports for the commit a1f78a6 to get more accurate results
[](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7319 +/- ##
============================================
+ Coverage 64.65% 69.55% +4.90%
+ Complexity 3303 3214 -89
============================================
Files 1496 1114 -382
Lines 74289 52738 -21551
Branches 10795 7906 -2889
============================================
- Hits 48033 36683 -11350
+ Misses 22826 13452 -9374
+ Partials 3430 2603 -827
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `69.55% <71.25%> (+0.08%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ata/manager/realtime/RealtimeTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVUYWJsZURhdGFNYW5hZ2VyLmphdmE=) | `10.52% <0.00%> (-56.42%)` | :arrow_down: |
| [...t/segment/local/data/manager/TableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9kYXRhL21hbmFnZXIvVGFibGVEYXRhTWFuYWdlci5qYXZh) | `0.00% <0.00%> (ø)` | |
| [.../pinot/core/data/manager/BaseTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvQmFzZVRhYmxlRGF0YU1hbmFnZXIuamF2YQ==) | `81.09% <74.02%> (-6.26%)` | :arrow_down: |
| [...a/org/apache/pinot/common/metrics/MinionMeter.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25NZXRlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...g/apache/pinot/common/metrics/ControllerMeter.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Db250cm9sbGVyTWV0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/BrokerQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Ccm9rZXJRdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/MinionQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25RdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pache/pinot/common/utils/grpc/GrpcQueryClient.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvZ3JwYy9HcnBjUXVlcnlDbGllbnQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...he/pinot/common/messages/SegmentReloadMessage.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWVzc2FnZXMvU2VnbWVudFJlbG9hZE1lc3NhZ2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...t/core/startree/plan/StarTreeDocIdSetPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zdGFydHJlZS9wbGFuL1N0YXJUcmVlRG9jSWRTZXRQbGFuTm9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [584 more](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [bb34406...a1f78a6](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7319: Move offline seg download code to OfflineTableManager
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#issuecomment-901313573
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7319](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5290e6a) into [master](https://codecov.io/gh/apache/pinot/commit/c15249484a5bbe0e9fbbee311218c163a26dd263?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (c152494) will **decrease** coverage by `2.35%`.
> The diff coverage is `73.41%`.
[](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7319 +/- ##
============================================
- Coverage 71.49% 69.13% -2.36%
+ Complexity 3282 3194 -88
============================================
Files 1504 1110 -394
Lines 74063 52074 -21989
Branches 10726 7777 -2949
============================================
- Hits 52949 36001 -16948
+ Misses 17515 13489 -4026
+ Partials 3599 2584 -1015
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `69.13% <73.41%> (+0.18%)` | :arrow_up: |
| unittests2 | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ata/manager/realtime/RealtimeTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVUYWJsZURhdGFNYW5hZ2VyLmphdmE=) | `10.61% <ø> (-56.33%)` | :arrow_down: |
| [.../pinot/core/data/manager/BaseTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvQmFzZVRhYmxlRGF0YU1hbmFnZXIuamF2YQ==) | `86.51% <50.00%> (-0.84%)` | :arrow_down: |
| [.../data/manager/offline/OfflineTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvb2ZmbGluZS9PZmZsaW5lVGFibGVEYXRhTWFuYWdlci5qYXZh) | `76.19% <74.02%> (-23.81%)` | :arrow_down: |
| [...a/org/apache/pinot/common/metrics/MinionMeter.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25NZXRlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...g/apache/pinot/common/metrics/ControllerMeter.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Db250cm9sbGVyTWV0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/BrokerQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Ccm9rZXJRdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/MinionQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25RdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pache/pinot/common/utils/grpc/GrpcQueryClient.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvZ3JwYy9HcnBjUXVlcnlDbGllbnQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...he/pinot/common/messages/SegmentReloadMessage.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWVzc2FnZXMvU2VnbWVudFJlbG9hZE1lc3NhZ2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...t/core/startree/plan/StarTreeDocIdSetPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zdGFydHJlZS9wbGFuL1N0YXJUcmVlRG9jSWRTZXRQbGFuTm9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [688 more](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [c152494...5290e6a](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7319: Move offline seg download code to OfflineTableManager
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#issuecomment-901313573
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7319](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5290e6a) into [master](https://codecov.io/gh/apache/pinot/commit/c15249484a5bbe0e9fbbee311218c163a26dd263?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (c152494) will **decrease** coverage by `1.30%`.
> The diff coverage is `84.03%`.
[](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7319 +/- ##
============================================
- Coverage 71.49% 70.18% -1.31%
- Complexity 3282 3286 +4
============================================
Files 1504 1502 -2
Lines 74063 73858 -205
Branches 10726 10694 -32
============================================
- Hits 52949 51840 -1109
- Misses 17515 18432 +917
+ Partials 3599 3586 -13
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `29.29% <63.02%> (-0.05%)` | :arrow_down: |
| unittests1 | `69.14% <73.41%> (+0.20%)` | :arrow_up: |
| unittests2 | `14.59% <0.00%> (-0.05%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ata/manager/realtime/RealtimeTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVUYWJsZURhdGFNYW5hZ2VyLmphdmE=) | `65.71% <ø> (-1.23%)` | :arrow_down: |
| [.../pinot/core/data/manager/BaseTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvQmFzZVRhYmxlRGF0YU1hbmFnZXIuamF2YQ==) | `86.51% <50.00%> (-0.84%)` | :arrow_down: |
| [...er/starter/helix/SegmentMessageHandlerFactory.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9TZWdtZW50TWVzc2FnZUhhbmRsZXJGYWN0b3J5LmphdmE=) | `63.63% <75.00%> (-14.57%)` | :arrow_down: |
| [.../data/manager/offline/OfflineTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvb2ZmbGluZS9PZmZsaW5lVGFibGVEYXRhTWFuYWdlci5qYXZh) | `79.76% <77.92%> (-20.24%)` | :arrow_down: |
| [.../pinot/server/starter/helix/BaseServerStarter.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9CYXNlU2VydmVyU3RhcnRlci5qYXZh) | `49.70% <100.00%> (-5.06%)` | :arrow_down: |
| [...server/starter/helix/HelixInstanceDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9IZWxpeEluc3RhbmNlRGF0YU1hbmFnZXIuamF2YQ==) | `79.09% <100.00%> (-3.49%)` | :arrow_down: |
| [...r/helix/SegmentOnlineOfflineStateModelFactory.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9TZWdtZW50T25saW5lT2ZmbGluZVN0YXRlTW9kZWxGYWN0b3J5LmphdmE=) | `60.18% <100.00%> (+1.30%)` | :arrow_up: |
| [...pinot/minion/exception/TaskCancelledException.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vZXhjZXB0aW9uL1Rhc2tDYW5jZWxsZWRFeGNlcHRpb24uamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...ore/startree/executor/StarTreeGroupByExecutor.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zdGFydHJlZS9leGVjdXRvci9TdGFyVHJlZUdyb3VwQnlFeGVjdXRvci5qYXZh) | `0.00% <0.00%> (-86.67%)` | :arrow_down: |
| [.../transform/function/MapValueTransformFunction.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vTWFwVmFsdWVUcmFuc2Zvcm1GdW5jdGlvbi5qYXZh) | `0.00% <0.00%> (-85.30%)` | :arrow_down: |
| ... and [279 more](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [c152494...5290e6a](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on a change in pull request #7319: Move offline seg download code to OfflineTableManager
Posted by GitBox <gi...@apache.org>.
klsince commented on a change in pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#discussion_r704593059
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
##########
@@ -256,4 +267,147 @@ public void addSegmentError(String segmentName, SegmentErrorInfo segmentErrorInf
.collect(Collectors.toMap(map -> map.getKey().getSecond(), Map.Entry::getValue));
}
}
+
+ @Override
+ public void addOrReplaceSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentMetadata localMetadata, SegmentZKMetadata zkMetadata, boolean forceDownload)
+ throws Exception {
+ if (!forceDownload && !isNewSegment(zkMetadata, localMetadata)) {
+ LOGGER.info("Segment: {} of table: {} has crc: {} same as before, already loaded, do nothing", segmentName,
+ _tableNameWithType, localMetadata.getCrc());
+ return;
+ }
+
+ // If not forced to download, then try to recover if no local metadata is provided.
+ if (!forceDownload && localMetadata == null) {
+ LOGGER.info("Segment: {} of table: {} is not loaded, checking disk", segmentName, _tableNameWithType);
+ localMetadata = recoverSegmentQuietly(segmentName);
+ if (!isNewSegment(zkMetadata, localMetadata)) {
+ LOGGER.info("Segment: {} of table {} has crc: {} same as before, loading", segmentName, _tableNameWithType,
+ localMetadata.getCrc());
+ if (loadSegmentQuietly(segmentName, indexLoadingConfig)) {
Review comment:
basically, method with `quietly` suffix takes care of exceptions and may return a boolean to indicate whether the operation is done or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7319: Move offline seg download code to OfflineTableManager
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#issuecomment-901313573
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7319](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (46fd9b5) into [master](https://codecov.io/gh/apache/pinot/commit/bb34406aaa205f5c85b88c928d477fd267eda1b4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bb34406) will **increase** coverage by `4.84%`.
> The diff coverage is `72.15%`.
> :exclamation: Current head 46fd9b5 differs from pull request most recent head 331392f. Consider uploading reports for the commit 331392f to get more accurate results
[](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7319 +/- ##
============================================
+ Coverage 64.65% 69.50% +4.84%
+ Complexity 3303 3210 -93
============================================
Files 1496 1114 -382
Lines 74289 52737 -21552
Branches 10795 7906 -2889
============================================
- Hits 48033 36655 -11378
+ Misses 22826 13476 -9350
+ Partials 3430 2606 -824
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `69.50% <72.15%> (+0.03%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ata/manager/realtime/RealtimeTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVUYWJsZURhdGFNYW5hZ2VyLmphdmE=) | `10.56% <0.00%> (-56.37%)` | :arrow_down: |
| [...t/segment/local/data/manager/TableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9kYXRhL21hbmFnZXIvVGFibGVEYXRhTWFuYWdlci5qYXZh) | `0.00% <0.00%> (ø)` | |
| [.../pinot/core/data/manager/BaseTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvQmFzZVRhYmxlRGF0YU1hbmFnZXIuamF2YQ==) | `81.09% <74.02%> (-6.26%)` | :arrow_down: |
| [...a/org/apache/pinot/common/metrics/MinionMeter.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25NZXRlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...g/apache/pinot/common/metrics/ControllerMeter.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Db250cm9sbGVyTWV0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/BrokerQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Ccm9rZXJRdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/MinionQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25RdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pache/pinot/common/utils/grpc/GrpcQueryClient.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvZ3JwYy9HcnBjUXVlcnlDbGllbnQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...he/pinot/common/messages/SegmentReloadMessage.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWVzc2FnZXMvU2VnbWVudFJlbG9hZE1lc3NhZ2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...t/core/startree/plan/StarTreeDocIdSetPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zdGFydHJlZS9wbGFuL1N0YXJUcmVlRG9jSWRTZXRQbGFuTm9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [563 more](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [bb34406...331392f](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7319: Move offline seg download code to OfflineTableManager
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#discussion_r704600925
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
##########
@@ -256,4 +267,147 @@ public void addSegmentError(String segmentName, SegmentErrorInfo segmentErrorInf
.collect(Collectors.toMap(map -> map.getKey().getSecond(), Map.Entry::getValue));
}
}
+
+ @Override
+ public void addOrReplaceSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentMetadata localMetadata, SegmentZKMetadata zkMetadata, boolean forceDownload)
+ throws Exception {
+ if (!forceDownload && !isNewSegment(zkMetadata, localMetadata)) {
+ LOGGER.info("Segment: {} of table: {} has crc: {} same as before, already loaded, do nothing", segmentName,
+ _tableNameWithType, localMetadata.getCrc());
+ return;
+ }
+
+ // If not forced to download, then try to recover if no local metadata is provided.
+ if (!forceDownload && localMetadata == null) {
+ LOGGER.info("Segment: {} of table: {} is not loaded, checking disk", segmentName, _tableNameWithType);
+ localMetadata = recoverSegmentQuietly(segmentName);
+ if (!isNewSegment(zkMetadata, localMetadata)) {
+ LOGGER.info("Segment: {} of table {} has crc: {} same as before, loading", segmentName, _tableNameWithType,
+ localMetadata.getCrc());
+ if (loadSegmentQuietly(segmentName, indexLoadingConfig)) {
+ return;
+ }
+ localMetadata = null;
+ }
+ }
+
+ // Download segment and replace the local one, either due to being forced to download, or the
+ // local segment is not able to get loaded, or the segment data is updated and has new CRC now.
+ if (forceDownload) {
+ LOGGER.info("Force to download segment: {} of table: {}", segmentName, _tableNameWithType);
+ } else if (localMetadata == null) {
+ LOGGER.info("Download segment: {} of table: {} as no one exists locally", segmentName, _tableNameWithType);
+ } else {
+ LOGGER.info("Download segment: {} of table: {} as local crc: {} mismatches remote crc: {}.", segmentName,
+ _tableNameWithType, localMetadata.getCrc(), zkMetadata.getCrc());
+ }
+ File indexDir = downloadSegmentFromDeepStore(segmentName, zkMetadata);
+ SegmentMetadata segmentMetadata = new SegmentMetadataImpl(indexDir);
+ addSegment(indexDir, indexLoadingConfig);
+ LOGGER.info("Downloaded and replaced segment: {} of table: {} with crc: {}", segmentName, _tableNameWithType,
+ segmentMetadata.getCrc());
+ }
+
+ /**
+ * Server restart during segment reload might leave segment directory in inconsistent state, like the index
+ * directory might not exist but segment backup directory existed. This method tries to recover from reload
+ * failure before checking the existence of the index directory and loading segment metadata from it.
+ */
+ private SegmentMetadata recoverSegmentQuietly(String segmentName) {
Review comment:
I think naming conventions like this are helpful, and help reduce paranoid error handling. Without reading the method body itself, the author is communicating to the reader that the method doesn't throw.
If the method _does_ throw unexpectedly, it's revealing in stack traces, either in logs or in profiles, drawing one's eye to something which shouldn't have happened.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on a change in pull request #7319: Move offline seg download code to OfflineTableManager
Posted by GitBox <gi...@apache.org>.
klsince commented on a change in pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#discussion_r704593059
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
##########
@@ -256,4 +267,147 @@ public void addSegmentError(String segmentName, SegmentErrorInfo segmentErrorInf
.collect(Collectors.toMap(map -> map.getKey().getSecond(), Map.Entry::getValue));
}
}
+
+ @Override
+ public void addOrReplaceSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentMetadata localMetadata, SegmentZKMetadata zkMetadata, boolean forceDownload)
+ throws Exception {
+ if (!forceDownload && !isNewSegment(zkMetadata, localMetadata)) {
+ LOGGER.info("Segment: {} of table: {} has crc: {} same as before, already loaded, do nothing", segmentName,
+ _tableNameWithType, localMetadata.getCrc());
+ return;
+ }
+
+ // If not forced to download, then try to recover if no local metadata is provided.
+ if (!forceDownload && localMetadata == null) {
+ LOGGER.info("Segment: {} of table: {} is not loaded, checking disk", segmentName, _tableNameWithType);
+ localMetadata = recoverSegmentQuietly(segmentName);
+ if (!isNewSegment(zkMetadata, localMetadata)) {
+ LOGGER.info("Segment: {} of table {} has crc: {} same as before, loading", segmentName, _tableNameWithType,
+ localMetadata.getCrc());
+ if (loadSegmentQuietly(segmentName, indexLoadingConfig)) {
Review comment:
basically, method with `quietly` suffix takes care of exceptions and may return a boolean to indicate whether the operation is done or not.
e.g. FileUtils.deleteQuietly()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #7319: Move offline seg download code to OfflineTableManager
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#discussion_r699715552
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
##########
@@ -256,4 +267,147 @@ public void addSegmentError(String segmentName, SegmentErrorInfo segmentErrorInf
.collect(Collectors.toMap(map -> map.getKey().getSecond(), Map.Entry::getValue));
}
}
+
+ @Override
+ public void addOrReplaceSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentMetadata localMetadata, SegmentZKMetadata zkMetadata, boolean forceDownload)
Review comment:
annotate local metadata as nullable
##########
File path: pinot-core/src/test/java/org/apache/pinot/core/data/manager/BaseTableDataManagerAcquireSegmentTest.java
##########
@@ -240,7 +239,7 @@ public void testReplace()
runStorageServer(numQueryThreads, runTimeSec, tableDataManager); // replaces segments while online
-// System.out.println("Nops = " + _numQueries + ",nDrops=" + _nDestroys + ",nCreates=" + _allSegments.size());
+ // System.out.println("Nops = " + _numQueries + ",nDrops=" + _nDestroys + ",nCreates=" + _allSegments.size());
Review comment:
Remove it
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java
##########
@@ -248,19 +249,27 @@ public boolean isPartialUpsertEnabled() {
return _upsertMode == UpsertConfig.Mode.PARTIAL;
}
+ /**
+ * Add or replace an immutable segment created by the REALTIME table.
+ * Skipping segment still in consuming state.
+ */
+ @Override
+ public void addOrReplaceSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentMetadata localMetadata, SegmentZKMetadata zkMetadata, boolean forceDownload)
+ throws Exception {
+ throw new UnsupportedOperationException();
Review comment:
Why do we throw exception here?
##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/data/manager/TableDataManager.java
##########
@@ -146,4 +156,12 @@ void addSegment(String segmentName, TableConfig tableConfig, IndexLoadingConfig
* @return List of {@link SegmentErrorInfo}
*/
Map<String, SegmentErrorInfo> getSegmentErrors();
+
+ /**
+ * Get the dir for the given table segment, when TableDataManager object is not
+ * yet initialized for the given table.
+ */
+ static File getSegmentDataDir(String instanceDataDir, String tableNameWithType, String segmentName) {
Review comment:
This is instance level method, and seems only used in `InstanceDataManager`. Can we inline it instead of putting it as an interface method?
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java
##########
@@ -248,19 +249,27 @@ public boolean isPartialUpsertEnabled() {
return _upsertMode == UpsertConfig.Mode.PARTIAL;
}
+ /**
+ * Add or replace an immutable segment created by the REALTIME table.
+ * Skipping segment still in consuming state.
+ */
+ @Override
+ public void addOrReplaceSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentMetadata localMetadata, SegmentZKMetadata zkMetadata, boolean forceDownload)
Review comment:
Annotate as nullable
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/InstanceDataManager.java
##########
@@ -86,11 +86,28 @@ void reloadSegment(String tableNameWithType, String segmentName)
throws Exception;
/**
- * Reloads all segment in a table.
+ * Reloads all segments in a table.
*/
void reloadAllSegments(String tableNameWithType)
throws Exception;
+ /**
+ * Adds or replaces a segment in a table. Different from segment reloading,
+ * 1. this method doesn't assume the existence of TableDataManager object and it
+ * can actually initialize the TableDataManager for the segment;
+ * 2. this method can download a new segment to replace the local one before loading.
+ * Download is conducted when local segment's CRC is different from the one of the
+ * remote segment, but can also be forced to do regardless of CRC difference.
+ */
+ void addOrReplaceSegment(String tableNameWithType, String segmentName, boolean forceDownload)
Review comment:
The `forceDownload` should not be part of this API. We can add separate APIs for `redownloadSegment` and `redownloadAllSegments`, or add the flag into the `reload` apis.
##########
File path: pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixInstanceDataManager.java
##########
@@ -311,6 +310,47 @@ private void reloadSegment(String tableNameWithType, SegmentMetadata segmentMeta
}
}
+ @Override
+ public void addOrReplaceSegment(String tableNameWithType, String segmentName, boolean forceDownload)
+ throws Exception {
+ LOGGER.info("Adding or replacing segment: {} for table: {} downloadIsForced: {}", segmentName, tableNameWithType,
+ forceDownload);
+
+ // Get updated table config and segment metadata from Zookeeper.
+ TableConfig tableConfig = ZKMetadataProvider.getTableConfig(_propertyStore, tableNameWithType);
+ Preconditions.checkNotNull(tableConfig);
+ SegmentZKMetadata zkMetadata =
+ ZKMetadataProvider.getSegmentZKMetadata(_propertyStore, tableNameWithType, segmentName);
+ Preconditions.checkNotNull(zkMetadata);
+
+ // This method might modify the file on disk. Use segment lock to prevent race condition
+ Lock segmentLock = SegmentLocks.getSegmentLock(tableNameWithType, segmentName);
+ try {
+ // Lock the segment to get its metadata, so that no other threads are modifying
+ // the disk files of this segment.
+ segmentLock.lock();
+ SegmentMetadata localMetadata = getSegmentMetadata(tableNameWithType, segmentName);
+
+ _tableDataManagerMap.computeIfAbsent(tableNameWithType, k -> createTableDataManager(k, tableConfig))
+ .addOrReplaceSegment(segmentName, new IndexLoadingConfig(_instanceDataManagerConfig, tableConfig),
+ localMetadata, zkMetadata, forceDownload);
+ LOGGER.info("Added or replaced segment: {} of table: {}", segmentName, tableNameWithType);
+ } finally {
+ segmentLock.unlock();
+ }
+ }
+
+ @Override
+ public void addOrReplaceAllSegments(String tableNameWithType, boolean forceDownload)
+ throws Exception {
+ LOGGER.info("Adding or replacing all segments in table: {} downloadIsForced: {}", tableNameWithType, forceDownload);
+ List<SegmentMetadata> segMds = getAllSegmentsMetadata(tableNameWithType);
+ for (SegmentMetadata segMd : segMds) {
+ addOrReplaceSegment(tableNameWithType, segMd.getName(), forceDownload);
Review comment:
Add a helper method with `TableConfig` and `SegmentZKMetadata` as the arguments, and make `addOrReplaceSegment()` and `addOrReplaceAllSegments()` calling the helper method. We should avoid reading them for each segment for bulk replace
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
##########
@@ -256,4 +267,147 @@ public void addSegmentError(String segmentName, SegmentErrorInfo segmentErrorInf
.collect(Collectors.toMap(map -> map.getKey().getSecond(), Map.Entry::getValue));
}
}
+
+ @Override
+ public void addOrReplaceSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentMetadata localMetadata, SegmentZKMetadata zkMetadata, boolean forceDownload)
+ throws Exception {
+ if (!forceDownload && !isNewSegment(zkMetadata, localMetadata)) {
+ LOGGER.info("Segment: {} of table: {} has crc: {} same as before, already loaded, do nothing", segmentName,
+ _tableNameWithType, localMetadata.getCrc());
+ return;
+ }
+
+ // If not forced to download, then try to recover if no local metadata is provided.
+ if (!forceDownload && localMetadata == null) {
+ LOGGER.info("Segment: {} of table: {} is not loaded, checking disk", segmentName, _tableNameWithType);
+ localMetadata = recoverSegmentQuietly(segmentName);
+ if (!isNewSegment(zkMetadata, localMetadata)) {
+ LOGGER.info("Segment: {} of table {} has crc: {} same as before, loading", segmentName, _tableNameWithType,
+ localMetadata.getCrc());
+ if (loadSegmentQuietly(segmentName, indexLoadingConfig)) {
+ return;
+ }
+ localMetadata = null;
+ }
+ }
+
+ // Download segment and replace the local one, either due to being forced to download, or the
+ // local segment is not able to get loaded, or the segment data is updated and has new CRC now.
+ if (forceDownload) {
+ LOGGER.info("Force to download segment: {} of table: {}", segmentName, _tableNameWithType);
+ } else if (localMetadata == null) {
+ LOGGER.info("Download segment: {} of table: {} as no one exists locally", segmentName, _tableNameWithType);
+ } else {
+ LOGGER.info("Download segment: {} of table: {} as local crc: {} mismatches remote crc: {}.", segmentName,
+ _tableNameWithType, localMetadata.getCrc(), zkMetadata.getCrc());
+ }
+ File indexDir = downloadSegmentFromDeepStore(segmentName, zkMetadata);
+ SegmentMetadata segmentMetadata = new SegmentMetadataImpl(indexDir);
+ addSegment(indexDir, indexLoadingConfig);
+ LOGGER.info("Downloaded and replaced segment: {} of table: {} with crc: {}", segmentName, _tableNameWithType,
+ segmentMetadata.getCrc());
+ }
+
+ /**
+ * Server restart during segment reload might leave segment directory in inconsistent state, like the index
+ * directory might not exist but segment backup directory existed. This method tries to recover from reload
+ * failure before checking the existence of the index directory and loading segment metadata from it.
+ */
+ private SegmentMetadata recoverSegmentQuietly(String segmentName) {
Review comment:
(nit) I don't think we need to name it as `quietly`, same for `loadSegmentQuietly`
##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/data/manager/TableDataManager.java
##########
@@ -78,6 +80,14 @@ void addSegment(File indexDir, IndexLoadingConfig indexLoadingConfig)
void addSegment(String segmentName, TableConfig tableConfig, IndexLoadingConfig indexLoadingConfig)
throws Exception;
+ /**
+ * Add or replace an immutable segment for current table, which can be either
+ * OFFLINE or REALTIME table.
+ */
+ void addOrReplaceSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ @Nullable SegmentMetadata localMetadata, SegmentZKMetadata zkMetadata, boolean forceDownload)
Review comment:
(nit) for slightly better readability:
```suggestion
SegmentZKMetadata remoteZKMetadata, @Nullable SegmentMetadata localMetadata, boolean forceDownload)
```
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
##########
@@ -256,4 +267,147 @@ public void addSegmentError(String segmentName, SegmentErrorInfo segmentErrorInf
.collect(Collectors.toMap(map -> map.getKey().getSecond(), Map.Entry::getValue));
}
}
+
+ @Override
+ public void addOrReplaceSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentMetadata localMetadata, SegmentZKMetadata zkMetadata, boolean forceDownload)
+ throws Exception {
+ if (!forceDownload && !isNewSegment(zkMetadata, localMetadata)) {
+ LOGGER.info("Segment: {} of table: {} has crc: {} same as before, already loaded, do nothing", segmentName,
+ _tableNameWithType, localMetadata.getCrc());
+ return;
+ }
+
+ // If not forced to download, then try to recover if no local metadata is provided.
+ if (!forceDownload && localMetadata == null) {
+ LOGGER.info("Segment: {} of table: {} is not loaded, checking disk", segmentName, _tableNameWithType);
+ localMetadata = recoverSegmentQuietly(segmentName);
+ if (!isNewSegment(zkMetadata, localMetadata)) {
+ LOGGER.info("Segment: {} of table {} has crc: {} same as before, loading", segmentName, _tableNameWithType,
+ localMetadata.getCrc());
+ if (loadSegmentQuietly(segmentName, indexLoadingConfig)) {
Review comment:
What does quietly stand for?
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
##########
@@ -256,4 +267,147 @@ public void addSegmentError(String segmentName, SegmentErrorInfo segmentErrorInf
.collect(Collectors.toMap(map -> map.getKey().getSecond(), Map.Entry::getValue));
}
}
+
+ @Override
+ public void addOrReplaceSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentMetadata localMetadata, SegmentZKMetadata zkMetadata, boolean forceDownload)
+ throws Exception {
+ if (!forceDownload && !isNewSegment(zkMetadata, localMetadata)) {
+ LOGGER.info("Segment: {} of table: {} has crc: {} same as before, already loaded, do nothing", segmentName,
+ _tableNameWithType, localMetadata.getCrc());
+ return;
Review comment:
This is not correct. The segment is on local disk, but we still need to load the segment into the memory.
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
##########
@@ -256,4 +267,147 @@ public void addSegmentError(String segmentName, SegmentErrorInfo segmentErrorInf
.collect(Collectors.toMap(map -> map.getKey().getSecond(), Map.Entry::getValue));
}
}
+
+ @Override
+ public void addOrReplaceSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentMetadata localMetadata, SegmentZKMetadata zkMetadata, boolean forceDownload)
+ throws Exception {
+ if (!forceDownload && !isNewSegment(zkMetadata, localMetadata)) {
+ LOGGER.info("Segment: {} of table: {} has crc: {} same as before, already loaded, do nothing", segmentName,
+ _tableNameWithType, localMetadata.getCrc());
+ return;
Review comment:
nvm. The `localMetadata` is from memory, not loaded from disk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7319: Move segment download and segment reload into TableManager
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#issuecomment-901313573
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7319](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2b4488d) into [master](https://codecov.io/gh/apache/pinot/commit/0e66d688c6f2c9bc41507d35450bfaaf8c12f7de?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0e66d68) will **decrease** coverage by `5.62%`.
> The diff coverage is `78.89%`.
[](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7319 +/- ##
============================================
- Coverage 70.70% 65.08% -5.63%
+ Complexity 3352 3343 -9
============================================
Files 1517 1471 -46
Lines 75105 73291 -1814
Branches 10940 10743 -197
============================================
- Hits 53100 47698 -5402
- Misses 18381 22209 +3828
+ Partials 3624 3384 -240
```
| Flag | Coverage Δ | |
|---|---|---|
| integration2 | `?` | |
| unittests1 | `69.78% <78.89%> (-0.03%)` | :arrow_down: |
| unittests2 | `14.50% <0.00%> (-0.04%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../pinot/core/data/manager/BaseTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvQmFzZVRhYmxlRGF0YU1hbmFnZXIuamF2YQ==) | `82.19% <77.88%> (-5.16%)` | :arrow_down: |
| [...ata/manager/realtime/RealtimeTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVUYWJsZURhdGFNYW5hZ2VyLmphdmE=) | `11.78% <100.00%> (-55.29%)` | :arrow_down: |
| [...va/org/apache/pinot/spi/utils/CommonConstants.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQ29tbW9uQ29uc3RhbnRzLmphdmE=) | `26.31% <100.00%> (ø)` | |
| [...a/org/apache/pinot/common/metrics/MinionMeter.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25NZXRlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...g/apache/pinot/common/metrics/ControllerMeter.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Db250cm9sbGVyTWV0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/BrokerQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Ccm9rZXJRdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/MinionQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25RdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...he/pinot/common/messages/SegmentReloadMessage.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWVzc2FnZXMvU2VnbWVudFJlbG9hZE1lc3NhZ2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pinot/core/data/manager/realtime/TimerService.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvVGltZXJTZXJ2aWNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...t/core/startree/plan/StarTreeDocIdSetPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zdGFydHJlZS9wbGFuL1N0YXJUcmVlRG9jSWRTZXRQbGFuTm9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [318 more](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [0e66d68...2b4488d](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on a change in pull request #7319: Move segment download and segment reload into TableManager
Posted by GitBox <gi...@apache.org>.
klsince commented on a change in pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#discussion_r708489746
##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/data/manager/TableDataManager.java
##########
@@ -78,6 +81,27 @@ void addSegment(File indexDir, IndexLoadingConfig indexLoadingConfig)
void addSegment(String segmentName, TableConfig tableConfig, IndexLoadingConfig indexLoadingConfig)
throws Exception;
+ /**
+ * Reloads an existing immutable segment for the table, which can be an OFFLINE or REALTIME table.
+ * A new segment may be downloaded if the local one has a different CRC; or can be forced to download
+ * if forceDownload flag is true. This operation is conducted within a failure handling framework
+ * and made transparent to ongoing queries, because the segment is in online serving state.
+ */
+ void reloadSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentZKMetadata zkMetadata, SegmentMetadata localMetadata, Schema schema, boolean forceDownload)
Review comment:
yup, it's annotated nullable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7319: Move offline seg download code to OfflineTableManager
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#issuecomment-901313573
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7319](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5290e6a) into [master](https://codecov.io/gh/apache/pinot/commit/c15249484a5bbe0e9fbbee311218c163a26dd263?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (c152494) will **decrease** coverage by `6.91%`.
> The diff coverage is `73.41%`.
[](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7319 +/- ##
============================================
- Coverage 71.49% 64.57% -6.92%
- Complexity 3282 3286 +4
============================================
Files 1504 1457 -47
Lines 74063 72035 -2028
Branches 10726 10501 -225
============================================
- Hits 52949 46520 -6429
- Misses 17515 22169 +4654
+ Partials 3599 3346 -253
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `69.14% <73.41%> (+0.20%)` | :arrow_up: |
| unittests2 | `14.59% <0.00%> (-0.05%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ata/manager/realtime/RealtimeTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVUYWJsZURhdGFNYW5hZ2VyLmphdmE=) | `10.61% <ø> (-56.33%)` | :arrow_down: |
| [.../pinot/core/data/manager/BaseTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvQmFzZVRhYmxlRGF0YU1hbmFnZXIuamF2YQ==) | `86.51% <50.00%> (-0.84%)` | :arrow_down: |
| [.../data/manager/offline/OfflineTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvb2ZmbGluZS9PZmZsaW5lVGFibGVEYXRhTWFuYWdlci5qYXZh) | `76.19% <74.02%> (-23.81%)` | :arrow_down: |
| [...a/org/apache/pinot/common/metrics/MinionMeter.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25NZXRlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...g/apache/pinot/common/metrics/ControllerMeter.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Db250cm9sbGVyTWV0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/BrokerQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Ccm9rZXJRdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/MinionQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25RdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pache/pinot/common/utils/grpc/GrpcQueryClient.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvZ3JwYy9HcnBjUXVlcnlDbGllbnQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...he/pinot/common/messages/SegmentReloadMessage.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWVzc2FnZXMvU2VnbWVudFJlbG9hZE1lc3NhZ2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pinot/minion/exception/TaskCancelledException.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vZXhjZXB0aW9uL1Rhc2tDYW5jZWxsZWRFeGNlcHRpb24uamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [484 more](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [c152494...5290e6a](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on a change in pull request #7319: Move offline seg download code to OfflineTableManager
Posted by GitBox <gi...@apache.org>.
klsince commented on a change in pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#discussion_r704605056
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java
##########
@@ -248,19 +249,27 @@ public boolean isPartialUpsertEnabled() {
return _upsertMode == UpsertConfig.Mode.PARTIAL;
}
+ /**
+ * Add or replace an immutable segment created by the REALTIME table.
+ * Skipping segment still in consuming state.
+ */
+ @Override
+ public void addOrReplaceSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentMetadata localMetadata, SegmentZKMetadata zkMetadata, boolean forceDownload)
+ throws Exception {
+ throw new UnsupportedOperationException();
Review comment:
I was planning to add support for RT table in next PR, and make this one focus on refactoring existing logic. As reload...() APIs are to be extended with forceDownload flag, I'll double check if changes are still needed here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on a change in pull request #7319: Move segment download and segment reload into TableManager
Posted by GitBox <gi...@apache.org>.
klsince commented on a change in pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#discussion_r708357041
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
##########
@@ -256,4 +270,219 @@ public void addSegmentError(String segmentName, SegmentErrorInfo segmentErrorInf
.collect(Collectors.toMap(map -> map.getKey().getSecond(), Map.Entry::getValue));
}
}
+
+ @Override
+ public void reloadSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentZKMetadata zkMetadata, SegmentMetadata localMetadata, Schema schema, boolean forceDownload)
+ throws Exception {
+ File indexDir = localMetadata.getIndexDir();
+ Preconditions.checkState(indexDir.isDirectory(), "Index directory: %s is not a directory", indexDir);
+
+ File parentFile = indexDir.getParentFile();
+ File segmentBackupDir =
+ new File(parentFile, indexDir.getName() + CommonConstants.Segment.SEGMENT_BACKUP_DIR_SUFFIX);
+
+ try {
+ // First rename index directory to segment backup directory so that original segment have all file descriptors
+ // point to the segment backup directory to ensure original segment serves queries properly
+
+ // Rename index directory to segment backup directory (atomic)
+ Preconditions.checkState(indexDir.renameTo(segmentBackupDir),
+ "Failed to rename index directory: %s to segment backup directory: %s", indexDir, segmentBackupDir);
+
+ // Download from remote or copy from local backup directory into index directory,
+ // and then continue to load the segment from index directory.
+ boolean shouldDownload = forceDownload || !hasSameCRC(zkMetadata, localMetadata);
+ if (shouldDownload && allowDownload(segmentName, zkMetadata)) {
+ if (forceDownload) {
+ LOGGER.info("Segment: {} of table: {} is forced to download", segmentName, _tableNameWithType);
+ } else {
+ LOGGER.info("Download segment:{} of table: {} as local crc:{} mismatches remote crc: {}", segmentName,
Review comment:
sharp eye!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7319: Move offline seg download code to OfflineTableManager
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#issuecomment-901313573
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7319](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (791cefd) into [master](https://codecov.io/gh/apache/pinot/commit/6294d1be67c465f907aafa8ad74f8c949c193991?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6294d1b) will **decrease** coverage by `2.16%`.
> The diff coverage is `74.75%`.
> :exclamation: Current head 791cefd differs from pull request most recent head 0cb37d0. Consider uploading reports for the commit 0cb37d0 to get more accurate results
[](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7319 +/- ##
============================================
- Coverage 71.89% 69.73% -2.17%
+ Complexity 3430 3252 -178
============================================
Files 1517 1123 -394
Lines 75039 53051 -21988
Branches 10921 7977 -2944
============================================
- Hits 53950 36995 -16955
+ Misses 17457 13433 -4024
+ Partials 3632 2623 -1009
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `69.73% <74.75%> (+0.03%)` | :arrow_up: |
| unittests2 | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ata/manager/realtime/RealtimeTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVUYWJsZURhdGFNYW5hZ2VyLmphdmE=) | `10.61% <ø> (-56.33%)` | :arrow_down: |
| [.../pinot/core/data/manager/BaseTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvQmFzZVRhYmxlRGF0YU1hbmFnZXIuamF2YQ==) | `80.52% <74.75%> (-6.83%)` | :arrow_down: |
| [...a/org/apache/pinot/common/metrics/MinionMeter.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25NZXRlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...g/apache/pinot/common/metrics/ControllerMeter.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Db250cm9sbGVyTWV0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/BrokerQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Ccm9rZXJRdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/MinionQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25RdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pache/pinot/common/utils/grpc/GrpcQueryClient.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvZ3JwYy9HcnBjUXVlcnlDbGllbnQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...he/pinot/common/messages/SegmentReloadMessage.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWVzc2FnZXMvU2VnbWVudFJlbG9hZE1lc3NhZ2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pinot/core/data/manager/realtime/TimerService.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvVGltZXJTZXJ2aWNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...t/core/startree/plan/StarTreeDocIdSetPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zdGFydHJlZS9wbGFuL1N0YXJUcmVlRG9jSWRTZXRQbGFuTm9kZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [591 more](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [6294d1b...0cb37d0](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7319: Move offline seg download code to OfflineTableManager
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#issuecomment-901313573
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7319](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (dd46cf1) into [master](https://codecov.io/gh/apache/pinot/commit/12e5bcd37350a3e85a9e9630a7f6de05d91b5a24?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (12e5bcd) will **decrease** coverage by `42.00%`.
> The diff coverage is `34.17%`.
> :exclamation: Current head dd46cf1 differs from pull request most recent head 15eb4e8. Consider uploading reports for the commit 15eb4e8 to get more accurate results
[](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7319 +/- ##
=============================================
- Coverage 71.32% 29.32% -42.01%
+ Complexity 3367 83 -3284
=============================================
Files 1503 1493 -10
Lines 74030 73510 -520
Branches 10718 10656 -62
=============================================
- Hits 52803 21556 -31247
- Misses 17622 49996 +32374
+ Partials 3605 1958 -1647
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `29.32% <34.17%> (-0.01%)` | :arrow_down: |
| unittests1 | `?` | |
| unittests2 | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../pinot/common/function/DateTimePatternHandler.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vRGF0ZVRpbWVQYXR0ZXJuSGFuZGxlci5qYXZh) | `60.00% <ø> (-23.34%)` | :arrow_down: |
| [...rg/apache/pinot/common/function/DateTimeUtils.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vRGF0ZVRpbWVVdGlscy5qYXZh) | `0.00% <0.00%> (-80.00%)` | :arrow_down: |
| [...ot/common/function/FunctionDefinitionRegistry.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vRnVuY3Rpb25EZWZpbml0aW9uUmVnaXN0cnkuamF2YQ==) | `50.00% <ø> (-38.89%)` | :arrow_down: |
| [.../org/apache/pinot/common/function/TimeZoneKey.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vVGltZVpvbmVLZXkuamF2YQ==) | `0.00% <0.00%> (-38.94%)` | :arrow_down: |
| [...n/function/scalar/DataTypeConversionFunctions.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vc2NhbGFyL0RhdGFUeXBlQ29udmVyc2lvbkZ1bmN0aW9ucy5qYXZh) | `0.00% <ø> (ø)` | |
| [...inot/common/function/scalar/DateTimeFunctions.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vc2NhbGFyL0RhdGVUaW1lRnVuY3Rpb25zLmphdmE=) | `6.94% <0.00%> (-91.67%)` | :arrow_down: |
| [...he/pinot/common/function/scalar/HashFunctions.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vc2NhbGFyL0hhc2hGdW5jdGlvbnMuamF2YQ==) | `0.00% <ø> (-100.00%)` | :arrow_down: |
| [.../org/apache/pinot/common/lineage/LineageEntry.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbGluZWFnZS9MaW5lYWdlRW50cnkuamF2YQ==) | `0.00% <0.00%> (-64.71%)` | :arrow_down: |
| [...not/common/lineage/SegmentLineageAccessHelper.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbGluZWFnZS9TZWdtZW50TGluZWFnZUFjY2Vzc0hlbHBlci5qYXZh) | `50.00% <ø> (+3.33%)` | :arrow_up: |
| [...ot/common/messages/RoutingTableRebuildMessage.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWVzc2FnZXMvUm91dGluZ1RhYmxlUmVidWlsZE1lc3NhZ2UuamF2YQ==) | `0.00% <ø> (ø)` | |
| ... and [1205 more](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [12e5bcd...15eb4e8](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on a change in pull request #7319: Move segment download and segment reload into TableManager
Posted by GitBox <gi...@apache.org>.
klsince commented on a change in pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#discussion_r708503938
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
##########
@@ -256,4 +270,219 @@ public void addSegmentError(String segmentName, SegmentErrorInfo segmentErrorInf
.collect(Collectors.toMap(map -> map.getKey().getSecond(), Map.Entry::getValue));
}
}
+
+ @Override
+ public void reloadSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentZKMetadata zkMetadata, SegmentMetadata localMetadata, Schema schema, boolean forceDownload)
+ throws Exception {
+ File indexDir = localMetadata.getIndexDir();
+ Preconditions.checkState(indexDir.isDirectory(), "Index directory: %s is not a directory", indexDir);
+
+ File parentFile = indexDir.getParentFile();
+ File segmentBackupDir =
+ new File(parentFile, indexDir.getName() + CommonConstants.Segment.SEGMENT_BACKUP_DIR_SUFFIX);
+
+ try {
+ // First rename index directory to segment backup directory so that original segment have all file descriptors
+ // point to the segment backup directory to ensure original segment serves queries properly
+
+ // Rename index directory to segment backup directory (atomic)
+ Preconditions.checkState(indexDir.renameTo(segmentBackupDir),
+ "Failed to rename index directory: %s to segment backup directory: %s", indexDir, segmentBackupDir);
+
+ // Download from remote or copy from local backup directory into index directory,
+ // and then continue to load the segment from index directory.
+ boolean shouldDownload = forceDownload || !hasSameCRC(zkMetadata, localMetadata);
+ if (shouldDownload && allowDownload(segmentName, zkMetadata)) {
+ if (forceDownload) {
+ LOGGER.info("Segment: {} of table: {} is forced to download", segmentName, _tableNameWithType);
+ } else {
+ LOGGER.info("Download segment:{} of table: {} as local crc:{} mismatches remote crc: {}", segmentName,
+ _tableNameWithType, localMetadata.getCrc(), zkMetadata.getCrc());
+ }
+ indexDir = downloadSegment(segmentName, zkMetadata);
+ } else {
+ LOGGER.info("Reload the local copy of segment: {} of table: {}", segmentName, _tableNameWithType);
+ FileUtils.copyDirectory(segmentBackupDir, indexDir);
+ }
+
+ // Load from index directory and replace the old segment in memory.
+ addSegment(ImmutableSegmentLoader.load(indexDir, indexLoadingConfig, schema));
+
+ // Rename segment backup directory to segment temporary directory (atomic)
+ // The reason to first rename then delete is that, renaming is an atomic operation, but deleting is not. When we
+ // rename the segment backup directory to segment temporary directory, we know the reload already succeeded, so
+ // that we can safely delete the segment temporary directory
+ File segmentTempDir = new File(parentFile, indexDir.getName() + CommonConstants.Segment.SEGMENT_TEMP_DIR_SUFFIX);
+ Preconditions.checkState(segmentBackupDir.renameTo(segmentTempDir),
+ "Failed to rename segment backup directory: %s to segment temporary directory: %s", segmentBackupDir,
+ segmentTempDir);
+ FileUtils.deleteDirectory(segmentTempDir);
+ } catch (Exception reloadFailureException) {
+ try {
+ LoaderUtils.reloadFailureRecovery(indexDir);
+ } catch (Exception recoveryFailureException) {
+ LOGGER.error("Failed to recover after reload failure", recoveryFailureException);
+ reloadFailureException.addSuppressed(recoveryFailureException);
+ }
+ throw reloadFailureException;
+ }
+ }
+
+ @Override
+ public void addOrReplaceSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentZKMetadata zkMetadata, @Nullable SegmentMetadata localMetadata)
+ throws Exception {
+ if (!isNewSegment(zkMetadata, localMetadata)) {
+ LOGGER.info("Segment: {} of table: {} has crc: {} same as before, already loaded, do nothing", segmentName,
+ _tableNameWithType, localMetadata.getCrc());
+ return;
+ }
+
+ // Try to recover if no local metadata is provided.
+ if (localMetadata == null) {
+ LOGGER.info("Segment: {} of table: {} is not loaded, checking disk", segmentName, _tableNameWithType);
+ localMetadata = recoverSegmentQuietly(segmentName);
+ if (!isNewSegment(zkMetadata, localMetadata)) {
+ LOGGER.info("Segment: {} of table {} has crc: {} same as before, loading", segmentName, _tableNameWithType,
+ localMetadata.getCrc());
+ if (loadSegmentQuietly(segmentName, indexLoadingConfig)) {
+ return;
+ }
+ localMetadata = null;
+ }
+ }
+
+ Preconditions.checkState(allowDownload(segmentName, zkMetadata), "Segment: %s of table: %s does not allow download",
+ segmentName, _tableNameWithType);
+
+ // Download segment and replace the local one, either due to failure to recover local segment,
+ // or the segment data is updated and has new CRC now.
+ if (localMetadata == null) {
+ LOGGER.info("Download segment: {} of table: {} as no good one exists locally", segmentName, _tableNameWithType);
+ } else {
+ LOGGER.info("Download segment: {} of table: {} as local crc: {} mismatches remote crc: {}.", segmentName,
+ _tableNameWithType, localMetadata.getCrc(), zkMetadata.getCrc());
+ }
+ File indexDir = downloadSegment(segmentName, zkMetadata);
+ addSegment(indexDir, indexLoadingConfig);
+ LOGGER.info("Downloaded and replaced segment: {} of table: {} with local crc now becomes: {} and remote crc: {}",
Review comment:
simplified to `Downloaded and loaded segment ... with crc: {}`, as the previous log msgs can tell if it's added or replaced.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang merged pull request #7319: Move segment download and segment reload into TableManager
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang merged pull request #7319:
URL: https://github.com/apache/pinot/pull/7319
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7319: Move offline seg download code to OfflineTableManager
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#issuecomment-901313573
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7319](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e8f676d) into [master](https://codecov.io/gh/apache/pinot/commit/2d1f7dc9060d45ac9f4ff0621da62790ea9b8e64?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2d1f7dc) will **decrease** coverage by `2.12%`.
> The diff coverage is `72.15%`.
[](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7319 +/- ##
============================================
- Coverage 71.65% 69.53% -2.13%
+ Complexity 3305 3213 -92
============================================
Files 1515 1122 -393
Lines 74992 53005 -21987
Branches 10903 7964 -2939
============================================
- Hits 53736 36855 -16881
+ Misses 17629 13530 -4099
+ Partials 3627 2620 -1007
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `69.53% <72.15%> (+0.05%)` | :arrow_up: |
| unittests2 | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ata/manager/realtime/RealtimeTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVUYWJsZURhdGFNYW5hZ2VyLmphdmE=) | `10.56% <0.00%> (-57.19%)` | :arrow_down: |
| [...t/segment/local/data/manager/TableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9kYXRhL21hbmFnZXIvVGFibGVEYXRhTWFuYWdlci5qYXZh) | `0.00% <0.00%> (ø)` | |
| [.../pinot/core/data/manager/BaseTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvQmFzZVRhYmxlRGF0YU1hbmFnZXIuamF2YQ==) | `81.09% <74.02%> (-6.26%)` | :arrow_down: |
| [...a/org/apache/pinot/common/metrics/MinionMeter.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25NZXRlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...g/apache/pinot/common/metrics/ControllerMeter.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Db250cm9sbGVyTWV0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/BrokerQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Ccm9rZXJRdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/MinionQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25RdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pache/pinot/common/utils/grpc/GrpcQueryClient.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvZ3JwYy9HcnBjUXVlcnlDbGllbnQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...he/pinot/common/messages/SegmentReloadMessage.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWVzc2FnZXMvU2VnbWVudFJlbG9hZE1lc3NhZ2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pinot/core/data/manager/realtime/TimerService.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvVGltZXJTZXJ2aWNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [595 more](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [2d1f7dc...e8f676d](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #7319: Move offline seg download code to OfflineTableManager
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#discussion_r699751733
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
##########
@@ -256,4 +267,147 @@ public void addSegmentError(String segmentName, SegmentErrorInfo segmentErrorInf
.collect(Collectors.toMap(map -> map.getKey().getSecond(), Map.Entry::getValue));
}
}
+
+ @Override
+ public void addOrReplaceSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentMetadata localMetadata, SegmentZKMetadata zkMetadata, boolean forceDownload)
+ throws Exception {
+ if (!forceDownload && !isNewSegment(zkMetadata, localMetadata)) {
+ LOGGER.info("Segment: {} of table: {} has crc: {} same as before, already loaded, do nothing", segmentName,
+ _tableNameWithType, localMetadata.getCrc());
+ return;
Review comment:
nvm. The `localMetadata` is from memory, not loaded from disk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on a change in pull request #7319: Move offline seg download code to OfflineTableManager
Posted by GitBox <gi...@apache.org>.
klsince commented on a change in pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#discussion_r704597692
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/InstanceDataManager.java
##########
@@ -86,11 +86,28 @@ void reloadSegment(String tableNameWithType, String segmentName)
throws Exception;
/**
- * Reloads all segment in a table.
+ * Reloads all segments in a table.
*/
void reloadAllSegments(String tableNameWithType)
throws Exception;
+ /**
+ * Adds or replaces a segment in a table. Different from segment reloading,
+ * 1. this method doesn't assume the existence of TableDataManager object and it
+ * can actually initialize the TableDataManager for the segment;
+ * 2. this method can download a new segment to replace the local one before loading.
+ * Download is conducted when local segment's CRC is different from the one of the
+ * remote segment, but can also be forced to do regardless of CRC difference.
+ */
+ void addOrReplaceSegment(String tableNameWithType, String segmentName, boolean forceDownload)
Review comment:
as discussed, I'm adding the forceDownload flag to reload...() APIs, and refactor this addOrReplace...().
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7319: Move offline seg download code to OfflineTableManager
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#issuecomment-901313573
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7319](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3fe8d5d) into [master](https://codecov.io/gh/apache/pinot/commit/c15249484a5bbe0e9fbbee311218c163a26dd263?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (c152494) will **decrease** coverage by `2.15%`.
> The diff coverage is `66.47%`.
> :exclamation: Current head 3fe8d5d differs from pull request most recent head 603310e. Consider uploading reports for the commit 603310e to get more accurate results
[](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7319 +/- ##
============================================
- Coverage 71.49% 69.33% -2.16%
+ Complexity 3282 3196 -86
============================================
Files 1504 1111 -393
Lines 74063 52530 -21533
Branches 10726 7877 -2849
============================================
- Hits 52949 36422 -16527
+ Misses 17515 13499 -4016
+ Partials 3599 2609 -990
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `69.33% <66.47%> (+0.38%)` | :arrow_up: |
| unittests2 | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...mmon/assignment/InstanceAssignmentConfigUtils.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vYXNzaWdubWVudC9JbnN0YW5jZUFzc2lnbm1lbnRDb25maWdVdGlscy5qYXZh) | `0.00% <0.00%> (-15.79%)` | :arrow_down: |
| [...he/pinot/common/assignment/InstancePartitions.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vYXNzaWdubWVudC9JbnN0YW5jZVBhcnRpdGlvbnMuamF2YQ==) | `0.00% <0.00%> (-41.18%)` | :arrow_down: |
| [...not/common/assignment/InstancePartitionsUtils.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vYXNzaWdubWVudC9JbnN0YW5jZVBhcnRpdGlvbnNVdGlscy5qYXZh) | `0.00% <0.00%> (-67.50%)` | :arrow_down: |
| [.../pinot/common/function/DateTimePatternHandler.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vRGF0ZVRpbWVQYXR0ZXJuSGFuZGxlci5qYXZh) | `100.00% <ø> (+16.66%)` | :arrow_up: |
| [...ot/common/function/FunctionDefinitionRegistry.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vRnVuY3Rpb25EZWZpbml0aW9uUmVnaXN0cnkuamF2YQ==) | `100.00% <ø> (+11.11%)` | :arrow_up: |
| [...n/function/scalar/DataTypeConversionFunctions.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vc2NhbGFyL0RhdGFUeXBlQ29udmVyc2lvbkZ1bmN0aW9ucy5qYXZh) | `0.00% <ø> (ø)` | |
| [...he/pinot/common/function/scalar/HashFunctions.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vc2NhbGFyL0hhc2hGdW5jdGlvbnMuamF2YQ==) | `100.00% <ø> (ø)` | |
| [.../org/apache/pinot/common/lineage/LineageEntry.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbGluZWFnZS9MaW5lYWdlRW50cnkuamF2YQ==) | `64.70% <0.00%> (ø)` | |
| [...not/common/lineage/SegmentLineageAccessHelper.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbGluZWFnZS9TZWdtZW50TGluZWFnZUFjY2Vzc0hlbHBlci5qYXZh) | `0.00% <ø> (-93.34%)` | :arrow_down: |
| [...ot/common/messages/RoutingTableRebuildMessage.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWVzc2FnZXMvUm91dGluZ1RhYmxlUmVidWlsZE1lc3NhZ2UuamF2YQ==) | `0.00% <ø> (-81.82%)` | :arrow_down: |
| ... and [732 more](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [c152494...603310e](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7319: Move offline seg download code to OfflineTableManager
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#issuecomment-901313573
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7319](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5290e6a) into [master](https://codecov.io/gh/apache/pinot/commit/c15249484a5bbe0e9fbbee311218c163a26dd263?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (c152494) will **increase** coverage by `0.00%`.
> The diff coverage is `87.39%`.
[](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7319 +/- ##
==========================================
Coverage 71.49% 71.49%
- Complexity 3282 3286 +4
==========================================
Files 1504 1502 -2
Lines 74063 73858 -205
Branches 10726 10694 -32
==========================================
- Hits 52949 52805 -144
+ Misses 17515 17462 -53
+ Partials 3599 3591 -8
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `30.83% <78.99%> (-0.01%)` | :arrow_down: |
| integration2 | `29.29% <63.02%> (-0.05%)` | :arrow_down: |
| unittests1 | `69.14% <73.41%> (+0.20%)` | :arrow_up: |
| unittests2 | `14.59% <0.00%> (-0.05%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ata/manager/realtime/RealtimeTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVUYWJsZURhdGFNYW5hZ2VyLmphdmE=) | `66.93% <ø> (ø)` | |
| [.../pinot/core/data/manager/BaseTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvQmFzZVRhYmxlRGF0YU1hbmFnZXIuamF2YQ==) | `86.51% <50.00%> (-0.84%)` | :arrow_down: |
| [.../data/manager/offline/OfflineTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvb2ZmbGluZS9PZmZsaW5lVGFibGVEYXRhTWFuYWdlci5qYXZh) | `83.33% <81.81%> (-16.67%)` | :arrow_down: |
| [.../pinot/server/starter/helix/BaseServerStarter.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9CYXNlU2VydmVyU3RhcnRlci5qYXZh) | `55.81% <100.00%> (+1.05%)` | :arrow_up: |
| [...server/starter/helix/HelixInstanceDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9IZWxpeEluc3RhbmNlRGF0YU1hbmFnZXIuamF2YQ==) | `80.22% <100.00%> (-2.36%)` | :arrow_down: |
| [...er/starter/helix/SegmentMessageHandlerFactory.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9TZWdtZW50TWVzc2FnZUhhbmRsZXJGYWN0b3J5LmphdmE=) | `77.92% <100.00%> (-0.29%)` | :arrow_down: |
| [...r/helix/SegmentOnlineOfflineStateModelFactory.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9TZWdtZW50T25saW5lT2ZmbGluZVN0YXRlTW9kZWxGYWN0b3J5LmphdmE=) | `61.11% <100.00%> (+2.23%)` | :arrow_up: |
| [...lix/core/minion/PinotHelixTaskResourceManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL21pbmlvbi9QaW5vdEhlbGl4VGFza1Jlc291cmNlTWFuYWdlci5qYXZh) | `46.53% <0.00%> (-30.14%)` | :arrow_down: |
| [...a/manager/realtime/RealtimeSegmentDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVTZWdtZW50RGF0YU1hbmFnZXIuamF2YQ==) | `50.00% <0.00%> (-25.00%)` | :arrow_down: |
| [...pinot/common/utils/fetcher/HttpSegmentFetcher.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvZmV0Y2hlci9IdHRwU2VnbWVudEZldGNoZXIuamF2YQ==) | `48.00% <0.00%> (-5.34%)` | :arrow_down: |
| ... and [220 more](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [c152494...5290e6a](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7319: Move offline seg download code to OfflineTableManager
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#issuecomment-901313573
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7319](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e479f0e) into [master](https://codecov.io/gh/apache/pinot/commit/12e5bcd37350a3e85a9e9630a7f6de05d91b5a24?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (12e5bcd) will **increase** coverage by `0.13%`.
> The diff coverage is `81.51%`.
[](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7319 +/- ##
============================================
+ Coverage 71.32% 71.45% +0.13%
+ Complexity 3367 3281 -86
============================================
Files 1503 1502 -1
Lines 74030 73849 -181
Branches 10718 10692 -26
============================================
- Hits 52803 52771 -32
+ Misses 17622 17487 -135
+ Partials 3605 3591 -14
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `30.77% <78.99%> (+0.69%)` | :arrow_up: |
| integration2 | `29.32% <63.02%> (-0.01%)` | :arrow_down: |
| unittests1 | `69.06% <44.30%> (+0.04%)` | :arrow_up: |
| unittests2 | `14.58% <0.00%> (-0.07%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ata/manager/realtime/RealtimeTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVUYWJsZURhdGFNYW5hZ2VyLmphdmE=) | `67.75% <ø> (+0.81%)` | :arrow_up: |
| [.../pinot/core/data/manager/BaseTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvQmFzZVRhYmxlRGF0YU1hbmFnZXIuamF2YQ==) | `86.51% <50.00%> (-0.84%)` | :arrow_down: |
| [.../data/manager/offline/OfflineTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvb2ZmbGluZS9PZmZsaW5lVGFibGVEYXRhTWFuYWdlci5qYXZh) | `75.00% <72.72%> (-25.00%)` | :arrow_down: |
| [.../pinot/server/starter/helix/BaseServerStarter.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9CYXNlU2VydmVyU3RhcnRlci5qYXZh) | `55.81% <100.00%> (+1.05%)` | :arrow_up: |
| [...server/starter/helix/HelixInstanceDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9IZWxpeEluc3RhbmNlRGF0YU1hbmFnZXIuamF2YQ==) | `80.22% <100.00%> (-2.36%)` | :arrow_down: |
| [...er/starter/helix/SegmentMessageHandlerFactory.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9TZWdtZW50TWVzc2FnZUhhbmRsZXJGYWN0b3J5LmphdmE=) | `77.92% <100.00%> (-0.29%)` | :arrow_down: |
| [...r/helix/SegmentOnlineOfflineStateModelFactory.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9TZWdtZW50T25saW5lT2ZmbGluZVN0YXRlTW9kZWxGYWN0b3J5LmphdmE=) | `61.11% <100.00%> (+2.23%)` | :arrow_up: |
| [...ache/pinot/segment/spi/store/SegmentDirectory.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL3N0b3JlL1NlZ21lbnREaXJlY3RvcnkuamF2YQ==) | `50.00% <0.00%> (-33.34%)` | :arrow_down: |
| [.../pinot/segment/spi/store/ColumnIndexDirectory.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL3N0b3JlL0NvbHVtbkluZGV4RGlyZWN0b3J5LmphdmE=) | `40.00% <0.00%> (-26.67%)` | :arrow_down: |
| [...lix/core/minion/PinotHelixTaskResourceManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL21pbmlvbi9QaW5vdEhlbGl4VGFza1Jlc291cmNlTWFuYWdlci5qYXZh) | `46.53% <0.00%> (-25.47%)` | :arrow_down: |
| ... and [260 more](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [12e5bcd...e479f0e](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7319: Move offline seg download code to OfflineTableManager
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#issuecomment-901313573
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7319](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5290e6a) into [master](https://codecov.io/gh/apache/pinot/commit/c15249484a5bbe0e9fbbee311218c163a26dd263?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (c152494) will **decrease** coverage by `6.93%`.
> The diff coverage is `73.41%`.
[](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7319 +/- ##
============================================
- Coverage 71.49% 64.55% -6.94%
- Complexity 3282 3286 +4
============================================
Files 1504 1457 -47
Lines 74063 72035 -2028
Branches 10726 10501 -225
============================================
- Hits 52949 46501 -6448
- Misses 17515 22183 +4668
+ Partials 3599 3351 -248
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `69.13% <73.41%> (+0.18%)` | :arrow_up: |
| unittests2 | `14.57% <0.00%> (-0.06%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ata/manager/realtime/RealtimeTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVUYWJsZURhdGFNYW5hZ2VyLmphdmE=) | `10.61% <ø> (-56.33%)` | :arrow_down: |
| [.../pinot/core/data/manager/BaseTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvQmFzZVRhYmxlRGF0YU1hbmFnZXIuamF2YQ==) | `86.51% <50.00%> (-0.84%)` | :arrow_down: |
| [.../data/manager/offline/OfflineTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvb2ZmbGluZS9PZmZsaW5lVGFibGVEYXRhTWFuYWdlci5qYXZh) | `76.19% <74.02%> (-23.81%)` | :arrow_down: |
| [...a/org/apache/pinot/common/metrics/MinionMeter.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25NZXRlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...g/apache/pinot/common/metrics/ControllerMeter.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Db250cm9sbGVyTWV0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/BrokerQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Ccm9rZXJRdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/MinionQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25RdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pache/pinot/common/utils/grpc/GrpcQueryClient.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvZ3JwYy9HcnBjUXVlcnlDbGllbnQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...he/pinot/common/messages/SegmentReloadMessage.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWVzc2FnZXMvU2VnbWVudFJlbG9hZE1lc3NhZ2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pinot/minion/exception/TaskCancelledException.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vZXhjZXB0aW9uL1Rhc2tDYW5jZWxsZWRFeGNlcHRpb24uamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [486 more](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [c152494...5290e6a](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #7319: Move offline seg download code to OfflineTableManager
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#discussion_r699715552
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
##########
@@ -256,4 +267,147 @@ public void addSegmentError(String segmentName, SegmentErrorInfo segmentErrorInf
.collect(Collectors.toMap(map -> map.getKey().getSecond(), Map.Entry::getValue));
}
}
+
+ @Override
+ public void addOrReplaceSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentMetadata localMetadata, SegmentZKMetadata zkMetadata, boolean forceDownload)
Review comment:
annotate local metadata as nullable
##########
File path: pinot-core/src/test/java/org/apache/pinot/core/data/manager/BaseTableDataManagerAcquireSegmentTest.java
##########
@@ -240,7 +239,7 @@ public void testReplace()
runStorageServer(numQueryThreads, runTimeSec, tableDataManager); // replaces segments while online
-// System.out.println("Nops = " + _numQueries + ",nDrops=" + _nDestroys + ",nCreates=" + _allSegments.size());
+ // System.out.println("Nops = " + _numQueries + ",nDrops=" + _nDestroys + ",nCreates=" + _allSegments.size());
Review comment:
Remove it
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java
##########
@@ -248,19 +249,27 @@ public boolean isPartialUpsertEnabled() {
return _upsertMode == UpsertConfig.Mode.PARTIAL;
}
+ /**
+ * Add or replace an immutable segment created by the REALTIME table.
+ * Skipping segment still in consuming state.
+ */
+ @Override
+ public void addOrReplaceSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentMetadata localMetadata, SegmentZKMetadata zkMetadata, boolean forceDownload)
+ throws Exception {
+ throw new UnsupportedOperationException();
Review comment:
Why do we throw exception here?
##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/data/manager/TableDataManager.java
##########
@@ -146,4 +156,12 @@ void addSegment(String segmentName, TableConfig tableConfig, IndexLoadingConfig
* @return List of {@link SegmentErrorInfo}
*/
Map<String, SegmentErrorInfo> getSegmentErrors();
+
+ /**
+ * Get the dir for the given table segment, when TableDataManager object is not
+ * yet initialized for the given table.
+ */
+ static File getSegmentDataDir(String instanceDataDir, String tableNameWithType, String segmentName) {
Review comment:
This is instance level method, and seems only used in `InstanceDataManager`. Can we inline it instead of putting it as an interface method?
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java
##########
@@ -248,19 +249,27 @@ public boolean isPartialUpsertEnabled() {
return _upsertMode == UpsertConfig.Mode.PARTIAL;
}
+ /**
+ * Add or replace an immutable segment created by the REALTIME table.
+ * Skipping segment still in consuming state.
+ */
+ @Override
+ public void addOrReplaceSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentMetadata localMetadata, SegmentZKMetadata zkMetadata, boolean forceDownload)
Review comment:
Annotate as nullable
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/InstanceDataManager.java
##########
@@ -86,11 +86,28 @@ void reloadSegment(String tableNameWithType, String segmentName)
throws Exception;
/**
- * Reloads all segment in a table.
+ * Reloads all segments in a table.
*/
void reloadAllSegments(String tableNameWithType)
throws Exception;
+ /**
+ * Adds or replaces a segment in a table. Different from segment reloading,
+ * 1. this method doesn't assume the existence of TableDataManager object and it
+ * can actually initialize the TableDataManager for the segment;
+ * 2. this method can download a new segment to replace the local one before loading.
+ * Download is conducted when local segment's CRC is different from the one of the
+ * remote segment, but can also be forced to do regardless of CRC difference.
+ */
+ void addOrReplaceSegment(String tableNameWithType, String segmentName, boolean forceDownload)
Review comment:
The `forceDownload` should not be part of this API. We can add separate APIs for `redownloadSegment` and `redownloadAllSegments`, or add the flag into the `reload` apis.
##########
File path: pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixInstanceDataManager.java
##########
@@ -311,6 +310,47 @@ private void reloadSegment(String tableNameWithType, SegmentMetadata segmentMeta
}
}
+ @Override
+ public void addOrReplaceSegment(String tableNameWithType, String segmentName, boolean forceDownload)
+ throws Exception {
+ LOGGER.info("Adding or replacing segment: {} for table: {} downloadIsForced: {}", segmentName, tableNameWithType,
+ forceDownload);
+
+ // Get updated table config and segment metadata from Zookeeper.
+ TableConfig tableConfig = ZKMetadataProvider.getTableConfig(_propertyStore, tableNameWithType);
+ Preconditions.checkNotNull(tableConfig);
+ SegmentZKMetadata zkMetadata =
+ ZKMetadataProvider.getSegmentZKMetadata(_propertyStore, tableNameWithType, segmentName);
+ Preconditions.checkNotNull(zkMetadata);
+
+ // This method might modify the file on disk. Use segment lock to prevent race condition
+ Lock segmentLock = SegmentLocks.getSegmentLock(tableNameWithType, segmentName);
+ try {
+ // Lock the segment to get its metadata, so that no other threads are modifying
+ // the disk files of this segment.
+ segmentLock.lock();
+ SegmentMetadata localMetadata = getSegmentMetadata(tableNameWithType, segmentName);
+
+ _tableDataManagerMap.computeIfAbsent(tableNameWithType, k -> createTableDataManager(k, tableConfig))
+ .addOrReplaceSegment(segmentName, new IndexLoadingConfig(_instanceDataManagerConfig, tableConfig),
+ localMetadata, zkMetadata, forceDownload);
+ LOGGER.info("Added or replaced segment: {} of table: {}", segmentName, tableNameWithType);
+ } finally {
+ segmentLock.unlock();
+ }
+ }
+
+ @Override
+ public void addOrReplaceAllSegments(String tableNameWithType, boolean forceDownload)
+ throws Exception {
+ LOGGER.info("Adding or replacing all segments in table: {} downloadIsForced: {}", tableNameWithType, forceDownload);
+ List<SegmentMetadata> segMds = getAllSegmentsMetadata(tableNameWithType);
+ for (SegmentMetadata segMd : segMds) {
+ addOrReplaceSegment(tableNameWithType, segMd.getName(), forceDownload);
Review comment:
Add a helper method with `TableConfig` and `SegmentZKMetadata` as the arguments, and make `addOrReplaceSegment()` and `addOrReplaceAllSegments()` calling the helper method. We should avoid reading them for each segment for bulk replace
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
##########
@@ -256,4 +267,147 @@ public void addSegmentError(String segmentName, SegmentErrorInfo segmentErrorInf
.collect(Collectors.toMap(map -> map.getKey().getSecond(), Map.Entry::getValue));
}
}
+
+ @Override
+ public void addOrReplaceSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentMetadata localMetadata, SegmentZKMetadata zkMetadata, boolean forceDownload)
+ throws Exception {
+ if (!forceDownload && !isNewSegment(zkMetadata, localMetadata)) {
+ LOGGER.info("Segment: {} of table: {} has crc: {} same as before, already loaded, do nothing", segmentName,
+ _tableNameWithType, localMetadata.getCrc());
+ return;
+ }
+
+ // If not forced to download, then try to recover if no local metadata is provided.
+ if (!forceDownload && localMetadata == null) {
+ LOGGER.info("Segment: {} of table: {} is not loaded, checking disk", segmentName, _tableNameWithType);
+ localMetadata = recoverSegmentQuietly(segmentName);
+ if (!isNewSegment(zkMetadata, localMetadata)) {
+ LOGGER.info("Segment: {} of table {} has crc: {} same as before, loading", segmentName, _tableNameWithType,
+ localMetadata.getCrc());
+ if (loadSegmentQuietly(segmentName, indexLoadingConfig)) {
+ return;
+ }
+ localMetadata = null;
+ }
+ }
+
+ // Download segment and replace the local one, either due to being forced to download, or the
+ // local segment is not able to get loaded, or the segment data is updated and has new CRC now.
+ if (forceDownload) {
+ LOGGER.info("Force to download segment: {} of table: {}", segmentName, _tableNameWithType);
+ } else if (localMetadata == null) {
+ LOGGER.info("Download segment: {} of table: {} as no one exists locally", segmentName, _tableNameWithType);
+ } else {
+ LOGGER.info("Download segment: {} of table: {} as local crc: {} mismatches remote crc: {}.", segmentName,
+ _tableNameWithType, localMetadata.getCrc(), zkMetadata.getCrc());
+ }
+ File indexDir = downloadSegmentFromDeepStore(segmentName, zkMetadata);
+ SegmentMetadata segmentMetadata = new SegmentMetadataImpl(indexDir);
+ addSegment(indexDir, indexLoadingConfig);
+ LOGGER.info("Downloaded and replaced segment: {} of table: {} with crc: {}", segmentName, _tableNameWithType,
+ segmentMetadata.getCrc());
+ }
+
+ /**
+ * Server restart during segment reload might leave segment directory in inconsistent state, like the index
+ * directory might not exist but segment backup directory existed. This method tries to recover from reload
+ * failure before checking the existence of the index directory and loading segment metadata from it.
+ */
+ private SegmentMetadata recoverSegmentQuietly(String segmentName) {
Review comment:
(nit) I don't think we need to name it as `quietly`, same for `loadSegmentQuietly`
##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/data/manager/TableDataManager.java
##########
@@ -78,6 +80,14 @@ void addSegment(File indexDir, IndexLoadingConfig indexLoadingConfig)
void addSegment(String segmentName, TableConfig tableConfig, IndexLoadingConfig indexLoadingConfig)
throws Exception;
+ /**
+ * Add or replace an immutable segment for current table, which can be either
+ * OFFLINE or REALTIME table.
+ */
+ void addOrReplaceSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ @Nullable SegmentMetadata localMetadata, SegmentZKMetadata zkMetadata, boolean forceDownload)
Review comment:
(nit) for slightly better readability:
```suggestion
SegmentZKMetadata remoteZKMetadata, @Nullable SegmentMetadata localMetadata, boolean forceDownload)
```
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
##########
@@ -256,4 +267,147 @@ public void addSegmentError(String segmentName, SegmentErrorInfo segmentErrorInf
.collect(Collectors.toMap(map -> map.getKey().getSecond(), Map.Entry::getValue));
}
}
+
+ @Override
+ public void addOrReplaceSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentMetadata localMetadata, SegmentZKMetadata zkMetadata, boolean forceDownload)
+ throws Exception {
+ if (!forceDownload && !isNewSegment(zkMetadata, localMetadata)) {
+ LOGGER.info("Segment: {} of table: {} has crc: {} same as before, already loaded, do nothing", segmentName,
+ _tableNameWithType, localMetadata.getCrc());
+ return;
+ }
+
+ // If not forced to download, then try to recover if no local metadata is provided.
+ if (!forceDownload && localMetadata == null) {
+ LOGGER.info("Segment: {} of table: {} is not loaded, checking disk", segmentName, _tableNameWithType);
+ localMetadata = recoverSegmentQuietly(segmentName);
+ if (!isNewSegment(zkMetadata, localMetadata)) {
+ LOGGER.info("Segment: {} of table {} has crc: {} same as before, loading", segmentName, _tableNameWithType,
+ localMetadata.getCrc());
+ if (loadSegmentQuietly(segmentName, indexLoadingConfig)) {
Review comment:
What does quietly stand for?
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java
##########
@@ -256,4 +267,147 @@ public void addSegmentError(String segmentName, SegmentErrorInfo segmentErrorInf
.collect(Collectors.toMap(map -> map.getKey().getSecond(), Map.Entry::getValue));
}
}
+
+ @Override
+ public void addOrReplaceSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentMetadata localMetadata, SegmentZKMetadata zkMetadata, boolean forceDownload)
+ throws Exception {
+ if (!forceDownload && !isNewSegment(zkMetadata, localMetadata)) {
+ LOGGER.info("Segment: {} of table: {} has crc: {} same as before, already loaded, do nothing", segmentName,
+ _tableNameWithType, localMetadata.getCrc());
+ return;
Review comment:
This is not correct. The segment is on local disk, but we still need to load the segment into the memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7319: Move segment download and segment reload into TableManager
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#issuecomment-901313573
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7319](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2e009c4) into [master](https://codecov.io/gh/apache/pinot/commit/6294d1be67c465f907aafa8ad74f8c949c193991?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6294d1b) will **decrease** coverage by `1.19%`.
> The diff coverage is `87.50%`.
> :exclamation: Current head 2e009c4 differs from pull request most recent head 072b5d8. Consider uploading reports for the commit 072b5d8 to get more accurate results
[](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7319 +/- ##
============================================
- Coverage 71.89% 70.70% -1.20%
+ Complexity 3430 3348 -82
============================================
Files 1517 1516 -1
Lines 75039 75042 +3
Branches 10921 10931 +10
============================================
- Hits 53950 53057 -893
- Misses 17457 18358 +901
+ Partials 3632 3627 -5
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `29.11% <58.85%> (+0.02%)` | :arrow_up: |
| unittests1 | `69.76% <82.11%> (+0.06%)` | :arrow_up: |
| unittests2 | `14.51% <0.00%> (-0.03%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...er/starter/helix/SegmentMessageHandlerFactory.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9TZWdtZW50TWVzc2FnZUhhbmRsZXJGYWN0b3J5LmphdmE=) | `57.53% <75.00%> (-14.27%)` | :arrow_down: |
| [.../pinot/core/data/manager/BaseTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvQmFzZVRhYmxlRGF0YU1hbmFnZXIuamF2YQ==) | `82.38% <78.30%> (-4.98%)` | :arrow_down: |
| [...a/org/apache/pinot/common/utils/PinotDataType.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvUGlub3REYXRhVHlwZS5qYXZh) | `79.25% <100.00%> (+0.68%)` | :arrow_up: |
| [...ata/manager/realtime/RealtimeTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVUYWJsZURhdGFNYW5hZ2VyLmphdmE=) | `66.12% <100.00%> (-0.81%)` | :arrow_down: |
| [.../pinot/server/starter/helix/BaseServerStarter.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9CYXNlU2VydmVyU3RhcnRlci5qYXZh) | `49.70% <100.00%> (-5.06%)` | :arrow_down: |
| [...server/starter/helix/HelixInstanceDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9IZWxpeEluc3RhbmNlRGF0YU1hbmFnZXIuamF2YQ==) | `83.97% <100.00%> (+3.97%)` | :arrow_up: |
| [...r/helix/SegmentOnlineOfflineStateModelFactory.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9TZWdtZW50T25saW5lT2ZmbGluZVN0YXRlTW9kZWxGYWN0b3J5LmphdmE=) | `58.49% <100.00%> (-0.39%)` | :arrow_down: |
| [...va/org/apache/pinot/spi/utils/CommonConstants.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQ29tbW9uQ29uc3RhbnRzLmphdmE=) | `26.31% <100.00%> (ø)` | |
| [...pinot/minion/exception/TaskCancelledException.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtbWluaW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9taW5pb24vZXhjZXB0aW9uL1Rhc2tDYW5jZWxsZWRFeGNlcHRpb24uamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...nverttorawindex/ConvertToRawIndexTaskExecutor.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvY29udmVydHRvcmF3aW5kZXgvQ29udmVydFRvUmF3SW5kZXhUYXNrRXhlY3V0b3IuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [108 more](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [6294d1b...072b5d8](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] klsince commented on a change in pull request #7319: Move offline seg download code to OfflineTableManager
Posted by GitBox <gi...@apache.org>.
klsince commented on a change in pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#discussion_r704818424
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java
##########
@@ -248,19 +249,27 @@ public boolean isPartialUpsertEnabled() {
return _upsertMode == UpsertConfig.Mode.PARTIAL;
}
+ /**
+ * Add or replace an immutable segment created by the REALTIME table.
+ * Skipping segment still in consuming state.
+ */
+ @Override
+ public void addOrReplaceSegment(String segmentName, IndexLoadingConfig indexLoadingConfig,
+ SegmentMetadata localMetadata, SegmentZKMetadata zkMetadata, boolean forceDownload)
+ throws Exception {
+ throw new UnsupportedOperationException();
Review comment:
it looks like RealTime table doesn't support RefreshSegmentMsg [yet](https://github.com/apache/pinot/blob/master/pinot-controller/src/main/java/org/apache/pinot/controller/api/upload/ZKOperator.java#L75) thus no chance to call addOrReplaceSegment() on RT table mgr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7319: Move offline seg download code to OfflineTableManager
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#issuecomment-901313573
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7319](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (13a1795) into [master](https://codecov.io/gh/apache/pinot/commit/bb34406aaa205f5c85b88c928d477fd267eda1b4?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bb34406) will **increase** coverage by `4.84%`.
> The diff coverage is `72.50%`.
> :exclamation: Current head 13a1795 differs from pull request most recent head a499317. Consider uploading reports for the commit a499317 to get more accurate results
[](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7319 +/- ##
============================================
+ Coverage 64.65% 69.50% +4.84%
+ Complexity 3303 3209 -94
============================================
Files 1496 1114 -382
Lines 74289 52738 -21551
Branches 10795 7906 -2889
============================================
- Hits 48033 36655 -11378
+ Misses 22826 13478 -9348
+ Partials 3430 2605 -825
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `69.50% <72.50%> (+0.03%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ata/manager/realtime/RealtimeTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVUYWJsZURhdGFNYW5hZ2VyLmphdmE=) | `10.61% <ø> (-56.33%)` | :arrow_down: |
| [...t/segment/local/data/manager/TableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9kYXRhL21hbmFnZXIvVGFibGVEYXRhTWFuYWdlci5qYXZh) | `0.00% <0.00%> (ø)` | |
| [.../pinot/core/data/manager/BaseTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvQmFzZVRhYmxlRGF0YU1hbmFnZXIuamF2YQ==) | `86.51% <50.00%> (-0.84%)` | :arrow_down: |
| [.../data/manager/offline/OfflineTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvb2ZmbGluZS9PZmZsaW5lVGFibGVEYXRhTWFuYWdlci5qYXZh) | `76.19% <74.02%> (-23.81%)` | :arrow_down: |
| [...a/org/apache/pinot/common/metrics/MinionMeter.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25NZXRlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...g/apache/pinot/common/metrics/ControllerMeter.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Db250cm9sbGVyTWV0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/BrokerQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Ccm9rZXJRdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/MinionQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25RdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...pache/pinot/common/utils/grpc/GrpcQueryClient.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvZ3JwYy9HcnBjUXVlcnlDbGllbnQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...he/pinot/common/messages/SegmentReloadMessage.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWVzc2FnZXMvU2VnbWVudFJlbG9hZE1lc3NhZ2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [583 more](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [bb34406...a499317](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mcvsubbu commented on a change in pull request #7319: Move offline seg download code to OfflineTableManager
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on a change in pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#discussion_r704738027
##########
File path: pinot-core/src/test/java/org/apache/pinot/core/data/manager/BaseTableDataManagerAcquireSegmentTest.java
##########
@@ -240,7 +239,7 @@ public void testReplace()
runStorageServer(numQueryThreads, runTimeSec, tableDataManager); // replaces segments while online
-// System.out.println("Nops = " + _numQueries + ",nDrops=" + _nDestroys + ",nCreates=" + _allSegments.size());
+ // System.out.println("Nops = " + _numQueries + ",nDrops=" + _nDestroys + ",nCreates=" + _allSegments.size());
Review comment:
Probably un-intended. Random is random, no matter if there is one or many. I suggest you set the seed to be the current time, and print out that value once in the test. Refactor the test to use the same random everywhere
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7319: Move offline seg download code to OfflineTableManager
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#issuecomment-901313573
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7319](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3fe8d5d) into [master](https://codecov.io/gh/apache/pinot/commit/c15249484a5bbe0e9fbbee311218c163a26dd263?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (c152494) will **decrease** coverage by `6.78%`.
> The diff coverage is `59.57%`.
> :exclamation: Current head 3fe8d5d differs from pull request most recent head 603310e. Consider uploading reports for the commit 603310e to get more accurate results
[](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7319 +/- ##
============================================
- Coverage 71.49% 64.70% -6.79%
- Complexity 3282 3288 +6
============================================
Files 1504 1458 -46
Lines 74063 72475 -1588
Branches 10726 10600 -126
============================================
- Hits 52949 46897 -6052
- Misses 17515 22202 +4687
+ Partials 3599 3376 -223
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `69.33% <66.47%> (+0.38%)` | :arrow_up: |
| unittests2 | `14.45% <3.71%> (-0.19%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../org/apache/pinot/client/ResultTableResultSet.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L1Jlc3VsdFRhYmxlUmVzdWx0U2V0LmphdmE=) | `88.00% <ø> (ø)` | |
| [...mmon/assignment/InstanceAssignmentConfigUtils.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vYXNzaWdubWVudC9JbnN0YW5jZUFzc2lnbm1lbnRDb25maWdVdGlscy5qYXZh) | `0.00% <0.00%> (-15.79%)` | :arrow_down: |
| [...he/pinot/common/assignment/InstancePartitions.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vYXNzaWdubWVudC9JbnN0YW5jZVBhcnRpdGlvbnMuamF2YQ==) | `0.00% <0.00%> (-41.18%)` | :arrow_down: |
| [...not/common/assignment/InstancePartitionsUtils.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vYXNzaWdubWVudC9JbnN0YW5jZVBhcnRpdGlvbnNVdGlscy5qYXZh) | `0.00% <0.00%> (-67.50%)` | :arrow_down: |
| [.../pinot/common/function/DateTimePatternHandler.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vRGF0ZVRpbWVQYXR0ZXJuSGFuZGxlci5qYXZh) | `100.00% <ø> (+16.66%)` | :arrow_up: |
| [...ot/common/function/FunctionDefinitionRegistry.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vRnVuY3Rpb25EZWZpbml0aW9uUmVnaXN0cnkuamF2YQ==) | `100.00% <ø> (+11.11%)` | :arrow_up: |
| [...n/function/scalar/DataTypeConversionFunctions.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vc2NhbGFyL0RhdGFUeXBlQ29udmVyc2lvbkZ1bmN0aW9ucy5qYXZh) | `0.00% <ø> (ø)` | |
| [...he/pinot/common/function/scalar/HashFunctions.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vc2NhbGFyL0hhc2hGdW5jdGlvbnMuamF2YQ==) | `100.00% <ø> (ø)` | |
| [.../org/apache/pinot/common/lineage/LineageEntry.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbGluZWFnZS9MaW5lYWdlRW50cnkuamF2YQ==) | `64.70% <0.00%> (ø)` | |
| [...not/common/lineage/SegmentLineageAccessHelper.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbGluZWFnZS9TZWdtZW50TGluZWFnZUFjY2Vzc0hlbHBlci5qYXZh) | `0.00% <ø> (-93.34%)` | :arrow_down: |
| ... and [558 more](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [c152494...603310e](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7319: Move offline seg download code to OfflineTableManager
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7319:
URL: https://github.com/apache/pinot/pull/7319#issuecomment-901313573
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7319](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (91e3244) into [master](https://codecov.io/gh/apache/pinot/commit/2d1f7dc9060d45ac9f4ff0621da62790ea9b8e64?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2d1f7dc) will **decrease** coverage by `42.69%`.
> The diff coverage is `62.18%`.
> :exclamation: Current head 91e3244 differs from pull request most recent head e8f676d. Consider uploading reports for the commit e8f676d to get more accurate results
[](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7319 +/- ##
=============================================
- Coverage 71.65% 28.95% -42.70%
+ Complexity 3305 84 -3221
=============================================
Files 1515 1506 -9
Lines 74992 74630 -362
Branches 10903 10860 -43
=============================================
- Hits 53736 21611 -32125
- Misses 17629 51057 +33428
+ Partials 3627 1962 -1665
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `28.95% <62.18%> (-0.13%)` | :arrow_down: |
| unittests1 | `?` | |
| unittests2 | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ata/manager/realtime/RealtimeTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVUYWJsZURhdGFNYW5hZ2VyLmphdmE=) | `65.44% <0.00%> (-2.31%)` | :arrow_down: |
| [...t/segment/local/data/manager/TableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9kYXRhL21hbmFnZXIvVGFibGVEYXRhTWFuYWdlci5qYXZh) | `0.00% <0.00%> (ø)` | |
| [.../pinot/core/data/manager/BaseTableDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvQmFzZVRhYmxlRGF0YU1hbmFnZXIuamF2YQ==) | `65.85% <45.45%> (-21.51%)` | :arrow_down: |
| [...er/starter/helix/SegmentMessageHandlerFactory.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9TZWdtZW50TWVzc2FnZUhhbmRsZXJGYWN0b3J5LmphdmE=) | `63.63% <75.00%> (-14.57%)` | :arrow_down: |
| [.../pinot/server/starter/helix/BaseServerStarter.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9CYXNlU2VydmVyU3RhcnRlci5qYXZh) | `49.70% <100.00%> (-5.06%)` | :arrow_down: |
| [...server/starter/helix/HelixInstanceDataManager.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9IZWxpeEluc3RhbmNlRGF0YU1hbmFnZXIuamF2YQ==) | `79.77% <100.00%> (-2.81%)` | :arrow_down: |
| [...r/helix/SegmentOnlineOfflineStateModelFactory.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zZXJ2ZXIvc3RhcnRlci9oZWxpeC9TZWdtZW50T25saW5lT2ZmbGluZVN0YXRlTW9kZWxGYWN0b3J5LmphdmE=) | `58.49% <100.00%> (-0.39%)` | :arrow_down: |
| [...c/main/java/org/apache/pinot/common/tier/Tier.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdGllci9UaWVyLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../java/org/apache/pinot/spi/utils/BooleanUtils.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQm9vbGVhblV0aWxzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...ava/org/apache/pinot/spi/data/MetricFieldSpec.java](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9NZXRyaWNGaWVsZFNwZWMuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [1058 more](https://codecov.io/gh/apache/pinot/pull/7319/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [2d1f7dc...e8f676d](https://codecov.io/gh/apache/pinot/pull/7319?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org