You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "J-HowHuang (via GitHub)" <gi...@apache.org> on 2025/04/11 21:58:25 UTC
[PR] Fix internal logic of pre-check minimizeDataMovement [pinot]
J-HowHuang opened a new pull request, #15527:
URL: https://github.com/apache/pinot/pull/15527
## Description
For table config:
```
...
"instanceAssignmentConfigMap": {
"CONSUMING": {
"tagPoolConfig": {
"tag": "DefaultTenant_OFFLINE",
"poolBased": false,
"numPools": 0
},
"replicaGroupPartitionConfig": {
},
"partitionSelector": "INSTANCE_REPLICA_GROUP_PARTITION_SELECTOR",
"minimizeDataMovement": false
}
},
...
```
When it's set only for `CONSUMING` or `COMPLETED`, a warning shows up even if `minimizeDataMovement=ENABLE`, which is not ture.
```
WARN: minimizeDataMovement may not enabled for consuming or completed but instance assignment is allowed for at least one
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fix internal logic of pre-check minimizeDataMovement [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on code in PR #15527:
URL: https://github.com/apache/pinot/pull/15527#discussion_r2040352447
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/DefaultRebalancePreChecker.java:
##########
@@ -227,9 +228,31 @@ private RebalancePreCheckerResult checkIsMinimizeDataMovement(String rebalanceJo
return RebalancePreCheckerResult.warn(
"minimizeDataMovement may not be enabled for consuming or completed, but instance assigment is allowed "
Review Comment:
nit: not a mistake in your PR, but can you fix spelling `assigment` to `assignment`
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/DefaultRebalancePreChecker.java:
##########
@@ -227,9 +228,31 @@ private RebalancePreCheckerResult checkIsMinimizeDataMovement(String rebalanceJo
return RebalancePreCheckerResult.warn(
"minimizeDataMovement may not be enabled for consuming or completed, but instance assigment is allowed "
+ "for both");
+ } else if (instanceAssignmentConfigConsuming != null) {
+ if (rebalanceConfig.getMinimizeDataMovement() == RebalanceConfig.MinimizeDataMovementOptions.ENABLE) {
+ return RebalancePreCheckerResult.pass("minimizeDataMovement is enabled");
+ }
+ if (instanceAssignmentConfigConsuming.isMinimizeDataMovement()) {
+ return rebalanceConfig.getMinimizeDataMovement() == RebalanceConfig.MinimizeDataMovementOptions.DISABLE
+ ? RebalancePreCheckerResult.warn("minimizeDataMovement is enabled in table config but it's overridden "
+ + "with disabled")
+ : RebalancePreCheckerResult.pass("minimizeDataMovement is enabled");
+ }
+ return RebalancePreCheckerResult.warn(
+ "minimizeDataMovement is not enabled for consuming segments, but instance assignment is allowed");
+ } else {
+ if (rebalanceConfig.getMinimizeDataMovement() == RebalanceConfig.MinimizeDataMovementOptions.ENABLE) {
+ return RebalancePreCheckerResult.pass("minimizeDataMovement is enabled");
+ }
+ if (instanceAssignmentConfigCompleted.isMinimizeDataMovement()) {
+ return rebalanceConfig.getMinimizeDataMovement() == RebalanceConfig.MinimizeDataMovementOptions.DISABLE
+ ? RebalancePreCheckerResult.warn("minimizeDataMovement is enabled in table config but it's overridden "
Review Comment:
same, let's call out this is for COMPLETED
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/DefaultRebalancePreChecker.java:
##########
@@ -227,9 +228,31 @@ private RebalancePreCheckerResult checkIsMinimizeDataMovement(String rebalanceJo
return RebalancePreCheckerResult.warn(
"minimizeDataMovement may not be enabled for consuming or completed, but instance assigment is allowed "
+ "for both");
+ } else if (instanceAssignmentConfigConsuming != null) {
+ if (rebalanceConfig.getMinimizeDataMovement() == RebalanceConfig.MinimizeDataMovementOptions.ENABLE) {
+ return RebalancePreCheckerResult.pass("minimizeDataMovement is enabled");
+ }
+ if (instanceAssignmentConfigConsuming.isMinimizeDataMovement()) {
+ return rebalanceConfig.getMinimizeDataMovement() == RebalanceConfig.MinimizeDataMovementOptions.DISABLE
+ ? RebalancePreCheckerResult.warn("minimizeDataMovement is enabled in table config but it's overridden "
Review Comment:
shall we call out this is for CONSUMING?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fix internal logic of pre-check minimizeDataMovement [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #15527:
URL: https://github.com/apache/pinot/pull/15527#issuecomment-2798171562
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15527?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 55.87%. Comparing base [(`196de1d`)](https://app.codecov.io/gh/apache/pinot/commit/196de1d54f2af4550949f821b9adf99cccc3d3fc?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`b78cdc1`)](https://app.codecov.io/gh/apache/pinot/commit/b78cdc17549e13229819d25a65c2efd94ee79043?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 1 commits behind head on master.
> :exclamation: There is a different number of reports uploaded between BASE (196de1d) and HEAD (b78cdc1). Click for more details.
>
> <details><summary>HEAD has 67 uploads less than BASE</summary>
>
>| Flag | BASE (196de1d) | HEAD (b78cdc1) |
>|------|------|------|
>|unittests|6|1|
>|unittests1|3|1|
>|java-21|10|1|
>|skip-bytebuffers-false|8|1|
>|temurin|15|1|
>|skip-bytebuffers-true|4|0|
>|unittests2|3|0|
>|java-11|5|0|
>|integration|9|0|
>|integration2|3|0|
>|integration1|3|0|
>|custom-integration1|3|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #15527 +/- ##
============================================
- Coverage 62.98% 55.87% -7.11%
+ Complexity 1382 681 -701
============================================
Files 2848 2223 -625
Lines 161601 120087 -41514
Branches 24754 19187 -5567
============================================
- Hits 101785 67104 -34681
+ Misses 52132 47455 -4677
+ Partials 7684 5528 -2156
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/15527/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/15527/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/15527/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/15527/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/15527/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/15527/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/15527/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.87% <ø> (-7.07%)` | :arrow_down: |
| [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/15527/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.87% <ø> (-7.10%)` | :arrow_down: |
| [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/15527/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/15527/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.87% <ø> (-7.11%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/15527/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.87% <ø> (-7.11%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/15527/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.87% <ø> (-0.03%)` | :arrow_down: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/15527/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/15527?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fix internal logic of pre-check minimizeDataMovement [pinot]
Posted by "klsince (via GitHub)" <gi...@apache.org>.
klsince merged PR #15527:
URL: https://github.com/apache/pinot/pull/15527
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org