You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "praveenc7 (via GitHub)" <gi...@apache.org> on 2025/08/19 19:00:00 UTC
[PR] [Query Resource Isolation] Fix Refresh message [pinot]
praveenc7 opened a new pull request, #16636:
URL: https://github.com/apache/pinot/pull/16636
## Summary
Dual registration of `MessageHandlerFactory` leads to a No-op for the second registration. Moving the QueryWorkloadRefresh to the existing MessageHandler
## Testing done
Quick starter test to check if message if received on config change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [Query Resource Isolation] Fix Refresh message [pinot]
Posted by "vvivekiyer (via GitHub)" <gi...@apache.org>.
vvivekiyer merged PR #16636:
URL: https://github.com/apache/pinot/pull/16636
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [Query Resource Isolation] Fix Refresh message [pinot]
Posted by "vvivekiyer (via GitHub)" <gi...@apache.org>.
vvivekiyer commented on code in PR #16636:
URL: https://github.com/apache/pinot/pull/16636#discussion_r2304871155
##########
pinot-spi/src/main/java/org/apache/pinot/spi/accounting/WorkloadBudgetManager.java:
##########
@@ -113,6 +113,11 @@ public void addOrUpdateWorkload(String workload, long cpuBudgetNs, long memoryBu
memoryBudgetBytes);
}
+ public void deleteWorkload(String workload) {
+ _workloadBudgets.remove(workload);
Review Comment:
Can we initialize _workloadBudgets only if _isEnabled is true?
Also handle isEnabled in this function before removing.
##########
pinot-spi/src/main/java/org/apache/pinot/spi/accounting/WorkloadBudgetManager.java:
##########
@@ -168,7 +173,7 @@ private void startBudgetResetTask() {
// Also print the budget used in the last enforcement window.
_workloadBudgets.forEach((workload, budget) -> {
BudgetStats stats = budget.getStats();
- LOGGER.debug("Workload: {} -> CPU: {}ns, Memory: {} bytes", workload, stats._cpuRemaining,
+ LOGGER.info("Workload: {} -> CPU: {}ns, Memory: {} bytes", workload, stats._cpuRemaining,
Review Comment:
Can we please keep this as debug? This will log profusely during every refresh if enforcementWindow is small (say 1s).
(Unrelated) can you remove the TODO at the beginning of the function?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [Query Resource Isolation] Fix Refresh message [pinot]
Posted by "praveenc7 (via GitHub)" <gi...@apache.org>.
praveenc7 commented on code in PR #16636:
URL: https://github.com/apache/pinot/pull/16636#discussion_r2286097499
##########
pinot-spi/src/main/java/org/apache/pinot/core/accounting/WorkloadBudgetManager.java:
##########
@@ -137,7 +137,7 @@ private void startBudgetResetTask() {
// Also print the budget used in the last enforcement window.
_workloadBudgets.forEach((workload, budget) -> {
BudgetStats stats = budget.getStats();
- LOGGER.debug("Workload: {} -> CPU: {}ns, Memory: {} bytes", workload, stats._cpuRemaining,
+ LOGGER.info("Workload: {} -> CPU: {}ns, Memory: {} bytes", workload, stats._cpuRemaining,
Review Comment:
Found this log to be useful.
Given it is not per query, would be useful to make it info
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [Query Resource Isolation] Fix Refresh message [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #16636:
URL: https://github.com/apache/pinot/pull/16636#issuecomment-3202198759
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/16636?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `50.00000%` with `1 line` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 56.40%. Comparing base ([`72bf3bf`](https://app.codecov.io/gh/apache/pinot/commit/72bf3bf26fa3d269daf5fd15a6dfbc43db6c8b66?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`985eada`](https://app.codecov.io/gh/apache/pinot/commit/985eada36f334fc93ac5638511ab588d0d00a00a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/16636?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...t/common/messages/QueryWorkloadRefreshMessage.java](https://app.codecov.io/gh/apache/pinot/pull/16636?src=pr&el=tree&filepath=pinot-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcommon%2Fmessages%2FQueryWorkloadRefreshMessage.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWVzc2FnZXMvUXVlcnlXb3JrbG9hZFJlZnJlc2hNZXNzYWdlLmphdmE=) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/16636?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (72bf3bf) and HEAD (985eada). Click for more details.
>
> <details><summary>HEAD has 32 uploads less than BASE</summary>
>
>| Flag | BASE (72bf3bf) | HEAD (985eada) |
>|------|------|------|
>|java-21|4|1|
>|unittests1|2|1|
>|unittests|4|1|
>|temurin|9|1|
>|unittests2|2|0|
>|java-11|5|0|
>|integration|5|0|
>|integration1|2|0|
>|custom-integration1|2|0|
>|integration2|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #16636 +/- ##
============================================
- Coverage 63.32% 56.40% -6.93%
+ Complexity 1380 699 -681
============================================
Files 3026 2370 -656
Lines 176434 131442 -44992
Branches 27073 20954 -6119
============================================
- Hits 111734 74138 -37596
+ Misses 56133 51157 -4976
+ Partials 8567 6147 -2420
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/16636/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/16636/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/16636/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/16636/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/16636/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/16636/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/16636/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.40% <50.00%> (-6.89%)` | :arrow_down: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/16636/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.40% <50.00%> (-6.93%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/16636/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.40% <50.00%> (-6.93%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/16636/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.40% <50.00%> (-0.07%)` | :arrow_down: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/16636/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/16636?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [Query Resource Isolation] Fix Refresh message [pinot]
Posted by "praveenc7 (via GitHub)" <gi...@apache.org>.
praveenc7 commented on PR #16636:
URL: https://github.com/apache/pinot/pull/16636#issuecomment-3234394329
The failure is due to unrelated change : `OfflineClusterMemBasedServerQueryKillingTest.testDigestOOM4MSE:397 [{"errorCode":450,"message":"Received 1 error from stage 2 on Server_localhost_22001: Found an error on stage 2 while reading from a child stage"}] expected [true] but found [false]`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [Query Resource Isolation] Fix Refresh message [pinot]
Posted by "vvivekiyer (via GitHub)" <gi...@apache.org>.
vvivekiyer commented on code in PR #16636:
URL: https://github.com/apache/pinot/pull/16636#discussion_r2306185119
##########
pinot-spi/src/main/java/org/apache/pinot/spi/accounting/WorkloadBudgetManager.java:
##########
@@ -113,6 +113,11 @@ public void addOrUpdateWorkload(String workload, long cpuBudgetNs, long memoryBu
memoryBudgetBytes);
}
+ public void deleteWorkload(String workload) {
+ _workloadBudgets.remove(workload);
Review Comment:
Can you also move the initialization of _workloadBudgets such that it's done after the isEnabled check?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [Query Resource Isolation] Fix Refresh message [pinot]
Posted by "praveenc7 (via GitHub)" <gi...@apache.org>.
praveenc7 commented on code in PR #16636:
URL: https://github.com/apache/pinot/pull/16636#discussion_r2286097499
##########
pinot-spi/src/main/java/org/apache/pinot/core/accounting/WorkloadBudgetManager.java:
##########
@@ -137,7 +137,7 @@ private void startBudgetResetTask() {
// Also print the budget used in the last enforcement window.
_workloadBudgets.forEach((workload, budget) -> {
BudgetStats stats = budget.getStats();
- LOGGER.debug("Workload: {} -> CPU: {}ns, Memory: {} bytes", workload, stats._cpuRemaining,
+ LOGGER.info("Workload: {} -> CPU: {}ns, Memory: {} bytes", workload, stats._cpuRemaining,
Review Comment:
Found this log to be useful.
Given it is not per query having, would be useful to make it info
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [Query Resource Isolation] Fix Refresh message [pinot]
Posted by "praveenc7 (via GitHub)" <gi...@apache.org>.
praveenc7 commented on code in PR #16636:
URL: https://github.com/apache/pinot/pull/16636#discussion_r2305725771
##########
pinot-spi/src/main/java/org/apache/pinot/spi/accounting/WorkloadBudgetManager.java:
##########
@@ -168,7 +173,7 @@ private void startBudgetResetTask() {
// Also print the budget used in the last enforcement window.
_workloadBudgets.forEach((workload, budget) -> {
BudgetStats stats = budget.getStats();
Review Comment:
The Updates/deletes logs are spare, will update the rest
##########
pinot-spi/src/main/java/org/apache/pinot/spi/accounting/WorkloadBudgetManager.java:
##########
@@ -113,6 +113,11 @@ public void addOrUpdateWorkload(String workload, long cpuBudgetNs, long memoryBu
memoryBudgetBytes);
}
+ public void deleteWorkload(String workload) {
+ _workloadBudgets.remove(workload);
Review Comment:
was initially thinking of keeping delete lenient. For example, if the feature is disabled and we try to delete a workload, the operation wouldn’t succeed. However, since the state is in-memory, it would eventually get cleaned up on restart.
On second thought, it’s probably better to support delete only when the feature is enabled — it’s more useful when the feature is toggled on and off and on.
##########
pinot-spi/src/main/java/org/apache/pinot/spi/accounting/WorkloadBudgetManager.java:
##########
@@ -168,7 +173,7 @@ private void startBudgetResetTask() {
// Also print the budget used in the last enforcement window.
_workloadBudgets.forEach((workload, budget) -> {
BudgetStats stats = budget.getStats();
- LOGGER.debug("Workload: {} -> CPU: {}ns, Memory: {} bytes", workload, stats._cpuRemaining,
+ LOGGER.info("Workload: {} -> CPU: {}ns, Memory: {} bytes", workload, stats._cpuRemaining,
Review Comment:
I see some value in quickly debugging. But given we have metrics soon, we can keep it a debug if we anticipate aggresive enforcementWindow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] [Query Resource Isolation] Fix Refresh message [pinot]
Posted by "vvivekiyer (via GitHub)" <gi...@apache.org>.
vvivekiyer commented on code in PR #16636:
URL: https://github.com/apache/pinot/pull/16636#discussion_r2304830151
##########
pinot-spi/src/main/java/org/apache/pinot/spi/accounting/WorkloadBudgetManager.java:
##########
@@ -168,7 +173,7 @@ private void startBudgetResetTask() {
// Also print the budget used in the last enforcement window.
_workloadBudgets.forEach((workload, budget) -> {
BudgetStats stats = budget.getStats();
Review Comment:
Can we change all loggers here to debug?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org