You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "praveenc7 (via GitHub)" <gi...@apache.org> on 2025/08/19 19:00:00 UTC

[PR] [Query Resource Isolation] Fix Refresh message [pinot]

praveenc7 opened a new pull request, #16636:
URL: https://github.com/apache/pinot/pull/16636

   ## Summary
   Dual registration of `MessageHandlerFactory` leads to a No-op for the second registration. Moving the QueryWorkloadRefresh to the existing MessageHandler
   
   ## Testing done
   Quick starter test to check if message if received on config change


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [Query Resource Isolation] Fix Refresh message [pinot]

Posted by "vvivekiyer (via GitHub)" <gi...@apache.org>.
vvivekiyer merged PR #16636:
URL: https://github.com/apache/pinot/pull/16636


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [Query Resource Isolation] Fix Refresh message [pinot]

Posted by "vvivekiyer (via GitHub)" <gi...@apache.org>.
vvivekiyer commented on code in PR #16636:
URL: https://github.com/apache/pinot/pull/16636#discussion_r2304871155


##########
pinot-spi/src/main/java/org/apache/pinot/spi/accounting/WorkloadBudgetManager.java:
##########
@@ -113,6 +113,11 @@ public void addOrUpdateWorkload(String workload, long cpuBudgetNs, long memoryBu
         memoryBudgetBytes);
   }
 
+  public void deleteWorkload(String workload) {
+    _workloadBudgets.remove(workload);

Review Comment:
   Can we initialize _workloadBudgets only if _isEnabled is true? 
   Also handle isEnabled in this function before removing. 



##########
pinot-spi/src/main/java/org/apache/pinot/spi/accounting/WorkloadBudgetManager.java:
##########
@@ -168,7 +173,7 @@ private void startBudgetResetTask() {
       // Also print the budget used in the last enforcement window.
       _workloadBudgets.forEach((workload, budget) -> {
         BudgetStats stats = budget.getStats();
-        LOGGER.debug("Workload: {} -> CPU: {}ns, Memory: {} bytes", workload, stats._cpuRemaining,
+        LOGGER.info("Workload: {} -> CPU: {}ns, Memory: {} bytes", workload, stats._cpuRemaining,

Review Comment:
   Can we please keep this as debug? This will log profusely during every refresh if enforcementWindow is small (say 1s).
   
   (Unrelated) can you remove the TODO at the beginning of the function? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [Query Resource Isolation] Fix Refresh message [pinot]

Posted by "praveenc7 (via GitHub)" <gi...@apache.org>.
praveenc7 commented on code in PR #16636:
URL: https://github.com/apache/pinot/pull/16636#discussion_r2286097499


##########
pinot-spi/src/main/java/org/apache/pinot/core/accounting/WorkloadBudgetManager.java:
##########
@@ -137,7 +137,7 @@ private void startBudgetResetTask() {
       // Also print the budget used in the last enforcement window.
       _workloadBudgets.forEach((workload, budget) -> {
         BudgetStats stats = budget.getStats();
-        LOGGER.debug("Workload: {} -> CPU: {}ns, Memory: {} bytes", workload, stats._cpuRemaining,
+        LOGGER.info("Workload: {} -> CPU: {}ns, Memory: {} bytes", workload, stats._cpuRemaining,

Review Comment:
   Found this log to be useful.
   
   Given it is not per query, would be useful to make it info



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [Query Resource Isolation] Fix Refresh message [pinot]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #16636:
URL: https://github.com/apache/pinot/pull/16636#issuecomment-3202198759

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/16636?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `50.00000%` with `1 line` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 56.40%. Comparing base ([`72bf3bf`](https://app.codecov.io/gh/apache/pinot/commit/72bf3bf26fa3d269daf5fd15a6dfbc43db6c8b66?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`985eada`](https://app.codecov.io/gh/apache/pinot/commit/985eada36f334fc93ac5638511ab588d0d00a00a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/16636?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [...t/common/messages/QueryWorkloadRefreshMessage.java](https://app.codecov.io/gh/apache/pinot/pull/16636?src=pr&el=tree&filepath=pinot-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcommon%2Fmessages%2FQueryWorkloadRefreshMessage.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWVzc2FnZXMvUXVlcnlXb3JrbG9hZFJlZnJlc2hNZXNzYWdlLmphdmE=) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/16636?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   > :exclamation:  There is a different number of reports uploaded between BASE (72bf3bf) and HEAD (985eada). Click for more details.
   > 
   > <details><summary>HEAD has 32 uploads less than BASE</summary>
   >
   >| Flag | BASE (72bf3bf) | HEAD (985eada) |
   >|------|------|------|
   >|java-21|4|1|
   >|unittests1|2|1|
   >|unittests|4|1|
   >|temurin|9|1|
   >|unittests2|2|0|
   >|java-11|5|0|
   >|integration|5|0|
   >|integration1|2|0|
   >|custom-integration1|2|0|
   >|integration2|1|0|
   ></details>
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master   #16636      +/-   ##
   ============================================
   - Coverage     63.32%   56.40%   -6.93%     
   + Complexity     1380      699     -681     
   ============================================
     Files          3026     2370     -656     
     Lines        176434   131442   -44992     
     Branches      27073    20954    -6119     
   ============================================
   - Hits         111734    74138   -37596     
   + Misses        56133    51157    -4976     
   + Partials       8567     6147    -2420     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/16636/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/16636/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/16636/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/16636/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration2](https://app.codecov.io/gh/apache/pinot/pull/16636/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-11](https://app.codecov.io/gh/apache/pinot/pull/16636/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-21](https://app.codecov.io/gh/apache/pinot/pull/16636/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.40% <50.00%> (-6.89%)` | :arrow_down: |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/16636/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.40% <50.00%> (-6.93%)` | :arrow_down: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/16636/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.40% <50.00%> (-6.93%)` | :arrow_down: |
   | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/16636/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.40% <50.00%> (-0.07%)` | :arrow_down: |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/16636/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/16636?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [Query Resource Isolation] Fix Refresh message [pinot]

Posted by "praveenc7 (via GitHub)" <gi...@apache.org>.
praveenc7 commented on PR #16636:
URL: https://github.com/apache/pinot/pull/16636#issuecomment-3234394329

   The failure is due to unrelated change :    `OfflineClusterMemBasedServerQueryKillingTest.testDigestOOM4MSE:397 [{"errorCode":450,"message":"Received 1 error from stage 2 on Server_localhost_22001: Found an error on stage 2 while reading from a child stage"}] expected [true] but found [false]`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [Query Resource Isolation] Fix Refresh message [pinot]

Posted by "vvivekiyer (via GitHub)" <gi...@apache.org>.
vvivekiyer commented on code in PR #16636:
URL: https://github.com/apache/pinot/pull/16636#discussion_r2306185119


##########
pinot-spi/src/main/java/org/apache/pinot/spi/accounting/WorkloadBudgetManager.java:
##########
@@ -113,6 +113,11 @@ public void addOrUpdateWorkload(String workload, long cpuBudgetNs, long memoryBu
         memoryBudgetBytes);
   }
 
+  public void deleteWorkload(String workload) {
+    _workloadBudgets.remove(workload);

Review Comment:
   Can you also move the initialization of _workloadBudgets such that it's done after the isEnabled check? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [Query Resource Isolation] Fix Refresh message [pinot]

Posted by "praveenc7 (via GitHub)" <gi...@apache.org>.
praveenc7 commented on code in PR #16636:
URL: https://github.com/apache/pinot/pull/16636#discussion_r2286097499


##########
pinot-spi/src/main/java/org/apache/pinot/core/accounting/WorkloadBudgetManager.java:
##########
@@ -137,7 +137,7 @@ private void startBudgetResetTask() {
       // Also print the budget used in the last enforcement window.
       _workloadBudgets.forEach((workload, budget) -> {
         BudgetStats stats = budget.getStats();
-        LOGGER.debug("Workload: {} -> CPU: {}ns, Memory: {} bytes", workload, stats._cpuRemaining,
+        LOGGER.info("Workload: {} -> CPU: {}ns, Memory: {} bytes", workload, stats._cpuRemaining,

Review Comment:
   Found this log to be useful.
   
   Given it is not per query having, would be useful to make it info



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [Query Resource Isolation] Fix Refresh message [pinot]

Posted by "praveenc7 (via GitHub)" <gi...@apache.org>.
praveenc7 commented on code in PR #16636:
URL: https://github.com/apache/pinot/pull/16636#discussion_r2305725771


##########
pinot-spi/src/main/java/org/apache/pinot/spi/accounting/WorkloadBudgetManager.java:
##########
@@ -168,7 +173,7 @@ private void startBudgetResetTask() {
       // Also print the budget used in the last enforcement window.
       _workloadBudgets.forEach((workload, budget) -> {
         BudgetStats stats = budget.getStats();

Review Comment:
   The Updates/deletes logs are spare, will update the rest



##########
pinot-spi/src/main/java/org/apache/pinot/spi/accounting/WorkloadBudgetManager.java:
##########
@@ -113,6 +113,11 @@ public void addOrUpdateWorkload(String workload, long cpuBudgetNs, long memoryBu
         memoryBudgetBytes);
   }
 
+  public void deleteWorkload(String workload) {
+    _workloadBudgets.remove(workload);

Review Comment:
   was initially thinking of keeping delete lenient. For example, if the feature is disabled and we try to delete a workload, the operation wouldn’t succeed. However, since the state is in-memory, it would eventually get cleaned up on restart.
   
   On second thought, it’s probably better to support delete only when the feature is enabled — it’s more useful when the feature is toggled on and off and on. 



##########
pinot-spi/src/main/java/org/apache/pinot/spi/accounting/WorkloadBudgetManager.java:
##########
@@ -168,7 +173,7 @@ private void startBudgetResetTask() {
       // Also print the budget used in the last enforcement window.
       _workloadBudgets.forEach((workload, budget) -> {
         BudgetStats stats = budget.getStats();
-        LOGGER.debug("Workload: {} -> CPU: {}ns, Memory: {} bytes", workload, stats._cpuRemaining,
+        LOGGER.info("Workload: {} -> CPU: {}ns, Memory: {} bytes", workload, stats._cpuRemaining,

Review Comment:
   I see some value in quickly debugging. But given we have metrics soon, we can keep it a debug if we anticipate aggresive enforcementWindow



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [Query Resource Isolation] Fix Refresh message [pinot]

Posted by "vvivekiyer (via GitHub)" <gi...@apache.org>.
vvivekiyer commented on code in PR #16636:
URL: https://github.com/apache/pinot/pull/16636#discussion_r2304830151


##########
pinot-spi/src/main/java/org/apache/pinot/spi/accounting/WorkloadBudgetManager.java:
##########
@@ -168,7 +173,7 @@ private void startBudgetResetTask() {
       // Also print the budget used in the last enforcement window.
       _workloadBudgets.forEach((workload, budget) -> {
         BudgetStats stats = budget.getStats();

Review Comment:
   Can we change all loggers here to debug? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org