You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "timothy-e (via GitHub)" <gi...@apache.org> on 2026/04/08 19:14:24 UTC

[PR] Export Adaptive Routing Stats as broker metrics [pinot]

timothy-e opened a new pull request, #18134:
URL: https://github.com/apache/pinot/pull/18134

   When multistage adaptive server selector stats collection is enabled (`pinot.broker.adaptive.server.selector.enable.stats.collection=true`), the broker tracks per-server routing stats (in-flight request count, latency EMA, hybrid score) in memory but has no way to observe them externally without hitting the broker's debug API. This makes it difficult to monitor the health and behavior of adaptive routing in production.
   
   We add new metrics for the stats:
   * adaptiveServerNumInFlightRequests
   * adaptiveServerLatencyEma
   * adaptiveServerHybridScore
   
   This introduces 3 new metrics for each broker x server combination on a tenant. Not all users / usecases will want or warrant this, **so we disable it by default.** 
   
   The resulting Pinot metric names are of the form: `pinot.broker.adaptiveServerLatencyEma.server.Server_pinotdb1_8098`
   
   ### Hybrid Score Range
   `ServerRoutingStatsEntry.java::computeHybridScore` defines a hybrid score as:
   
   ```java
     public double computeHybridScore() {
       double estimatedQSize = _numInFlightRequests + _inFlighRequestsEMA.getAverage();
       return Math.pow(estimatedQSize, _hybridScoreExponent) * _latencyMsEMA.getAverage();
     }
   ```
   _hybridScoreExponent defaults to 3.
   
   The hybrid score could be as low as 0 if there are no inflight requests, but as soon as there is an inflight request, we jump to at least _latencyMsEMA.getAverage(). An unhealthy server would be much higher because it would have an increased number of inflight requests (~ `(inflight_reqs * infight_reqs_ema) ^ 3` => a server with 5 inflight requests would have it's latency_ema multiplied by  ~ `(5 + 5) ^ 3 = 1000`. 
   
   Using a decimal data type would only help differentiate scores among servers with very low inflight requests, which isn't too helpful. 
   
   ### Testing
   Unit tests verify that we:
   * don't export when adaptive routing stats are disabled
   * don't export when adaptive routing stats metric export is disabled
   
   We deployed this to an internal stripe cluster with adaptive routing stats enabled, and saw metrics appearing in Grafana. 
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Export Adaptive Routing Stats as broker metrics [pinot]

Posted by "timothye-gh (via GitHub)" <gi...@apache.org>.
timothye-gh commented on PR #18134:
URL: https://github.com/apache/pinot/pull/18134#issuecomment-4253990926

   @xiangfu0 can you take another look at this when you have a chance? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Export Adaptive Routing Stats as broker metrics [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on PR #18134:
URL: https://github.com/apache/pinot/pull/18134#issuecomment-4320562014

   Documentation PR opened: https://github.com/pinot-contrib/pinot-docs/pull/763


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Export Adaptive Routing Stats as broker metrics [pinot]

Posted by "timothy-e (via GitHub)" <gi...@apache.org>.
timothy-e commented on PR #18134:
URL: https://github.com/apache/pinot/pull/18134#issuecomment-4209186806

   @Jackie-Jiang @vvivekiyer can you please take a look at this when you have a chance? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Export Adaptive Routing Stats as broker metrics [pinot]

Posted by "timothy-e (via GitHub)" <gi...@apache.org>.
timothy-e commented on PR #18134:
URL: https://github.com/apache/pinot/pull/18134#issuecomment-4226707136

   > I found a potential thread-safety issue in the `exportStatsAsMetrics()` method:
   > 
   > The iteration over `_serverQueryStatsMap.entrySet()` at line 408 is not protected. While ConcurrentHashMap is used, iterating over entrySet() while the map is being concurrently modified (via `recordStatsForQuerySubmission()`) can result in:
   > 
   > 1. ConcurrentModificationException
   > 2. Missed entries in the metrics export
   > 3. Inconsistent snapshots across iterations
   
   I don't think this is actually a problem. The [concurrent hash map java doc](https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ConcurrentHashMap.html) says:
   > Similarly, Iterators, Spliterators and Enumerations return elements reflecting the state of the hash table at some point at or since the creation of the iterator/enumeration. They do _not_ throw [ConcurrentModificationException](https://docs.oracle.com/javase/8/docs/api/java/util/ConcurrentModificationException.html).
   
   There are already 4 patterns like this in the file: `getServerRoutingStats`, `fetchNumInFlightRequestsForAllServers`. `fetchEMALatencyForAllServers`, `fetchHybridScoreForAllServers`. 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Export Adaptive Routing Stats as broker metrics [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 merged PR #18134:
URL: https://github.com/apache/pinot/pull/18134


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Export Adaptive Routing Stats as broker metrics [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on PR #18134:
URL: https://github.com/apache/pinot/pull/18134#issuecomment-4323342286

   Documentation PR has been created: https://github.com/pinot-contrib/pinot-docs/pull/764
   
   This documents the new adaptive routing metrics exported as broker metrics:
   - `adaptiveServerNumInFlightRequests`
   - `adaptiveServerLatencyEma`
   - `adaptiveServerHybridScore`
   
   Documentation covers:
   - Prerequisites and configuration requirements
   - Metric naming format and cardinality implications
   - Hybrid score calculation and interpretation
   - Reference to the new broker metrics section


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Export Adaptive Routing Stats as broker metrics [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on PR #18134:
URL: https://github.com/apache/pinot/pull/18134#issuecomment-4297890482

   Opened docs update: https://github.com/pinot-contrib/pinot-docs/pull/754


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Export Adaptive Routing Stats as broker metrics [pinot]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18134:
URL: https://github.com/apache/pinot/pull/18134#issuecomment-4209313875

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/18134?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `84.61538%` with `4 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 55.84%. Comparing base ([`739043c`](https://app.codecov.io/gh/apache/pinot/commit/739043c09b41d1ce69310f2688204da6fcade8d7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`bd68256`](https://app.codecov.io/gh/apache/pinot/commit/bd68256c2585e6a1a5fa93ef1d56fcd7ee204bf5?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 304 commits behind head on master.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/18134?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [...erver/routing/stats/ServerRoutingStatsManager.java](https://app.codecov.io/gh/apache/pinot/pull/18134?src=pr&el=tree&filepath=pinot-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcore%2Ftransport%2Fserver%2Frouting%2Fstats%2FServerRoutingStatsManager.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS90cmFuc3BvcnQvc2VydmVyL3JvdXRpbmcvc3RhdHMvU2VydmVyUm91dGluZ1N0YXRzTWFuYWdlci5qYXZh) | 90.47% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18134?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...va/org/apache/pinot/spi/utils/CommonConstants.java](https://app.codecov.io/gh/apache/pinot/pull/18134?src=pr&el=tree&filepath=pinot-spi%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fspi%2Futils%2FCommonConstants.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQ29tbW9uQ29uc3RhbnRzLmphdmE=) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18134?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   > :exclamation:  There is a different number of reports uploaded between BASE (739043c) and HEAD (bd68256). Click for more details.
   > 
   > <details><summary>HEAD has 28 uploads less than BASE</summary>
   >
   >| Flag | BASE (739043c) | HEAD (bd68256) |
   >|------|------|------|
   >|java-21|5|1|
   >|unittests1|2|1|
   >|unittests|4|1|
   >|temurin|8|1|
   >|java-11|3|0|
   >|unittests2|2|0|
   >|integration|4|0|
   >|integration1|1|0|
   >|custom-integration1|1|0|
   >|integration2|2|0|
   ></details>
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master   #18134      +/-   ##
   ============================================
   - Coverage     63.22%   55.84%   -7.38%     
   + Complexity     1454      822     -632     
   ============================================
     Files          3176     2481     -695     
     Lines        191002   142679   -48323     
     Branches      29204    22976    -6228     
   ============================================
   - Hits         120763    79683   -41080     
   + Misses        60818    56167    -4651     
   + Partials       9421     6829    -2592     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/18134/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/18134/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/18134/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/18134/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration2](https://app.codecov.io/gh/apache/pinot/pull/18134/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-11](https://app.codecov.io/gh/apache/pinot/pull/18134/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-21](https://app.codecov.io/gh/apache/pinot/pull/18134/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.84% <84.61%> (-7.34%)` | :arrow_down: |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/18134/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.84% <84.61%> (-7.38%)` | :arrow_down: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/18134/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.84% <84.61%> (-7.38%)` | :arrow_down: |
   | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/18134/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.84% <84.61%> (+0.29%)` | :arrow_up: |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/18134/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/18134?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org