You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "maobaolong (via GitHub)" <gi...@apache.org> on 2024/09/13 02:33:11 UTC

[PR] [#2117] feat(server): Introduce block count in buffer pool metrics [incubator-uniffle]

maobaolong opened a new pull request, #2118:
URL: https://github.com/apache/incubator-uniffle/pull/2118

   <!--
   1. Title: [#<issue>] <type>(<scope>): <subject>
      Examples:
        - "[#123] feat(operator): support xxx"
        - "[#233] fix: check null before access result in xxx"
        - "[MINOR] refactor: fix typo in variable name"
        - "[MINOR] docs: fix typo in README"
        - "[#255] test: fix flaky test NameOfTheTest"
      Reference: https://www.conventionalcommits.org/en/v1.0.0/
   2. Contributor guidelines:
      https://github.com/apache/incubator-uniffle/blob/master/CONTRIBUTING.md
   3. If the PR is unfinished, please mark this PR as draft.
   -->
   
   ### What changes were proposed in this pull request?
   
   Introduce block count in buffer pool metrics.
   
   ### Why are the changes needed?
   
   (Please clarify why the changes are needed. For instance,
     1. If you propose a new API, clarify the use case for a new API.
     2. If you fix a bug, describe the bug.)
   
   Fix: #2117
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes. `block_count_in_buffer_pool` metrics introduced.
   
   ### How was this patch tested?
   
   By dashboard server metrics popup page.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#2117] feat(server): Introduce bufferPool related metrics(block/buffer/shuffle count) [incubator-uniffle]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #2118:
URL: https://github.com/apache/incubator-uniffle/pull/2118#issuecomment-2370124365

   ## [Codecov](https://app.codecov.io/gh/apache/incubator-uniffle/pull/2118?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   Attention: Patch coverage is `51.72414%` with `14 lines` in your changes missing coverage. Please review.
   > Project coverage is 52.37%. Comparing base [(`7e84a1a`)](https://app.codecov.io/gh/apache/incubator-uniffle/commit/7e84a1a6d826a3c705b7896fde18e340022b4bff?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`79df183`)](https://app.codecov.io/gh/apache/incubator-uniffle/commit/79df183623331e28073e54e697d3ec0b6eba089e?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   > Report is 1 commits behind head on master.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/incubator-uniffle/pull/2118?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [...g/apache/uniffle/common/metrics/SupplierGauge.java](https://app.codecov.io/gh/apache/incubator-uniffle/pull/2118?src=pr&el=tree&filepath=common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Funiffle%2Fcommon%2Fmetrics%2FSupplierGauge.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS91bmlmZmxlL2NvbW1vbi9tZXRyaWNzL1N1cHBsaWVyR2F1Z2UuamF2YQ==) | 0.00% | [10 Missing :warning: ](https://app.codecov.io/gh/apache/incubator-uniffle/pull/2118?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [.../apache/uniffle/common/metrics/MetricsManager.java](https://app.codecov.io/gh/apache/incubator-uniffle/pull/2118?src=pr&el=tree&filepath=common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Funiffle%2Fcommon%2Fmetrics%2FMetricsManager.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS91bmlmZmxlL2NvbW1vbi9tZXRyaWNzL01ldHJpY3NNYW5hZ2VyLmphdmE=) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/incubator-uniffle/pull/2118?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...rg/apache/uniffle/server/ShuffleServerMetrics.java](https://app.codecov.io/gh/apache/incubator-uniffle/pull/2118?src=pr&el=tree&filepath=server%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Funiffle%2Fserver%2FShuffleServerMetrics.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS91bmlmZmxlL3NlcnZlci9TaHVmZmxlU2VydmVyTWV0cmljcy5qYXZh) | 50.00% | [1 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/incubator-uniffle/pull/2118?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #2118      +/-   ##
   ============================================
   + Coverage     43.83%   52.37%   +8.53%     
   - Complexity        0     2730    +2730     
   ============================================
     Files            21      444     +423     
     Lines          2537    20693   +18156     
     Branches          0     1896    +1896     
   ============================================
   + Hits           1112    10837    +9725     
   - Misses         1352     9169    +7817     
   - Partials         73      687     +614     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/incubator-uniffle/pull/2118/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [](https://app.codecov.io/gh/apache/incubator-uniffle/pull/2118/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `52.37% <51.72%> (+8.53%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/incubator-uniffle/pull/2118?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#2117] feat(server): Introduce block count in buffer pool metrics [incubator-uniffle]

Posted by "maobaolong (via GitHub)" <gi...@apache.org>.
maobaolong closed pull request #2118: [#2117] feat(server): Introduce block count in buffer pool metrics
URL: https://github.com/apache/incubator-uniffle/pull/2118


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#2117] feat(server): Introduce bufferPool related metrics(block/buffer/shuffle count) [incubator-uniffle]

Posted by "maobaolong (via GitHub)" <gi...@apache.org>.
maobaolong commented on PR #2118:
URL: https://github.com/apache/incubator-uniffle/pull/2118#issuecomment-2371487609

   I countered the flaky test `AccessClusterTest`, same with https://github.com/apache/incubator-uniffle/actions/runs/11008932091/job/30567534037?pr=2138


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#2117] feat(server): Introduce bufferPool related metrics(block/buffer/shuffle count) [incubator-uniffle]

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston merged PR #2118:
URL: https://github.com/apache/incubator-uniffle/pull/2118


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#2117] feat(server): Introduce bufferPool related metrics(block/buffer/shuffle count) [incubator-uniffle]

Posted by "maobaolong (via GitHub)" <gi...@apache.org>.
maobaolong closed pull request #2118: [#2117] feat(server): Introduce bufferPool related metrics(block/buffer/shuffle count)
URL: https://github.com/apache/incubator-uniffle/pull/2118


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#2117] feat(server): Introduce bufferPool related metrics(block/buffer/shuffle count) [incubator-uniffle]

Posted by "maobaolong (via GitHub)" <gi...@apache.org>.
maobaolong closed pull request #2118: [#2117] feat(server): Introduce bufferPool related metrics(block/buffer/shuffle count)
URL: https://github.com/apache/incubator-uniffle/pull/2118


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#2117] feat(server): Introduce bufferPool related metrics(block/buffer/shuffle count) [incubator-uniffle]

Posted by "maobaolong (via GitHub)" <gi...@apache.org>.
maobaolong closed pull request #2118: [#2117] feat(server): Introduce bufferPool related metrics(block/buffer/shuffle count)
URL: https://github.com/apache/incubator-uniffle/pull/2118


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#2117] feat(server): Introduce block count in buffer pool metrics [incubator-uniffle]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #2118:
URL: https://github.com/apache/incubator-uniffle/pull/2118#issuecomment-2347942400

   ## Test Results
    2 881 files  +20   2 881 suites  +20   5h 55m 52s :stopwatch: + 8m 41s
    1 019 tests ± 0   1 017 :white_check_mark: ± 0   1 :zzz: ±0  1 :x: ±0 
   12 867 runs  +49  12 851 :white_check_mark: +49  15 :zzz: ±0  1 :x: ±0 
   
   For more details on these failures, see [this check](https://github.com/apache/incubator-uniffle/runs/30087575481).
   
   Results for commit a7c9335a. ± Comparison against base commit c36d8c8d.
   
   [test-results]:data:application/gzip;base64,H4sIAESq42YC/1XMyQ7CIBSF4VdpWLsAKpMvY24ZEmJbDMPK+O7S2iIuz3dv/hdyfrYJ3QYqJbkMKBWf+21KhOzDugkZGa1U73n7IJioc95T0fpr4mcP/9ysgQM//4GNMcQquEos616lkotjtiqVjDQ7quyErrrvPqrDsvhcBwKh1TgyMGCuzsFkNQelnJkUFcoyjJ2wduIcvT8kTx5/FAEAAA==
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#2117] feat(server): Introduce block count in buffer pool metrics [incubator-uniffle]

Posted by "maobaolong (via GitHub)" <gi...@apache.org>.
maobaolong commented on PR #2118:
URL: https://github.com/apache/incubator-uniffle/pull/2118#issuecomment-2347925832

   @kuszz This PR base on the `addLabeledCacheGauge` API finished by you, you can create a new PR about that code.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#2117] feat(server): Introduce bufferPool related metrics(block/buffer/shuffle count) [incubator-uniffle]

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #2118:
URL: https://github.com/apache/incubator-uniffle/pull/2118#discussion_r1776474126


##########
common/src/main/java/org/apache/uniffle/common/metrics/SupplierGauge.java:
##########
@@ -36,27 +36,46 @@ class SupplierGauge<T extends Number> extends Collector implements Collector.Des
   private Supplier<T> supplier;
   private List<String> labelNames;
   private List<String> labelValues;
+  private long updateInterval;
+  private long lastUpdateTime;
+  private T lastValue;
 
   SupplierGauge(

Review Comment:
   Oh, I like this gauge. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org