You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "rickyma (via GitHub)" <gi...@apache.org> on 2024/02/13 05:20:05 UTC
[PR] [#1472][part-4] feature(server): Introduce metrics for Netty's pinnedDirectMemory and usedDirectMemory [incubator-uniffle]
rickyma opened a new pull request, #1524:
URL: https://github.com/apache/incubator-uniffle/pull/1524
### What changes were proposed in this pull request?
We need to know the exact direct memory usage of `PooledByteBufAllocator` in `Netty`.
So we should introduce metrics for Netty's `pinnedDirectMemory` and `usedDirectMemory`.
### Why are the changes needed?
A sub PR for: https://github.com/apache/incubator-uniffle/pull/1519
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing UTs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1472][part-4] feature(server): Introduce metrics for Netty's pinnedDirectMemory and usedDirectMemory [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on code in PR #1524:
URL: https://github.com/apache/incubator-uniffle/pull/1524#discussion_r1487778150
##########
server/src/main/java/org/apache/uniffle/server/ShuffleServerMetrics.java:
##########
@@ -79,6 +79,8 @@ public class ShuffleServerMetrics {
private static final String USED_BUFFER_SIZE = "used_buffer_size";
private static final String READ_USED_BUFFER_SIZE = "read_used_buffer_size";
private static final String USED_DIRECT_MEMORY_SIZE = "used_direct_memory_size";
+ private static final String ALLOCATED_DIRECT_MEMORY_SIZE = "allocated_direct_memory_size";
+ private static final String PINNED_DIRECT_MEMORY_SIZE = "pinned_direct_memory_size";
Review Comment:
Could you give it a better name?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1472][part-4] feature(server): Introduce metrics for Netty's pinnedDirectMemory and usedDirectMemory [incubator-uniffle]
Posted by "rickyma (via GitHub)" <gi...@apache.org>.
rickyma commented on code in PR #1524:
URL: https://github.com/apache/incubator-uniffle/pull/1524#discussion_r1487390047
##########
server/src/main/java/org/apache/uniffle/server/ShuffleServerMetrics.java:
##########
@@ -79,6 +79,8 @@ public class ShuffleServerMetrics {
private static final String USED_BUFFER_SIZE = "used_buffer_size";
private static final String READ_USED_BUFFER_SIZE = "read_used_buffer_size";
private static final String USED_DIRECT_MEMORY_SIZE = "used_direct_memory_size";
+ private static final String ALLOCATED_DIRECT_MEMORY_SIZE = "allocated_direct_memory_size";
+ private static final String PINNED_DIRECT_MEMORY_SIZE = "pinned_direct_memory_size";
Review Comment:
Refer to https://github.com/netty/netty/pull/1166.
To be brief, it means the direct memory actually used by `PooledByteBufAllocator`, which cannot be used anymore. If we use the old metric `usedDirectMemory`, we cannot tell whether the direct memory is cached by `PooledByteBufAllocator` or actually exhausted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1472][part-4] feature(server): Introduce metrics for Netty's pinnedDirectMemory and usedDirectMemory [incubator-uniffle]
Posted by "rickyma (via GitHub)" <gi...@apache.org>.
rickyma commented on code in PR #1524:
URL: https://github.com/apache/incubator-uniffle/pull/1524#discussion_r1487390047
##########
server/src/main/java/org/apache/uniffle/server/ShuffleServerMetrics.java:
##########
@@ -79,6 +79,8 @@ public class ShuffleServerMetrics {
private static final String USED_BUFFER_SIZE = "used_buffer_size";
private static final String READ_USED_BUFFER_SIZE = "read_used_buffer_size";
private static final String USED_DIRECT_MEMORY_SIZE = "used_direct_memory_size";
+ private static final String ALLOCATED_DIRECT_MEMORY_SIZE = "allocated_direct_memory_size";
+ private static final String PINNED_DIRECT_MEMORY_SIZE = "pinned_direct_memory_size";
Review Comment:
Refer to https://github.com/netty/netty/pull/1166.
To be brief, it means the direct memory actually used by `PooledByteBufAllocator`, which cannot be used anymore. If we use the old metric `usedDirectMemory`, we cannot tell whether the direct memory is cached by `PooledByteBufAllocator` or actually used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1472][part-4] feature(server): Introduce metrics for Netty's pinnedDirectMemory and usedDirectMemory [incubator-uniffle]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #1524:
URL: https://github.com/apache/incubator-uniffle/pull/1524#issuecomment-1940475097
## Test Results
2 287 files ±0 2 287 suites ±0 4h 28m 15s :stopwatch: -45s
819 tests ±0 818 :white_check_mark: ±0 1 :zzz: ±0 0 :x: ±0
9 086 runs ±0 9 073 :white_check_mark: ±0 13 :zzz: ±0 0 :x: ±0
Results for commit 49a3c723. ± Comparison against base commit d87dc900.
[test-results]:data:application/gzip;base64,H4sIAGcFy2UC/03MQQ6DIBCF4asY1l2AosP0Mg2FISFVaRBWTe9eNEhd/h/D+zDnZ9rYvet7BbeObdmna9scdfJhLSImjmOh8p72CyXwrMeWjTlI/enl3/uvBk77uQBvQDGGWCXmdd9ErqZa5yZyGBrVzQaXzaOvkyYsi08lmEQ9GOgHIA1EpUYLaPRTOQLpJBogMUkL7PsDi2la6RABAAA=
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1472][part-4] feature(server): Introduce metrics for Netty's pinnedDirectMemory and usedDirectMemory [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi merged PR #1524:
URL: https://github.com/apache/incubator-uniffle/pull/1524
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1472][part-4] feature(server): Introduce metrics for Netty's pinnedDirectMemory and usedDirectMemory [incubator-uniffle]
Posted by "rickyma (via GitHub)" <gi...@apache.org>.
rickyma commented on PR #1524:
URL: https://github.com/apache/incubator-uniffle/pull/1524#issuecomment-1940444496
@jerqi PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1472][part-4] feature(server): Introduce metrics for Netty's pinnedDirectMemory and usedDirectMemory [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on code in PR #1524:
URL: https://github.com/apache/incubator-uniffle/pull/1524#discussion_r1487317252
##########
server/src/main/java/org/apache/uniffle/server/ShuffleServerMetrics.java:
##########
@@ -79,6 +79,8 @@ public class ShuffleServerMetrics {
private static final String USED_BUFFER_SIZE = "used_buffer_size";
private static final String READ_USED_BUFFER_SIZE = "read_used_buffer_size";
private static final String USED_DIRECT_MEMORY_SIZE = "used_direct_memory_size";
+ private static final String ALLOCATED_DIRECT_MEMORY_SIZE = "allocated_direct_memory_size";
+ private static final String PINNED_DIRECT_MEMORY_SIZE = "pinned_direct_memory_size";
Review Comment:
What's the meaning of `pinned`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1472][part-4] feature(server): Introduce metrics for Netty's pinnedDirectMemory and usedDirectMemory [incubator-uniffle]
Posted by "rickyma (via GitHub)" <gi...@apache.org>.
rickyma commented on code in PR #1524:
URL: https://github.com/apache/incubator-uniffle/pull/1524#discussion_r1487390047
##########
server/src/main/java/org/apache/uniffle/server/ShuffleServerMetrics.java:
##########
@@ -79,6 +79,8 @@ public class ShuffleServerMetrics {
private static final String USED_BUFFER_SIZE = "used_buffer_size";
private static final String READ_USED_BUFFER_SIZE = "read_used_buffer_size";
private static final String USED_DIRECT_MEMORY_SIZE = "used_direct_memory_size";
+ private static final String ALLOCATED_DIRECT_MEMORY_SIZE = "allocated_direct_memory_size";
+ private static final String PINNED_DIRECT_MEMORY_SIZE = "pinned_direct_memory_size";
Review Comment:
Refer to https://github.com/netty/netty/pull/1166.
To be brief, it means the direct memory actually used by `PooledByteBufAllocator`, which cannot be used anymore. If we use the old metric `usedDirectMemory`, we cannot tell whether the diect memory is cached by `PooledByteBufAllocator` or actually used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1472][part-4] feature(server): Introduce metrics for Netty's pinnedDirectMemory and usedDirectMemory [incubator-uniffle]
Posted by "rickyma (via GitHub)" <gi...@apache.org>.
rickyma commented on code in PR #1524:
URL: https://github.com/apache/incubator-uniffle/pull/1524#discussion_r1487390047
##########
server/src/main/java/org/apache/uniffle/server/ShuffleServerMetrics.java:
##########
@@ -79,6 +79,8 @@ public class ShuffleServerMetrics {
private static final String USED_BUFFER_SIZE = "used_buffer_size";
private static final String READ_USED_BUFFER_SIZE = "read_used_buffer_size";
private static final String USED_DIRECT_MEMORY_SIZE = "used_direct_memory_size";
+ private static final String ALLOCATED_DIRECT_MEMORY_SIZE = "allocated_direct_memory_size";
+ private static final String PINNED_DIRECT_MEMORY_SIZE = "pinned_direct_memory_size";
Review Comment:
Refer to https://github.com/netty/netty/pull/1166.
To be brief, it means the direct memory actually used by `PooledByteBufAllocator`, cannot be used anymore. If we use the old metric `usedDirectMemory`, we cannot tell whether the direct memory is cached by `PooledByteBufAllocator` or if it's truly exhausted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1472][part-4] feature(server): Introduce metrics for Netty's pinnedDirectMemory and usedDirectMemory [incubator-uniffle]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #1524:
URL: https://github.com/apache/incubator-uniffle/pull/1524#issuecomment-1940452047
## [Codecov](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1524?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: `4 lines` in your changes are missing coverage. Please review.
> Comparison is base [(`d87dc90`)](https://app.codecov.io/gh/apache/incubator-uniffle/commit/d87dc900407d4faa9ba5f80cd19d08513e71082d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 54.20% compared to head [(`49a3c72`)](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1524?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 54.70%.
| [Files](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1524?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...pache/uniffle/server/NettyDirectMemoryTracker.java](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1524?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS91bmlmZmxlL3NlcnZlci9OZXR0eURpcmVjdE1lbW9yeVRyYWNrZXIuamF2YQ==) | 50.00% | [4 Missing :warning: ](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1524?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #1524 +/- ##
============================================
+ Coverage 54.20% 54.70% +0.50%
+ Complexity 2808 2271 -537
============================================
Files 430 353 -77
Lines 24410 15821 -8589
Branches 2082 1451 -631
============================================
- Hits 13231 8655 -4576
+ Misses 10349 6670 -3679
+ Partials 830 496 -334
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1524?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1472][part-4] feature(server): Introduce metrics for Netty's pinnedDirectMemory and usedDirectMemory [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on code in PR #1524:
URL: https://github.com/apache/incubator-uniffle/pull/1524#discussion_r1487778150
##########
server/src/main/java/org/apache/uniffle/server/ShuffleServerMetrics.java:
##########
@@ -79,6 +79,8 @@ public class ShuffleServerMetrics {
private static final String USED_BUFFER_SIZE = "used_buffer_size";
private static final String READ_USED_BUFFER_SIZE = "read_used_buffer_size";
private static final String USED_DIRECT_MEMORY_SIZE = "used_direct_memory_size";
+ private static final String ALLOCATED_DIRECT_MEMORY_SIZE = "allocated_direct_memory_size";
+ private static final String PINNED_DIRECT_MEMORY_SIZE = "pinned_direct_memory_size";
Review Comment:
Could you give it a better name?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1472][part-4] feature(server): Introduce metrics for Netty's pinnedDirectMemory and usedDirectMemory [incubator-uniffle]
Posted by "rickyma (via GitHub)" <gi...@apache.org>.
rickyma commented on code in PR #1524:
URL: https://github.com/apache/incubator-uniffle/pull/1524#discussion_r1487390047
##########
server/src/main/java/org/apache/uniffle/server/ShuffleServerMetrics.java:
##########
@@ -79,6 +79,8 @@ public class ShuffleServerMetrics {
private static final String USED_BUFFER_SIZE = "used_buffer_size";
private static final String READ_USED_BUFFER_SIZE = "read_used_buffer_size";
private static final String USED_DIRECT_MEMORY_SIZE = "used_direct_memory_size";
+ private static final String ALLOCATED_DIRECT_MEMORY_SIZE = "allocated_direct_memory_size";
+ private static final String PINNED_DIRECT_MEMORY_SIZE = "pinned_direct_memory_size";
Review Comment:
Refer to https://github.com/netty/netty/pull/11667.
To be brief, it means the direct memory actually used by `PooledByteBufAllocator`, cannot be used anymore. If we use the old metric `usedDirectMemory`, we cannot tell whether the direct memory is cached by `PooledByteBufAllocator` or if it's truly exhausted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org