You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "zwangsheng (via GitHub)" <gi...@apache.org> on 2024/06/17 07:02:59 UTC
[PR] [CELEBORN-1463][FOLLOWUP] Respeact client/server threads num to avoid competitiveness [celeborn]
zwangsheng opened a new pull request, #2571:
URL: https://github.com/apache/celeborn/pull/2571
<!--
Thanks for sending a pull request! Here are some tips for you:
- Make sure the PR title start w/ a JIRA ticket, e.g. '[CELEBORN-XXXX] Your PR title ...'.
- Be sure to keep the PR description updated to reflect all changes.
- Please write your PR title to summarize what this PR proposes.
- If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
Respect client/server threads num when create transport client/server memory allocator.
### Why are the changes needed?
Follow up, as title.
Respeact client/server threads num to avoid competitiveness
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1463][FOLLOWUP] Respeact client/server threads num to avoid competitiveness [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #2571:
URL: https://github.com/apache/celeborn/pull/2571#discussion_r1642618935
##########
common/src/main/java/org/apache/celeborn/common/network/util/NettyUtils.java:
##########
@@ -143,15 +143,25 @@ public static synchronized PooledByteBufAllocator getSharedPooledByteBufAllocato
public static PooledByteBufAllocator getPooledByteBufAllocator(
TransportConf conf, AbstractSource source, boolean allowCache) {
+ return getPooledByteBufAllocator(conf, source, allowCache, 0);
+ }
+
+ public static PooledByteBufAllocator getPooledByteBufAllocator(
+ TransportConf conf, AbstractSource source, boolean allowCache, int coreNum) {
if (conf.getCelebornConf().networkShareMemoryAllocator()) {
return getSharedPooledByteBufAllocator(
conf.getCelebornConf(),
source,
allowCache && conf.getCelebornConf().networkMemoryAllocatorAllowCache());
}
+ int arenas;
+ if (coreNum != 0) {
+ arenas = coreNum;
+ } else {
+ arenas = conf.getCelebornConf().networkAllocatorArenas();
Review Comment:
Could this limit the arena with netty max thread num to avoid the config value is too large?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1463][FOLLOWUP] Respeact client/server threads num to avoid competitiveness [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #2571:
URL: https://github.com/apache/celeborn/pull/2571#discussion_r1642618935
##########
common/src/main/java/org/apache/celeborn/common/network/util/NettyUtils.java:
##########
@@ -143,15 +143,25 @@ public static synchronized PooledByteBufAllocator getSharedPooledByteBufAllocato
public static PooledByteBufAllocator getPooledByteBufAllocator(
TransportConf conf, AbstractSource source, boolean allowCache) {
+ return getPooledByteBufAllocator(conf, source, allowCache, 0);
+ }
+
+ public static PooledByteBufAllocator getPooledByteBufAllocator(
+ TransportConf conf, AbstractSource source, boolean allowCache, int coreNum) {
if (conf.getCelebornConf().networkShareMemoryAllocator()) {
return getSharedPooledByteBufAllocator(
conf.getCelebornConf(),
source,
allowCache && conf.getCelebornConf().networkMemoryAllocatorAllowCache());
}
+ int arenas;
+ if (coreNum != 0) {
+ arenas = coreNum;
+ } else {
+ arenas = conf.getCelebornConf().networkAllocatorArenas();
Review Comment:
Could this limit the arena with netty max thread num?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1463][FOLLOWUP] Respeact client/server threads num to avoid competitiveness [celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on code in PR #2571:
URL: https://github.com/apache/celeborn/pull/2571#discussion_r1643636485
##########
common/src/main/java/org/apache/celeborn/common/network/util/NettyUtils.java:
##########
@@ -143,15 +143,25 @@ public static synchronized PooledByteBufAllocator getSharedPooledByteBufAllocato
public static PooledByteBufAllocator getPooledByteBufAllocator(
TransportConf conf, AbstractSource source, boolean allowCache) {
+ return getPooledByteBufAllocator(conf, source, allowCache, 0);
+ }
+
+ public static PooledByteBufAllocator getPooledByteBufAllocator(
+ TransportConf conf, AbstractSource source, boolean allowCache, int coreNum) {
if (conf.getCelebornConf().networkShareMemoryAllocator()) {
return getSharedPooledByteBufAllocator(
conf.getCelebornConf(),
source,
allowCache && conf.getCelebornConf().networkMemoryAllocatorAllowCache());
}
+ int arenas;
+ if (coreNum != 0) {
+ arenas = coreNum;
+ } else {
+ arenas = conf.getCelebornConf().networkAllocatorArenas();
Review Comment:
User environments are various, there won't be a proper limit for all environments. I think this should have no limit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1463][FOLLOWUP] Respeact client/server threads num to avoid competitiveness [celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX closed pull request #2571: [CELEBORN-1463][FOLLOWUP] Respeact client/server threads num to avoid competitiveness
URL: https://github.com/apache/celeborn/pull/2571
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1463][FOLLOWUP] Respeact client/server threads num to avoid competitiveness [celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on PR #2571:
URL: https://github.com/apache/celeborn/pull/2571#issuecomment-2174855286
Merged into main(v0.6.0), branch-0.5(v0.5.0), branch-0.4(v0.4.2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1463][FOLLOWUP] Respeact client/server threads num to avoid competitiveness [celeborn]
Posted by "zwangsheng (via GitHub)" <gi...@apache.org>.
zwangsheng commented on code in PR #2571:
URL: https://github.com/apache/celeborn/pull/2571#discussion_r1642885004
##########
common/src/main/java/org/apache/celeborn/common/network/util/NettyUtils.java:
##########
@@ -143,15 +143,25 @@ public static synchronized PooledByteBufAllocator getSharedPooledByteBufAllocato
public static PooledByteBufAllocator getPooledByteBufAllocator(
TransportConf conf, AbstractSource source, boolean allowCache) {
+ return getPooledByteBufAllocator(conf, source, allowCache, 0);
+ }
+
+ public static PooledByteBufAllocator getPooledByteBufAllocator(
+ TransportConf conf, AbstractSource source, boolean allowCache, int coreNum) {
if (conf.getCelebornConf().networkShareMemoryAllocator()) {
return getSharedPooledByteBufAllocator(
conf.getCelebornConf(),
source,
allowCache && conf.getCelebornConf().networkMemoryAllocatorAllowCache());
}
+ int arenas;
+ if (coreNum != 0) {
+ arenas = coreNum;
+ } else {
+ arenas = conf.getCelebornConf().networkAllocatorArenas();
Review Comment:
Should we limit this max value in `conf.getCelebornConf().networkAllocatorArenas()` or just in this un-shared pool memory allocator case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org