You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "waitinfuture (via GitHub)" <gi...@apache.org> on 2023/10/10 14:19:19 UTC
[PR] [CELEBORN-1034] Offer slots uses random range of available workers instead of shuffling [incubator-celeborn]
waitinfuture opened a new pull request, #1975:
URL: https://github.com/apache/incubator-celeborn/pull/1975
<!--
Thanks for sending a pull request! Here are some tips for you:
- Make sure the PR title start w/ a JIRA ticket, e.g. '[CELEBORN-XXXX] Your PR title ...'.
- Be sure to keep the PR description updated to reflect all changes.
- Please write your PR title to summarize what this PR proposes.
- If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
In original design, (primary worker, replica worker) pairs tends to stay stable, for example,
for primary PartitionLocations on Worker A, their replica PartitionLocations will all be on
Worker B in general scenarios, i.e. all workers are healthy and works well. This way, one Worker
will have only one (or very few) connections to other workers' replicate netty server.
However, https://github.com/apache/incubator-celeborn/pull/1790 calls `Collections.shuffle(availableWorkers)`,
causing the number of replica connections increases dramatically:
### Why are the changes needed?
This PR refine the logic of selecting limited number of workers, instead of shuffling,
Master just randomly picks a range of available workers.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manual test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1034] Offer slots uses random range of available workers instead of shuffling [incubator-celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on PR #1975:
URL: https://github.com/apache/incubator-celeborn/pull/1975#issuecomment-1767996120
I tested this patch on a four worker cluster and ran TPC-DS and terasort.
Before this patch,
<img width="1366" alt="截屏2023-10-18 14 34 26" src="https://github.com/apache/incubator-celeborn/assets/4150993/55c03045-eee4-4474-89ad-9be72f6ff3f9">
After this patch,
<img width="1285" alt="截屏2023-10-18 16 55 06" src="https://github.com/apache/incubator-celeborn/assets/4150993/dd68a043-7149-402c-b97d-015a36b2fc46">
Ignore the worker core-1-2, it's working but http server crashed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1034] Offer slots uses random range of available workers instead of shuffling [incubator-celeborn]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #1975:
URL: https://github.com/apache/incubator-celeborn/pull/1975#issuecomment-1757663420
## [Codecov](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1975?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#1975](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1975?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (d3c0503) into [main](https://app.codecov.io/gh/apache/incubator-celeborn/commit/bcf89da7ddb38756fe8a988ffb557cf65f1c7018?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (bcf89da) will **decrease** coverage by `0.00%`.
> Report is 4 commits behind head on main.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## main #1975 +/- ##
==========================================
- Coverage 47.02% 47.01% -0.00%
==========================================
Files 164 164
Lines 10368 10369 +1
Branches 957 957
==========================================
Hits 4874 4874
Misses 5178 5178
- Partials 316 317 +1
```
[see 1 file with indirect coverage changes](https://app.codecov.io/gh/apache/incubator-celeborn/pull/1975/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1034] Offer slots uses random range of available workers instead of shuffling [incubator-celeborn]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on code in PR #1975:
URL: https://github.com/apache/incubator-celeborn/pull/1975#discussion_r1354917276
##########
master/src/main/scala/org/apache/celeborn/service/deploy/master/Master.scala:
##########
@@ -616,21 +616,30 @@ private[celeborn] class Master(
val shuffleKey = Utils.makeShuffleKey(requestSlots.applicationId, requestSlots.shuffleId)
var availableWorkers = workersAvailable()
- Collections.shuffle(availableWorkers)
+ val numAvailableWorkers = availableWorkers.size()
val numWorkers = Math.min(
Math.max(
if (requestSlots.shouldReplicate) 2 else 1,
if (requestSlots.maxWorkers <= 0) slotsAssignMaxWorkers
else Math.min(slotsAssignMaxWorkers, requestSlots.maxWorkers)),
- availableWorkers.size())
- availableWorkers = availableWorkers.subList(0, numWorkers)
+ numAvailableWorkers)
+ val startIndex = Random.nextInt(availableWorkers.size())
Review Comment:
```suggestion
val startIndex = Random.nextInt(numAvailableWorkers)
```
##########
master/src/main/scala/org/apache/celeborn/service/deploy/master/Master.scala:
##########
@@ -616,21 +616,30 @@ private[celeborn] class Master(
val shuffleKey = Utils.makeShuffleKey(requestSlots.applicationId, requestSlots.shuffleId)
var availableWorkers = workersAvailable()
Review Comment:
```suggestion
val availableWorkers = workersAvailable()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1034] Offer slots uses random range of available workers instead of shuffling [incubator-celeborn]
Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture commented on PR #1975:
URL: https://github.com/apache/incubator-celeborn/pull/1975#issuecomment-1755533074
cc @liying919
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1034] Offer slots uses random range of available workers instead of shuffling [incubator-celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX closed pull request #1975: [CELEBORN-1034] Offer slots uses random range of available workers instead of shuffling
URL: https://github.com/apache/incubator-celeborn/pull/1975
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org