You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "leixm (via GitHub)" <gi...@apache.org> on 2024/07/09 11:28:54 UTC
[PR] [CELEBORN-1487] CongestionController support control traffic by user/worker traffic speed [celeborn]
leixm opened a new pull request, #2608:
URL: https://github.com/apache/celeborn/pull/2608
### What changes were proposed in this pull request?
Introduce support control traffic by user/worker traffic speed.
### Why are the changes needed?
Currently, Celeborn only supports quota management based on disk file bytes/count, and this quota management cannot cope with sudden increases in traffic, which will cause corrupt to the cluster.
### Does this PR introduce _any_ user-facing change?
Yes.
### How was this patch tested?
UTs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1487] CongestionController support control traffic by user/worker traffic speed [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #2608:
URL: https://github.com/apache/celeborn/pull/2608#issuecomment-2293078373
This issue was closed because it has been staled for 10 days with no activity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1487] CongestionController support control traffic by user/worker traffic speed [celeborn]
Posted by "mridulm (via GitHub)" <gi...@apache.org>.
mridulm commented on PR #2608:
URL: https://github.com/apache/celeborn/pull/2608#issuecomment-2225998932
Can you open up access to the doc @leixm ? Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1487] CongestionController support control traffic by user/worker traffic speed [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #2608: [CELEBORN-1487] CongestionController support control traffic by user/worker traffic speed
URL: https://github.com/apache/celeborn/pull/2608
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1487] CongestionController support control traffic by user/worker traffic speed [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #2608:
URL: https://github.com/apache/celeborn/pull/2608#issuecomment-2268484244
This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1487] CongestionController support control traffic by user/worker traffic speed [celeborn]
Posted by "leixm (via GitHub)" <gi...@apache.org>.
leixm commented on PR #2608:
URL: https://github.com/apache/celeborn/pull/2608#issuecomment-2227423269
> Can you open up access to the doc @leixm ? Thanks
Sorry, I have updated the doc access, can you access now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1487] CongestionController support control traffic by user/worker traffic speed [celeborn]
Posted by "AngersZhuuuu (via GitHub)" <gi...@apache.org>.
AngersZhuuuu commented on code in PR #2608:
URL: https://github.com/apache/celeborn/pull/2608#discussion_r1672038378
##########
worker/src/main/java/org/apache/celeborn/service/deploy/worker/congestcontrol/CongestionController.java:
##########
@@ -217,15 +268,22 @@ private void removeInactiveUsers() {
protected void checkCongestion() {
try {
long pendingConsume = getTotalPendingBytes();
- if (pendingConsume < lowWatermark) {
+ long workerProduceSpeed = consumedBufferStatusHub.avgBytesPerSec();
Review Comment:
Maybe we should use produce bytes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1487] CongestionController support control traffic by user/worker traffic speed [celeborn]
Posted by "leixm (via GitHub)" <gi...@apache.org>.
leixm commented on PR #2608:
URL: https://github.com/apache/celeborn/pull/2608#issuecomment-2225293268
https://docs.google.com/document/d/15ALP-Wy0CktO7PX39oXpZjIi0llFxos0HQ53KnYFq5E/edit?usp=sharing cc @RexXiong @AngersZhuuuu Can you help review design doc?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1487] CongestionController support control traffic by user/worker traffic speed [celeborn]
Posted by "leixm (via GitHub)" <gi...@apache.org>.
leixm commented on code in PR #2608:
URL: https://github.com/apache/celeborn/pull/2608#discussion_r1678728486
##########
worker/src/main/java/org/apache/celeborn/service/deploy/worker/congestcontrol/CongestionController.java:
##########
@@ -217,15 +268,22 @@ private void removeInactiveUsers() {
protected void checkCongestion() {
try {
long pendingConsume = getTotalPendingBytes();
- if (pendingConsume < lowWatermark) {
+ long workerProduceSpeed = consumedBufferStatusHub.avgBytesPerSec();
Review Comment:
Sure, I have updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1487] CongestionController support control traffic by user/worker traffic speed [celeborn]
Posted by "leixm (via GitHub)" <gi...@apache.org>.
leixm commented on PR #2608:
URL: https://github.com/apache/celeborn/pull/2608#issuecomment-2229975963
Flaky test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org