You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "r7raul1984 (via GitHub)" <gi...@apache.org> on 2025/10/23 03:03:36 UTC
[PR] [CELEBORN-2183]Client always tries to start a connection with exclude… [celeborn]
r7raul1984 opened a new pull request, #3517:
URL: https://github.com/apache/celeborn/pull/3517
### What changes were proposed in this pull request?
Do not attempt to fetch from any worker that has already been added to fetchExcludedWorkers. Instead, try fetching from a peer worker within the same PartitionLocation.
### Why are the changes needed?
With the configuration celeborn.client.fetch.excludeWorkerOnFailure.enabled=true and celeborn.client.push.replicate.enabled=true,
when a fetch operation fails for a specific worker, that worker is added to fetchExcludedWorkers. However, subsequent attempts still try to fetch from this worker, causing repeated errors and slowing down shuffle fetch performance.
The logic should be modified so that any worker already in fetchExcludedWorkers is completely excluded from further fetch operations.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manual test by debug
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2183] Fix client tries to start a connection with excluded workers [celeborn]
Posted by "r7raul1984 (via GitHub)" <gi...@apache.org>.
r7raul1984 commented on PR #3517:
URL: https://github.com/apache/celeborn/pull/3517#issuecomment-3544793304
> @r7raul1984, the CI run failed. PTAL.
how to rerun these checks?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2183] Fix client tries to start a connection with excluded workers [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on PR #3517:
URL: https://github.com/apache/celeborn/pull/3517#issuecomment-3541591070
@r7raul1984, the CI run failed. PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2183] Fix client tries to start a connection with excluded workers [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #3517: [CELEBORN-2183] Fix client tries to start a connection with excluded workers
URL: https://github.com/apache/celeborn/pull/3517
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2183] Fix client tries to start a connection with excluded workers [celeborn]
Posted by "r7raul1984 (via GitHub)" <gi...@apache.org>.
r7raul1984 commented on code in PR #3517:
URL: https://github.com/apache/celeborn/pull/3517#discussion_r2501509939
##########
client/src/main/java/org/apache/celeborn/client/read/CelebornInputStream.java:
##########
@@ -467,7 +467,9 @@ private PartitionReader createReaderWithRetry(
lastException = e;
shuffleClient.excludeFailedFetchLocation(location.hostAndFetchPort(), e);
fetchChunkRetryCnt++;
- if (location.hasPeer() && !readSkewPartitionWithoutMapRange) {
+ if (location.hasPeer()
+ && !isExcluded(location.getPeer())
Review Comment:
My idea is to add this here to prevent throwing a new CelebornIOException at line 456.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2183] Fix client tries to start a connection with excluded workers [celeborn]
Posted by "r7raul1984 (via GitHub)" <gi...@apache.org>.
r7raul1984 commented on PR #3517:
URL: https://github.com/apache/celeborn/pull/3517#issuecomment-3551084619
> @r7raul1984, you'd better to close this pull request and reopen.
It’s very strange — there are three checks that always fail randomly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2183] Fix client tries to start a connection with excluded workers [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3517:
URL: https://github.com/apache/celeborn/pull/3517#issuecomment-3713715655
This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2183] Fix client tries to start a connection with excluded workers [celeborn]
Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong commented on code in PR #3517:
URL: https://github.com/apache/celeborn/pull/3517#discussion_r2454488479
##########
client/src/main/java/org/apache/celeborn/client/read/CelebornInputStream.java:
##########
@@ -479,6 +481,7 @@ private PartitionReader createReaderWithRetry(
fetchChunkMaxRetry,
location,
e);
+ location = location.getPeer();
Review Comment:
Why change location to peer at this position?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2183] Fix client tries to start a connection with excluded workers [celeborn]
Posted by "r7raul1984 (via GitHub)" <gi...@apache.org>.
r7raul1984 commented on code in PR #3517:
URL: https://github.com/apache/celeborn/pull/3517#discussion_r2454573234
##########
client/src/main/java/org/apache/celeborn/client/read/CelebornInputStream.java:
##########
@@ -479,6 +481,7 @@ private PartitionReader createReaderWithRetry(
fetchChunkMaxRetry,
location,
e);
+ location = location.getPeer();
Review Comment:
Since the previous location has already been excluded, I think we need to try fetching from peers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2183] Fix client tries to start a connection with excluded workers [celeborn]
Posted by "r7raul1984 (via GitHub)" <gi...@apache.org>.
r7raul1984 commented on code in PR #3517:
URL: https://github.com/apache/celeborn/pull/3517#discussion_r2485274752
##########
client/src/main/java/org/apache/celeborn/client/read/CelebornInputStream.java:
##########
@@ -479,6 +481,7 @@ private PartitionReader createReaderWithRetry(
fetchChunkMaxRetry,
location,
e);
+ location = location.getPeer();
Review Comment:
> > Since the previous location has already been excluded, I think we need to try fetching from peers.
>
> Should send bufferStreamEnd first before change location to peer
yes, you are right
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2183] Fix client tries to start a connection with excluded workers [celeborn]
Posted by "r7raul1984 (via GitHub)" <gi...@apache.org>.
r7raul1984 closed pull request #3517: [CELEBORN-2183] Fix client tries to start a connection with excluded workers
URL: https://github.com/apache/celeborn/pull/3517
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2183] Fix client tries to start a connection with excluded workers [celeborn]
Posted by "r7raul1984 (via GitHub)" <gi...@apache.org>.
r7raul1984 commented on PR #3517:
URL: https://github.com/apache/celeborn/pull/3517#issuecomment-3541615614
> PTAL
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2183] Fix client tries to start a connection with excluded workers [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on PR #3517:
URL: https://github.com/apache/celeborn/pull/3517#issuecomment-3479044998
@r7raul1984, no any changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2183] Client always tries to start a connection with exclude wrokers [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on PR #3517:
URL: https://github.com/apache/celeborn/pull/3517#issuecomment-3434924235
@r7raul1984, thanks for contribution. Please run `dev/reformat` to format.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2183] Fix client tries to start a connection with excluded workers [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3517:
URL: https://github.com/apache/celeborn/pull/3517#issuecomment-3631033008
This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2183] Fix client tries to start a connection with excluded workers [celeborn]
Posted by "r7raul1984 (via GitHub)" <gi...@apache.org>.
r7raul1984 commented on PR #3517:
URL: https://github.com/apache/celeborn/pull/3517#issuecomment-3500295695
> @r7raul1984, any update?
No more update
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2183] Fix client tries to start a connection with excluded workers [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #3517:
URL: https://github.com/apache/celeborn/pull/3517#discussion_r2471858948
##########
client/src/main/java/org/apache/celeborn/client/read/CelebornInputStream.java:
##########
@@ -479,6 +481,7 @@ private PartitionReader createReaderWithRetry(
fetchChunkMaxRetry,
location,
e);
+ location = location.getPeer();
Review Comment:
@r7raul1984, does the changed location have the corresponding shuffle data to read?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2183] Client always tries to start a connection with exclude wrokers [celeborn]
Posted by "r7raul1984 (via GitHub)" <gi...@apache.org>.
r7raul1984 commented on PR #3517:
URL: https://github.com/apache/celeborn/pull/3517#issuecomment-3434969009
> @r7raul1984, thanks for contribution. Please run `dev/reformat` to format.
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2183] Fix client tries to start a connection with excluded workers [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on PR #3517:
URL: https://github.com/apache/celeborn/pull/3517#issuecomment-3500155454
@r7raul1984, any update?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2183] Fix client tries to start a connection with excluded workers [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on PR #3517:
URL: https://github.com/apache/celeborn/pull/3517#issuecomment-3547837045
@r7raul1984, you'd better to close this pull request and reopen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2183] Fix client tries to start a connection with excluded workers [celeborn]
Posted by "r7raul1984 (via GitHub)" <gi...@apache.org>.
r7raul1984 commented on code in PR #3517:
URL: https://github.com/apache/celeborn/pull/3517#discussion_r2476483235
##########
client/src/main/java/org/apache/celeborn/client/read/CelebornInputStream.java:
##########
@@ -479,6 +481,7 @@ private PartitionReader createReaderWithRetry(
fetchChunkMaxRetry,
location,
e);
+ location = location.getPeer();
Review Comment:
After reviewing the code more carefully, I think the issue might not be in those areas. Maybe the problem is as follows:
In CelebornShuffleReader, when building makeOpenStreamList, suppose:
workerRequestMap uses hostAndFetchPort as the key.
There are two workers(hostAndFetchPort): h1:p1 and h1:p2.
Before down scale by kubectl scale:
shuffleClient.getDataClientFactory().createClient(h1, p1) succeeds.
So in CelebornShuffleReader A, workerRequestMap contains an entry for h1:p1.
After down scale by kubectl scale:
The worker h1:p1 is removed.
In another CelebornShuffleReader B, trying to create a client for h1:p1 fails.
As a result, h1:p1 is added to the excluded worker list.
However, in CelebornShuffleReader A, workerRequestMap still contains h1:p1, so it keeps retrying the removed worker, causing unnecessary retries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2183] Fix client tries to start a connection with excluded workers [celeborn]
Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong commented on code in PR #3517:
URL: https://github.com/apache/celeborn/pull/3517#discussion_r2489029546
##########
client/src/main/java/org/apache/celeborn/client/read/CelebornInputStream.java:
##########
@@ -467,7 +467,9 @@ private PartitionReader createReaderWithRetry(
lastException = e;
shuffleClient.excludeFailedFetchLocation(location.hostAndFetchPort(), e);
fetchChunkRetryCnt++;
- if (location.hasPeer() && !readSkewPartitionWithoutMapRange) {
+ if (location.hasPeer()
+ && !isExcluded(location.getPeer())
Review Comment:
It appears that `isExcluded(location.getPeer())` now serves the same purpose as `isExcluded(location)` at line 455.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2183] Fix client tries to start a connection with excluded workers [celeborn]
Posted by "r7raul1984 (via GitHub)" <gi...@apache.org>.
r7raul1984 closed pull request #3517: [CELEBORN-2183] Fix client tries to start a connection with excluded workers
URL: https://github.com/apache/celeborn/pull/3517
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2183] Fix client tries to start a connection with excluded workers [celeborn]
Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong commented on code in PR #3517:
URL: https://github.com/apache/celeborn/pull/3517#discussion_r2485200923
##########
client/src/main/java/org/apache/celeborn/client/read/CelebornInputStream.java:
##########
@@ -479,6 +481,7 @@ private PartitionReader createReaderWithRetry(
fetchChunkMaxRetry,
location,
e);
+ location = location.getPeer();
Review Comment:
> Since the previous location has already been excluded, I think we need to try fetching from peers.
Should send bufferStreamEnd first before change location to peer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2183] Fix client tries to start a connection with excluded workers [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3517:
URL: https://github.com/apache/celeborn/pull/3517#issuecomment-3762908524
This issue was closed because it has been staled for 10 days with no activity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org