You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "littlexyw (via GitHub)" <gi...@apache.org> on 2025/07/11 10:37:48 UTC
[PR] [CELEBORN-2064] Fix the issue where reading replica partition that returns zero chunk causes tasks to hang [celeborn]
littlexyw opened a new pull request, #3364:
URL: https://github.com/apache/celeborn/pull/3364
<!--
Thanks for sending a pull request! Here are some tips for you:
- Make sure the PR title start w/ a JIRA ticket, e.g. '[CELEBORN-XXXX] Your PR title ...'.
- Be sure to keep the PR description updated to reflect all changes.
- Please write your PR title to summarize what this PR proposes.
- If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
Re-validate hasNextChunk within getNextChunk.
### Why are the changes needed?
Fix the issue where reading replica partition that returns zero chunk causes tasks to hang
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
manual test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2064] Fix the issue where reading replica partition that returns zero chunk causes tasks to hang [celeborn]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei closed pull request #3364: [CELEBORN-2064] Fix the issue where reading replica partition that returns zero chunk causes tasks to hang
URL: https://github.com/apache/celeborn/pull/3364
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2064] Fix the issue where reading replica partition that returns zero chunk causes tasks to hang [celeborn]
Posted by "littlexyw (via GitHub)" <gi...@apache.org>.
littlexyw commented on PR #3364:
URL: https://github.com/apache/celeborn/pull/3364#issuecomment-3105545905
> Under which case might a replica return an empty chunk? As far as I know, only files with length > 0 would return.
It occurs when partition data is skewed, and a write operation from the primary to the replica fails. In this case, replica file is not empty but may not contain chunks within specific map range.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2064] Fix the issue where reading replica partition that returns zero chunk causes tasks to hang [celeborn]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on PR #3364:
URL: https://github.com/apache/celeborn/pull/3364#issuecomment-3145730536
thanks, merging to main(0.7.0) and branch-0.6(0.6.1)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2064] Fix the issue where reading replica partition that returns zero chunk causes tasks to hang [celeborn]
Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong commented on PR #3364:
URL: https://github.com/apache/celeborn/pull/3364#issuecomment-3120749256
> > Under which case might a replica return an empty chunk? As far as I know, only files with length > 0 would return.
>
> It occurs when partition data is skewed, and a write operation from the primary to the replica fails. In this case, replica file is not empty but may not contain chunks within specific map range.
Thank you for the clarification， I see.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2064] Fix the issue where reading replica partition that returns zero chunk causes tasks to hang [celeborn]
Posted by "littlexyw (via GitHub)" <gi...@apache.org>.
littlexyw commented on code in PR #3364:
URL: https://github.com/apache/celeborn/pull/3364#discussion_r2224262416
##########
client/src/main/java/org/apache/celeborn/client/read/CelebornInputStream.java:
##########
@@ -526,6 +523,9 @@ private ByteBuf getNextChunk() throws IOException {
throw new CelebornIOException(
"Fetch data from excluded worker! " + currentReader.getLocation());
}
+ if (!currentReader.hasNext()) {
Review Comment:
> I think it is not a good idea to just do this fix. Which version are you using? If you are using 0.5, this situation may be caused by inconsistent data between the primary and replica partition. It is likely that hard splits has occurred when pushing data. The fix submitted here can really solve the hang task problem, but it may cause missing data, which is more dangerous. You can try to turn off either the push replicate or `celeborn.client.shuffle.batchHandleCommitPartition.enabled` first, which may solve your problem. Further solutions may require code modifications. Internally, we are trying to do some optimizations such as checking commit result and so on.
Under which case will it cause missing data?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2064] Fix the issue where reading replica partition that returns zero chunk causes tasks to hang [celeborn]
Posted by "onebox-li (via GitHub)" <gi...@apache.org>.
onebox-li commented on PR #3364:
URL: https://github.com/apache/celeborn/pull/3364#issuecomment-3143662387
> LGTM, ping @onebox-li Could you please explain again why data might be lost in this scenario?
Sorry for the late reply and thanks for @RexXiong offline explanation. The data lost case I thought before should not exist. LGTM! Thanks for @littlexyw the fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2064] Fix the issue where reading replica partition that returns zero chunk causes tasks to hang [celeborn]
Posted by "onebox-li (via GitHub)" <gi...@apache.org>.
onebox-li commented on code in PR #3364:
URL: https://github.com/apache/celeborn/pull/3364#discussion_r2212925067
##########
client/src/main/java/org/apache/celeborn/client/read/CelebornInputStream.java:
##########
@@ -526,6 +523,9 @@ private ByteBuf getNextChunk() throws IOException {
throw new CelebornIOException(
"Fetch data from excluded worker! " + currentReader.getLocation());
}
+ if (!currentReader.hasNext()) {
Review Comment:
I think it is not a good idea to just do this fix.
Which version are you using? If you are using 0.5, this situation may be caused by inconsistent data between the primary and replica partition. It is likely that hard splits has occurred when pushing data.
The fix submitted here can really solve the hang task problem, but it may cause missing data, which is more dangerous.
You can try to turn off either the push replicate or `celeborn.client.shuffle.batchHandleCommitPartition.enabled` first, which may solve your problem.
Further solutions may require code modifications. Internally, we are trying to do some optimizations such as checking commit result and so on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2064] Fix the issue where reading replica partition that returns zero chunk causes tasks to hang [celeborn]
Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong commented on PR #3364:
URL: https://github.com/apache/celeborn/pull/3364#issuecomment-3101916428
Under which case might a replica return an empty chunk? As far as I know, only files with length > 0 would return.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org