You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "Z1Wu (via GitHub)" <gi...@apache.org> on 2025/02/28 14:48:21 UTC
[PR] [CELEBORN-1886]Enable read from replicated partitionLocation in sort-free skew optimization when replicated partitionLocation and primaryLocation have same chunk range [celeborn]
Z1Wu opened a new pull request, #3125:
URL: https://github.com/apache/celeborn/pull/3125
…ation and primaryLocation share same chunk range.
<!--
Thanks for sending a pull request! Here are some tips for you:
- Make sure the PR title start w/ a JIRA ticket, e.g. '[CELEBORN-XXXX] Your PR title ...'.
- Be sure to keep the PR description updated to reflect all changes.
- Please write your PR title to summarize what this PR proposes.
- If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
In current implementation, we disable reading from replicated partitionLocation when we enable sort-free skew handling and read from a skew partition (i.e. `readSkewPartitionWithoutMapRange`).
However, when replicated partitionLocation and primary partitionLocation have the same chunks from StartChunkIndex to EndChunkIndex, it is safe to read from replicated partitionLocation, because PushDataHandler ensures that there is exactly the same data in replicated partitionLocation and primary partitionLocation in all scenarios and the same chunk offsets in most of scenarios. ( Please correct me if I have any misunderstanding. )
### Why are the changes needed?
enable read from replicated partitionLocation when primary partitionLocation lost in sort-free skew handling
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
Local cluster test and simulate partitionLocation lost by injecting exception in `org.apache.celeborn.client.read.CelebornInputStream.CelebornInputStreamImpl#createReaderWithRetry` manually
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1886]Enable read from replicated partitionLocation in sort-free skew optimization when replicated partitionLocation and primary partitionLocation have same chunk range [celeborn]
Posted by "Z1Wu (via GitHub)" <gi...@apache.org>.
Z1Wu closed pull request #3125: [CELEBORN-1886]Enable read from replicated partitionLocation in sort-free skew optimization when replicated partitionLocation and primary partitionLocation have same chunk range
URL: https://github.com/apache/celeborn/pull/3125
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1886]Enable read from replicated partitionLocation in sort-free skew optimization when replicated partitionLocation and primary partitionLocation have same chunk range [celeborn]
Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong commented on PR #3125:
URL: https://github.com/apache/celeborn/pull/3125#issuecomment-2693125809
Thanks @Z1Wu, Currently, both Push and replicate data are unordered, so there is no way to guarantee that the order of the replicated data is exactly the same as the order of the pushed data. When reading by chunk in a skewed partition, even if the chunk offsets are completely identical, it cannot be ensured that the data in the two replicas is exactly the same unless we can verify the checksums of both replicas. However, my understanding is that in most cases, they should be different.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1886]Enable read from replicated partitionLocation in sort-free skew optimization when replicated partitionLocation and primary partitionLocation have same chunk range [celeborn]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on PR #3125:
URL: https://github.com/apache/celeborn/pull/3125#issuecomment-2693287577
cc @RexXiong In sort-free skew optimization case, If we enable shuffle replicate, and then during the process of reading primary shuffle data, the node where the primary shuffle data is located goes offline, causing shuffle data lost, in this scenario, will the spark app fallback to reading replicate shuffle data, or will it cause stage failure/rerun?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1886]Enable read from replicated partitionLocation in sort-free skew optimization when replicated partitionLocation and primary partitionLocation have same chunk range [celeborn]
Posted by "Z1Wu (via GitHub)" <gi...@apache.org>.
Z1Wu commented on PR #3125:
URL: https://github.com/apache/celeborn/pull/3125#issuecomment-2690833598
cc @wangshengjie123 @RexXiong
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1886]Enable read from replicated partitionLocation in sort-free skew optimization when replicated partitionLocation and primary partitionLocation have same chunk range [celeborn]
Posted by "Z1Wu (via GitHub)" <gi...@apache.org>.
Z1Wu commented on PR #3125:
URL: https://github.com/apache/celeborn/pull/3125#issuecomment-2693264836
> Push and replicate data are unordered
Thanks for your reply. You are correct, I overlooked that the pushData may arrive at the Primary Worker and the Replicated Worker in different order when there are multiple PushDataRequests being handled simultaneously.
> However, my understanding is that in most cases, they should be different.
I agree with this too, there are lots of simultaneous PushDataRequests handling in real workload. And the pushData order in Primary PartitionLocation and Replicated PartitionLocation is very likely to be different.
Thank you for pointing out this problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1886]Enable read from replicated partitionLocation in sort-free skew optimization when replicated partitionLocation and primary partitionLocation have same chunk range [celeborn]
Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong commented on PR #3125:
URL: https://github.com/apache/celeborn/pull/3125#issuecomment-2693369740
> will the spark app fallback to reading replicate shuffle data
We cannot fallback to the replica because some sub reducer tasks may have already successfully read data from the primary copy. If a task that encounters an error fallback to the replica, it may read duplicate data, which is caused by the different order of data between the primary and replica. In this scenario, trigger stage rerun would be better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org