You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "attilapiros (via GitHub)" <gi...@apache.org> on 2025/04/15 22:29:51 UTC
[PR] [SPARK-51638][CORE][3.5] Fix fetching the remote disk stored RDD blocks via the external shuffle service [spark]
attilapiros opened a new pull request, #50596:
URL: https://github.com/apache/spark/pull/50596
### What changes were proposed in this pull request?
Fix remote fetching of disk stored RDD blocks via the external shuffle service when `spark.shuffle.service.fetch.rdd.enabled` is set.
### Why are the changes needed?
After https://issues.apache.org/jira/browse/SPARK-43221 remote fetching was handled in `BlockManagerMasterEndpoint#getLocationsAndStatus` at one place where all the location was used along with the `blockManagerInfo` map but this map only includes information about the active executors which are not already killed (after for example downscaling in dynamic allocation or just killed because of a failures).
This PR extend the search to all the remote external shuffle services where the `blockStatusByShuffleService` map is used. That map contains block infos even for the killed executors.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
An existing unit test was extended.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51638][CORE][3.5] Fix fetching the remote disk stored RDD blocks via the external shuffle service [spark]
Posted by "attilapiros (via GitHub)" <gi...@apache.org>.
attilapiros commented on PR #50596:
URL: https://github.com/apache/spark/pull/50596#issuecomment-2807781919
@mridulm this is a cherry-pick to branch-3.5 as a separate PR as it was conflicting. Could you please review this, too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51638][CORE][3.5] Fix fetching the remote disk stored RDD blocks via the external shuffle service [spark]
Posted by "attilapiros (via GitHub)" <gi...@apache.org>.
attilapiros commented on PR #50596:
URL: https://github.com/apache/spark/pull/50596#issuecomment-2808218219
Thanks @mridulm and @yaooqinn. It is merged!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-51638][CORE][3.5] Fix fetching the remote disk stored RDD blocks via the external shuffle service [spark]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn closed pull request #50596: [SPARK-51638][CORE][3.5] Fix fetching the remote disk stored RDD blocks via the external shuffle service
URL: https://github.com/apache/spark/pull/50596
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org