You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "karuppayya (via GitHub)" <gi...@apache.org> on 2025/11/13 00:34:14 UTC
[PR] [SHUFFLE][WIP]Prototype using remote storage for shuffle [spark]
karuppayya opened a new pull request, #53028:
URL: https://github.com/apache/spark/pull/53028
### What changes were proposed in this pull request?
This change([design doc)](https://docs.google.com/document/d/1tuWyXAaIBR0oVD5KZwYvz7JLyn6jB55_35xeslUEu7s/edit?usp=sharing) adds support to use remote storage for shuffle data storage.
The primary goal is to enhance the elasticity and resilience of Spark workloads, leading to substantial cost optimization opportunities.
This is a PoC to elicit feedback from community.
### Why are the changes needed?
This change decouples storage from compute, therein helping to minimize shuffle failure and also in better scaling of the cluster.
### Does this PR introduce any user-facing change?
This change adds 3 SQL configs to enable the feature
Remote storage location for shuffle
`spark.shuffle.remote.storage.path=<remote storage path>`
Config that determines if the feature needs to be used
`spark.sql.shuffle.consolidation.enabled=true|false`
Shuffle plugin to use when the feature is enabled(This needs to be configured currently, but we can switch to this automatically when feature is enabled TBD )
`spark.shuffle.sort.io.plugin.class=org.apache.spark.shuffle.sort.remote.HybridShuffleDataIO` -
### How was this patch tested?
Manual testing. Unit test to be added.
Trying to get feedback from community before writing elaborate tests.
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SHUFFLE][WIP]Prototype using remote storage for shuffle [spark]
Posted by "karuppayya (via GitHub)" <gi...@apache.org>.
karuppayya commented on PR #53028:
URL: https://github.com/apache/spark/pull/53028#issuecomment-3725488965
> Quick question: why are they called SQL configs?
@holdenk I guess your are referring to the "SQL configs" in PR description?
Only one of them is actually SQL which actual enables the feature and uses the other 2 configs which are Spark session configs. I will have them reworded.
Also In case if you are wondering why i havent handled some few of your review comments yet,
I am yet to restart the discussion to resolve the pending concerns on design/SPIP from @mridulm, since had some sick timeoff and was on holiday break.
Will start that soon and address the reviews comments here once the discussion settles.
I would also highly appreciate and value your thoughts on SPIP and design doc, if you havent taken a look already.
Thank you for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SHUFFLE][WIP]Prototype using remote storage for shuffle [spark]
Posted by "holdenk (via GitHub)" <gi...@apache.org>.
holdenk commented on code in PR #53028:
URL: https://github.com/apache/spark/pull/53028#discussion_r2551006131
##########
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##########
@@ -626,6 +626,13 @@ package object config {
.checkValue(_.endsWith(java.io.File.separator), "Path should end with separator.")
.createOptional
+ private[spark] val SHUFFLE_REMOTE_STORAGE_CLEANUP =
+ ConfigBuilder("spark.shuffle.remote.storage.cleanUp")
+ .doc("If true, Spark cleans up its fallback storage data during shutting down.")
+ .version("3.2.0")
Review Comment:
version
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SHUFFLE][WIP]Prototype using remote storage for shuffle [spark]
Posted by "holdenk (via GitHub)" <gi...@apache.org>.
holdenk commented on PR #53028:
URL: https://github.com/apache/spark/pull/53028#issuecomment-3725492435
Cool just wanted to make sure :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SHUFFLE][WIP]Prototype using remote storage for shuffle [spark]
Posted by "holdenk (via GitHub)" <gi...@apache.org>.
holdenk commented on PR #53028:
URL: https://github.com/apache/spark/pull/53028#issuecomment-3725005265
Quick question: why are they called SQL configs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SHUFFLE][WIP]Prototype using remote storage for shuffle [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #53028:
URL: https://github.com/apache/spark/pull/53028#issuecomment-4636714163
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SHUFFLE][WIP]Prototype using remote storage for shuffle [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #53028: [SHUFFLE][WIP]Prototype using remote storage for shuffle
URL: https://github.com/apache/spark/pull/53028
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org