You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2019/05/30 17:01:27 UTC
[GitHub] [spark] squito commented on issue #24740: [SPARK-27876][CORE] Split
large shuffle partition to multi-segments to enable transfer oversize
shuffle partition block
squito commented on issue #24740: [SPARK-27876][CORE] Split large shuffle partition to multi-segments to enable transfer oversize shuffle partition block
URL: https://github.com/apache/spark/pull/24740#issuecomment-497403804
You shouldn't have this limit anymore (from spark 2.4 onwards) as long as you're also running a recent shuffle service. this uses fetching shuffle blocks to disk, instead of memory, which should be enabled by default for large blocks (https://issues.apache.org/jira/browse/SPARK-24297).
But if you're seeing a failure with that, can you share some more details?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org