You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dale Richardson (Jira)" <ji...@apache.org> on 2020/04/21 11:06:00 UTC
[jira] [Commented] (SPARK-31329) Modify executor monitor to allow
for moving shuffle blocks
[ https://issues.apache.org/jira/browse/SPARK-31329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17088569#comment-17088569 ]
Dale Richardson commented on SPARK-31329:
-----------------------------------------
Hi Holden,
I've been thinking about this issue as well.
Blocks can't move, but they can be replicated. Any reason we can't just replicate the blocks out, allow the existing code paths to update the block locations with the block manager master, then unregister the current blocks?
> Modify executor monitor to allow for moving shuffle blocks
> ----------------------------------------------------------
>
> Key: SPARK-31329
> URL: https://issues.apache.org/jira/browse/SPARK-31329
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes, Spark Core
> Affects Versions: 3.1.0
> Reporter: Holden Karau
> Assignee: Holden Karau
> Priority: Minor
>
> To enable Spark-20629 we need to revisit code that assumes shuffle blocks don't move. Currently, the executor monitor assumes that shuffle blocks are immovable. We should modify this code.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org