You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Holden Karau (Jira)" <ji...@apache.org> on 2020/05/18 17:55:00 UTC
[jira] [Updated] (SPARK-31555) Improve cache block migration
[ https://issues.apache.org/jira/browse/SPARK-31555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Holden Karau updated SPARK-31555:
---------------------------------
Description:
We should explore the following improvements to cache block migration:
1) Peer selection (right now may overbalance on certain peers)
2) Do we need to configure the number of blocks to be migrated at the same time
3) Are there any blocks we don't need to replicate (e.g. they are already stored on the desired number of executors even once we remove the executors slated for decommissioning).
4) Do we want to prioritize migrating blocks with no replicas
was:
We should explore the following improvements to cache block migration:
1) Peer selection (right now may overbalance on certain peers)
2) Do we need to configure the number of blocks to be migrated at the same time
3) Do we want to prioritize migrating blocks with no replicas
> Improve cache block migration
> -----------------------------
>
> Key: SPARK-31555
> URL: https://issues.apache.org/jira/browse/SPARK-31555
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 3.1.0
> Reporter: Holden Karau
> Priority: Major
>
> We should explore the following improvements to cache block migration:
> 1) Peer selection (right now may overbalance on certain peers)
> 2) Do we need to configure the number of blocks to be migrated at the same time
> 3) Are there any blocks we don't need to replicate (e.g. they are already stored on the desired number of executors even once we remove the executors slated for decommissioning).
> 4) Do we want to prioritize migrating blocks with no replicas
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org