You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Thomas Graves (JIRA)" <ji...@apache.org> on 2018/08/06 19:00:00 UTC
[jira] [Resolved] (SPARK-24992) spark should randomize yarn local
dir selection
[ https://issues.apache.org/jira/browse/SPARK-24992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Graves resolved SPARK-24992.
-----------------------------------
Resolution: Fixed
Assignee: Hieu Tri Huynh
Fix Version/s: 2.4.0
> spark should randomize yarn local dir selection
> -----------------------------------------------
>
> Key: SPARK-24992
> URL: https://issues.apache.org/jira/browse/SPARK-24992
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.3.1
> Reporter: Hieu Tri Huynh
> Assignee: Hieu Tri Huynh
> Priority: Minor
> Fix For: 2.4.0
>
>
> Utils.getLocalDir is used to get path of a temporary directory. However, it always returns the the same directory, which is the first element in the array _localRootDirs_. When running on YARN, this might causes the case that we always write to one disk, which makes it busy while other disks are free. We should randomize the selection to spread out the loads.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org