You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2016/11/13 13:51:59 UTC

[jira] [Commented] (SPARK-18421) Dynamic disk allocation

    [ https://issues.apache.org/jira/browse/SPARK-18421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15661520#comment-15661520 ] 

Sean Owen commented on SPARK-18421:
-----------------------------------

Spark doesn't manage storage at all. I don't think this could be in scope therefore especially because it could only apply to cloud and would be cloud specific.

> Dynamic disk allocation
> -----------------------
>
>                 Key: SPARK-18421
>                 URL: https://issues.apache.org/jira/browse/SPARK-18421
>             Project: Spark
>          Issue Type: New Feature
>          Components: Spark Core
>    Affects Versions: 2.0.1
>            Reporter: Aniket Bhatnagar
>            Priority: Minor
>
> Dynamic allocation feature allows you to add executors and scale computation power. This is great, however, I feel like we also need a way to dynamically scale storage. Currently, if the disk is not able to hold the spilled/shuffle data, the job is aborted (in yarn, the node manager kills the container) causing frustration and loss of time. In deployments like AWS EMR, it is possible to run an agent that add disks on the fly if it sees that the disks are running out of space and it would be great if Spark could immediately start using the added disks just as it does when new executors are added.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org