You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Ilya Ganelin (JIRA)" <ji...@apache.org> on 2015/06/30 22:19:04 UTC

[jira] [Commented] (SPARK-3153) shuffle will run out of space when disks have different free space

    [ https://issues.apache.org/jira/browse/SPARK-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14608993#comment-14608993 ] 

Ilya Ganelin commented on SPARK-3153:
-------------------------------------

cc [~davies] Believe this issue can be closed given duplication of SPARK-5418, no?

> shuffle will run out of space when disks have different free space
> ------------------------------------------------------------------
>
>                 Key: SPARK-3153
>                 URL: https://issues.apache.org/jira/browse/SPARK-3153
>             Project: Spark
>          Issue Type: Bug
>          Components: Shuffle
>            Reporter: Davies Liu
>
> If we have several disks in SPARK_LOCAL_DIRS, and one of them is much smaller than others (maybe added in my mistake, or special disk, SSD), them the shuffle will meet the problem of run out of space with this smaller disk.
> PySpark also has this issue during spilling.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org