You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/05/21 05:37:40 UTC

[jira] [Resolved] (SPARK-3153) shuffle will run out of space when disks have different free space

     [ https://issues.apache.org/jira/browse/SPARK-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hyukjin Kwon resolved SPARK-3153.
---------------------------------
    Resolution: Incomplete

> shuffle will run out of space when disks have different free space
> ------------------------------------------------------------------
>
>                 Key: SPARK-3153
>                 URL: https://issues.apache.org/jira/browse/SPARK-3153
>             Project: Spark
>          Issue Type: Bug
>          Components: Shuffle
>            Reporter: Davies Liu
>            Priority: Major
>              Labels: bulk-closed
>
> If we have several disks in SPARK_LOCAL_DIRS, and one of them is much smaller than others (maybe added in my mistake, or special disk, SSD), them the shuffle will meet the problem of run out of space with this smaller disk.
> PySpark also has this issue during spilling.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org