You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Weizhong (JIRA)" <ji...@apache.org> on 2015/05/04 14:27:06 UTC

[jira] [Created] (SPARK-7339) PySpark shuffle spill memory sometimes are not correct

Weizhong created SPARK-7339:
-------------------------------

             Summary: PySpark shuffle spill memory sometimes are not correct
                 Key: SPARK-7339
                 URL: https://issues.apache.org/jira/browse/SPARK-7339
             Project: Spark
          Issue Type: Improvement
          Components: PySpark
    Affects Versions: 1.3.1
            Reporter: Weizhong
            Priority: Minor


In PySpark we get memory used before and after spill, then use the difference of these two value as memorySpilled, but if the before value is small than after value, then we will get a negative value, but this scenario 0 value may be more reasonable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org