You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2015/05/04 14:33:06 UTC

[jira] [Assigned] (SPARK-7339) PySpark shuffle spill memory sometimes are not correct

     [ https://issues.apache.org/jira/browse/SPARK-7339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Apache Spark reassigned SPARK-7339:
-----------------------------------

    Assignee:     (was: Apache Spark)

> PySpark shuffle spill memory sometimes are not correct
> ------------------------------------------------------
>
>                 Key: SPARK-7339
>                 URL: https://issues.apache.org/jira/browse/SPARK-7339
>             Project: Spark
>          Issue Type: Improvement
>          Components: PySpark
>    Affects Versions: 1.3.1
>            Reporter: Weizhong
>            Priority: Minor
>
> In PySpark we get memory used before and after spill, then use the difference of these two value as memorySpilled, but if the before value is small than after value, then we will get a negative value, but this scenario 0 value may be more reasonable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org