You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Davies Liu (JIRA)" <ji...@apache.org> on 2014/07/29 00:23:39 UTC

[jira] [Resolved] (SPARK-1343) PySpark OOMs without caching

     [ https://issues.apache.org/jira/browse/SPARK-1343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Davies Liu resolved SPARK-1343.
-------------------------------

          Resolution: Fixed
       Fix Version/s: 0.9.0
                      1.0.0
    Target Version/s: 1.1.0

> PySpark OOMs without caching
> ----------------------------
>
>                 Key: SPARK-1343
>                 URL: https://issues.apache.org/jira/browse/SPARK-1343
>             Project: Spark
>          Issue Type: Bug
>          Components: PySpark
>    Affects Versions: 0.9.0
>            Reporter: Matei Zaharia
>            Assignee: Davies Liu
>             Fix For: 1.0.0, 0.9.0
>
>
> There have been several reports on the list of PySpark 0.9 OOMing even if it does simple maps and counts, whereas 0.9 didn't. This may be due to either the batching added to serialization, or due to invalid serialized data which makes the Java side allocate an overly large array. Needs investigating for 1.0.



--
This message was sent by Atlassian JIRA
(v6.2#6252)