You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Michael Armbrust (JIRA)" <ji...@apache.org> on 2014/08/12 05:23:11 UTC

[jira] [Resolved] (SPARK-2650) Caching tables larger than memory causes OOMs

     [ https://issues.apache.org/jira/browse/SPARK-2650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael Armbrust resolved SPARK-2650.
-------------------------------------

          Resolution: Fixed
       Fix Version/s: 1.1.0
            Assignee: Michael Armbrust  (was: Cheng Lian)
    Target Version/s: 1.1.0  (was: 1.2.0)

> Caching tables larger than memory causes OOMs
> ---------------------------------------------
>
>                 Key: SPARK-2650
>                 URL: https://issues.apache.org/jira/browse/SPARK-2650
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.0.0, 1.0.1
>            Reporter: Michael Armbrust
>            Assignee: Michael Armbrust
>            Priority: Critical
>             Fix For: 1.1.0
>
>
> The logic for setting up the initial column buffers is different for Spark SQL compared to Shark and I'm seeing OOMs when caching tables that are larger than available memory (where shark was okay).
> Two suspicious things: the intialSize is always set to 0 so we always go with the default.  The default looks like it was copied from code like 10 * 1024 * 1024... but in Spark SQL its 10 * 102 * 1024.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org