You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@systemml.apache.org by "Matthias Boehm (JIRA)" <ji...@apache.org> on 2018/03/17 00:10:00 UTC

[jira] [Closed] (SYSTEMML-2188) Unnecessary evictions on rdd collect

     [ https://issues.apache.org/jira/browse/SYSTEMML-2188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matthias Boehm closed SYSTEMML-2188.
------------------------------------
       Resolution: Fixed
         Assignee: Matthias Boehm
    Fix Version/s: SystemML 1.1

> Unnecessary evictions on rdd collect 
> -------------------------------------
>
>                 Key: SYSTEMML-2188
>                 URL: https://issues.apache.org/jira/browse/SYSTEMML-2188
>             Project: SystemML
>          Issue Type: Sub-task
>            Reporter: Matthias Boehm
>            Assignee: Matthias Boehm
>            Priority: Major
>             Fix For: SystemML 1.1
>
>
> For robustness regarding potential OOMs we already have functionality for guarded collects that write the RDD to hdfs and read it into memory instead of collect because the latter requires twice the memory of a simple read. However, there are scenarios, where we collect an RDD and because its size exceeds the buffer pool, we immediately evict to local file system in a single-threaded manner. This task aims to consolidate this and use the guarded collect whenever the data is known to exceed the buffer pool size.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)