[jira] [Resolved] (SYSTEMML-1430) Robust memory handling broadcasts

["Matthias Boehm (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SYSTEMML-1430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matthias Boehm resolved SYSTEMML-1430.
--------------------------------------
    Resolution: Done
      Assignee: Matthias Boehm

> Robust memory handling broadcasts
> ---------------------------------
>
>                 Key: SYSTEMML-1430
>                 URL: https://issues.apache.org/jira/browse/SYSTEMML-1430
>             Project: SystemML
>          Issue Type: Sub-task
>          Components: APIs, Runtime
>            Reporter: Matthias Boehm
>            Assignee: Matthias Boehm
>             Fix For: SystemML 1.0
>
>
> This task aims to make the memory handling of re-used and temporary broadcast variables more robust in order to avoid unnecessary OOMs. 
> 1) Explicitly destroy temporary broadcast variables because we would otherwise never clean them up and Spark's ContextCleaner seems to be a best effort daemon that triggers every 30min or on garbage collection (which might be too late if a large object is allocated)
> 2) Keep track of currently softly reachable (re-used) broadcasts in order to take their size into account when deciding on guarded collect.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)