[jira] [Updated] (SYSTEMML-1430) Robust memory handling broadcasts

["Matthias Boehm (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SYSTEMML-1430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matthias Boehm updated SYSTEMML-1430:
-------------------------------------
    Description: 
This task aims to make the memory handling of re-used and temporary broadcast variables more robust in order to avoid unnecessary OOMs. 

1) Explicitly destroy temporary broadcast variables because we would otherwise never clean them up and Spark's ContextCleaner seems to be a best effort daemon that triggers every 30min or on garbage collection (which might be too late if a large object is allocated)

2) Keep track of currently softly reachable (re-used) broadcasts in order to take their size into account when deciding on guarded collect.

> Robust memory handling broadcasts
> ---------------------------------
>
>                 Key: SYSTEMML-1430
>                 URL: https://issues.apache.org/jira/browse/SYSTEMML-1430
>             Project: SystemML
>          Issue Type: Sub-task
>          Components: APIs, Runtime
>            Reporter: Matthias Boehm
>             Fix For: SystemML 1.0
>
>
> This task aims to make the memory handling of re-used and temporary broadcast variables more robust in order to avoid unnecessary OOMs. 
> 1) Explicitly destroy temporary broadcast variables because we would otherwise never clean them up and Spark's ContextCleaner seems to be a best effort daemon that triggers every 30min or on garbage collection (which might be too late if a large object is allocated)
> 2) Keep track of currently softly reachable (re-used) broadcasts in order to take their size into account when deciding on guarded collect.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)