[jira] [Closed] (SYSTEMML-1309) Parfor spark buffer pool handling

["Matthias Boehm (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SYSTEMML-1309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matthias Boehm closed SYSTEMML-1309.
------------------------------------

> Parfor spark buffer pool handling
> ---------------------------------
>
>                 Key: SYSTEMML-1309
>                 URL: https://issues.apache.org/jira/browse/SYSTEMML-1309
>             Project: SystemML
>          Issue Type: Sub-task
>          Components: APIs, Runtime
>            Reporter: Matthias Boehm
>            Assignee: Matthias Boehm
>             Fix For: SystemML 1.0
>
>
> In contrast to parfor mr jobs, where every task has its own, process-local buffer pool, on spark with multi-threaded executors, multiple tasks share a common buffer pool. This is advantageous because common inputs are just read once. However, it also requires a synchronized buffer pool initialization and cleanup per executor. Especially the cleanup (e.g., of created cache directories) is tricky because spark does not provide an executor close call. Hence, our approach is to use a robust version of deleteOnExit that is independent of the exit code and also removes remaining files that are unknown during delete registration.  



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)