[jira] [Updated] (SYSTEMML-1349) Parfor optimizer support for shared reads (lower memory req)

["Berthold Reinwald (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SYSTEMML-1349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Berthold Reinwald updated SYSTEMML-1349:
----------------------------------------
    Fix Version/s:     (was: SystemML 1.0)
                   SystemML 1.1

> Parfor optimizer support for shared reads (lower memory req)
> ------------------------------------------------------------
>
>                 Key: SYSTEMML-1349
>                 URL: https://issues.apache.org/jira/browse/SYSTEMML-1349
>             Project: SystemML
>          Issue Type: Sub-task
>          Components: Compiler
>            Reporter: Matthias Boehm
>            Assignee: Matthias Boehm
>             Fix For: SystemML 1.1
>
>
> Let's assume the following example script and focus on local parfor with a memory budget of 10GB and k=16 threads (but this issue similarly applies to parfor spark).
> {code}
> X = rand(rows=1000000, cols=1000); #8GB
> parfor(i in 1:ncol(X)) 
>     print(sum(X[,i]));
> {code}
> The memory estimate of the right indexing operation is 8GB and the parfor optimizer tries to set k=min(floor(10GB/8GB), 16) and hence falls back to a single-threaded plan with k=1. However, this is completely unnecessary because X is a shared-read (i.e., variable that is never updated in the parfor body). Accordingly, we should determine the degree of parallelism as k=min(floor((10GB-8GB)/8MB), 16).
> The challenge is to do this for arbitrary control flow and multiple shared reads of different dimension and sparsity. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)