[jira] [Resolved] (SYSTEMML-1753) OOM on parfor local in-memory result merge

["Matthias Boehm (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SYSTEMML-1753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matthias Boehm resolved SYSTEMML-1753.
--------------------------------------
       Resolution: Fixed
         Assignee: Matthias Boehm
    Fix Version/s: SystemML 1.0

> OOM on parfor local in-memory result merge
> ------------------------------------------
>
>                 Key: SYSTEMML-1753
>                 URL: https://issues.apache.org/jira/browse/SYSTEMML-1753
>             Project: SystemML
>          Issue Type: Bug
>            Reporter: Matthias Boehm
>            Assignee: Matthias Boehm
>             Fix For: SystemML 1.0
>
>
> Consider a scenario with relatively large parfor result variable (e.g., many iterations with relatively large output vector each). There are conditions under which the parfor local in-memory result merge runs unnecessarily out of memory (as shown below) due to allocating the result in sparse, collecting all (sparse) outputs and finally converting this result to dense. For result merge without compare, the target number of non-zeros are exactly known, allowing us to directly allocate the result in the correct format, which reduces memory pressure by more than 2x (dense matrix in sparse format).
> {code}
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
>         at org.apache.sysml.runtime.matrix.data.MatrixBlock.allocateDenseBlock(MatrixBlock.java:362)
>         at org.apache.sysml.runtime.matrix.data.MatrixBlock.sparseToDense(MatrixBlock.java:1136)
>         at org.apache.sysml.runtime.matrix.data.MatrixBlock.examSparsity(MatrixBlock.java:1019)
>         at org.apache.sysml.runtime.controlprogram.parfor.ResultMergeLocalMemory.executeSerialMerge(ResultMergeLocalMemory.java:114)
>         at org.apache.sysml.runtime.controlprogram.ParForProgramBlock.consolidateAndCheckResults(ParForProgramBlock.java:1751)
>         at org.apache.sysml.runtime.controlprogram.ParForProgramBlock.executeLocalParFor(ParForProgramBlock.java:814)
>         at org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute(ParForProgramBlock.java:635)
>         at org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:123)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)