[jira] [Created] (SYSTEMML-1753) OOM on parfor local in-memory result merge

["Matthias Boehm (JIRA)" <ji...@apache.org>]

Matthias Boehm created SYSTEMML-1753:
----------------------------------------

             Summary: OOM on parfor local in-memory result merge
                 Key: SYSTEMML-1753
                 URL: https://issues.apache.org/jira/browse/SYSTEMML-1753
             Project: SystemML
          Issue Type: Bug
            Reporter: Matthias Boehm


Consider a scenario with relatively large parfor result variable (e.g., many iterations with relatively large output vector each). There are conditions under which the parfor local in-memory result merge runs unnecessarily out of memory (as shown below) due to allocating the result in sparse, collecting all (sparse) outputs and finally converting this result to dense. For result merge without compare, the target number of non-zeros are exactly known, allowing us to directly allocate the result in the correct format, which reduces memory pressure by more than 2x (dense matrix in sparse format).

{code}
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at org.apache.sysml.runtime.matrix.data.MatrixBlock.allocateDenseBlock(MatrixBlock.java:362)
        at org.apache.sysml.runtime.matrix.data.MatrixBlock.sparseToDense(MatrixBlock.java:1136)
        at org.apache.sysml.runtime.matrix.data.MatrixBlock.examSparsity(MatrixBlock.java:1019)
        at org.apache.sysml.runtime.controlprogram.parfor.ResultMergeLocalMemory.executeSerialMerge(ResultMergeLocalMemory.java:114)
        at org.apache.sysml.runtime.controlprogram.ParForProgramBlock.consolidateAndCheckResults(ParForProgramBlock.java:1751)
        at org.apache.sysml.runtime.controlprogram.ParForProgramBlock.executeLocalParFor(ParForProgramBlock.java:814)
        at org.apache.sysml.runtime.controlprogram.ParForProgramBlock.execute(ParForProgramBlock.java:635)
        at org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:123)
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)