[jira] [Commented] (SYSTEMML-1762) Improve the matrix reshape function for the Spark mode

["Matthias Boehm (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SYSTEMML-1762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16085249#comment-16085249 ] 

Matthias Boehm commented on SYSTEMML-1762:
------------------------------------------

thanks [~Tenma] for catching this issue. As it turned out, this issue occurs in the special case if, for a given input block, we create at least three output blocks and the first and last output block have the same row index. For example, if we have an output matrix of 13 column blocks and we computed (1,12) and (1,1) as the first and last output block index, we missed the middle index (1,13).

> Improve the matrix reshape function for the Spark mode
> ------------------------------------------------------
>
>                 Key: SYSTEMML-1762
>                 URL: https://issues.apache.org/jira/browse/SYSTEMML-1762
>             Project: SystemML
>          Issue Type: Bug
>          Components: Algorithms, ParFor, Runtime
>            Reporter: Fei Hu
>            Assignee: Fei Hu
>         Attachments: MNIST_Distrib_Sgd.scala
>
>
> When running the [distributed MNIST LeNet example | https://github.com/apache/systemml/blob/master/scripts/nn/examples/mnist_lenet_distrib_sgd.dml], it works well in the hybrid mode. But in the Spark mode, there are some errors about
> {{java.lang.NullPointerException}}  and {{java.lang.ArrayIndexOutOfBoundsException: 1000}} when reshaping the matrix. The involved functions are {{org.apache.sysml.runtime.matrix.data.LibMatrixReorg#reshapeSparse}} and {{org.apache.sysml.runtime.matrix.data.LibMatrixReorg#reshapeDense}}. The reason is that the output matrix index computed by {{org.apache.sysml.runtime.matrix.data.LibMatrixReorg#computeResultBlockIndex}} does not match the keys in the {{HashMap<MatrixIndexes,MatrixBlock> rix}}. 
> To reproduce the error, the attached scala file {{MNIST_Distrib_Sgd.scala}} could be used to run the distributed MNIST example.  
> In addition, if adding some codes to ignore the null output matrix block from {{MatrixBlock out = rix.get(ixtmp)}},  the distributed MNIST example could run in the Spark mode, but the result may not be right. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)