[jira] [Closed] (SYSTEMML-1029) Perftest: Data generator for descriptive statistics (slowdown due to update in-place)

["Matthias Boehm (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SYSTEMML-1029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matthias Boehm closed SYSTEMML-1029.
------------------------------------

> Perftest: Data generator for descriptive statistics (slowdown due to update in-place)
> -------------------------------------------------------------------------------------
>
>                 Key: SYSTEMML-1029
>                 URL: https://issues.apache.org/jira/browse/SYSTEMML-1029
>             Project: SystemML
>          Issue Type: Bug
>            Reporter: Matthias Boehm
>            Assignee: Matthias Boehm
>            Priority: Blocker
>             Fix For: SystemML 0.11
>
>
> The data generator for descriptive statistics shows severe performance issues, due to update-in-place. In order to avoid buffer pool serialization and evictions via shallow serialize, we represent sparse matrices which are subject to update in-place in CSR. This works very well for row major updates but has severe problems on column-wise updates (due to shifting of values). 
> There are three potential options:
> * (1) Workaround: Change the data generator script from column-wise indexing to row-wise indexing and transpose after the parfor loop.
> * (2) Parfor rewrite: Disable update in-place, if sparse input, no parfor update in-place, and column-wise access pattern.
> * (3) Improved left indexing over CSR: Reallocate and shift the output once. This is feasible as left indexing (specifically sparse-sparse and dense-sparse) are the only operations which update the CSR representation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)