[jira] [Closed] (SYSTEMML-1019) Perftest: MSVM, L sparse, unncessary under-utilization (parfor dop)

["Matthias Boehm (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SYSTEMML-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matthias Boehm closed SYSTEMML-1019.
------------------------------------

> Perftest: MSVM, L sparse, unncessary under-utilization (parfor dop)
> -------------------------------------------------------------------
>
>                 Key: SYSTEMML-1019
>                 URL: https://issues.apache.org/jira/browse/SYSTEMML-1019
>             Project: SystemML
>          Issue Type: Bug
>            Reporter: Matthias Boehm
>            Assignee: Matthias Boehm
>             Fix For: SystemML 0.11
>
>
> In execution modes spark and hybrid_spark, the parfor optimizer uses a smaller memory budget (50% of normal memory budget) for deciding on the local parfor degree of parallelism in order to avoid unnecessary "guarded collect" over hdfs. However, "guarded collect" can only happen in case of existing spark instructions; otherwise this decision leads to unnecessary under-utilization if the memory requirement is limiting the degree of parallelism. 
> We should determine, if there are existing spark instructions in the body and accordingly adapt the memory budget the parfor optimizer works with. On this MSVM usecase, the performance differences are as follows: 
> {code}
> #before the change
> MSVM train ict=0 on mbperftest/multinomial/X10M_1k_sparse_k150: 144
> MSVM train ict=1 on mbperftest/multinomial/X10M_1k_sparse_k150: 156
> #after the change
> MSVM train ict=0 on mbperftest/multinomial/X10M_1k_sparse_k150: 99
> MSVM train ict=1 on mbperftest/multinomial/X10M_1k_sparse_k150: 117
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)