[jira] [Closed] (SYSTEMML-1388) Load imbalance parfor spark w/ factoring task partitioning

["Matthias Boehm (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SYSTEMML-1388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matthias Boehm closed SYSTEMML-1388.
------------------------------------

> Load imbalance parfor spark w/ factoring task partitioning
> ----------------------------------------------------------
>
>                 Key: SYSTEMML-1388
>                 URL: https://issues.apache.org/jira/browse/SYSTEMML-1388
>             Project: SystemML
>          Issue Type: Bug
>            Reporter: Matthias Boehm
>            Assignee: Matthias Boehm
>             Fix For: SystemML 1.0
>
>
> For large numbers of iterations, the parfor optmizer uses factoring for task partitioning, i.e., tasks w/ exponentially decreasing sizes, which is supposed to achieve good load balance while keeping the number of tasks low. 
> However, our current spark runtime parallelizes theses parfor tasks to a number of partitions equal to the number of cores. This leads to multiple parfor tasks falling into one physical partition and thus spark task. Due to exponentially decreasing task sizes, we see huge imbalance. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)