[jira] [Created] (SYSTEMML-1388) Load imbalance parfor spark w/ factoring task partitioning

["Matthias Boehm (JIRA)" <ji...@apache.org>]

Matthias Boehm created SYSTEMML-1388:
----------------------------------------

             Summary: Load imbalance parfor spark w/ factoring task partitioning
                 Key: SYSTEMML-1388
                 URL: https://issues.apache.org/jira/browse/SYSTEMML-1388
             Project: SystemML
          Issue Type: Bug
            Reporter: Matthias Boehm


For large numbers of iterations, the parfor optmizer uses factoring for task partitioning, i.e., tasks w/ exponentially decreasing sizes, which is supposed to achieve good load balance while keeping the number of tasks low. 

However, our current spark runtime parallelizes theses parfor tasks to a number of partitions equal to the number of cores. This leads to multiple parfor tasks falling into one physical partition and thus spark task. Due to exponentially decreasing task sizes, we see huge imbalance. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)