[jira] [Assigned] (SYSTEMML-1025) Perftest: Large performance variability on scenario L dense (80GB)

["Matthias Boehm (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SYSTEMML-1025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matthias Boehm reassigned SYSTEMML-1025:
----------------------------------------

    Assignee: Matthias Boehm

> Perftest: Large performance variability on scenario L dense (80GB)
> ------------------------------------------------------------------
>
>                 Key: SYSTEMML-1025
>                 URL: https://issues.apache.org/jira/browse/SYSTEMML-1025
>             Project: SystemML
>          Issue Type: Bug
>            Reporter: Matthias Boehm
>            Assignee: Matthias Boehm
>            Priority: Blocker
>
> During many runs of our entire performance testsuite, we've seen quite some performance variability, especially for scenario L dense (80GB) where spark operations are the dominating factor for end-to-end performance. These issues showed up over all algorithms and configurations but especially for multinomial classification and parfor scripts. 
> Let's take for example Naive Bayes over the dense 10M x 1K input with 20 classes. Below are the results of 7 consecutive runs:
> {code}
> NaiveBayes train on mbperftest/multinomial/X10M_1k_dense_k150: 67
> NaiveBayes train on mbperftest/multinomial/X10M_1k_dense_k150: 362
> NaiveBayes train on mbperftest/multinomial/X10M_1k_dense_k150: 484
> NaiveBayes train on mbperftest/multinomial/X10M_1k_dense_k150: 64
> NaiveBayes train on mbperftest/multinomial/X10M_1k_dense_k150: 310
> NaiveBayes train on mbperftest/multinomial/X10M_1k_dense_k150: 91
> NaiveBayes train on mbperftest/multinomial/X10M_1k_dense_k150: 68
> {code} 
> After a detailed investigation, it seems that imbalance, garbage collection, and poor data locality are the reasons:
> * First, we generated the inputs with our Spark backend. Apparently, the rand operation caused imbalance due to garbage collection of some nodes. However, this is a very realistic scenario as we cannot always assume perfect balance.
> * Second, especially for multinomial classification and parfor scripts, the intermediates are not just vectors but larger matrices or there are simply more intermediates. This led again to more garbage collection.
> * Third, the scheduler delay of 3s for pending tasks was exceeded due to garbage collection, leading to remote execution which significantly slowed down the overall execution.
> To resolve these issues, we should make the following two changes:
> * (1) More conservative configuration of spark.locality.wait in systemml's preferred spark configuration, where we did not consider this at all so far.
> * (2) Improvements of reduce-all operations which current unnecessarily create intermediate pair outputs and hence unnecessary Tuple2 and MatrixIndexes objects. 
> With a default scheduler delay of 5s instead of the default 3s as well as improved reduce-all for mapmm, groupedagg, tsmm, tsmm2, zipmm, and uagg, we got the following promising results (which include spark context creation and initial read):
> {code}
> NaiveBayes train on mbperftest/multinomial/X10M_1k_dense_k150: 52
> NaiveBayes train on mbperftest/multinomial/X10M_1k_dense_k150: 45
> NaiveBayes train on mbperftest/multinomial/X10M_1k_dense_k150: 44
> NaiveBayes train on mbperftest/multinomial/X10M_1k_dense_k150: 44
> NaiveBayes train on mbperftest/multinomial/X10M_1k_dense_k150: 51
> NaiveBayes train on mbperftest/multinomial/X10M_1k_dense_k150: 50
> NaiveBayes train on mbperftest/multinomial/X10M_1k_dense_k150: 47
> {code}
> cc [~reinwald] [~niketanpansare] [~freiss]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)