[jira] [Resolved] (SYSTEMML-845) Compare Performance of LeNet Scripts With & Without Using SystemML-NN

["Mike Dusenberry (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SYSTEMML-845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mike Dusenberry resolved SYSTEMML-845.
--------------------------------------
       Resolution: Fixed
    Fix Version/s: SystemML 0.11

> Compare Performance of LeNet Scripts With & Without Using SystemML-NN
> ---------------------------------------------------------------------
>
>                 Key: SYSTEMML-845
>                 URL: https://issues.apache.org/jira/browse/SYSTEMML-845
>             Project: SystemML
>          Issue Type: Improvement
>          Components: Algorithms, Compiler
>    Affects Versions: SystemML 0.11
>            Reporter: Mike Dusenberry
>            Assignee: Mike Dusenberry
>             Fix For: SystemML 0.11
>
>         Attachments: convert.dml, lenet-train-spark-explain-recompile-hops.log, lenet-train-spark-explain.log, log08.03.16-1470268602.txt, mnist_lenet-train-spark-explain-recompile-hops.log, mnist_lenet-train-spark-explain.log, perf.sh, run.sh
>
>
> This JIRA issue tracks the comparison of the performance of the LeNet scripts with & without using SystemML-NN.  The goal is that they should have equal performance in terms of both accuracy and time.  Any difference will be indicate areas of engine improvement.
> Scripts:
> * [mnist_lenet-train.dml | https://github.com/apache/incubator-systemml/blob/master/scripts/staging/SystemML-NN/examples/mnist_lenet-train.dml] - LeNet script that *does* use the SystemML-NN library.
> * [lenet-train.dml | https://github.com/apache/incubator-systemml/blob/master/scripts/staging/lenet-train.dml] - LeNet script that *does not* use the SystemML-NN library.
> *Current Status - Forced Singlenode:*
> Equal performance when running the scripts in standalone mode with the {{-exec singlenode}} flag, 20GB of memory, and using data inputs in the SystemML binary format -- see {{run.sh}} and {{perf.sh}} for information.
> Results:
> - Run #1:
> || Script | Time (s) | Accuracy ||
> | mnist_lenet-train.dml | 2987.400704441 | 99.32% |
> | lenet-train.dml | 2816.369435579 | 99.28% |
> - Run #2:
> || Script | Time (s) | Accuracy ||
> | mnist_lenet-train.dml | 2847.790531812 | 99.16% |
> | lenet-train.dml | 2950.520494210 | 99.18% |
> So, same accuracy, and same runtime in singlenode mode!
> To fully reproduce, I basically created a directory, placed the two attached bash scripts in it, grabbed a copy of the NN library and placed it into the directory, ran the examples/get_mnist_data.sh script from the library to get the data (placed into examples/data), then used the attached convert.dml to create binary copies of the data for both scripts, then ran run.sh. Also, I copied examples/data to the base directory as well.  Adjust the {{EXEC}} and related variables in {{perf.sh}} to switch between standalone, Spark, memory sizes, explain, stats, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)