You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@systemml.apache.org by na...@apache.org on 2017/08/14 22:19:23 UTC
systemml git commit: [DOC][HOTFIX] updatest to the performance test
scripts
Repository: systemml
Updated Branches:
refs/heads/master 54e809898 -> 667aeb2b7
[DOC][HOTFIX] updatest to the performance test scripts
Closes #616
Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/667aeb2b
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/667aeb2b
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/667aeb2b
Branch: refs/heads/master
Commit: 667aeb2b7f38b76b1ff85138426f215a03a4dfc4
Parents: 54e8098
Author: krishnakalyan3 <kr...@gmail.com>
Authored: Mon Aug 14 15:18:50 2017 -0700
Committer: Nakul Jindal <na...@gmail.com>
Committed: Mon Aug 14 15:18:50 2017 -0700
----------------------------------------------------------------------
docs/python-performance-test.md | 15 ++++++++++++++-
scripts/perftest/python/utils_fs.py | 4 ++--
2 files changed, 16 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/systemml/blob/667aeb2b/docs/python-performance-test.md
----------------------------------------------------------------------
diff --git a/docs/python-performance-test.md b/docs/python-performance-test.md
index 3d29f01..ce36c2d 100644
--- a/docs/python-performance-test.md
+++ b/docs/python-performance-test.md
@@ -177,7 +177,20 @@ In the example above `--tag` can be a major/minor systemml version and `--auth`
Currently we only support time difference between algorithms in different versions. This can be obtained by running the script below
`./stats.py --auth client_json.json --exec-mode singlenode --tags 1.0 2.0`
-Note: Please pip install `https://github.com/burnash/gspread` to use google docs client.
+We pass different `matrix shapes` using `--mat-shape` argument.
+
+Matrix Shape | Approximate Data Size
+--- | --- |
+10k_1k|80MB
+100k_1k|800MB
+1M_1k|8GB
+10M_1k|80GB
+100M_1k|800GB
+
+For example the command below runs performance test for all data sizes described above
+`run_perftest.py --family binomial clustering multinomial regression1 regression2 stats1 stats2 --mat-shape 10k_1k 100k_1k 1M_1k 10M_1k 100M_1k --master yarn-client --temp-dir hdfs://localhost:9000/user/systemml`
+
+Note: Please use this command `pip3 install -r requirements.txt` before using the perftest scripts.
## Troubleshooting
http://git-wip-us.apache.org/repos/asf/systemml/blob/667aeb2b/scripts/perftest/python/utils_fs.py
----------------------------------------------------------------------
diff --git a/scripts/perftest/python/utils_fs.py b/scripts/perftest/python/utils_fs.py
index 977c4f4..7e04907 100755
--- a/scripts/perftest/python/utils_fs.py
+++ b/scripts/perftest/python/utils_fs.py
@@ -134,12 +134,12 @@ def relevant_folders(path, algo, family, matrix_type, matrix_shape, mode):
if mode == 'data-gen':
sub_folder_name = '.'.join([family, current_matrix_type, current_matrix_shape])
cmd = ['hdfs', 'dfs', '-ls', path]
- path_subdir = subprocess_exec(' '.join(cmd), 'dir')
+ path_subdir = subprocess_exec(' '.join(cmd), extract='dir')
if mode == 'train':
sub_folder_name = '.'.join([algo, family, current_matrix_type, current_matrix_shape])
cmd = ['hdfs', 'dfs', '-ls', path]
- path_subdir = subprocess_exec(' '.join(cmd), 'dir')
+ path_subdir = subprocess_exec(' '.join(cmd), extract='dir')
path_folders = list(filter(lambda x: contains_dir(x, sub_folder_name), path_subdir))