You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@systemml.apache.org by du...@apache.org on 2015/12/02 02:05:19 UTC
[41/47] incubator-systemml git commit: [SYSML-347] Invocation Tabs on
Algorithms Reference
[SYSML-347] Invocation Tabs on Algorithms Reference
Added Hadoop and Spark tabs showing SystemML invocations to
Algorithms Reference.
Updated main.js for handling tabs.
Updated DML and PyDML Programming Guide with respect to tab handling.
Highlighted reference keys in Algorithms Bibliography.
Fixed links in Quick Start Guide.
Project: http://git-wip-us.apache.org/repos/asf/incubator-systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-systemml/commit/b8055201
Tree: http://git-wip-us.apache.org/repos/asf/incubator-systemml/tree/b8055201
Diff: http://git-wip-us.apache.org/repos/asf/incubator-systemml/diff/b8055201
Branch: refs/heads/gh-pages
Commit: b8055201425c0a4073568d6e07d38106138c5d80
Parents: 2a2520b
Author: Deron Eriksson <de...@us.ibm.com>
Authored: Mon Oct 26 09:40:25 2015 -0700
Committer: Luciano Resende <lr...@apache.org>
Committed: Mon Oct 26 09:40:25 2015 -0700
----------------------------------------------------------------------
algorithms-bibliography.md | 50 +--
algorithms-classification.md | 518 +++++++++++++++++++++++-
algorithms-clustering.md | 136 ++++++-
algorithms-descriptive-statistics.md | 139 ++++++-
algorithms-matrix-factorization.md | 194 ++++++++-
algorithms-regression.md | 641 ++++++++++++++++++++++++------
algorithms-survival-analysis.md | 202 +++++++++-
dml-and-pydml-programming-guide.md | 26 +-
js/main.js | 70 +++-
quick-start-guide.md | 54 +--
10 files changed, 1800 insertions(+), 230 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/b8055201/algorithms-bibliography.md
----------------------------------------------------------------------
diff --git a/algorithms-bibliography.md b/algorithms-bibliography.md
index 1012d09..d529c42 100644
--- a/algorithms-bibliography.md
+++ b/algorithms-bibliography.md
@@ -6,103 +6,103 @@ displayTitle: <a href="algorithms-reference.html">SystemML Algorithms Reference<
# 7. Bibliography
-AcockStavig1979, Alan C. Acock and Gordon
+**[AcockStavig1979]** Alan C. Acock and Gordon
R. Stavig, A Measure of Association for Nonparametric
Statistics, Social Forces, Oxford University
Press, Volume 57, Number 4, June, 1979,
1381--1386.
-AgrawalKSX2002, Rakesh Agrawal and
+**[AgrawalKSX2002]** Rakesh Agrawal and
Jerry Kiernan and Ramakrishnan Srikant and Yirong Xu,
Hippocratic Databases, Proceedings of the 28-th
International Conference on Very Large Data Bases (VLDB 2002),
Hong Kong, China, August 20--23, 2002,
143--154.
-Agresti2002, Alan Agresti, Categorical
+**[Agresti2002]** Alan Agresti, Categorical
Data Analysis, Second Edition, Wiley Series in
Probability and Statistics, Wiley-Interscience
2002, 710.
-AloiseDHP2009, Daniel Aloise and Amit
+**[AloiseDHP2009]** Daniel Aloise and Amit
Deshpande and Pierre Hansen and Preyas Popat, NP-hardness of
Euclidean Sum-of-squares Clustering, Machine Learning,
Kluwer Academic Publishers, Volume 75, Number 2,
May, 2009, 245--248.
-ArthurVassilvitskii2007,
+**[ArthurVassilvitskii2007]**
k-means++: The Advantages of Careful Seeding, David
Arthur and Sergei Vassilvitskii, Proceedings of the 18th
Annual ACM-SIAM Symposium on Discrete Algorithms (SODA 2007),
January 7--9, 2007, New Orleans, LA,
USA, 1027--1035.
-Breiman2001, L. Breiman. Random forests. Machine Learning, 45(1):5–32, 2001.
+**[Breiman2001]** L. Breiman. Random forests. Machine Learning, 45(1):5–32, 2001.
-BreimanFOS1984, L. Breiman, J. H. Friedman, R. A. Olshen, and C. J. Stone. Classification and Regression Trees. Wadsworth, 1984.
+**[BreimanFOS1984]** L. Breiman, J. H. Friedman, R. A. Olshen, and C. J. Stone. Classification and Regression Trees. Wadsworth, 1984.
-Chapelle2007, Olivier Chapelle, Training a Support Vector Machine in the Primal, Neural Computation, 2007.
+**[Chapelle2007]** Olivier Chapelle, Training a Support Vector Machine in the Primal, Neural Computation, 2007.
-Cochran1954, William G. Cochran,
+**[Cochran1954]** William G. Cochran,
Some Methods for Strengthening the Common $\chi^2$ Tests,
Biometrics, Volume 10, Number 4, December
1954, 417--451.
-Collett2003, D. Collett. Modelling Survival Data in Medical Research, Second Edition. Chapman & Hall/CRC Texts in Statistical Science. Taylor & Francis, 2003.
+**[Collett2003]** D. Collett. Modelling Survival Data in Medical Research, Second Edition. Chapman & Hall/CRC Texts in Statistical Science. Taylor & Francis, 2003.
-Gill2000, Jeff Gill, Generalized Linear
+**[Gill2000]** Jeff Gill, Generalized Linear
Models: A Unified Approach, Sage University Papers Series on
Quantitative Applications in the Social Sciences, Number 07-134,
2000, Sage Publications, 101.
-Hartigan1975, John A. Hartigan,
+**[Hartigan1975]** John A. Hartigan,
Clustering Algorithms, John Wiley~&~Sons Inc.,
Probability and Mathematical Statistics, April
1975, 365.
-Hsieh2008, C-J Hsieh, K-W Chang, C-J Lin, S. S. Keerthi and S. Sundararajan, A Dual Coordinate Descent Method for Large-scale Linear SVM, International Conference of Machine Learning (ICML), 2008.
+**[Hsieh2008]** C-J Hsieh, K-W Chang, C-J Lin, S. S. Keerthi and S. Sundararajan, A Dual Coordinate Descent Method for Large-scale Linear SVM, International Conference of Machine Learning (ICML), 2008.
-Lin2008, Chih-Jen Lin and Ruby C. Weng and
+**[Lin2008]** Chih-Jen Lin and Ruby C. Weng and
S. Sathiya Keerthi, Trust Region Newton Method for
Large-Scale Logistic Regression, Journal of Machine Learning
Research, April, 2008, Volume 9, 627--650.
-McCallum1998, A. McCallum and K. Nigam, A comparison of event models for naive bayes text classification, AAAI-98 workshop on learning for text categorization, 1998.
+**[McCallum1998]** A. McCallum and K. Nigam, A comparison of event models for naive bayes text classification, AAAI-98 workshop on learning for text categorization, 1998.
-McCullagh1989, Peter McCullagh and John Ashworth
+**[McCullagh1989]** Peter McCullagh and John Ashworth
Nelder, Generalized Linear Models, Second Edition,
Monographs on Statistics and Applied Probability, Number 37,
1989, Chapman & Hall/CRC, 532.
-Nelder1972, John Ashworth Nelder and Robert
+**[Nelder1972]** John Ashworth Nelder and Robert
William Maclagan Wedderburn, Generalized Linear Models,
Journal of the Royal Statistical Society, Series A
(General), 1972, Volume 135, Number 3,
370--384.
-Nocedal1999, J. Nocedal and S. J. Wright, Numerical Optimization, Springer-Verlag, 1999.
+**[Nocedal1999]** J. Nocedal and S. J. Wright, Numerical Optimization, Springer-Verlag, 1999.
-Nocedal2006, Optimization Numerical Optimization,
+**[Nocedal2006]** Optimization Numerical Optimization,
Jorge Nocedal and Stephen Wright, Springer Series
in Operations Research and Financial Engineering, 664,
Second Edition, Springer, 2006.
-PandaHBB2009, B. Panda, J. Herbach, S. Basu, and R. J. Bayardo. PLANET: massively parallel learning of tree ensembles with mapreduce. PVLDB, 2(2):1426– 1437, 2009.
+**[PandaHBB2009]** B. Panda, J. Herbach, S. Basu, and R. J. Bayardo. PLANET: massively parallel learning of tree ensembles with mapreduce. PVLDB, 2(2):1426– 1437, 2009.
-Russell2009, S. Russell and P. Norvig, Artificial Intelligence: A Modern Approach, Prentice Hall, 2009.
+**[Russell2009]** S. Russell and P. Norvig, Artificial Intelligence: A Modern Approach, Prentice Hall, 2009.
-Scholkopf1995, B. Scholkopf, C. Burges and V. Vapnik, Extracting Support Data for a Given Task, International Conference on Knowledge Discovery and Data Mining (ICDM), 1995.
+**[Scholkopf1995]** B. Scholkopf, C. Burges and V. Vapnik, Extracting Support Data for a Given Task, International Conference on Knowledge Discovery and Data Mining (ICDM), 1995.
-Stevens1946, Stanley Smith Stevens,
+**[Stevens1946]** Stanley Smith Stevens,
On the Theory of Scales of Measurement, Science
June 7, 1946, Volume 103, Number 2684,
677--680.
-Vetterling1992,
+**[Vetterling1992]**
W. T. Vetterling and B. P. Flannery,
Multidimensions in Numerical Recipes in C - The Art in Scientific Computing, W. H. Press and S. A. Teukolsky (eds.), Cambridge University Press, 1992.
-ZhouWSP08,
+**[ZhouWSP08]**
Y. Zhou, D. M. Wilkinson, R. Schreiber, and R. Pan. Large-scale parallel collaborative filtering for the Netflix prize.
In Algorithmic Aspects in Information and Management, 4th International Conference, AAIM 2008, Shanghai, China, June 23-25, 2008. Proceedings, pages 337–348, 2008.
http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/b8055201/algorithms-classification.md
----------------------------------------------------------------------
diff --git a/algorithms-classification.md b/algorithms-classification.md
index d6215df..b076017 100644
--- a/algorithms-classification.md
+++ b/algorithms-classification.md
@@ -108,6 +108,8 @@ Eqs. (1) and (2).
### Usage
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f MultiLogReg.dml
-nvargs X=<file>
Y=<file>
@@ -119,7 +121,27 @@ Eqs. (1) and (2).
moi=[int]
mii=[int]
fmt=[format]
-
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f MultiLogReg.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs X=<file>
+ Y=<file>
+ B=<file>
+ Log=[file]
+ icpt=[int]
+ reg=[double]
+ tol=[double]
+ moi=[int]
+ mii=[int]
+ fmt=[format]
+</div>
+</div>
### Arguments
@@ -173,6 +195,8 @@ SystemML Language Reference for details.
### Examples
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f MultiLogReg.dml
-nvargs X=/user/ml/X.mtx
Y=/user/ml/Y.mtx
@@ -184,6 +208,27 @@ SystemML Language Reference for details.
moi=100
mii=10
Log=/user/ml/log.csv
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f MultiLogReg.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs X=/user/ml/X.mtx
+ Y=/user/ml/Y.mtx
+ B=/user/ml/B.mtx
+ fmt=csv
+ icpt=2
+ reg=1.0
+ tol=0.0001
+ moi=100
+ mii=10
+ Log=/user/ml/log.csv
+</div>
+</div>
* * *
@@ -329,6 +374,8 @@ support vector machine (`y` with domain size `2`).
**Binary-Class Support Vector Machines**:
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f l2-svm.dml
-nvargs X=<file>
Y=<file>
@@ -339,9 +386,31 @@ support vector machine (`y` with domain size `2`).
model=<file>
Log=<file>
fmt=[format]
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f l2-svm.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs X=<file>
+ Y=<file>
+ icpt=[int]
+ tol=[double]
+ reg=[double]
+ maxiter=[int]
+ model=<file>
+ Log=<file>
+ fmt=[format]
+</div>
+</div>
**Binary-Class Support Vector Machines Prediction**:
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f l2-svm-predict.dml
-nvargs X=<file>
Y=[file]
@@ -351,7 +420,25 @@ support vector machine (`y` with domain size `2`).
accuracy=[file]
confusion=[file]
fmt=[format]
-
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f l2-svm-predict.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs X=<file>
+ Y=[file]
+ icpt=[int]
+ model=<file>
+ scores=[file]
+ accuracy=[file]
+ confusion=[file]
+ fmt=[format]
+</div>
+</div>
#### Arguments
@@ -403,6 +490,8 @@ using a held-out test set. Note that this is an optional argument.
**Binary-Class Support Vector Machines**:
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f l2-svm.dml
-nvargs X=/user/ml/X.mtx
Y=/user/ml/y.mtx
@@ -413,9 +502,31 @@ using a held-out test set. Note that this is an optional argument.
maxiter=100
model=/user/ml/weights.csv
Log=/user/ml/Log.csv
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f l2-svm.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs X=/user/ml/X.mtx
+ Y=/user/ml/y.mtx
+ icpt=0
+ tol=0.001
+ fmt=csv
+ reg=1.0
+ maxiter=100
+ model=/user/ml/weights.csv
+ Log=/user/ml/Log.csv
+</div>
+</div>
**Binary-Class Support Vector Machines Prediction**:
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f l2-svm-predict.dml
-nvargs X=/user/ml/X.mtx
Y=/user/ml/y.mtx
@@ -425,6 +536,25 @@ using a held-out test set. Note that this is an optional argument.
scores=/user/ml/scores.csv
accuracy=/user/ml/accuracy.csv
confusion=/user/ml/confusion.csv
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f l2-svm-predict.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs X=/user/ml/X.mtx
+ Y=/user/ml/y.mtx
+ icpt=0
+ fmt=csv
+ model=/user/ml/weights.csv
+ scores=/user/ml/scores.csv
+ accuracy=/user/ml/accuracy.csv
+ confusion=/user/ml/confusion.csv
+</div>
+</div>
#### Details
@@ -481,6 +611,8 @@ class labels.
**Multi-Class Support Vector Machines**:
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f m-svm.dml
-nvargs X=<file>
Y=<file>
@@ -491,9 +623,31 @@ class labels.
model=<file>
Log=<file>
fmt=[format]
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f m-svm.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs X=<file>
+ Y=<file>
+ icpt=[int]
+ tol=[double]
+ reg=[double]
+ maxiter=[int]
+ model=<file>
+ Log=<file>
+ fmt=[format]
+</div>
+</div>
**Multi-Class Support Vector Machines Prediction**:
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f m-svm-predict.dml
-nvargs X=<file>
Y=[file]
@@ -503,6 +657,25 @@ class labels.
accuracy=[file]
confusion=[file]
fmt=[format]
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f m-svm-predict.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs X=<file>
+ Y=[file]
+ icpt=[int]
+ model=<file>
+ scores=[file]
+ accuracy=[file]
+ confusion=[file]
+ fmt=[format]
+</div>
+</div>
#### Arguments
@@ -555,28 +728,71 @@ SystemML Language Reference for details.
**Multi-Class Support Vector Machines**:
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f m-svm.dml
-nvargs X=/user/ml/X.mtx
- Y=/user/ml/y.mtx
+ Y=/user/ml/y.mtx
icpt=0
tol=0.001
- reg=1.0
- maxiter=100
- fmt=csv
+ reg=1.0
+ maxiter=100
+ fmt=csv
model=/user/ml/weights.csv
Log=/user/ml/Log.csv
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f m-svm.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs X=/user/ml/X.mtx
+ Y=/user/ml/y.mtx
+ icpt=0
+ tol=0.001
+ reg=1.0
+ maxiter=100
+ fmt=csv
+ model=/user/ml/weights.csv
+ Log=/user/ml/Log.csv
+</div>
+</div>
**Multi-Class Support Vector Machines Prediction**:
- hadoop jar SystemML.jar -f m-svm-predict.dml
- -nvargs X=/user/ml/X.mtx
- Y=/user/ml/y.mtx
- icpt=0
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
+ hadoop jar SystemML.jar -f m-svm-predict.dml
+ -nvargs X=/user/ml/X.mtx
+ Y=/user/ml/y.mtx
+ icpt=0
fmt=csv
model=/user/ml/weights.csv
scores=/user/ml/scores.csv
accuracy=/user/ml/accuracy.csv
confusion=/user/ml/confusion.csv
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f m-svm-predict.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs X=/user/ml/X.mtx
+ Y=/user/ml/y.mtx
+ icpt=0
+ fmt=csv
+ model=/user/ml/weights.csv
+ scores=/user/ml/scores.csv
+ accuracy=/user/ml/accuracy.csv
+ confusion=/user/ml/confusion.csv
+</div>
+</div>
#### Details
@@ -636,6 +852,8 @@ applicable when all features are counts of categorical values.
**Naive Bayes**:
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f naive-bayes.dml
-nvargs X=<file>
Y=<file>
@@ -644,9 +862,29 @@ applicable when all features are counts of categorical values.
conditionals=<file>
accuracy=<file>
fmt=[format]
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f naive-bayes.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs X=<file>
+ Y=<file>
+ laplace=[double]
+ prior=<file>
+ conditionals=<file>
+ accuracy=<file>
+ fmt=[format]
+</div>
+</div>
**Naive Bayes Prediction**:
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f naive-bayes-predict.dml
-nvargs X=<file>
Y=[file]
@@ -656,6 +894,25 @@ applicable when all features are counts of categorical values.
accuracy=[file]
confusion=[file]
probabilities=[file]
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f naive-bayes-predict.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs X=<file>
+ Y=[file]
+ prior=<file>
+ conditionals=<file>
+ fmt=[format]
+ accuracy=[file]
+ confusion=[file]
+ probabilities=[file]
+</div>
+</div>
### Arguments
@@ -698,25 +955,67 @@ SystemML Language Reference for details.
**Naive Bayes**:
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f naive-bayes.dml
- -nvargs X=/user/ml/X.mtx
- Y=/user/ml/y.mtx
- laplace=1 fmt=csv
+ -nvargs X=/user/ml/X.mtx
+ Y=/user/ml/y.mtx
+ laplace=1
+ fmt=csv
prior=/user/ml/prior.csv
conditionals=/user/ml/conditionals.csv
accuracy=/user/ml/accuracy.csv
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f naive-bayes.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs X=/user/ml/X.mtx
+ Y=/user/ml/y.mtx
+ laplace=1
+ fmt=csv
+ prior=/user/ml/prior.csv
+ conditionals=/user/ml/conditionals.csv
+ accuracy=/user/ml/accuracy.csv
+</div>
+</div>
**Naive Bayes Prediction**:
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f naive-bayes-predict.dml
- -nvargs X=/user/ml/X.mtx
- Y=/user/ml/y.mtx
+ -nvargs X=/user/ml/X.mtx
+ Y=/user/ml/y.mtx
prior=/user/ml/prior.csv
conditionals=/user/ml/conditionals.csv
fmt=csv
accuracy=/user/ml/accuracy.csv
probabilities=/user/ml/probabilities.csv
confusion=/user/ml/confusion.csv
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f naive-bayes-predict.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs X=/user/ml/X.mtx
+ Y=/user/ml/y.mtx
+ prior=/user/ml/prior.csv
+ conditionals=/user/ml/conditionals.csv
+ fmt=csv
+ accuracy=/user/ml/accuracy.csv
+ probabilities=/user/ml/probabilities.csv
+ confusion=/user/ml/confusion.csv
+</div>
+</div>
### Details
@@ -781,6 +1080,8 @@ implementation is well-suited to handle large-scale data and builds a
**Decision Tree**:
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f decision-tree.dml
-nvargs X=<file>
Y=<file>
@@ -795,9 +1096,35 @@ implementation is well-suited to handle large-scale data and builds a
S_map=[file]
C_map=[file]
fmt=[format]
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f decision-tree.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs X=<file>
+ Y=<file>
+ R=[file]
+ M=<file>
+ bins=[int]
+ depth=[int]
+ num_leaf=[int]
+ num_samples=[int]
+ impurity=[Gini|entropy]
+ O=[file]
+ S_map=[file]
+ C_map=[file]
+ fmt=[format]
+</div>
+</div>
**Decision Tree Prediction**:
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f decision-tree-predict.dml
-nvargs X=<file>
Y=[file]
@@ -807,6 +1134,25 @@ implementation is well-suited to handle large-scale data and builds a
A=[file]
CM=[file]
fmt=[format]
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f decision-tree-predict.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs X=<file>
+ Y=[file]
+ R=[file]
+ M=<file>
+ P=<file>
+ A=[file]
+ CM=[file]
+ fmt=[format]
+</div>
+</div>
### Arguments
@@ -875,6 +1221,8 @@ SystemML Language Reference for details.
**Decision Tree**:
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f decision-tree.dml
-nvargs X=/user/ml/X.mtx
Y=/user/ml/Y.mtx
@@ -886,9 +1234,32 @@ SystemML Language Reference for details.
num_samples=3000
impurity=Gini
fmt=csv
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f decision-tree.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs X=/user/ml/X.mtx
+ Y=/user/ml/Y.mtx
+ R=/user/ml/R.csv
+ M=/user/ml/model.csv
+ bins=20
+ depth=25
+ num_leaf=10
+ num_samples=3000
+ impurity=Gini
+ fmt=csv
+</div>
+</div>
**Decision Tree Prediction**:
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f decision-tree-predict.dml
-nvargs X=/user/ml/X.mtx
Y=/user/ml/Y.mtx
@@ -898,7 +1269,25 @@ SystemML Language Reference for details.
A=/user/ml/accuracy.csv
CM=/user/ml/confusion.csv
fmt=csv
-
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f decision-tree-predict.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs X=/user/ml/X.mtx
+ Y=/user/ml/Y.mtx
+ R=/user/ml/R.csv
+ M=/user/ml/model.csv
+ P=/user/ml/predictions.csv
+ A=/user/ml/accuracy.csv
+ CM=/user/ml/confusion.csv
+ fmt=csv
+</div>
+</div>
### Details
@@ -1096,6 +1485,8 @@ for classification in parallel.
**Random Forest**:
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f random-forest.dml
-nvargs X=<file>
Y=<file>
@@ -1113,9 +1504,38 @@ for classification in parallel.
S_map=[file]
C_map=[file]
fmt=[format]
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f random-forest.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs X=<file>
+ Y=<file>
+ R=[file]
+ M=<file>
+ bins=[int]
+ depth=[int]
+ num_leaf=[int]
+ num_samples=[int]
+ num_trees=[int]
+ subsamp_rate=[double]
+ feature_subset=[double]
+ impurity=[Gini|entropy]
+ C=[file]
+ S_map=[file]
+ C_map=[file]
+ fmt=[format]
+</div>
+</div>
**Random Forest Prediction**:
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f random-forest-predict.dml
-nvargs X=<file>
Y=[file]
@@ -1127,6 +1547,27 @@ for classification in parallel.
OOB=[file]
CM=[file]
fmt=[format]
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f random-forest-predict.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs X=<file>
+ Y=[file]
+ R=[file]
+ M=<file>
+ C=[file]
+ P=<file>
+ A=[file]
+ OOB=[file]
+ CM=[file]
+ fmt=[format]
+</div>
+</div>
### Arguments
@@ -1215,6 +1656,8 @@ SystemML Language Reference for details.
**Random Forest**:
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f random-forest.dml
-nvargs X=/user/ml/X.mtx
Y=/user/ml/Y.mtx
@@ -1227,11 +1670,35 @@ SystemML Language Reference for details.
num_trees=10
impurity=Gini
fmt=csv
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f random-forest.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs X=/user/ml/X.mtx
+ Y=/user/ml/Y.mtx
+ R=/user/ml/R.csv
+ M=/user/ml/model.csv
+ bins=20
+ depth=25
+ num_leaf=10
+ num_samples=3000
+ num_trees=10
+ impurity=Gini
+ fmt=csv
+</div>
+</div>
**Random Forest Prediction**:
To compute predictions:
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f random-forest-predict.dml
-nvargs X=/user/ml/X.mtx
Y=/user/ml/Y.mtx
@@ -1241,6 +1708,25 @@ To compute predictions:
A=/user/ml/accuracy.csv
CM=/user/ml/confusion.csv
fmt=csv
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f random-forest-predict.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs X=/user/ml/X.mtx
+ Y=/user/ml/Y.mtx
+ R=/user/ml/R.csv
+ M=/user/ml/model.csv
+ P=/user/ml/predictions.csv
+ A=/user/ml/accuracy.csv
+ CM=/user/ml/confusion.csv
+ fmt=csv
+</div>
+</div>
### Details
http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/b8055201/algorithms-clustering.md
----------------------------------------------------------------------
diff --git a/algorithms-clustering.md b/algorithms-clustering.md
index a443083..31a31e2 100644
--- a/algorithms-clustering.md
+++ b/algorithms-clustering.md
@@ -95,6 +95,8 @@ apart is a “false negative” etc.
**K-Means**:
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f Kmeans.dml
-nvargs X=<file>
C=[file]
@@ -107,9 +109,33 @@ apart is a “false negative” etc.
Y=[file]
fmt=[format]
verb=[int]
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f Kmeans.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs X=<file>
+ C=[file]
+ k=<int>
+ runs=[int]
+ maxi=[int]
+ tol=[double]
+ samp=[int]
+ isY=[int]
+ Y=[file]
+ fmt=[format]
+ verb=[int]
+</div>
+</div>
**K-Means Prediction**:
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f Kmeans-predict.dml
-nvargs X=[file]
C=[file]
@@ -117,7 +143,23 @@ apart is a “false negative” etc.
prY=[file]
fmt=[format]
O=[file]
-
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f Kmeans-predict.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs X=[file]
+ C=[file]
+ spY=[file]
+ prY=[file]
+ fmt=[format]
+ O=[file]
+</div>
+</div>
### Arguments - K-Means
@@ -186,12 +228,31 @@ standard output
**K-Means**:
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f Kmeans.dml
-nvargs X=/user/ml/X.mtx
k=5
C=/user/ml/centroids.mtx
fmt=csv
-
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f Kmeans.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs X=/user/ml/X.mtx
+ k=5
+ C=/user/ml/centroids.mtx
+ fmt=csv
+</div>
+</div>
+
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f Kmeans.dml
-nvargs X=/user/ml/X.mtx
k=5
@@ -203,33 +264,104 @@ standard output
isY=1
Y=/user/ml/Yout.mtx
verb=1
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f Kmeans.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs X=/user/ml/X.mtx
+ k=5
+ runs=100
+ maxi=5000
+ tol=0.00000001
+ samp=20
+ C=/user/ml/centroids.mtx
+ isY=1
+ Y=/user/ml/Yout.mtx
+ verb=1
+</div>
+</div>
**K-Means Prediction**:
To predict Y given X and C:
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f Kmeans-predict.dml
-nvargs X=/user/ml/X.mtx
C=/user/ml/C.mtx
prY=/user/ml/PredY.mtx
O=/user/ml/stats.csv
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f Kmeans-predict.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs X=/user/ml/X.mtx
+ C=/user/ml/C.mtx
+ prY=/user/ml/PredY.mtx
+ O=/user/ml/stats.csv
+</div>
+</div>
To compare “actual” labels `spY` with “predicted” labels
given X and C:
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f Kmeans-predict.dml
-nvargs X=/user/ml/X.mtx
C=/user/ml/C.mtx
spY=/user/ml/Y.mtx
O=/user/ml/stats.csv
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f Kmeans-predict.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs X=/user/ml/X.mtx
+ C=/user/ml/C.mtx
+ spY=/user/ml/Y.mtx
+ O=/user/ml/stats.csv
+</div>
+</div>
To compare “actual” labels `spY` with given “predicted”
labels prY:
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f Kmeans-predict.dml
-nvargs spY=/user/ml/Y.mtx
prY=/user/ml/PredY.mtx
O=/user/ml/stats.csv
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f Kmeans-predict.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs spY=/user/ml/Y.mtx
+ prY=/user/ml/PredY.mtx
+ O=/user/ml/stats.csv
+</div>
+</div>
* * *
http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/b8055201/algorithms-descriptive-statistics.md
----------------------------------------------------------------------
diff --git a/algorithms-descriptive-statistics.md b/algorithms-descriptive-statistics.md
index eafb88e..a0dd133 100644
--- a/algorithms-descriptive-statistics.md
+++ b/algorithms-descriptive-statistics.md
@@ -99,10 +99,26 @@ to compute the mean of a categorical attribute like ‘Hair Color’.
### Usage
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f Univar-Stats.dml
- -nvargs X=<file>
+ -nvargs X=<file>
TYPES=<file>
STATS=<file>
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f Univar-Stats.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs X=<file>
+ TYPES=<file>
+ STATS=<file>
+</div>
+</div>
### Arguments
@@ -122,10 +138,26 @@ be stored. The format of the output matrix is defined by
### Examples
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f Univar-Stats.dml
-nvargs X=/user/ml/X.mtx
TYPES=/user/ml/types.mtx
STATS=/user/ml/stats.mtx
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f Univar-Stats.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs X=/user/ml/X.mtx
+ TYPES=/user/ml/types.mtx
+ STATS=/user/ml/stats.mtx
+</div>
+</div>
* * *
@@ -524,6 +556,8 @@ attributes like ‘Hair Color’.
### Usage
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f bivar-stats.dml
-nvargs X=<file>
index1=<file>
@@ -531,6 +565,23 @@ attributes like ‘Hair Color’.
types1=<file>
types2=<file>
OUTDIR=<directory>
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f bivar-stats.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs X=<file>
+ index1=<file>
+ index2=<file>
+ types1=<file>
+ types2=<file>
+ OUTDIR=<directory>
+</div>
+</div>
### Arguments
@@ -574,6 +625,8 @@ are defined in [**Table 2**](algorithms-descriptive-statistics.html#table2).
### Examples
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f bivar-stats.dml
-nvargs X=/user/ml/X.mtx
index1=/user/ml/S1.mtx
@@ -581,7 +634,24 @@ are defined in [**Table 2**](algorithms-descriptive-statistics.html#table2).
types1=/user/ml/K1.mtx
types2=/user/ml/K2.mtx
OUTDIR=/user/ml/stats.mtx
-
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f bivar-stats.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs X=/user/ml/X.mtx
+ index1=/user/ml/S1.mtx
+ index2=/user/ml/S2.mtx
+ types1=/user/ml/K1.mtx
+ types2=/user/ml/K2.mtx
+ OUTDIR=/user/ml/stats.mtx
+</div>
+</div>
+
* * *
@@ -1046,6 +1116,8 @@ becomes reversed and amplified (from $+0.1$ to $-0.5$) if we ignore the months.
### Usage
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f stratstats.dml
-nvargs X=<file>
Xcid=[file]
@@ -1055,6 +1127,25 @@ becomes reversed and amplified (from $+0.1$ to $-0.5$) if we ignore the months.
Scid=[int]
O=<file>
fmt=[format]
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f stratstats.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs X=<file>
+ Xcid=[file]
+ Y=[file]
+ Ycid=[file]
+ S=[file]
+ Scid=[int]
+ O=<file>
+ fmt=[format]
+</div>
+</div>
### Arguments
@@ -1233,21 +1324,61 @@ SystemML Language Reference for details.
### Examples
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f stratstats.dml
-nvargs X=/user/ml/X.mtx
Xcid=/user/ml/Xcid.mtx
Y=/user/ml/Y.mtx
Ycid=/user/ml/Ycid.mtx
- S=/user/ml/S.mtx Scid=2
+ S=/user/ml/S.mtx
+ Scid=2
O=/user/ml/Out.mtx
fmt=csv
-
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f stratstats.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs X=/user/ml/X.mtx
+ Xcid=/user/ml/Xcid.mtx
+ Y=/user/ml/Y.mtx
+ Ycid=/user/ml/Ycid.mtx
+ S=/user/ml/S.mtx
+ Scid=2
+ O=/user/ml/Out.mtx
+ fmt=csv
+</div>
+</div>
+
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f stratstats.dml
-nvargs X=/user/ml/Data.mtx
Xcid=/user/ml/Xcid.mtx
Ycid=/user/ml/Ycid.mtx
Scid=7
O=/user/ml/Out.mtx
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f stratstats.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs X=/user/ml/Data.mtx
+ Xcid=/user/ml/Xcid.mtx
+ Ycid=/user/ml/Ycid.mtx
+ Scid=7
+ O=/user/ml/Out.mtx
+</div>
+</div>
### Details
http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/b8055201/algorithms-matrix-factorization.md
----------------------------------------------------------------------
diff --git a/algorithms-matrix-factorization.md b/algorithms-matrix-factorization.md
index a46a2cd..94627e3 100644
--- a/algorithms-matrix-factorization.md
+++ b/algorithms-matrix-factorization.md
@@ -25,6 +25,8 @@ top-$K$ (for a given value of $K$) principle components.
### Usage
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f PCA.dml
-nvargs INPUT=<file>
K=<int>
@@ -34,6 +36,25 @@ top-$K$ (for a given value of $K$) principle components.
OFMT=[format]
MODEL=<file>
OUTPUT=<file>
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f PCA.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs INPUT=<file>
+ K=<int>
+ CENTER=[int]
+ SCALE=[int]
+ PROJDATA=<int>
+ OFMT=[format]
+ MODEL=<file>
+ OUTPUT=<file>
+</div>
+</div>
#### Arguments
@@ -66,9 +87,10 @@ SystemML Language Reference for details.
vector space.
-
#### Examples
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f PCA.dml
-nvargs INPUT=/user/ml/input.mtx
K=10
@@ -77,7 +99,27 @@ SystemML Language Reference for details.
FMT=csv
PROJDATA=1
OUTPUT=/user/ml/pca_output/
-
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f PCA.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs INPUT=/user/ml/input.mtx
+ K=10
+ CENTER=1
+ SCALE=1O
+ FMT=csv
+ PROJDATA=1
+ OUTPUT=/user/ml/pca_output/
+</div>
+</div>
+
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f PCA.dml
-nvargs INPUT=/user/ml/test_input.mtx
K=10
@@ -87,7 +129,25 @@ SystemML Language Reference for details.
PROJDATA=1
MODEL=/user/ml/pca_output/
OUTPUT=/user/ml/test_output.mtx
-
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f PCA.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs INPUT=/user/ml/test_input.mtx
+ K=10
+ CENTER=1
+ SCALE=1O
+ FMT=csv
+ PROJDATA=1
+ MODEL=/user/ml/pca_output/
+ OUTPUT=/user/ml/test_output.mtx
+</div>
+</div>
#### Details
@@ -164,6 +224,8 @@ problems.
**ALS**:
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f ALS.dml
-nvargs V=<file>
L=<file>
@@ -175,9 +237,32 @@ problems.
check=[boolean]
thr=[double]
fmt=[format]
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f ALS.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs V=<file>
+ L=<file>
+ R=<file>
+ rank=[int]
+ reg=[L2|wL2]
+ lambda=[double]
+ maxi=[int]
+ check=[boolean]
+ thr=[double]
+ fmt=[format]
+</div>
+</div>
**ALS Prediction**:
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f ALS_predict.dml
-nvargs X=<file>
Y=<file>
@@ -186,9 +271,29 @@ problems.
Vrows=<int>
Vcols=<int>
fmt=[format]
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f ALS_predict.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs X=<file>
+ Y=<file>
+ L=<file>
+ R=<file>
+ Vrows=<int>
+ Vcols=<int>
+ fmt=[format]
+</div>
+</div>
**ALS Top-K Prediction**:
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f ALS_topk_predict.dml
-nvargs X=<file>
Y=<file>
@@ -197,6 +302,24 @@ problems.
V=<file>
K=[int]
fmt=[format]
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f ALS_topk_predict.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs X=<file>
+ Y=<file>
+ L=<file>
+ R=<file>
+ V=<file>
+ K=[int]
+ fmt=[format]
+</div>
+</div>
### Arguments - ALS
@@ -275,6 +398,8 @@ SystemML Language Reference for details.
**ALS**:
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f ALS.dml
-nvargs V=/user/ml/V
L=/user/ml/L
@@ -286,12 +411,34 @@ SystemML Language Reference for details.
check=TRUE
thr=0.001
fmt=csv
-
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f ALS.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs V=/user/ml/V
+ L=/user/ml/L
+ R=/user/ml/R
+ rank=10
+ reg="wL"
+ lambda=0.0001
+ maxi=50
+ check=TRUE
+ thr=0.001
+ fmt=csv
+</div>
+</div>
**ALS Prediction**:
To compute predicted ratings for a given list of users and items:
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f ALS_predict.dml
-nvargs X=/user/ml/X
Y=/user/ml/Y
@@ -300,13 +447,32 @@ To compute predicted ratings for a given list of users and items:
Vrows=100000
Vcols=10000
fmt=csv
-
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f ALS_predict.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs X=/user/ml/X
+ Y=/user/ml/Y
+ L=/user/ml/L
+ R=/user/ml/R
+ Vrows=100000
+ Vcols=10000
+ fmt=csv
+</div>
+</div>
**ALS Top-K Prediction**:
To compute top-K items with highest predicted ratings together with the
predicted ratings for a given list of users:
+<div class="codetabs">
+<div data-lang="Hadoop" markdown="1">
hadoop jar SystemML.jar -f ALS_topk_predict.dml
-nvargs X=/user/ml/X
Y=/user/ml/Y
@@ -315,6 +481,24 @@ predicted ratings for a given list of users:
V=/user/ml/V
K=10
fmt=csv
+</div>
+<div data-lang="Spark" markdown="1">
+ $SPARK_HOME/bin/spark-submit --master yarn-cluster
+ --conf spark.driver.maxResultSize=0
+ --conf spark.akka.frameSize=128
+ SystemML.jar
+ -f ALS_topk_predict.dml
+ -config=SystemML-config.xml
+ -exec hybrid_spark
+ -nvargs X=/user/ml/X
+ Y=/user/ml/Y
+ L=/user/ml/L
+ R=/user/ml/R
+ V=/user/ml/V
+ K=10
+ fmt=csv
+</div>
+</div>
### Details