You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@systemml.apache.org by de...@apache.org on 2016/01/12 22:59:32 UTC
[2/2] incubator-systemml git commit: Improve structure of doc
presentation
Improve structure of doc presentation
Update index.md and global.html to present available docs
in a more structured format, while additionally conveying
more information in index.md about the capabilities to be
found in the existing docs.
Add Apache license to hadoop-batch-mode.md.
Change MLContext to Spark MLContext on index.
Change ordering of execution modes.
Rename DML and PyDML Programming Guide to Beginner's Guide.
Include mention of Scala where relevant.
Rename MLContext Programming Guide to include Spark.
Update image paths.
Minor text updates.
Add analytics.
Closes #34.
Project: http://git-wip-us.apache.org/repos/asf/incubator-systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-systemml/commit/65844aa6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-systemml/tree/65844aa6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-systemml/diff/65844aa6
Branch: refs/heads/master
Commit: 65844aa6ddcde223a46a08a87ffd235165108648
Parents: 52fae50
Author: Deron Eriksson <de...@us.ibm.com>
Authored: Tue Jan 12 13:44:25 2016 -0800
Committer: Deron Eriksson <de...@us.ibm.com>
Committed: Tue Jan 12 13:44:25 2016 -0800
----------------------------------------------------------------------
docs/_config.yml | 5 +
docs/_layouts/global.html | 33 +-
docs/beginners-guide-to-dml-and-pydml.md | 889 +++++++++++++++++
docs/dml-and-pydml-programming-guide.md | 889 -----------------
docs/hadoop-batch-mode.md | 18 +
...bug-configuration-hello-world-main-class.png | Bin 0 -> 95393 bytes
...figuration-hello-world-program-arguments.png | Bin 0 -> 85243 bytes
...bug-configuration-hello-world-main-class.png | Bin 95393 -> 0 bytes
...figuration-hello-world-program-arguments.png | Bin 85243 -> 0 bytes
...elin-notebook-systemml-linear-regression.png | Bin 158819 -> 0 bytes
.../zeppelin-notebook.png | Bin 152132 -> 0 bytes
...elin-notebook-systemml-linear-regression.png | Bin 0 -> 158819 bytes
.../zeppelin-notebook.png | Bin 0 -> 152132 bytes
docs/index.md | 68 +-
docs/mlcontext-programming-guide.md | 996 -------------------
docs/spark-mlcontext-programming-guide.md | 996 +++++++++++++++++++
16 files changed, 1983 insertions(+), 1911 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/65844aa6/docs/_config.yml
----------------------------------------------------------------------
diff --git a/docs/_config.yml b/docs/_config.yml
index 0419bff..26ddc8a 100644
--- a/docs/_config.yml
+++ b/docs/_config.yml
@@ -14,3 +14,8 @@ include:
# These allow the documentation to be updated with newer releases
SYSTEMML_VERSION: 0.8.0
+
+# if 'analytics_on' is true, analytics section will be rendered on the HTML pages
+analytics_on: true
+analytics_provider: google_universal
+analytics_google_universal_tracking_id : UA-71553733-1
http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/65844aa6/docs/_layouts/global.html
----------------------------------------------------------------------
diff --git a/docs/_layouts/global.html b/docs/_layouts/global.html
index e6f6eed..89bb168 100644
--- a/docs/_layouts/global.html
+++ b/docs/_layouts/global.html
@@ -31,20 +31,26 @@
<ul class="nav">
<li><a href="index.html">Overview</a></li>
+ <li><a href="https://github.com/apache/incubator-systemml">GitHub</a></li>
+
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation<b class="caret"></b></a>
<ul class="dropdown-menu">
-
+ <li> <b>Running SystemML:</b></li>
<li><a href="https://github.com/apache/incubator-systemml">SystemML GitHub README</a></li>
<li><a href="quick-start-guide.html">Quick Start Guide</a></li>
- <li><a href="dml-and-pydml-programming-guide.html">DML and PyDML Programming Guide</a></li>
- <li><a href="mlcontext-programming-guide.html">MLContext Programming Guide</a></li>
<li><a href="hadoop-batch-mode.html">Hadoop Batch Mode</a>
- <li><a href="debugger-guide.html">Debugger Guide</a></li>
- <li><a href="algorithms-reference.html">Algorithms Reference</a></li>
+ <li><a href="spark-mlcontext-programming-guide.html">Spark MLContext Programming Guide</a></li>
+ <li class="divider"></li>
+ <li> <b>Language Guides:</b></li>
<li><a href="dml-language-reference.html">DML Language Reference</a></li>
+ <li><a href="beginners-guide-to-dml-and-pydml.html">Beginner's Guide to DML and PyDML</a></li>
+ <li class="divider"></li>
+ <li> <b>ML Algorithms:</b></li>
+ <li><a href="algorithms-reference.html">Algorithms Reference</a></li>
<li class="divider"></li>
- <li> More Coming Soon...</li>
+ <li> <b>Tools:</b></li>
+ <li><a href="debugger-guide.html">Debugger Guide</a></li>
</ul>
</li>
@@ -69,6 +75,21 @@
<script src="js/vendor/anchor.min.js"></script>
<script src="js/main.js"></script>
+{% if site.analytics_on == true %}
+{% case site.analytics_provider %}
+{% when "google_universal" %}
+ <!-- Analytics -->
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+ ga('create', '{{ site.analytics_google_universal_tracking_id }}', 'auto');
+ ga('send', 'pageview');
+ </script>
+{% endcase %}
+{% endif %}
+
<!-- MathJax Section -->
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/65844aa6/docs/beginners-guide-to-dml-and-pydml.md
----------------------------------------------------------------------
diff --git a/docs/beginners-guide-to-dml-and-pydml.md b/docs/beginners-guide-to-dml-and-pydml.md
new file mode 100644
index 0000000..611024d
--- /dev/null
+++ b/docs/beginners-guide-to-dml-and-pydml.md
@@ -0,0 +1,889 @@
+---
+layout: global
+title: Beginner's Guide to DML and PyDML
+description: Beginner's Guide to DML and PyDML
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+* This will become a table of contents (this text will be scraped).
+{:toc}
+
+<br/>
+
+
+# Overview
+
+SystemML enables *flexible*, scalable machine learning. This flexibility is achieved
+through the specification of a high-level declarative machine learning language
+that comes in two flavors, one with an R-like syntax (DML) and one with
+a Python-like syntax (PyDML).
+
+Algorithm scripts written in DML and PyDML can be run on Hadoop, on Spark, or
+in Standalone mode. No script modifications are required to change between modes.
+SystemML automatically performs advanced
+optimizations based on data and cluster characteristics, so much of the need to manually
+tweak algorithms is largely reduced or eliminated.
+
+This Beginner's Guide serves as a starting point for writing DML and PyDML
+scripts.
+
+
+# Script Invocation
+
+DML and PyDML scripts can be invoked in a variety of ways. Suppose that we have `hello.dml` and
+`hello.pydml` scripts containing the following:
+
+ print('hello ' + $1)
+
+One way to begin working with SystemML is to build the project and unpack the standalone distribution,
+which features the `runStandaloneSystemML.sh` and `runStandaloneSystemML.bat` scripts. The name of the DML or PyDML script
+is passed as the first argument to these scripts, along with a variety of arguments.
+
+ ./runStandaloneSystemML.sh hello.dml -args world
+ ./runStandaloneSystemML.sh hello.pydml -python -args world
+
+For DML and PyDML script invocations that take multiple arguments, a common technique is to create
+a standard script that invokes `runStandaloneSystemML.sh` or `runStandaloneSystemML.bat` with the arguments specified.
+
+SystemML itself is written in Java and is managed using Maven. As a result, SystemML can readily be
+imported into a standard development environment such as Eclipse.
+The `DMLScript` class serves as the main entrypoint to SystemML. Executing
+`DMLScript` with no arguments displays usage information. A script file can be specified using the `-f` argument.
+
+In Eclipse, a Debug Configuration can be created with `DMLScript` as the Main class and any arguments specified as
+Program arguments. A PyDML script requires the addition of a `-python` switch.
+
+<div class="codetabs2">
+
+<div data-lang="Eclipse Debug Configuration - Main" markdown="1">
+
+</div>
+
+<div data-lang="Eclipse Debug Configuration - Arguments" markdown="1">
+
+</div>
+
+</div>
+
+SystemML contains a default set of configuration information. In addition to this, SystemML looks for a default `./SystemML-config.xml` file in the working directory, where overriding configuration information can be specified. Furthermore, a config file can be specified using the `-config` argument, as in this example:
+
+ -f hello.dml -config=src/main/standalone/SystemML-config.xml -args world
+
+When operating in a distributed environment, it is *highly recommended* that cluster-specific configuration information
+is provided to SystemML via a configuration file for optimal performance.
+
+
+# Data Types
+
+SystemML has four value data types. In DML, these are: **double**, **integer**,
+**string**, and **boolean**. In PyDML, these are: **float**, **int**,
+**str**, and **bool**. In normal usage, the data type of a variable is implicit
+based on its value. Mathematical operations typically operate on
+doubles/floats, whereas integers/ints are typically useful for tasks such as
+iteration and accessing elements in a matrix.
+
+<div class="codetabs2">
+
+<div data-lang="DML" markdown="1">
+{% highlight r %}
+aDouble = 3.0
+bInteger = 2
+print('aDouble = ' + aDouble)
+print('bInteger = ' + bInteger)
+print('aDouble + bInteger = ' + (aDouble + bInteger))
+print('bInteger ^ 3 = ' + (bInteger ^ 3))
+print('aDouble ^ 2 = ' + (aDouble ^ 2))
+
+cBoolean = TRUE
+print('cBoolean = ' + cBoolean)
+print('(2 < 1) = ' + (2 < 1))
+
+dString = 'Hello'
+eString = dString + ' World'
+print('dString = ' + dString)
+print('eString = ' + eString)
+{% endhighlight %}
+</div>
+
+<div data-lang="PyDML" markdown="1">
+{% highlight python %}
+aFloat = 3.0
+bInt = 2
+print('aFloat = ' + aFloat)
+print('bInt = ' + bInt)
+print('aFloat + bInt = ' + (aFloat + bInt))
+print('bInt ** 3 = ' + (bInt ** 3))
+print('aFloat ** 2 = ' + (aFloat ** 2))
+
+cBool = True
+print('cBool = ' + cBool)
+print('(2 < 1) = ' + (2 < 1))
+
+dStr = 'Hello'
+eStr = dStr + ' World'
+print('dStr = ' + dStr)
+print('eStr = ' + eStr)
+{% endhighlight %}
+</div>
+
+<div data-lang="DML Result" markdown="1">
+ aDouble = 3.0
+ bInteger = 2
+ aDouble + bInteger = 5.0
+ bInteger ^ 3 = 8.0
+ aDouble ^ 2 = 9.0
+ cBoolean = TRUE
+ (2 < 1) = FALSE
+ dString = Hello
+ eString = Hello World
+</div>
+
+<div data-lang="PyDML Result" markdown="1">
+ aFloat = 3.0
+ bInt = 2
+ aFloat + bInt = 5.0
+ bInt ** 3 = 8.0
+ aFloat ** 2 = 9.0
+ cBool = TRUE
+ (2 < 1) = FALSE
+ dStr = Hello
+ eStr = Hello World
+</div>
+
+</div>
+
+
+# Matrix Basics
+
+## Creating a Matrix
+
+A matrix can be created in DML using the **`matrix()`** function and in PyDML using the **`full()`**
+function. In the example below, a matrix element is still considered to be of the matrix data type,
+so the value is cast to a scalar in order to print it. Matrix element values are of type **double**/**float**.
+*Note that matrices index from 1 in both DML and PyDML.*
+
+<div class="codetabs2">
+
+<div data-lang="DML" markdown="1">
+{% highlight r %}
+m = matrix("1 2 3 4 5 6 7 8 9 10 11 12", rows=4, cols=3)
+for (i in 1:nrow(m)) {
+ for (j in 1:ncol(m)) {
+ n = m[i,j]
+ print('[' + i + ',' + j + ']:' + as.scalar(n))
+ }
+}
+{% endhighlight %}
+</div>
+
+<div data-lang="PyDML" markdown="1">
+{% highlight python %}
+m = full("1 2 3 4 5 6 7 8 9 10 11 12", rows=4, cols=3)
+for (i in 1:nrow(m)):
+ for (j in 1:ncol(m)):
+ n = m[i,j]
+ print('[' + i + ',' + j + ']:' + scalar(n))
+{% endhighlight %}
+</div>
+
+<div data-lang="Result" markdown="1">
+ [1,1]:1.0
+ [1,2]:2.0
+ [1,3]:3.0
+ [2,1]:4.0
+ [2,2]:5.0
+ [2,3]:6.0
+ [3,1]:7.0
+ [3,2]:8.0
+ [3,3]:9.0
+ [4,1]:10.0
+ [4,2]:11.0
+ [4,3]:12.0
+</div>
+
+</div>
+
+For additional information about the **`matrix()`** and **`full()`** functions, please see the
+DML Language Reference ([Matrix Construction](dml-language-reference.html#matrix-construction-manipulation-and-aggregation-built-in-functions)) and the
+PyDML Language Reference (Matrix Construction).
+
+
+## Saving a Matrix
+
+A matrix can be saved using the **`write()`** function in DML and the **`save()`** function in PyDML. SystemML supports four
+different formats: **`text`** (`i,j,v`), **`mm`** (`Matrix Market`), **`csv`** (`delimiter-separated values`), and **`binary`**.
+
+<div class="codetabs2">
+
+<div data-lang="DML" markdown="1">
+{% highlight r %}
+m = matrix("1 2 3 0 0 0 7 8 9 0 0 0", rows=4, cols=3)
+write(m, "m.txt", format="text")
+write(m, "m.mm", format="mm")
+write(m, "m.csv", format="csv")
+write(m, "m.binary", format="binary")
+{% endhighlight %}
+</div>
+
+<div data-lang="PyDML" markdown="1">
+{% highlight python %}
+m = full("1 2 3 0 0 0 7 8 9 0 0 0", rows=4, cols=3)
+save(m, "m.txt", format="text")
+save(m, "m.mm", format="mm")
+save(m, "m.csv", format="csv")
+save(m, "m.binary", format="binary")
+{% endhighlight %}
+</div>
+
+</div>
+
+Saving a matrix automatically creates a metadata file for each format except for Matrix Market, since Matrix Market contains
+metadata within the *.mm file. All formats are text-based except binary. The contents of the resulting files are shown here.
+
+<div class="codetabs2">
+
+<div data-lang="m.txt" markdown="1">
+ 1 1 1.0
+ 1 2 2.0
+ 1 3 3.0
+ 3 1 7.0
+ 3 2 8.0
+ 3 3 9.0
+</div>
+
+<div data-lang="m.txt.mtd" markdown="1">
+ {
+ "data_type": "matrix"
+ ,"value_type": "double"
+ ,"rows": 4
+ ,"cols": 3
+ ,"nnz": 6
+ ,"format": "text"
+ ,"description": { "author": "SystemML" }
+ }
+</div>
+
+<div data-lang="m.mm" markdown="1">
+ %%MatrixMarket matrix coordinate real general
+ 4 3 6
+ 1 1 1.0
+ 1 2 2.0
+ 1 3 3.0
+ 3 1 7.0
+ 3 2 8.0
+ 3 3 9.0
+</div>
+
+<div data-lang="m.csv" markdown="1">
+ 1.0,2.0,3.0
+ 0,0,0
+ 7.0,8.0,9.0
+ 0,0,0
+</div>
+
+<div data-lang="m.csv.mtd" markdown="1">
+ {
+ "data_type": "matrix"
+ ,"value_type": "double"
+ ,"rows": 4
+ ,"cols": 3
+ ,"nnz": 6
+ ,"format": "csv"
+ ,"header": false
+ ,"sep": ","
+ ,"description": { "author": "SystemML" }
+ }
+</div>
+
+<div data-lang="m.binary" markdown="1">
+ Not text-based
+</div>
+
+<div data-lang="m.binary.mtd" markdown="1">
+ {
+ "data_type": "matrix"
+ ,"value_type": "double"
+ ,"rows": 4
+ ,"cols": 3
+ ,"rows_in_block": 1000
+ ,"cols_in_block": 1000
+ ,"nnz": 6
+ ,"format": "binary"
+ ,"description": { "author": "SystemML" }
+ }
+</div>
+
+</div>
+
+
+## Loading a Matrix
+
+A matrix can be loaded using the **`read()`** function in DML and the **`load()`** function in PyDML. As with saving, SystemML supports four
+formats: **`text`** (`i,j,v`), **`mm`** (`Matrix Market`), **`csv`** (`delimiter-separated values`), and **`binary`**. To read a file, a corresponding
+metadata file is required, except for the Matrix Market format.
+
+<div class="codetabs2">
+
+<div data-lang="DML" markdown="1">
+{% highlight r %}
+m = read("m.txt")
+print("min:" + min(m))
+print("max:" + max(m))
+print("sum:" + sum(m))
+mRowSums = rowSums(m)
+for (i in 1:nrow(mRowSums)) {
+ print("row " + i + " sum:" + as.scalar(mRowSums[i,1]))
+}
+mColSums = colSums(m)
+for (i in 1:ncol(mColSums)) {
+ print("col " + i + " sum:" + as.scalar(mColSums[1,i]))
+}
+{% endhighlight %}
+</div>
+
+<div data-lang="PyDML" markdown="1">
+{% highlight python %}
+m = load("m.txt")
+print("min:" + min(m))
+print("max:" + max(m))
+print("sum:" + sum(m))
+mRowSums = rowSums(m)
+for (i in 1:nrow(mRowSums)):
+ print("row " + i + " sum:" + scalar(mRowSums[i,1]))
+mColSums = colSums(m)
+for (i in 1:ncol(mColSums)):
+ print("col " + i + " sum:" + scalar(mColSums[1,i]))
+{% endhighlight %}
+</div>
+
+<div data-lang="Result" markdown="1">
+ min:0.0
+ max:9.0
+ sum:30.0
+ row 1 sum:6.0
+ row 2 sum:0.0
+ row 3 sum:24.0
+ row 4 sum:0.0
+ col 1 sum:8.0
+ col 2 sum:10.0
+ col 3 sum:12.0
+</div>
+
+</div>
+
+
+## Matrix Operations
+
+DML and PyDML offer a rich set of operators and built-in functions to perform various operations on matrices and scalars.
+Operators and built-in functions are described in great detail in the DML Language Reference
+([Expressions](dml-language-reference.html#expressions), [Built-In Functions](dml-language-reference.html#built-in-functions))
+and the PyDML Language Reference
+(Expressions, Built-In Functions).
+
+In this example, we create a matrix A. Next, we create another matrix B by adding 4 to each element in A. Next, we flip
+B by taking its transpose. We then multiply A and B, represented by matrix C. We create a matrix D with the same number
+of rows and columns as C, and initialize its elements to 5. We then subtract D from C and divide the values of its elements
+by 2 and assign the resulting matrix to D.
+
+This example also shows a user-defined function called `printMatrix()`, which takes a string and matrix as arguments and returns
+nothing.
+
+<div class="codetabs2">
+
+<div data-lang="DML" markdown="1">
+{% highlight r %}
+printMatrix = function(string which, matrix[double] mat) {
+ print(which)
+ for (i in 1:nrow(mat)) {
+ colVals = '| '
+ for (j in 1:ncol(mat)) {
+ n = mat[i,j]
+ colVals = colVals + as.scalar(n) + ' | '
+ }
+ print(colVals)
+ }
+}
+
+A = matrix("1 2 3 4 5 6", rows=3, cols=2)
+z = printMatrix('Matrix A:', A)
+B = A + 4
+B = t(B)
+z = printMatrix('Matrix B:', B)
+C = A %*% B
+z = printMatrix('Matrix C:', C)
+D = matrix(5, rows=nrow(C), cols=ncol(C))
+D = (C - D) / 2
+z = printMatrix('Matrix D:', D)
+
+{% endhighlight %}
+</div>
+
+<div data-lang="PyDML" markdown="1">
+{% highlight python %}
+def printMatrix(which: str, mat: matrix[float]):
+ print(which)
+ for (i in 1:nrow(mat)):
+ colVals = '| '
+ for (j in 1:ncol(mat)):
+ n = mat[i,j]
+ colVals = colVals + scalar(n) + ' | '
+ print(colVals)
+
+A = full("1 2 3 4 5 6", rows=3, cols=2)
+z = printMatrix('Matrix A:', A)
+B = A + 4
+B = transpose(B)
+z = printMatrix('Matrix B:', B)
+C = dot(A, B)
+z = printMatrix('Matrix C:', C)
+D = full(5, rows=nrow(C), cols=ncol(C))
+D = (C - D) / 2
+z = printMatrix('Matrix D:', D)
+
+{% endhighlight %}
+</div>
+
+<div data-lang="Result" markdown="1">
+ Matrix A:
+ | 1.0 | 2.0 |
+ | 3.0 | 4.0 |
+ | 5.0 | 6.0 |
+ Matrix B:
+ | 5.0 | 7.0 | 9.0 |
+ | 6.0 | 8.0 | 10.0 |
+ Matrix C:
+ | 17.0 | 23.0 | 29.0 |
+ | 39.0 | 53.0 | 67.0 |
+ | 61.0 | 83.0 | 105.0 |
+ Matrix D:
+ | 6.0 | 9.0 | 12.0 |
+ | 17.0 | 24.0 | 31.0 |
+ | 28.0 | 39.0 | 50.0 |
+</div>
+
+</div>
+
+
+## Matrix Indexing
+
+The elements in a matrix can be accessed by their row and column indices. In the example below, we have 3x3 matrix A.
+First, we access the element at the third row and third column. Next, we obtain a row slice (vector) of the matrix by
+specifying row 2 and leaving the column blank. We obtain a column slice (vector) by leaving the row blank and specifying
+column 3. After that, we obtain a submatrix via range indexing, where we specify rows 2 to 3, separated by a colon, and columns
+1 to 2, separated by a colon.
+
+<div class="codetabs2">
+
+<div data-lang="DML" markdown="1">
+{% highlight r %}
+printMatrix = function(string which, matrix[double] mat) {
+ print(which)
+ for (i in 1:nrow(mat)) {
+ colVals = '| '
+ for (j in 1:ncol(mat)) {
+ n = mat[i,j]
+ colVals = colVals + as.scalar(n) + ' | '
+ }
+ print(colVals)
+ }
+}
+
+A = matrix("1 2 3 4 5 6 7 8 9", rows=3, cols=3)
+z = printMatrix('Matrix A:', A)
+B = A[3,3]
+z = printMatrix('Matrix B:', B)
+C = A[2,]
+z = printMatrix('Matrix C:', C)
+D = A[,3]
+z = printMatrix('Matrix D:', D)
+E = A[2:3,1:2]
+z = printMatrix('Matrix E:', E)
+
+{% endhighlight %}
+</div>
+
+<div data-lang="PyDML" markdown="1">
+{% highlight python %}
+def printMatrix(which: str, mat: matrix[float]):
+ print(which)
+ for (i in 1:nrow(mat)):
+ colVals = '| '
+ for (j in 1:ncol(mat)):
+ n = mat[i,j]
+ colVals = colVals + scalar(n) + ' | '
+ print(colVals)
+
+A = full("1 2 3 4 5 6 7 8 9", rows=3, cols=3)
+z = printMatrix('Matrix A:', A)
+B = A[3,3]
+z = printMatrix('Matrix B:', B)
+C = A[2,]
+z = printMatrix('Matrix C:', C)
+D = A[,3]
+z = printMatrix('Matrix D:', D)
+E = A[2:3,1:2]
+z = printMatrix('Matrix E:', E)
+
+{% endhighlight %}
+</div>
+
+<div data-lang="Result" markdown="1">
+ Matrix A:
+ | 1.0 | 2.0 | 3.0 |
+ | 4.0 | 5.0 | 6.0 |
+ | 7.0 | 8.0 | 9.0 |
+ Matrix B:
+ | 9.0 |
+ Matrix C:
+ | 4.0 | 5.0 | 6.0 |
+ Matrix D:
+ | 3.0 |
+ | 6.0 |
+ | 9.0 |
+ Matrix E:
+ | 4.0 | 5.0 |
+ | 7.0 | 8.0 |
+</div>
+
+</div>
+
+
+# Control Statements
+
+DML and PyDML both feature `if` and `if-else` conditional statements. In addition, DML features `else-if` which avoids the
+need for nested conditional statements.
+
+DML and PyDML feature 3 loop statements: `while`, `for`, and `parfor` (parallel for). In the example, note that the
+`print` statements within the `parfor` loop can occur in any order since the iterations occur in parallel rather than
+sequentially as in a regular `for` loop. The `parfor` statement can include several optional parameters, as described
+in the DML Language Reference ([ParFor Statement](dml-language-reference.html#parfor-statement)) and PyDML Language Reference (ParFor Statement).
+
+<div class="codetabs2">
+
+<div data-lang="DML" markdown="1">
+{% highlight r %}
+i = 1
+while (i < 3) {
+ if (i == 1) {
+ print('hello')
+ } else {
+ print('world')
+ }
+ i = i + 1
+}
+
+A = matrix("1 2 3 4 5 6", rows=3, cols=2)
+
+for (i in 1:nrow(A)) {
+ print("for A[" + i + ",1]:" + as.scalar(A[i,1]))
+}
+
+parfor(i in 1:nrow(A)) {
+ print("parfor A[" + i + ",1]:" + as.scalar(A[i,1]))
+}
+{% endhighlight %}
+</div>
+
+<div data-lang="PyDML" markdown="1">
+{% highlight python %}
+i = 1
+while (i < 3):
+ if (i == 1):
+ print('hello')
+ else:
+ print('world')
+ i = i + 1
+
+A = full("1 2 3 4 5 6", rows=3, cols=2)
+
+for (i in 1:nrow(A)):
+ print("for A[" + i + ",1]:" + scalar(A[i,1]))
+
+parfor(i in 1:nrow(A)):
+ print("parfor A[" + i + ",1]:" + scalar(A[i,1]))
+{% endhighlight %}
+</div>
+
+<div data-lang="Result" markdown="1">
+ hello
+ world
+ for A[1,1]:1.0
+ for A[2,1]:3.0
+ for A[3,1]:5.0
+ parfor A[2,1]:3.0
+ parfor A[1,1]:1.0
+ parfor A[3,1]:5.0
+</div>
+
+</div>
+
+
+# User-Defined Functions
+
+Functions encapsulate useful functionality in SystemML. In addition to built-in functions, users can define their own functions.
+Functions take 0 or more parameters and return 0 or more values.
+Currently, if a function returns nothing, it still needs to be assigned to a variable.
+
+<div class="codetabs2">
+
+<div data-lang="DML" markdown="1">
+{% highlight r %}
+doSomething = function(matrix[double] mat) return (matrix[double] ret) {
+ additionalCol = matrix(1, rows=nrow(mat), cols=1) # 1x3 matrix with 1 values
+ ret = cbind(mat, additionalCol) # concatenate column to matrix
+ ret = cbind(ret, seq(0, 2, 1)) # concatenate column (0,1,2) to matrix
+ ret = cbind(ret, rowMaxs(ret)) # concatenate column of max row values to matrix
+ ret = cbind(ret, rowSums(ret)) # concatenate column of row sums to matrix
+}
+
+A = rand(rows=3, cols=2, min=0, max=2) # random 3x2 matrix with values 0 to 2
+B = doSomething(A)
+write(A, "A.csv", format="csv")
+write(B, "B.csv", format="csv")
+{% endhighlight %}
+</div>
+
+<div data-lang="PyDML" markdown="1">
+{% highlight python %}
+def doSomething(mat: matrix[float]) -> (ret: matrix[float]):
+ additionalCol = full(1, rows=nrow(mat), cols=1) # 1x3 matrix with 1 values
+ ret = cbind(mat, additionalCol) # concatenate column to matrix
+ ret = cbind(ret, seq(0, 2, 1)) # concatenate column (0,1,2) to matrix
+ ret = cbind(ret, rowMaxs(ret)) # concatenate column of max row values to matrix
+ ret = cbind(ret, rowSums(ret)) # concatenate column of row sums to matrix
+
+A = rand(rows=3, cols=2, min=0, max=2) # random 3x2 matrix with values 0 to 2
+B = doSomething(A)
+save(A, "A.csv", format="csv")
+save(B, "B.csv", format="csv")
+{% endhighlight %}
+</div>
+
+</div>
+
+In the above example, a 3x2 matrix of random doubles between 0 and 2 is created using the **`rand()`** function.
+Additional parameters can be passed to **`rand()`** to control sparsity and other matrix characteristics.
+
+Matrix A is passed to the `doSomething` function. A column of 1 values is concatenated to the matrix. A column
+consisting of the values `(0, 1, 2)` is concatenated to the matrix. Next, a column consisting of the maximum row values
+is concatenated to the matrix. A column consisting of the row sums is concatenated to the matrix, and this resulting
+matrix is returned to variable B. Matrix A is output to the `A.csv` file and matrix B is saved as the `B.csv` file.
+
+
+<div class="codetabs2">
+
+<div data-lang="A.csv" markdown="1">
+ 1.6091961493071,0.7088614208099939
+ 0.5984862383600267,1.5732118950764993
+ 0.2947607068519842,1.9081406573366781
+</div>
+
+<div data-lang="B.csv" markdown="1">
+ 1.6091961493071,0.7088614208099939,1.0,0,1.6091961493071,4.927253719424194
+ 0.5984862383600267,1.5732118950764993,1.0,1.0,1.5732118950764993,5.744910028513026
+ 0.2947607068519842,1.9081406573366781,1.0,2.0,2.0,7.202901364188662
+</div>
+
+</div>
+
+
+# Command-Line Arguments and Default Values
+
+Command-line arguments can be passed to DML and PyDML scripts either as named arguments or as positional arguments. Named
+arguments are the preferred technique. Named arguments can be passed utilizing the `-nvargs` switch, and positional arguments
+can be passed using the `-args` switch.
+
+Default values can be set using the **`ifdef()`** function.
+
+In the example below, a matrix is read from the file system using named argument `M`. The number of rows to print is specified
+using the `rowsToPrint` argument, which defaults to 2 if no argument is supplied. Likewise, the number of columns is
+specified using `colsToPrint` with a default value of 2.
+
+<div class="codetabs2">
+
+<div data-lang="DML" markdown="1">
+{% highlight r %}
+
+fileM = $M
+
+numRowsToPrint = ifdef($rowsToPrint, 2) # default to 2
+numColsToPrint = ifdef($colsToPrint, 2) # default to 2
+
+m = read(fileM)
+
+for (i in 1:numRowsToPrint) {
+ for (j in 1:numColsToPrint) {
+ print('[' + i + ',' + j + ']:' + as.scalar(m[i,j]))
+ }
+}
+
+{% endhighlight %}
+</div>
+
+<div data-lang="PyDML" markdown="1">
+{% highlight python %}
+
+fileM = $M
+
+numRowsToPrint = ifdef($rowsToPrint, 2) # default to 2
+numColsToPrint = ifdef($colsToPrint, 2) # default to 2
+
+m = load(fileM)
+
+for (i in 1:numRowsToPrint):
+ for (j in 1:numColsToPrint):
+ print('[' + i + ',' + j + ']:' + scalar(m[i,j]))
+
+{% endhighlight %}
+</div>
+
+<div data-lang="DML Named Arguments and Results" markdown="1">
+ Example #1 Arguments:
+ -f ex.dml -nvargs M=M.txt rowsToPrint=1 colsToPrint=3
+
+ Example #1 Results:
+ [1,1]:1.0
+ [1,2]:2.0
+ [1,3]:3.0
+
+ Example #2 Arguments:
+ -f ex.dml -nvargs M=M.txt
+
+ Example #2 Results:
+ [1,1]:1.0
+ [1,2]:2.0
+ [2,1]:0.0
+ [2,2]:0.0
+
+</div>
+
+<div data-lang="PyDML Named Arguments and Results" markdown="1">
+ Example #1 Arguments:
+ -f ex.pydml -python -nvargs M=M.txt rowsToPrint=1 colsToPrint=3
+
+ Example #1 Results:
+ [1,1]:1.0
+ [1,2]:2.0
+ [1,3]:3.0
+
+ Example #2 Arguments:
+ -f ex.pydml -python -nvargs M=M.txt
+
+ Example #2 Results:
+ [1,1]:1.0
+ [1,2]:2.0
+ [2,1]:0.0
+ [2,2]:0.0
+
+</div>
+
+</div>
+
+Here, we see identical functionality but with positional arguments.
+
+<div class="codetabs2">
+
+<div data-lang="DML" markdown="1">
+{% highlight r %}
+
+fileM = $1
+
+numRowsToPrint = ifdef($2, 2) # default to 2
+numColsToPrint = ifdef($3, 2) # default to 2
+
+m = read(fileM)
+
+for (i in 1:numRowsToPrint) {
+ for (j in 1:numColsToPrint) {
+ print('[' + i + ',' + j + ']:' + as.scalar(m[i,j]))
+ }
+}
+
+{% endhighlight %}
+</div>
+
+<div data-lang="PyDML" markdown="1">
+{% highlight python %}
+
+fileM = $1
+
+numRowsToPrint = ifdef($2, 2) # default to 2
+numColsToPrint = ifdef($3, 2) # default to 2
+
+m = load(fileM)
+
+for (i in 1:numRowsToPrint):
+ for (j in 1:numColsToPrint):
+ print('[' + i + ',' + j + ']:' + scalar(m[i,j]))
+
+{% endhighlight %}
+</div>
+
+<div data-lang="DML Positional Arguments and Results" markdown="1">
+ Example #1 Arguments:
+ -f ex.dml -args M.txt 1 3
+
+ Example #1 Results:
+ [1,1]:1.0
+ [1,2]:2.0
+ [1,3]:3.0
+
+ Example #2 Arguments:
+ -f ex.dml -args M.txt
+
+ Example #2 Results:
+ [1,1]:1.0
+ [1,2]:2.0
+ [2,1]:0.0
+ [2,2]:0.0
+
+</div>
+
+<div data-lang="PyDML Positional Arguments and Results" markdown="1">
+ Example #1 Arguments:
+ -f ex.pydml -python -args M.txt 1 3
+
+ Example #1 Results:
+ [1,1]:1.0
+ [1,2]:2.0
+ [1,3]:3.0
+
+ Example #2 Arguments:
+ -f ex.pydml -python -args M.txt
+
+ Example #2 Results:
+ [1,1]:1.0
+ [1,2]:2.0
+ [2,1]:0.0
+ [2,2]:0.0
+
+</div>
+
+</div>
+
+
+# Additional Information
+
+The [DML Language Reference](dml-language-reference.html) and PyDML Language Reference contain highly detailed information regard DML
+and PyDML.
+
+In addition, many excellent examples of DML and PyDML can be found in the `system-ml/scripts` and
+`system-ml/test/scripts/applications` directories.
+
http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/65844aa6/docs/dml-and-pydml-programming-guide.md
----------------------------------------------------------------------
diff --git a/docs/dml-and-pydml-programming-guide.md b/docs/dml-and-pydml-programming-guide.md
deleted file mode 100644
index 2745d3e..0000000
--- a/docs/dml-and-pydml-programming-guide.md
+++ /dev/null
@@ -1,889 +0,0 @@
----
-layout: global
-title: DML and PyDML Programming Guide
-description: DML and PyDML Programming Guide
----
-<!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-* This will become a table of contents (this text will be scraped).
-{:toc}
-
-<br/>
-
-
-# Overview
-
-SystemML enables *flexible*, scalable machine learning. This flexibility is achieved
-through the specification of a high-level declarative machine learning language
-that comes in two flavors, one with an R-like syntax (DML) and one with
-a Python-like syntax (PyDML).
-
-Algorithm scripts written in DML and PyDML can be run on Hadoop, on Spark, or
-in Standalone mode. No script modifications are required to change between modes.
-SystemML automatically performs advanced
-optimizations based on data and cluster characteristics, so much of the need to manually
-tweak algorithms is largely reduced or eliminated.
-
-This SystemML Programming Guide serves as a starting point for writing DML and PyDML
-scripts.
-
-
-# Script Invocation
-
-DML and PyDML scripts can be invoked in a variety of ways. Suppose that we have `hello.dml` and
-`hello.pydml` scripts containing the following:
-
- print('hello ' + $1)
-
-One way to begin working with SystemML is to build the project and unpack the standalone distribution,
-which features the `runStandaloneSystemML.sh` and `runStandaloneSystemML.bat` scripts. The name of the DML or PyDML script
-is passed as the first argument to these scripts, along with a variety of arguments.
-
- ./runStandaloneSystemML.sh hello.dml -args world
- ./runStandaloneSystemML.sh hello.pydml -python -args world
-
-For DML and PyDML script invocations that take multiple arguments, a common technique is to create
-a standard script that invokes `runStandaloneSystemML.sh` or `runStandaloneSystemML.bat` with the arguments specified.
-
-SystemML itself is written in Java and is managed using Maven. As a result, SystemML can readily be
-imported into a standard development environment such as Eclipse.
-The `DMLScript` class serves as the main entrypoint to SystemML. Executing
-`DMLScript` with no arguments displays usage information. A script file can be specified using the `-f` argument.
-
-In Eclipse, a Debug Configuration can be created with `DMLScript` as the Main class and any arguments specified as
-Program arguments. A PyDML script requires the addition of a `-python` switch.
-
-<div class="codetabs2">
-
-<div data-lang="Eclipse Debug Configuration - Main" markdown="1">
-
-</div>
-
-<div data-lang="Eclipse Debug Configuration - Arguments" markdown="1">
-
-</div>
-
-</div>
-
-SystemML contains a default set of configuration information. In addition to this, SystemML looks for a default `./SystemML-config.xml` file in the working directory, where overriding configuration information can be specified. Furthermore, a config file can be specified using the `-config` argument, as in this example:
-
- -f hello.dml -config=src/main/standalone/SystemML-config.xml -args world
-
-When operating in a distributed environment, it is *highly recommended* that cluster-specific configuration information
-is provided to SystemML via a configuration file for optimal performance.
-
-
-# Data Types
-
-SystemML has four value data types. In DML, these are: **double**, **integer**,
-**string**, and **boolean**. In PyDML, these are: **float**, **int**,
-**str**, and **bool**. In normal usage, the data type of a variable is implicit
-based on its value. Mathematical operations typically operate on
-doubles/floats, whereas integers/ints are typically useful for tasks such as
-iteration and accessing elements in a matrix.
-
-<div class="codetabs2">
-
-<div data-lang="DML" markdown="1">
-{% highlight r %}
-aDouble = 3.0
-bInteger = 2
-print('aDouble = ' + aDouble)
-print('bInteger = ' + bInteger)
-print('aDouble + bInteger = ' + (aDouble + bInteger))
-print('bInteger ^ 3 = ' + (bInteger ^ 3))
-print('aDouble ^ 2 = ' + (aDouble ^ 2))
-
-cBoolean = TRUE
-print('cBoolean = ' + cBoolean)
-print('(2 < 1) = ' + (2 < 1))
-
-dString = 'Hello'
-eString = dString + ' World'
-print('dString = ' + dString)
-print('eString = ' + eString)
-{% endhighlight %}
-</div>
-
-<div data-lang="PyDML" markdown="1">
-{% highlight python %}
-aFloat = 3.0
-bInt = 2
-print('aFloat = ' + aFloat)
-print('bInt = ' + bInt)
-print('aFloat + bInt = ' + (aFloat + bInt))
-print('bInt ** 3 = ' + (bInt ** 3))
-print('aFloat ** 2 = ' + (aFloat ** 2))
-
-cBool = True
-print('cBool = ' + cBool)
-print('(2 < 1) = ' + (2 < 1))
-
-dStr = 'Hello'
-eStr = dStr + ' World'
-print('dStr = ' + dStr)
-print('eStr = ' + eStr)
-{% endhighlight %}
-</div>
-
-<div data-lang="DML Result" markdown="1">
- aDouble = 3.0
- bInteger = 2
- aDouble + bInteger = 5.0
- bInteger ^ 3 = 8.0
- aDouble ^ 2 = 9.0
- cBoolean = TRUE
- (2 < 1) = FALSE
- dString = Hello
- eString = Hello World
-</div>
-
-<div data-lang="PyDML Result" markdown="1">
- aFloat = 3.0
- bInt = 2
- aFloat + bInt = 5.0
- bInt ** 3 = 8.0
- aFloat ** 2 = 9.0
- cBool = TRUE
- (2 < 1) = FALSE
- dStr = Hello
- eStr = Hello World
-</div>
-
-</div>
-
-
-# Matrix Basics
-
-## Creating a Matrix
-
-A matrix can be created in DML using the **`matrix()`** function and in PyDML using the **`full()`**
-function. In the example below, a matrix element is still considered to be of the matrix data type,
-so the value is cast to a scalar in order to print it. Matrix element values are of type **double**/**float**.
-*Note that matrices index from 1 in both DML and PyDML.*
-
-<div class="codetabs2">
-
-<div data-lang="DML" markdown="1">
-{% highlight r %}
-m = matrix("1 2 3 4 5 6 7 8 9 10 11 12", rows=4, cols=3)
-for (i in 1:nrow(m)) {
- for (j in 1:ncol(m)) {
- n = m[i,j]
- print('[' + i + ',' + j + ']:' + as.scalar(n))
- }
-}
-{% endhighlight %}
-</div>
-
-<div data-lang="PyDML" markdown="1">
-{% highlight python %}
-m = full("1 2 3 4 5 6 7 8 9 10 11 12", rows=4, cols=3)
-for (i in 1:nrow(m)):
- for (j in 1:ncol(m)):
- n = m[i,j]
- print('[' + i + ',' + j + ']:' + scalar(n))
-{% endhighlight %}
-</div>
-
-<div data-lang="Result" markdown="1">
- [1,1]:1.0
- [1,2]:2.0
- [1,3]:3.0
- [2,1]:4.0
- [2,2]:5.0
- [2,3]:6.0
- [3,1]:7.0
- [3,2]:8.0
- [3,3]:9.0
- [4,1]:10.0
- [4,2]:11.0
- [4,3]:12.0
-</div>
-
-</div>
-
-For additional information about the **`matrix()`** and **`full()`** functions, please see the
-DML Language Reference ([Matrix Construction](dml-language-reference.html#matrix-construction-manipulation-and-aggregation-built-in-functions)) and the
-PyDML Language Reference (Matrix Construction).
-
-
-## Saving a Matrix
-
-A matrix can be saved using the **`write()`** function in DML and the **`save()`** function in PyDML. SystemML supports four
-different formats: **`text`** (`i,j,v`), **`mm`** (`Matrix Market`), **`csv`** (`delimiter-separated values`), and **`binary`**.
-
-<div class="codetabs2">
-
-<div data-lang="DML" markdown="1">
-{% highlight r %}
-m = matrix("1 2 3 0 0 0 7 8 9 0 0 0", rows=4, cols=3)
-write(m, "m.txt", format="text")
-write(m, "m.mm", format="mm")
-write(m, "m.csv", format="csv")
-write(m, "m.binary", format="binary")
-{% endhighlight %}
-</div>
-
-<div data-lang="PyDML" markdown="1">
-{% highlight python %}
-m = full("1 2 3 0 0 0 7 8 9 0 0 0", rows=4, cols=3)
-save(m, "m.txt", format="text")
-save(m, "m.mm", format="mm")
-save(m, "m.csv", format="csv")
-save(m, "m.binary", format="binary")
-{% endhighlight %}
-</div>
-
-</div>
-
-Saving a matrix automatically creates a metadata file for each format except for Matrix Market, since Matrix Market contains
-metadata within the *.mm file. All formats are text-based except binary. The contents of the resulting files are shown here.
-
-<div class="codetabs2">
-
-<div data-lang="m.txt" markdown="1">
- 1 1 1.0
- 1 2 2.0
- 1 3 3.0
- 3 1 7.0
- 3 2 8.0
- 3 3 9.0
-</div>
-
-<div data-lang="m.txt.mtd" markdown="1">
- {
- "data_type": "matrix"
- ,"value_type": "double"
- ,"rows": 4
- ,"cols": 3
- ,"nnz": 6
- ,"format": "text"
- ,"description": { "author": "SystemML" }
- }
-</div>
-
-<div data-lang="m.mm" markdown="1">
- %%MatrixMarket matrix coordinate real general
- 4 3 6
- 1 1 1.0
- 1 2 2.0
- 1 3 3.0
- 3 1 7.0
- 3 2 8.0
- 3 3 9.0
-</div>
-
-<div data-lang="m.csv" markdown="1">
- 1.0,2.0,3.0
- 0,0,0
- 7.0,8.0,9.0
- 0,0,0
-</div>
-
-<div data-lang="m.csv.mtd" markdown="1">
- {
- "data_type": "matrix"
- ,"value_type": "double"
- ,"rows": 4
- ,"cols": 3
- ,"nnz": 6
- ,"format": "csv"
- ,"header": false
- ,"sep": ","
- ,"description": { "author": "SystemML" }
- }
-</div>
-
-<div data-lang="m.binary" markdown="1">
- Not text-based
-</div>
-
-<div data-lang="m.binary.mtd" markdown="1">
- {
- "data_type": "matrix"
- ,"value_type": "double"
- ,"rows": 4
- ,"cols": 3
- ,"rows_in_block": 1000
- ,"cols_in_block": 1000
- ,"nnz": 6
- ,"format": "binary"
- ,"description": { "author": "SystemML" }
- }
-</div>
-
-</div>
-
-
-## Loading a Matrix
-
-A matrix can be loaded using the **`read()`** function in DML and the **`load()`** function in PyDML. As with saving, SystemML supports four
-formats: **`text`** (`i,j,v`), **`mm`** (`Matrix Market`), **`csv`** (`delimiter-separated values`), and **`binary`**. To read a file, a corresponding
-metadata file is required, except for the Matrix Market format.
-
-<div class="codetabs2">
-
-<div data-lang="DML" markdown="1">
-{% highlight r %}
-m = read("m.txt")
-print("min:" + min(m))
-print("max:" + max(m))
-print("sum:" + sum(m))
-mRowSums = rowSums(m)
-for (i in 1:nrow(mRowSums)) {
- print("row " + i + " sum:" + as.scalar(mRowSums[i,1]))
-}
-mColSums = colSums(m)
-for (i in 1:ncol(mColSums)) {
- print("col " + i + " sum:" + as.scalar(mColSums[1,i]))
-}
-{% endhighlight %}
-</div>
-
-<div data-lang="PyDML" markdown="1">
-{% highlight python %}
-m = load("m.txt")
-print("min:" + min(m))
-print("max:" + max(m))
-print("sum:" + sum(m))
-mRowSums = rowSums(m)
-for (i in 1:nrow(mRowSums)):
- print("row " + i + " sum:" + scalar(mRowSums[i,1]))
-mColSums = colSums(m)
-for (i in 1:ncol(mColSums)):
- print("col " + i + " sum:" + scalar(mColSums[1,i]))
-{% endhighlight %}
-</div>
-
-<div data-lang="Result" markdown="1">
- min:0.0
- max:9.0
- sum:30.0
- row 1 sum:6.0
- row 2 sum:0.0
- row 3 sum:24.0
- row 4 sum:0.0
- col 1 sum:8.0
- col 2 sum:10.0
- col 3 sum:12.0
-</div>
-
-</div>
-
-
-## Matrix Operations
-
-DML and PyDML offer a rich set of operators and built-in functions to perform various operations on matrices and scalars.
-Operators and built-in functions are described in great detail in the DML Language Reference
-([Expressions](dml-language-reference.html#expressions), [Built-In Functions](dml-language-reference.html#built-in-functions))
-and the PyDML Language Reference
-(Expressions, Built-In Functions).
-
-In this example, we create a matrix A. Next, we create another matrix B by adding 4 to each element in A. Next, we flip
-B by taking its transpose. We then multiply A and B, represented by matrix C. We create a matrix D with the same number
-of rows and columns as C, and initialize its elements to 5. We then subtract D from C and divide the values of its elements
-by 2 and assign the resulting matrix to D.
-
-This example also shows a user-defined function called `printMatrix()`, which takes a string and matrix as arguments and returns
-nothing.
-
-<div class="codetabs2">
-
-<div data-lang="DML" markdown="1">
-{% highlight r %}
-printMatrix = function(string which, matrix[double] mat) {
- print(which)
- for (i in 1:nrow(mat)) {
- colVals = '| '
- for (j in 1:ncol(mat)) {
- n = mat[i,j]
- colVals = colVals + as.scalar(n) + ' | '
- }
- print(colVals)
- }
-}
-
-A = matrix("1 2 3 4 5 6", rows=3, cols=2)
-z = printMatrix('Matrix A:', A)
-B = A + 4
-B = t(B)
-z = printMatrix('Matrix B:', B)
-C = A %*% B
-z = printMatrix('Matrix C:', C)
-D = matrix(5, rows=nrow(C), cols=ncol(C))
-D = (C - D) / 2
-z = printMatrix('Matrix D:', D)
-
-{% endhighlight %}
-</div>
-
-<div data-lang="PyDML" markdown="1">
-{% highlight python %}
-def printMatrix(which: str, mat: matrix[float]):
- print(which)
- for (i in 1:nrow(mat)):
- colVals = '| '
- for (j in 1:ncol(mat)):
- n = mat[i,j]
- colVals = colVals + scalar(n) + ' | '
- print(colVals)
-
-A = full("1 2 3 4 5 6", rows=3, cols=2)
-z = printMatrix('Matrix A:', A)
-B = A + 4
-B = transpose(B)
-z = printMatrix('Matrix B:', B)
-C = dot(A, B)
-z = printMatrix('Matrix C:', C)
-D = full(5, rows=nrow(C), cols=ncol(C))
-D = (C - D) / 2
-z = printMatrix('Matrix D:', D)
-
-{% endhighlight %}
-</div>
-
-<div data-lang="Result" markdown="1">
- Matrix A:
- | 1.0 | 2.0 |
- | 3.0 | 4.0 |
- | 5.0 | 6.0 |
- Matrix B:
- | 5.0 | 7.0 | 9.0 |
- | 6.0 | 8.0 | 10.0 |
- Matrix C:
- | 17.0 | 23.0 | 29.0 |
- | 39.0 | 53.0 | 67.0 |
- | 61.0 | 83.0 | 105.0 |
- Matrix D:
- | 6.0 | 9.0 | 12.0 |
- | 17.0 | 24.0 | 31.0 |
- | 28.0 | 39.0 | 50.0 |
-</div>
-
-</div>
-
-
-## Matrix Indexing
-
-The elements in a matrix can be accessed by their row and column indices. In the example below, we have 3x3 matrix A.
-First, we access the element at the third row and third column. Next, we obtain a row slice (vector) of the matrix by
-specifying row 2 and leaving the column blank. We obtain a column slice (vector) by leaving the row blank and specifying
-column 3. After that, we obtain a submatrix via range indexing, where we specify rows 2 to 3, separated by a colon, and columns
-1 to 2, separated by a colon.
-
-<div class="codetabs2">
-
-<div data-lang="DML" markdown="1">
-{% highlight r %}
-printMatrix = function(string which, matrix[double] mat) {
- print(which)
- for (i in 1:nrow(mat)) {
- colVals = '| '
- for (j in 1:ncol(mat)) {
- n = mat[i,j]
- colVals = colVals + as.scalar(n) + ' | '
- }
- print(colVals)
- }
-}
-
-A = matrix("1 2 3 4 5 6 7 8 9", rows=3, cols=3)
-z = printMatrix('Matrix A:', A)
-B = A[3,3]
-z = printMatrix('Matrix B:', B)
-C = A[2,]
-z = printMatrix('Matrix C:', C)
-D = A[,3]
-z = printMatrix('Matrix D:', D)
-E = A[2:3,1:2]
-z = printMatrix('Matrix E:', E)
-
-{% endhighlight %}
-</div>
-
-<div data-lang="PyDML" markdown="1">
-{% highlight python %}
-def printMatrix(which: str, mat: matrix[float]):
- print(which)
- for (i in 1:nrow(mat)):
- colVals = '| '
- for (j in 1:ncol(mat)):
- n = mat[i,j]
- colVals = colVals + scalar(n) + ' | '
- print(colVals)
-
-A = full("1 2 3 4 5 6 7 8 9", rows=3, cols=3)
-z = printMatrix('Matrix A:', A)
-B = A[3,3]
-z = printMatrix('Matrix B:', B)
-C = A[2,]
-z = printMatrix('Matrix C:', C)
-D = A[,3]
-z = printMatrix('Matrix D:', D)
-E = A[2:3,1:2]
-z = printMatrix('Matrix E:', E)
-
-{% endhighlight %}
-</div>
-
-<div data-lang="Result" markdown="1">
- Matrix A:
- | 1.0 | 2.0 | 3.0 |
- | 4.0 | 5.0 | 6.0 |
- | 7.0 | 8.0 | 9.0 |
- Matrix B:
- | 9.0 |
- Matrix C:
- | 4.0 | 5.0 | 6.0 |
- Matrix D:
- | 3.0 |
- | 6.0 |
- | 9.0 |
- Matrix E:
- | 4.0 | 5.0 |
- | 7.0 | 8.0 |
-</div>
-
-</div>
-
-
-# Control Statements
-
-DML and PyDML both feature `if` and `if-else` conditional statements. In addition, DML features `else-if` which avoids the
-need for nested conditional statements.
-
-DML and PyDML feature 3 loop statements: `while`, `for`, and `parfor` (parallel for). In the example, note that the
-`print` statements within the `parfor` loop can occur in any order since the iterations occur in parallel rather than
-sequentially as in a regular `for` loop. The `parfor` statement can include several optional parameters, as described
-in the DML Language Reference ([ParFor Statement](dml-language-reference.html#parfor-statement)) and PyDML Language Reference (ParFor Statement).
-
-<div class="codetabs2">
-
-<div data-lang="DML" markdown="1">
-{% highlight r %}
-i = 1
-while (i < 3) {
- if (i == 1) {
- print('hello')
- } else {
- print('world')
- }
- i = i + 1
-}
-
-A = matrix("1 2 3 4 5 6", rows=3, cols=2)
-
-for (i in 1:nrow(A)) {
- print("for A[" + i + ",1]:" + as.scalar(A[i,1]))
-}
-
-parfor(i in 1:nrow(A)) {
- print("parfor A[" + i + ",1]:" + as.scalar(A[i,1]))
-}
-{% endhighlight %}
-</div>
-
-<div data-lang="PyDML" markdown="1">
-{% highlight python %}
-i = 1
-while (i < 3):
- if (i == 1):
- print('hello')
- else:
- print('world')
- i = i + 1
-
-A = full("1 2 3 4 5 6", rows=3, cols=2)
-
-for (i in 1:nrow(A)):
- print("for A[" + i + ",1]:" + scalar(A[i,1]))
-
-parfor(i in 1:nrow(A)):
- print("parfor A[" + i + ",1]:" + scalar(A[i,1]))
-{% endhighlight %}
-</div>
-
-<div data-lang="Result" markdown="1">
- hello
- world
- for A[1,1]:1.0
- for A[2,1]:3.0
- for A[3,1]:5.0
- parfor A[2,1]:3.0
- parfor A[1,1]:1.0
- parfor A[3,1]:5.0
-</div>
-
-</div>
-
-
-# User-Defined Functions
-
-Functions encapsulate useful functionality in SystemML. In addition to built-in functions, users can define their own functions.
-Functions take 0 or more parameters and return 0 or more values.
-Currently, if a function returns nothing, it still needs to be assigned to a variable.
-
-<div class="codetabs2">
-
-<div data-lang="DML" markdown="1">
-{% highlight r %}
-doSomething = function(matrix[double] mat) return (matrix[double] ret) {
- additionalCol = matrix(1, rows=nrow(mat), cols=1) # 1x3 matrix with 1 values
- ret = cbind(mat, additionalCol) # concatenate column to matrix
- ret = cbind(ret, seq(0, 2, 1)) # concatenate column (0,1,2) to matrix
- ret = cbind(ret, rowMaxs(ret)) # concatenate column of max row values to matrix
- ret = cbind(ret, rowSums(ret)) # concatenate column of row sums to matrix
-}
-
-A = rand(rows=3, cols=2, min=0, max=2) # random 3x2 matrix with values 0 to 2
-B = doSomething(A)
-write(A, "A.csv", format="csv")
-write(B, "B.csv", format="csv")
-{% endhighlight %}
-</div>
-
-<div data-lang="PyDML" markdown="1">
-{% highlight python %}
-def doSomething(mat: matrix[float]) -> (ret: matrix[float]):
- additionalCol = full(1, rows=nrow(mat), cols=1) # 1x3 matrix with 1 values
- ret = cbind(mat, additionalCol) # concatenate column to matrix
- ret = cbind(ret, seq(0, 2, 1)) # concatenate column (0,1,2) to matrix
- ret = cbind(ret, rowMaxs(ret)) # concatenate column of max row values to matrix
- ret = cbind(ret, rowSums(ret)) # concatenate column of row sums to matrix
-
-A = rand(rows=3, cols=2, min=0, max=2) # random 3x2 matrix with values 0 to 2
-B = doSomething(A)
-save(A, "A.csv", format="csv")
-save(B, "B.csv", format="csv")
-{% endhighlight %}
-</div>
-
-</div>
-
-In the above example, a 3x2 matrix of random doubles between 0 and 2 is created using the **`rand()`** function.
-Additional parameters can be passed to **`rand()`** to control sparsity and other matrix characteristics.
-
-Matrix A is passed to the `doSomething` function. A column of 1 values is concatenated to the matrix. A column
-consisting of the values `(0, 1, 2)` is concatenated to the matrix. Next, a column consisting of the maximum row values
-is concatenated to the matrix. A column consisting of the row sums is concatenated to the matrix, and this resulting
-matrix is returned to variable B. Matrix A is output to the `A.csv` file and matrix B is saved as the `B.csv` file.
-
-
-<div class="codetabs2">
-
-<div data-lang="A.csv" markdown="1">
- 1.6091961493071,0.7088614208099939
- 0.5984862383600267,1.5732118950764993
- 0.2947607068519842,1.9081406573366781
-</div>
-
-<div data-lang="B.csv" markdown="1">
- 1.6091961493071,0.7088614208099939,1.0,0,1.6091961493071,4.927253719424194
- 0.5984862383600267,1.5732118950764993,1.0,1.0,1.5732118950764993,5.744910028513026
- 0.2947607068519842,1.9081406573366781,1.0,2.0,2.0,7.202901364188662
-</div>
-
-</div>
-
-
-# Command-Line Arguments and Default Values
-
-Command-line arguments can be passed to DML and PyDML scripts either as named arguments or as positional arguments. Named
-arguments are the preferred technique. Named arguments can be passed utilizing the `-nvargs` switch, and positional arguments
-can be passed using the `-args` switch.
-
-Default values can be set using the **`ifdef()`** function.
-
-In the example below, a matrix is read from the file system using named argument `M`. The number of rows to print is specified
-using the `rowsToPrint` argument, which defaults to 2 if no argument is supplied. Likewise, the number of columns is
-specified using `colsToPrint` with a default value of 2.
-
-<div class="codetabs2">
-
-<div data-lang="DML" markdown="1">
-{% highlight r %}
-
-fileM = $M
-
-numRowsToPrint = ifdef($rowsToPrint, 2) # default to 2
-numColsToPrint = ifdef($colsToPrint, 2) # default to 2
-
-m = read(fileM)
-
-for (i in 1:numRowsToPrint) {
- for (j in 1:numColsToPrint) {
- print('[' + i + ',' + j + ']:' + as.scalar(m[i,j]))
- }
-}
-
-{% endhighlight %}
-</div>
-
-<div data-lang="PyDML" markdown="1">
-{% highlight python %}
-
-fileM = $M
-
-numRowsToPrint = ifdef($rowsToPrint, 2) # default to 2
-numColsToPrint = ifdef($colsToPrint, 2) # default to 2
-
-m = load(fileM)
-
-for (i in 1:numRowsToPrint):
- for (j in 1:numColsToPrint):
- print('[' + i + ',' + j + ']:' + scalar(m[i,j]))
-
-{% endhighlight %}
-</div>
-
-<div data-lang="DML Named Arguments and Results" markdown="1">
- Example #1 Arguments:
- -f ex.dml -nvargs M=M.txt rowsToPrint=1 colsToPrint=3
-
- Example #1 Results:
- [1,1]:1.0
- [1,2]:2.0
- [1,3]:3.0
-
- Example #2 Arguments:
- -f ex.dml -nvargs M=M.txt
-
- Example #2 Results:
- [1,1]:1.0
- [1,2]:2.0
- [2,1]:0.0
- [2,2]:0.0
-
-</div>
-
-<div data-lang="PyDML Named Arguments and Results" markdown="1">
- Example #1 Arguments:
- -f ex.pydml -python -nvargs M=M.txt rowsToPrint=1 colsToPrint=3
-
- Example #1 Results:
- [1,1]:1.0
- [1,2]:2.0
- [1,3]:3.0
-
- Example #2 Arguments:
- -f ex.pydml -python -nvargs M=M.txt
-
- Example #2 Results:
- [1,1]:1.0
- [1,2]:2.0
- [2,1]:0.0
- [2,2]:0.0
-
-</div>
-
-</div>
-
-Here, we see identical functionality but with positional arguments.
-
-<div class="codetabs2">
-
-<div data-lang="DML" markdown="1">
-{% highlight r %}
-
-fileM = $1
-
-numRowsToPrint = ifdef($2, 2) # default to 2
-numColsToPrint = ifdef($3, 2) # default to 2
-
-m = read(fileM)
-
-for (i in 1:numRowsToPrint) {
- for (j in 1:numColsToPrint) {
- print('[' + i + ',' + j + ']:' + as.scalar(m[i,j]))
- }
-}
-
-{% endhighlight %}
-</div>
-
-<div data-lang="PyDML" markdown="1">
-{% highlight python %}
-
-fileM = $1
-
-numRowsToPrint = ifdef($2, 2) # default to 2
-numColsToPrint = ifdef($3, 2) # default to 2
-
-m = load(fileM)
-
-for (i in 1:numRowsToPrint):
- for (j in 1:numColsToPrint):
- print('[' + i + ',' + j + ']:' + scalar(m[i,j]))
-
-{% endhighlight %}
-</div>
-
-<div data-lang="DML Positional Arguments and Results" markdown="1">
- Example #1 Arguments:
- -f ex.dml -args M.txt 1 3
-
- Example #1 Results:
- [1,1]:1.0
- [1,2]:2.0
- [1,3]:3.0
-
- Example #2 Arguments:
- -f ex.dml -args M.txt
-
- Example #2 Results:
- [1,1]:1.0
- [1,2]:2.0
- [2,1]:0.0
- [2,2]:0.0
-
-</div>
-
-<div data-lang="PyDML Positional Arguments and Results" markdown="1">
- Example #1 Arguments:
- -f ex.pydml -python -args M.txt 1 3
-
- Example #1 Results:
- [1,1]:1.0
- [1,2]:2.0
- [1,3]:3.0
-
- Example #2 Arguments:
- -f ex.pydml -python -args M.txt
-
- Example #2 Results:
- [1,1]:1.0
- [1,2]:2.0
- [2,1]:0.0
- [2,2]:0.0
-
-</div>
-
-</div>
-
-
-# Additional Information
-
-The [DML Language Reference](dml-language-reference.html) and PyDML Language Reference contain highly detailed information regard DML
-and PyDML.
-
-In addition, many excellent examples of DML and PyDML can be found in the `system-ml/scripts` and
-`system-ml/test/scripts/applications` directories.
-
http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/65844aa6/docs/hadoop-batch-mode.md
----------------------------------------------------------------------
diff --git a/docs/hadoop-batch-mode.md b/docs/hadoop-batch-mode.md
index 18c25ce..efce4b1 100644
--- a/docs/hadoop-batch-mode.md
+++ b/docs/hadoop-batch-mode.md
@@ -3,6 +3,24 @@ layout: global
title: Invoking SystemML in Hadoop Batch Mode
description: Invoking SystemML in Hadoop Batch Mode
---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
* This will become a table of contents (this text will be scraped).
{:toc}
http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/65844aa6/docs/img/beginners-guide-to-dml-and-pydml/dmlscript-debug-configuration-hello-world-main-class.png
----------------------------------------------------------------------
diff --git a/docs/img/beginners-guide-to-dml-and-pydml/dmlscript-debug-configuration-hello-world-main-class.png b/docs/img/beginners-guide-to-dml-and-pydml/dmlscript-debug-configuration-hello-world-main-class.png
new file mode 100644
index 0000000..2ee98b9
Binary files /dev/null and b/docs/img/beginners-guide-to-dml-and-pydml/dmlscript-debug-configuration-hello-world-main-class.png differ
http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/65844aa6/docs/img/beginners-guide-to-dml-and-pydml/dmlscript-debug-configuration-hello-world-program-arguments.png
----------------------------------------------------------------------
diff --git a/docs/img/beginners-guide-to-dml-and-pydml/dmlscript-debug-configuration-hello-world-program-arguments.png b/docs/img/beginners-guide-to-dml-and-pydml/dmlscript-debug-configuration-hello-world-program-arguments.png
new file mode 100644
index 0000000..b1e344a
Binary files /dev/null and b/docs/img/beginners-guide-to-dml-and-pydml/dmlscript-debug-configuration-hello-world-program-arguments.png differ
http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/65844aa6/docs/img/dml-and-pydml-programming-guide/dmlscript-debug-configuration-hello-world-main-class.png
----------------------------------------------------------------------
diff --git a/docs/img/dml-and-pydml-programming-guide/dmlscript-debug-configuration-hello-world-main-class.png b/docs/img/dml-and-pydml-programming-guide/dmlscript-debug-configuration-hello-world-main-class.png
deleted file mode 100644
index 2ee98b9..0000000
Binary files a/docs/img/dml-and-pydml-programming-guide/dmlscript-debug-configuration-hello-world-main-class.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/65844aa6/docs/img/dml-and-pydml-programming-guide/dmlscript-debug-configuration-hello-world-program-arguments.png
----------------------------------------------------------------------
diff --git a/docs/img/dml-and-pydml-programming-guide/dmlscript-debug-configuration-hello-world-program-arguments.png b/docs/img/dml-and-pydml-programming-guide/dmlscript-debug-configuration-hello-world-program-arguments.png
deleted file mode 100644
index b1e344a..0000000
Binary files a/docs/img/dml-and-pydml-programming-guide/dmlscript-debug-configuration-hello-world-program-arguments.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/65844aa6/docs/img/mlcontext-programming-guide/zeppelin-notebook-systemml-linear-regression.png
----------------------------------------------------------------------
diff --git a/docs/img/mlcontext-programming-guide/zeppelin-notebook-systemml-linear-regression.png b/docs/img/mlcontext-programming-guide/zeppelin-notebook-systemml-linear-regression.png
deleted file mode 100644
index bfadbc4..0000000
Binary files a/docs/img/mlcontext-programming-guide/zeppelin-notebook-systemml-linear-regression.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/65844aa6/docs/img/mlcontext-programming-guide/zeppelin-notebook.png
----------------------------------------------------------------------
diff --git a/docs/img/mlcontext-programming-guide/zeppelin-notebook.png b/docs/img/mlcontext-programming-guide/zeppelin-notebook.png
deleted file mode 100644
index 0e10cd1..0000000
Binary files a/docs/img/mlcontext-programming-guide/zeppelin-notebook.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/65844aa6/docs/img/spark-mlcontext-programming-guide/zeppelin-notebook-systemml-linear-regression.png
----------------------------------------------------------------------
diff --git a/docs/img/spark-mlcontext-programming-guide/zeppelin-notebook-systemml-linear-regression.png b/docs/img/spark-mlcontext-programming-guide/zeppelin-notebook-systemml-linear-regression.png
new file mode 100644
index 0000000..bfadbc4
Binary files /dev/null and b/docs/img/spark-mlcontext-programming-guide/zeppelin-notebook-systemml-linear-regression.png differ
http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/65844aa6/docs/img/spark-mlcontext-programming-guide/zeppelin-notebook.png
----------------------------------------------------------------------
diff --git a/docs/img/spark-mlcontext-programming-guide/zeppelin-notebook.png b/docs/img/spark-mlcontext-programming-guide/zeppelin-notebook.png
new file mode 100644
index 0000000..0e10cd1
Binary files /dev/null and b/docs/img/spark-mlcontext-programming-guide/zeppelin-notebook.png differ
http://git-wip-us.apache.org/repos/asf/incubator-systemml/blob/65844aa6/docs/index.md
----------------------------------------------------------------------
diff --git a/docs/index.md b/docs/index.md
index 0f88f04..3fa90cc 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -1,8 +1,8 @@
---
layout: global
-displayTitle: SystemML Overview
-title: SystemML Overview
-description: SystemML documentation homepage
+displayTitle: SystemML Documentation
+title: SystemML Documentation
+description: SystemML Documentation
---
<!--
{% comment %}
@@ -23,25 +23,53 @@ limitations under the License.
{% endcomment %}
-->
-SystemML is a flexible, scalable machine learning (ML) system.
-SystemML's distinguishing characteristics are: (1) algorithm customizability,
-(2) multiple execution modes, including Standalone, Hadoop Batch, and Spark Batch,
-and (3) automatic optimization.
-
SystemML is now an **Apache Incubator** project! Please see the [**Apache SystemML (incubating)**](http://systemml.apache.org/)
website for more information.
-## SystemML Documentation
+SystemML is a flexible, scalable machine learning system.
+SystemML's distinguishing characteristics are:
+ (1) **Algorithm customizability via R-like and Python-like languages**,
+ (2) **Multiple execution modes**, including Standalone, Spark Batch, Spark MLContext, Hadoop Batch, and JMLC, and
+ (3) **Automatic optimization** based on data and cluster characteristics to ensure both efficiency and scalability.
+
+The [SystemML GitHub README](https://github.com/apache/incubator-systemml) describes
+building, testing, and running SystemML.
+
+## Running SystemML
+
+* **Standalone** - Standalone mode allows data scientists to rapidly prototype algorithms on a single
+machine in R-like and Python-like declarative languages.
+ * The [SystemML GitHub README](https://github.com/apache/incubator-systemml) describes
+ a linear regression example in Standalone Mode.
+ * The [Quick Start Guide](quick-start-guide.html) provides additional examples of algorithm execution
+ in Standalone Mode.
+* **Spark Batch** - Algorithms are automatically optimized to run across Spark clusters.
+ * See **Invoking SystemML in Spark Batch Mode** **(Coming soon)**.
+* **Spark MLContext** - Spark MLContext is a programmatic API for running SystemML from Spark via Scala or Java.
+ * See the [Spark MLContext Programming Guide](spark-mlcontext-programming-guide.html) for
+ [**Spark Shell (Scala)**](spark-mlcontext-programming-guide.html#spark-shell-example),
+ [Java](spark-mlcontext-programming-guide.html#java-example), and
+ [**Zeppelin Notebook**](spark-mlcontext-programming-guide.html#zeppelin-notebook-example---linear-regression-algorithm)
+ examples.
+* **Hadoop Batch** - Algorithms are automatically optimized when distributed across Hadoop clusters.
+ * See [Invoking SystemML in Hadoop Batch Mode](hadoop-batch-mode.html) for detailed information.
+* **JMLC** - Java Machine Learning Connector.
+
+## Language Guides
+
+* [DML Language Reference](dml-language-reference.html) -
+DML is a high-level R-like declarative language for machine learning.
+* **PyDML Language Reference** **(Coming Soon)** -
+PyDML is a high-level Python-like declarative language for machine learning.
+* [Beginner's Guide to DML and PyDML](beginners-guide-to-dml-and-pydml.html) -
+An introduction to the basics of DML and PyDML.
+
+## ML Algorithms
+
+* [Algorithms Reference](algorithms-reference.html) - The Algorithms Reference describes the
+machine learning algorithms included with SystemML in detail.
-For more information about SystemML, please consult the following references:
+## Tools
-* [SystemML GitHub README](https://github.com/apache/incubator-systemml)
-* [Quick Start Guide](quick-start-guide.html)
-* [DML and PyDML Programming Guide](dml-and-pydml-programming-guide.html)
-* [MLContext Programming Guide](mlcontext-programming-guide.html)
-* [Hadoop Batch Mode](hadoop-batch-mode.html)
-* Spark Batch Mode - **Coming Soon**
-* [Debugger Guide](debugger-guide.html)
-* [Algorithms Reference](algorithms-reference.html)
-* [DML (R-like Declarative Machine Learning) Language Reference](dml-language-reference.html)
-* PyDML (Python-Like Declarative Machine Learning) Language Reference - **Coming Soon**
+* [Debugger Guide](debugger-guide.html) - SystemML supports DML script-level debugging through a
+command-line interface.