You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@systemml.apache.org by re...@apache.org on 2018/03/30 04:31:06 UTC
svn commit: r1828046 [16/20] - /systemml/site/docs/1.1.0/
Added: systemml/site/docs/1.1.0/python-reference.html
URL: http://svn.apache.org/viewvc/systemml/site/docs/1.1.0/python-reference.html?rev=1828046&view=auto
==============================================================================
--- systemml/site/docs/1.1.0/python-reference.html (added)
+++ systemml/site/docs/1.1.0/python-reference.html Fri Mar 30 04:31:05 2018
@@ -0,0 +1,1015 @@
+<!DOCTYPE html>
+<!--[if lt IE 7]> <html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]-->
+<!--[if IE 7]> <html class="no-js lt-ie9 lt-ie8"> <![endif]-->
+<!--[if IE 8]> <html class="no-js lt-ie9"> <![endif]-->
+<!--[if gt IE 8]><!--> <html class="no-js"> <!--<![endif]-->
+ <head>
+ <title>Reference Guide for Python Users - SystemML 1.1.0</title>
+ <meta charset="utf-8">
+ <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
+
+ <meta name="description" content="Reference Guide for Python Users">
+
+ <meta name="viewport" content="width=device-width">
+ <link rel="stylesheet" href="css/bootstrap.min.css">
+ <link rel="stylesheet" href="css/main.css">
+ <link rel="stylesheet" href="css/pygments-default.css">
+ <link rel="shortcut icon" href="img/favicon.png">
+ </head>
+ <body>
+ <!--[if lt IE 7]>
+ <p class="chromeframe">You are using an outdated browser. <a href="http://browsehappy.com/">Upgrade your browser today</a> or <a href="http://www.google.com/chromeframe/?redirect=true">install Google Chrome Frame</a> to better experience this site.</p>
+ <![endif]-->
+
+ <header class="navbar navbar-default navbar-fixed-top" id="topbar">
+ <div class="container">
+ <div class="navbar-header">
+ <div class="navbar-brand brand projectlogo">
+ <a href="http://systemml.apache.org/"><img class="logo" src="img/systemml-logo.png" alt="Apache SystemML" title="Apache SystemML"/></a>
+ </div>
+ <div class="navbar-brand brand projecttitle">
+ <a href="http://systemml.apache.org/">Apache SystemML<sup id="trademark">â¢</sup></a><br/>
+ <span class="version">1.1.0</span>
+ </div>
+ <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target=".navbar-collapse">
+ <span class="sr-only">Toggle navigation</span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ </div>
+ <nav class="navbar-collapse collapse">
+ <ul class="nav navbar-nav navbar-right">
+ <li><a href="index.html">Overview</a></li>
+ <li><a href="https://github.com/apache/systemml">GitHub</a></li>
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Documentation<b class="caret"></b></a>
+ <ul class="dropdown-menu" role="menu">
+ <li><b>Running SystemML:</b></li>
+ <li><a href="https://github.com/apache/systemml">SystemML GitHub README</a></li>
+ <li><a href="spark-mlcontext-programming-guide.html">Spark MLContext</a></li>
+ <li><a href="spark-batch-mode.html">Spark Batch Mode</a>
+ <li><a href="hadoop-batch-mode.html">Hadoop Batch Mode</a>
+ <li><a href="standalone-guide.html">Standalone Guide</a></li>
+ <li><a href="jmlc.html">Java Machine Learning Connector (JMLC)</a>
+ <li class="divider"></li>
+ <li><b>Language Guides:</b></li>
+ <li><a href="dml-language-reference.html">DML Language Reference</a></li>
+ <li><a href="beginners-guide-to-dml-and-pydml.html">Beginner's Guide to DML and PyDML</a></li>
+ <li><a href="beginners-guide-python.html">Beginner's Guide for Python Users</a></li>
+ <li><a href="python-reference.html">Reference Guide for Python Users</a></li>
+ <li class="divider"></li>
+ <li><b>ML Algorithms:</b></li>
+ <li><a href="algorithms-reference.html">Algorithms Reference</a></li>
+ <li class="divider"></li>
+ <li><b>Tools:</b></li>
+ <li><a href="debugger-guide.html">Debugger Guide</a></li>
+ <li><a href="developer-tools-systemml.html">IDE Guide</a></li>
+ <li class="divider"></li>
+ <li><b>Other:</b></li>
+ <li><a href="contributing-to-systemml.html">Contributing to SystemML</a></li>
+ <li><a href="engine-dev-guide.html">Engine Developer Guide</a></li>
+ <li><a href="troubleshooting-guide.html">Troubleshooting Guide</a></li>
+ <li><a href="release-process.html">Release Process</a></li>
+ </ul>
+ </li>
+
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">API Docs<b class="caret"></b></a>
+ <ul class="dropdown-menu" role="menu">
+ <li><a href="./api/java/index.html">Java</a></li>
+ <li><a href="./api/python/index.html">Python</a></li>
+ </ul>
+ </li>
+
+ <li class="dropdown">
+ <a href="#" class="dropdown-toggle" data-toggle="dropdown">Issues<b class="caret"></b></a>
+ <ul class="dropdown-menu" role="menu">
+ <li><b>JIRA:</b></li>
+ <li><a href="https://issues.apache.org/jira/browse/SYSTEMML">SystemML JIRA</a></li>
+
+ </ul>
+ </li>
+ </ul>
+ </nav>
+ </div>
+ </header>
+
+ <div class="container" id="content">
+
+ <h1 class="title">Reference Guide for Python Users</h1>
+
+
+ <!--
+
+-->
+
+<ul id="markdown-toc">
+ <li><a href="#introduction" id="markdown-toc-introduction">Introduction</a></li>
+ <li><a href="#matrix-class" id="markdown-toc-matrix-class">matrix class</a> <ul>
+ <li><a href="#operators" id="markdown-toc-operators">Operators</a></li>
+ <li><a href="#lazy-evaluation" id="markdown-toc-lazy-evaluation">Lazy evaluation</a></li>
+ <li><a href="#dealing-with-the-loops" id="markdown-toc-dealing-with-the-loops">Dealing with the loops</a></li>
+ <li><a href="#built-in-functions" id="markdown-toc-built-in-functions">Built-in functions</a></li>
+ <li><a href="#support-for-numpys-universal-functions" id="markdown-toc-support-for-numpys-universal-functions">Support for NumPy’s universal functions</a></li>
+ <li><a href="#design-decisions-of-matrix-class-developer-documentation" id="markdown-toc-design-decisions-of-matrix-class-developer-documentation">Design Decisions of matrix class (Developer documentation)</a></li>
+ </ul>
+ </li>
+ <li><a href="#mlcontext-api" id="markdown-toc-mlcontext-api">MLContext API</a> <ul>
+ <li><a href="#usage" id="markdown-toc-usage">Usage</a></li>
+ </ul>
+ </li>
+ <li><a href="#mllearn-api" id="markdown-toc-mllearn-api">mllearn API</a> <ul>
+ <li><a href="#passing-pyspark-dataframe" id="markdown-toc-passing-pyspark-dataframe">Passing PySpark DataFrame</a></li>
+ <li><a href="#mlpipeline-interface" id="markdown-toc-mlpipeline-interface">MLPipeline interface</a></li>
+ </ul>
+ </li>
+ <li><a href="#troubleshooting-python-apis" id="markdown-toc-troubleshooting-python-apis">Troubleshooting Python APIs</a> <ul>
+ <li><a href="#unable-to-load-systemmljar-into-current-pyspark-session" id="markdown-toc-unable-to-load-systemmljar-into-current-pyspark-session">Unable to load SystemML.jar into current pyspark session.</a></li>
+ <li><a href="#matrix-api-is-running-slow-when-setlazyfalse-or-when-eval-is-called-often" id="markdown-toc-matrix-api-is-running-slow-when-setlazyfalse-or-when-eval-is-called-often">matrix API is running slow when set_lazy(False) or when eval() is called often.</a></li>
+ <li><a href="#maximum-recursion-depth-exceeded" id="markdown-toc-maximum-recursion-depth-exceeded">maximum recursion depth exceeded</a></li>
+ </ul>
+ </li>
+</ul>
+
+<p><br /></p>
+
+<h2 id="introduction">Introduction</h2>
+
+<p>SystemML enables flexible, scalable machine learning. This flexibility is achieved through the specification of a high-level declarative machine learning language that comes in two flavors,
+one with an R-like syntax (DML) and one with a Python-like syntax (PyDML).</p>
+
+<p>Algorithm scripts written in DML and PyDML can be run on Hadoop, on Spark, or in Standalone mode.
+No script modifications are required to change between modes. SystemML automatically performs advanced optimizations
+based on data and cluster characteristics, so much of the need to manually tweak algorithms is largely reduced or eliminated.
+To understand more about DML and PyDML, we recommend that you read <a href="https://apache.github.io/systemml/beginners-guide-to-dml-and-pydml.html">Beginner’s Guide to DML and PyDML</a>.</p>
+
+<p>For convenience of Python users, SystemML exposes several language-level APIs that allow Python users to use SystemML
+and its algorithms without the need to know DML or PyDML. We explain these APIs in the below sections.</p>
+
+<h2 id="matrix-class">matrix class</h2>
+
+<p>The matrix class is an <strong>experimental</strong> feature that is often referred to as Python DSL.
+It allows the user to perform linear algebra operations in SystemML using a NumPy-like interface.
+It implements basic matrix operators, matrix functions as well as converters to common Python
+types (for example: Numpy arrays, PySpark DataFrame and Pandas
+DataFrame).</p>
+
+<p>The primary reason for supporting this API is to reduce the learning curve for an average Python user,
+who is more likely to know Numpy library, rather than the DML language.</p>
+
+<h3 id="operators">Operators</h3>
+
+<p>The operators supported are:</p>
+
+<ol>
+ <li>Arithmetic operators: +, -, <em>, /, //, %, *</em> as well as dot
+(i.e. matrix multiplication)</li>
+ <li>Indexing in the matrix</li>
+ <li>Relational/Boolean operators: <, <=, >, >=, ==, !=, &, |</li>
+</ol>
+
+<p>This class also supports several input/output formats such as NumPy arrays, Pandas DataFrame, SciPy sparse matrix and PySpark DataFrame.</p>
+
+<p>Here is a small example that demonstrates the usage:</p>
+
+<p><code>python
+>>> import systemml as sml
+>>> import numpy as np
+>>> m1 = sml.matrix(np.ones((3,3)) + 2)
+>>> m2 = sml.matrix(np.ones((3,3)) + 3)
+>>> m2 = m1 * (m2 + m1)
+>>> m4 = 1.0 - m2
+>>> m4.sum(axis=1).toNumPy()
+array([[-60.],
+ [-60.],
+ [-60.]])
+</code></p>
+
+<h3 id="lazy-evaluation">Lazy evaluation</h3>
+
+<p>By default, the operations are evaluated lazily to avoid conversion overhead and also to maximize optimization scope.
+To disable lazy evaluation, please us <code>set_lazy</code> method:</p>
+
+<p>```python
+»> import systemml as sml
+»> import numpy as np
+»> m1 = sml.matrix(np.ones((3,3)) + 2)</p>
+
+<p>Welcome to Apache SystemML!</p>
+
+<blockquote>
+ <blockquote>
+ <blockquote>
+ <p>m2 = sml.matrix(np.ones((3,3)) + 3)
+np.add(m1, m2) + m1
+# This matrix (mVar4) is backed by below given PyDML script (which is not yet evaluated). To fetch the data of this matrix, invoke toNumPy() or toDF() or toPandas() methods.
+mVar2 = load(“ “, format=”csv”)
+mVar1 = load(“ “, format=”csv”)
+mVar3 = mVar1 + mVar2
+mVar4 = mVar3 + mVar1
+save(mVar4, “ “)</p>
+ </blockquote>
+ </blockquote>
+</blockquote>
+
+<blockquote>
+ <blockquote>
+ <blockquote>
+ <p>sml.set_lazy(False)
+m1 = sml.matrix(np.ones((3,3)) + 2)
+m2 = sml.matrix(np.ones((3,3)) + 3)
+np.add(m1, m2) + m1
+# This matrix (mVar8) is backed by NumPy array. To fetch the NumPy array, invoke toNumPy() method.
+```</p>
+ </blockquote>
+ </blockquote>
+</blockquote>
+
+<p>Since matrix is backed by lazy evaluation and uses a recursive Depth First Search (DFS),
+you may run into <code>RuntimeError: maximum recursion depth exceeded</code>.
+Please see below <a href="http://apache.github.io/systemml/python-reference#maximum-recursion-depth-exceeded">troubleshooting steps</a></p>
+
+<h3 id="dealing-with-the-loops">Dealing with the loops</h3>
+
+<p>It is important to note that this API doesnot pushdown loop, which means the
+SystemML engine essentially gets an unrolled DML script.
+This can lead to two issues:</p>
+
+<ol>
+ <li>
+ <p>Since matrix is backed by lazy evaluation and uses a recursive Depth First Search (DFS),
+you may run into <code>RuntimeError: maximum recursion depth exceeded</code>.
+Please see below <a href="http://apache.github.io/systemml/python-reference#maximum-recursion-depth-exceeded">troubleshooting steps</a></p>
+ </li>
+ <li>
+ <p>Significant parsing/compilation overhead of potentially large unrolled DML script.</p>
+ </li>
+</ol>
+
+<p>The unrolling of the for loop can be demonstrated by the below example:</p>
+
+<p>```python
+»> import systemml as sml
+»> import numpy as np
+»> m1 = sml.matrix(np.ones((3,3)) + 2)</p>
+
+<p>Welcome to Apache SystemML!</p>
+
+<blockquote>
+ <blockquote>
+ <blockquote>
+ <p>m2 = sml.matrix(np.ones((3,3)) + 3)
+m3 = m1
+for i in range(5):
+… m3 = m1 * m3 + m1
+…
+m3
+# This matrix (mVar12) is backed by below given PyDML script (which is not yet evaluated). To fetch the data of this matrix, invoke toNumPy() or toDF() or toPandas() methods.
+mVar1 = load(“ “, format=”csv”)
+mVar3 = mVar1 * mVar1
+mVar4 = mVar3 + mVar1
+mVar5 = mVar1 * mVar4
+mVar6 = mVar5 + mVar1
+mVar7 = mVar1 * mVar6
+mVar8 = mVar7 + mVar1
+mVar9 = mVar1 * mVar8
+mVar10 = mVar9 + mVar1
+mVar11 = mVar1 * mVar10
+mVar12 = mVar11 + mVar1
+save(mVar12, “ “)
+```</p>
+ </blockquote>
+ </blockquote>
+</blockquote>
+
+<p>We can reduce the impact of this unrolling by eagerly evaluating the variables inside the loop:</p>
+
+<p>```python
+»> import systemml as sml
+»> import numpy as np
+»> m1 = sml.matrix(np.ones((3,3)) + 2)</p>
+
+<p>Welcome to Apache SystemML!</p>
+
+<blockquote>
+ <blockquote>
+ <blockquote>
+ <p>m2 = sml.matrix(np.ones((3,3)) + 3)
+m3 = m1
+for i in range(5):
+… m3 = m1 * m3 + m1
+… sml.eval(m3)</p>
+ </blockquote>
+ </blockquote>
+</blockquote>
+
+<p>```</p>
+
+<h3 id="built-in-functions">Built-in functions</h3>
+
+<p>In addition to the above mentioned operators, following functions are supported.</p>
+
+<ul>
+ <li>
+ <p>transpose: Transposes the input matrix.</p>
+ </li>
+ <li>
+ <p>Aggregation functions: prod, sum, mean, var, sd, max, min, argmin, argmax, cumsum</p>
+ </li>
+</ul>
+
+<table>
+ <thead>
+ <tr>
+ <th> </th>
+ <th>Description</th>
+ <th>Parameters</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>prod(self)</td>
+ <td>Return the product of all cells in matrix</td>
+ <td>self: input matrix object</td>
+ </tr>
+ <tr>
+ <td>sum(self, axis=None)</td>
+ <td>Compute the sum along the specified axis</td>
+ <td>axis : int, optional</td>
+ </tr>
+ <tr>
+ <td>mean(self, axis=None)</td>
+ <td>Compute the arithmetic mean along the specified axis</td>
+ <td>axis : int, optional</td>
+ </tr>
+ <tr>
+ <td>var(self, axis=None)</td>
+ <td>Compute the variance along the specified axis. We assume that delta degree of freedom is 1 (unlike NumPy which assumes ddof=0).</td>
+ <td>axis : int, optional</td>
+ </tr>
+ <tr>
+ <td>moment(self, moment=1, axis=None)</td>
+ <td>Calculates the nth moment about the mean</td>
+ <td>moment : int (can be 1, 2, 3 or 4), axis : int, optional</td>
+ </tr>
+ <tr>
+ <td>sd(self, axis=None)</td>
+ <td>Compute the standard deviation along the specified axis</td>
+ <td>axis : int, optional</td>
+ </tr>
+ <tr>
+ <td>max(self, other=None, axis=None)</td>
+ <td>Compute the maximum value along the specified axis</td>
+ <td>other: matrix or numpy array (& other supported types) or scalar, axis : int, optional</td>
+ </tr>
+ <tr>
+ <td>min(self, other=None, axis=None)</td>
+ <td>Compute the minimum value along the specified axis</td>
+ <td>other: matrix or numpy array (& other supported types) or scalar, axis : int, optional</td>
+ </tr>
+ <tr>
+ <td>argmin(self, axis=None)</td>
+ <td>Returns the indices of the minimum values along an axis.</td>
+ <td>axis : int, optional,(only axis=1, i.e. rowIndexMax is supported in this version)</td>
+ </tr>
+ <tr>
+ <td>argmax(self, axis=None)</td>
+ <td>Returns the indices of the maximum values along an axis.</td>
+ <td>axis : int, optional (only axis=1, i.e. rowIndexMax is supported in this version)</td>
+ </tr>
+ <tr>
+ <td>cumsum(self, axis=None)</td>
+ <td>Returns the indices of the maximum values along an axis.</td>
+ <td>axis : int, optional (only axis=0, i.e. cumsum along the rows is supported in this version)</td>
+ </tr>
+ </tbody>
+</table>
+
+<ul>
+ <li>Global statistical built-In functions: exp, log, abs, sqrt, round, floor, ceil, sin, cos, tan, sinh, cosh, tanh, asin, acos, atan, sign, solve</li>
+</ul>
+
+<table>
+ <thead>
+ <tr>
+ <th> </th>
+ <th>Description</th>
+ <th>Parameters</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>solve(A, b)</td>
+ <td>Computes the least squares solution for system of linear equations A %*% x = b</td>
+ <td>A, b: input matrices</td>
+ </tr>
+ </tbody>
+</table>
+
+<ul>
+ <li>Built-in sampling functions: normal, uniform, poisson</li>
+</ul>
+
+<table>
+ <thead>
+ <tr>
+ <th> </th>
+ <th>Description</th>
+ <th>Parameters</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>normal(loc=0.0, scale=1.0, size=(1,1), sparsity=1.0)</td>
+ <td>Draw random samples from a normal (Gaussian) distribution.</td>
+ <td>loc: Mean (“centre”) of the distribution, scale: Standard deviation (spread or “width”) of the distribution, size: Output shape (only tuple of length 2, i.e. (m, n), supported), sparsity: Sparsity (between 0.0 and 1.0).</td>
+ </tr>
+ <tr>
+ <td>uniform(low=0.0, high=1.0, size=(1,1), sparsity=1.0)</td>
+ <td>Draw samples from a uniform distribution.</td>
+ <td>low: Lower boundary of the output interval, high: Upper boundary of the output interval, size: Output shape (only tuple of length 2, i.e. (m, n), supported), sparsity: Sparsity (between 0.0 and 1.0).</td>
+ </tr>
+ <tr>
+ <td>poisson(lam=1.0, size=(1,1), sparsity=1.0)</td>
+ <td>Draw samples from a Poisson distribution.</td>
+ <td>lam: Expectation of interval, should be > 0, size: Output shape (only tuple of length 2, i.e. (m, n), supported), sparsity: Sparsity (between 0.0 and 1.0).</td>
+ </tr>
+ </tbody>
+</table>
+
+<ul>
+ <li>Other builtin functions: hstack, vstack, trace</li>
+</ul>
+
+<table>
+ <thead>
+ <tr>
+ <th> </th>
+ <th>Description</th>
+ <th>Parameters</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>hstack(self, other)</td>
+ <td>Stack matrices horizontally (column wise). Invokes cbind internally.</td>
+ <td>self: lhs matrix object, other: rhs matrix object</td>
+ </tr>
+ <tr>
+ <td>vstack(self, other)</td>
+ <td>Stack matrices vertically (row wise). Invokes rbind internally.</td>
+ <td>self: lhs matrix object, other: rhs matrix object</td>
+ </tr>
+ <tr>
+ <td>trace(self)</td>
+ <td>Return the sum of the cells of the main diagonal square matrix</td>
+ <td>self: input matrix</td>
+ </tr>
+ </tbody>
+</table>
+
+<p>Here is an example that uses the above functions and trains a simple linear regression model:</p>
+
+<p><code>python
+>>> import numpy as np
+>>> from sklearn import datasets
+>>> import systemml as sml
+>>> # Load the diabetes dataset
+>>> diabetes = datasets.load_diabetes()
+>>> # Use only one feature
+>>> diabetes_X = diabetes.data[:, np.newaxis, 2]
+>>> # Split the data into training/testing sets
+>>> X_train = diabetes_X[:-20]
+>>> X_test = diabetes_X[-20:]
+>>> # Split the targets into training/testing sets
+>>> y_train = diabetes.target[:-20]
+>>> y_test = diabetes.target[-20:]
+>>> # Train Linear Regression model
+>>> X = sml.matrix(X_train)
+>>> y = sml.matrix(np.matrix(y_train).T)
+>>> A = X.transpose().dot(X)
+>>> b = X.transpose().dot(y)
+>>> beta = sml.solve(A, b).toNumPy()
+>>> y_predicted = X_test.dot(beta)
+>>> print('Residual sum of squares: %.2f' % np.mean((y_predicted - y_test) ** 2))
+Residual sum of squares: 25282.12
+</code></p>
+
+<p>For all the above functions, we always return a two dimensional matrix, especially for aggregation functions with axis.
+For example: Assuming m1 is a matrix of (3, n), NumPy returns a 1d vector of dimension (3,) for operation m1.sum(axis=1)
+whereas SystemML returns a 2d matrix of dimension (3, 1).</p>
+
+<p>Note: an evaluated matrix contains a data field computed by eval
+method as DataFrame or NumPy array.</p>
+
+<h3 id="support-for-numpys-universal-functions">Support for NumPy’s universal functions</h3>
+
+<p>The matrix class also supports most of NumPy’s universal functions (i.e. ufuncs):</p>
+
+<p><code>bash
+pip install --ignore-installed 'numpy>=1.13.0rc2'
+</code></p>
+
+<p>This will enable NumPy’s functions to invoke matrix class:</p>
+
+<p><code>python
+import systemml as sml
+import numpy as np
+m1 = sml.matrix(np.ones((3,3)) + 2)
+m2 = sml.matrix(np.ones((3,3)) + 3)
+np.add(m1, m2)
+</code></p>
+
+<p>The matrix class doesnot support following ufuncs:</p>
+
+<ul>
+ <li>Complex number related ufunc (for example: <code>conj</code>)</li>
+ <li>Hyperbolic/inverse-hyperbolic functions (for example: sinh, arcsinh, cosh, …)</li>
+ <li>Bitwise operators</li>
+ <li>Xor operator</li>
+ <li>Infinite/Nan-checking (for example: isreal, iscomplex, isfinite, isinf, isnan)</li>
+ <li>Other ufuncs: copysign, nextafter, modf, frexp, trunc.</li>
+</ul>
+
+<h3 id="design-decisions-of-matrix-class-developer-documentation">Design Decisions of matrix class (Developer documentation)</h3>
+
+<ol>
+ <li>
+ <p>Until eval() method is invoked, we create an AST (not exposed to
+the user) that consist of unevaluated operations and data
+required by those operations. As an anology, a spark user can
+treat eval() method similar to calling RDD.persist() followed by
+RDD.count().</p>
+ </li>
+ <li>
+ <p>The AST consist of two kinds of nodes: either of type matrix or
+of type DMLOp. Both these classes expose _visit method, that
+helps in traversing the AST in DFS manner.</p>
+ </li>
+ <li>
+ <p>A matrix object can either be evaluated or not. If evaluated,
+the attribute ‘data’ is set to one of the supported types (for
+example: NumPy array or DataFrame). In this case, the attribute
+‘op’ is set to None. If not evaluated, the attribute ‘op’ which
+refers to one of the intermediate node of AST and if of type
+DMLOp. In this case, the attribute ‘data’ is set to None.</p>
+ </li>
+ <li>
+ <p>DMLOp has an attribute ‘inputs’ which contains list of matrix
+objects or DMLOp.</p>
+ </li>
+ <li>
+ <p>To simplify the traversal, every matrix object is considered
+immutable and an matrix operations creates a new matrix object.
+As an example: m1 = sml.matrix(np.ones((3,3))) creates a matrix
+object backed by ‘data=(np.ones((3,3))’. m1 = m1 * 2 will
+create a new matrix object which is now backed by ‘op=DMLOp( …)’
+whose input is earlier created matrix object.</p>
+ </li>
+ <li>
+ <p>Left indexing (implemented in __setitem__ method) is a
+special case, where Python expects the existing object to be
+mutated. To ensure the above property, we make deep copy of
+existing object and point any references to the left-indexed
+matrix to the newly created object. Then the left-indexed matrix
+is set to be backed by DMLOp consisting of following pydml:
+left-indexed-matrix = new-deep-copied-matrix
+left-indexed-matrix[index] = value</p>
+ </li>
+ <li>
+ <p>Please use m.print_ast() and/or type m for debugging. Here is a
+sample session:</p>
+ </li>
+</ol>
+
+<p><code>python
+>>> npm = np.ones((3,3))
+>>> m1 = sml.matrix(npm + 3)
+>>> m2 = sml.matrix(npm + 5)
+>>> m3 = m1 + m2
+>>> m3
+mVar2 = load(" ", format="csv")
+mVar1 = load(" ", format="csv")
+mVar3 = mVar1 + mVar2
+save(mVar3, " ")
+>>> m3.print_ast()
+- [mVar3] (op).
+ - [mVar1] (data).
+ - [mVar2] (data).
+</code></p>
+
+<h2 id="mlcontext-api">MLContext API</h2>
+
+<p>The Spark MLContext API offers a programmatic interface for interacting with SystemML from Spark using languages such as Scala, Java, and Python.
+As a result, it offers a convenient way to interact with SystemML from the Spark Shell and from Notebooks such as Jupyter and Zeppelin.</p>
+
+<h3 id="usage">Usage</h3>
+
+<p>The below example demonstrates how to invoke the algorithm <a href="https://github.com/apache/systemml/blob/master/scripts/algorithms/MultiLogReg.dml">scripts/algorithms/MultiLogReg.dml</a>
+using Python <a href="https://apache.github.io/systemml/spark-mlcontext-programming-guide">MLContext API</a>.</p>
+
+<p><code>python
+from sklearn import datasets, neighbors
+from pyspark.sql import DataFrame, SQLContext
+import systemml as sml
+import pandas as pd
+import os, imp
+sqlCtx = SQLContext(sc)
+digits = datasets.load_digits()
+X_digits = digits.data
+y_digits = digits.target + 1
+n_samples = len(X_digits)
+# Split the data into training/testing sets and convert to PySpark DataFrame
+X_df = sqlCtx.createDataFrame(pd.DataFrame(X_digits[:.9 * n_samples]))
+y_df = sqlCtx.createDataFrame(pd.DataFrame(y_digits[:.9 * n_samples]))
+ml = sml.MLContext(sc)
+# Get the path of MultiLogReg.dml
+scriptPath = os.path.join(imp.find_module("systemml")[1], 'systemml-java', 'scripts', 'algorithms', 'MultiLogReg.dml')
+script = sml.dml(scriptPath).input(X=X_df, Y_vec=y_df).output("B_out")
+beta = ml.execute(script).get('B_out').toNumPy()
+</code></p>
+
+<h2 id="mllearn-api">mllearn API</h2>
+
+<p>mllearn API is designed to be compatible with scikit-learn and MLLib.
+The classes that are part of mllearn API are LogisticRegression, LinearRegression, SVM, NaiveBayes
+and <a href="http://apache.github.io/systemml/beginners-guide-caffe2dml">Caffe2DML</a>.</p>
+
+<p>The below code describes how to use mllearn API for training:</p>
+
+<div class="codetabs">
+<div data-lang="sklearn way">
+
+ <div class="highlight"><pre><code class="language-python" data-lang="python"><span class="c"># Input: Two Python objects (X_train, y_train) of type numpy, pandas or scipy.</span>
+<span class="n">model</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_train</span><span class="p">,</span> <span class="n">y_train</span><span class="p">)</span></code></pre></div>
+
+ </div>
+<div data-lang="mllib way">
+
+ <div class="highlight"><pre><code class="language-python" data-lang="python"><span class="c"># Input: One LabeledPoint DataFrame with atleast two columns: features (of type Vector) and labels.</span>
+<span class="n">model</span><span class="o">.</span><span class="n">fit</span><span class="p">(</span><span class="n">X_df</span><span class="p">)</span></code></pre></div>
+
+ </div>
+</div>
+
+<p>The below code describes how to use mllearn API for prediction:</p>
+
+<div class="codetabs">
+<div data-lang="sklearn way">
+
+ <div class="highlight"><pre><code class="language-python" data-lang="python"><span class="c"># Input: One Python object (X_test) of type numpy, pandas or scipy.</span>
+<span class="n">model</span><span class="o">.</span><span class="n">predict</span><span class="p">(</span><span class="n">X_test</span><span class="p">)</span>
+<span class="c"># OR model.score(X_test, y_test)</span></code></pre></div>
+
+ </div>
+<div data-lang="mllib way">
+
+ <div class="highlight"><pre><code class="language-python" data-lang="python"><span class="c"># Input: One LabeledPoint DataFrame (df_test) with atleast one column: features (of type Vector).</span>
+<span class="n">model</span><span class="o">.</span><span class="n">transform</span><span class="p">(</span><span class="n">df_test</span><span class="p">)</span></code></pre></div>
+
+ </div>
+</div>
+
+<p>Please note that when training using mllearn API (i.e. <code>model.fit(X_df)</code>), SystemML
+expects that labels have been converted to 1-based value.
+This avoids unnecessary decoding overhead for large dataset if the label columns has already been decoded.
+For scikit-learn API, there is no such requirement.</p>
+
+<p>The table below describes the parameter available for mllearn algorithms:</p>
+
+<table>
+ <thead>
+ <tr>
+ <th>Parameters</th>
+ <th>Description of the Parameters</th>
+ <th>LogisticRegression</th>
+ <th>LinearRegression</th>
+ <th>SVM</th>
+ <th>NaiveBayes</th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>sparkSession</td>
+ <td>PySpark SparkSession</td>
+ <td>X</td>
+ <td>X</td>
+ <td>X</td>
+ <td>X</td>
+ </tr>
+ <tr>
+ <td>penalty</td>
+ <td>Used to specify the norm used in the penalization (default: ‘l2’)</td>
+ <td>only ‘l2’ supported</td>
+ <td>-</td>
+ <td>-</td>
+ <td>-</td>
+ </tr>
+ <tr>
+ <td>fit_intercept</td>
+ <td>Specifies whether to add intercept or not (default: True)</td>
+ <td>X</td>
+ <td>X</td>
+ <td>X</td>
+ <td>-</td>
+ </tr>
+ <tr>
+ <td>normalize</td>
+ <td>This parameter is ignored when fit_intercept is set to False. (default: False)</td>
+ <td>X</td>
+ <td>X</td>
+ <td>X</td>
+ <td>-</td>
+ </tr>
+ <tr>
+ <td>max_iter</td>
+ <td>Maximum number of iterations (default: 100)</td>
+ <td>X</td>
+ <td>X</td>
+ <td>X</td>
+ <td>-</td>
+ </tr>
+ <tr>
+ <td>max_inner_iter</td>
+ <td>Maximum number of inner iterations, or 0 if no maximum limit provided (default: 0)</td>
+ <td>X</td>
+ <td>-</td>
+ <td>-</td>
+ <td>-</td>
+ </tr>
+ <tr>
+ <td>tol</td>
+ <td>Tolerance used in the convergence criterion (default: 0.000001)</td>
+ <td>X</td>
+ <td>X</td>
+ <td>X</td>
+ <td>-</td>
+ </tr>
+ <tr>
+ <td>C</td>
+ <td>1/regularization parameter (default: 1.0). To disable regularization, please use float(“inf”)</td>
+ <td>X</td>
+ <td>X</td>
+ <td>X</td>
+ <td>-</td>
+ </tr>
+ <tr>
+ <td>solver</td>
+ <td>Algorithm to use in the optimization problem.</td>
+ <td>Only ‘newton-cg’ solver supported</td>
+ <td>Supports either ‘newton-cg’ or ‘direct-solve’ (default: ‘newton-cg’). Depending on the size and the sparsity of the feature matrix, one or the other solver may be more efficient. ‘direct-solve’ solver is more efficient when the number of features is relatively small (m < 1000) and input matrix X is either tall or fairly dense; otherwise ‘newton-cg’ solver is more efficient.</td>
+ <td>-</td>
+ <td>-</td>
+ </tr>
+ <tr>
+ <td>is_multi_class</td>
+ <td>Specifies whether to use binary-class or multi-class classifier (default: False)</td>
+ <td>-</td>
+ <td>-</td>
+ <td>X</td>
+ <td>-</td>
+ </tr>
+ <tr>
+ <td>laplace</td>
+ <td>Laplace smoothing specified by the user to avoid creation of 0 probabilities (default: 1.0)</td>
+ <td>-</td>
+ <td>-</td>
+ <td>-</td>
+ <td>X</td>
+ </tr>
+ </tbody>
+</table>
+
+<p>In the below example, we invoke SystemML’s <a href="https://apache.github.io/systemml/algorithms-classification.html#multinomial-logistic-regression">Logistic Regression</a>
+algorithm on digits datasets.</p>
+
+<p><code>python
+# Scikit-learn way
+from sklearn import datasets, neighbors
+from systemml.mllearn import LogisticRegression
+digits = datasets.load_digits()
+X_digits = digits.data
+y_digits = digits.target
+n_samples = len(X_digits)
+X_train = X_digits[:int(.9 * n_samples)]
+y_train = y_digits[:int(.9 * n_samples)]
+X_test = X_digits[int(.9 * n_samples):]
+y_test = y_digits[int(.9 * n_samples):]
+logistic = LogisticRegression(spark)
+print('LogisticRegression score: %f' % logistic.fit(X_train, y_train).score(X_test, y_test))
+</code></p>
+
+<p>Output:</p>
+
+<p><code>bash
+LogisticRegression score: 0.927778
+</code></p>
+
+<p>You can also save the trained model and load it later for prediction:</p>
+
+<p><code>python
+# Assuming logistic.fit(X_train, y_train) is already invoked
+logistic.save('logistic_model')
+new_logistic = LogisticRegression(spark)
+new_logistic.load('logistic_model')
+print('LogisticRegression score: %f' % new_logistic.score(X_test, y_test))
+</code></p>
+
+<h4 id="passing-pyspark-dataframe">Passing PySpark DataFrame</h4>
+
+<p>To train the above algorithm on larger dataset, we can load the dataset into DataFrame and pass it to the <code>fit</code> method:</p>
+
+<p><code>python
+from sklearn import datasets
+from systemml.mllearn import LogisticRegression
+import pandas as pd
+from sklearn.metrics import accuracy_score
+import systemml as sml
+digits = datasets.load_digits()
+X_digits = digits.data
+y_digits = digits.target
+n_samples = len(X_digits)
+# Split the data into training/testing sets and convert to PySpark DataFrame
+df_train = sml.convertToLabeledDF(sqlCtx, X_digits[:int(.9 * n_samples)], y_digits[:int(.9 * n_samples)])
+X_test = spark.createDataFrame(pd.DataFrame(X_digits[int(.9 * n_samples):]))
+logistic = LogisticRegression(spark)
+logistic.fit(df_train)
+y_predicted = logistic.predict(X_test)
+y_predicted = y_predicted.select('prediction').toPandas().as_matrix().flatten()
+y_test = y_digits[int(.9 * n_samples):]
+print('LogisticRegression score: %f' % accuracy_score(y_test, y_predicted))
+</code></p>
+
+<p>Output:</p>
+
+<p><code>bash
+LogisticRegression score: 0.922222
+</code></p>
+
+<h4 id="mlpipeline-interface">MLPipeline interface</h4>
+
+<p>In the below example, we demonstrate how the same <code>LogisticRegression</code> class can allow SystemML to fit seamlessly into
+large data pipelines.</p>
+
+<p><code>python
+# MLPipeline way
+from pyspark.ml import Pipeline
+from systemml.mllearn import LogisticRegression
+from pyspark.ml.feature import HashingTF, Tokenizer
+training = spark.createDataFrame([
+ (0, "a b c d e spark", 1.0),
+ (1, "b d", 2.0),
+ (2, "spark f g h", 1.0),
+ (3, "hadoop mapreduce", 2.0),
+ (4, "b spark who", 1.0),
+ (5, "g d a y", 2.0),
+ (6, "spark fly", 1.0),
+ (7, "was mapreduce", 2.0),
+ (8, "e spark program", 1.0),
+ (9, "a e c l", 2.0),
+ (10, "spark compile", 1.0),
+ (11, "hadoop software", 2.0)
+], ["id", "text", "label"])
+tokenizer = Tokenizer(inputCol="text", outputCol="words")
+hashingTF = HashingTF(inputCol="words", outputCol="features", numFeatures=20)
+lr = LogisticRegression(sqlCtx)
+pipeline = Pipeline(stages=[tokenizer, hashingTF, lr])
+model = pipeline.fit(training)
+test = spark.createDataFrame([
+ (12, "spark i j k"),
+ (13, "l m n"),
+ (14, "mapreduce spark"),
+ (15, "apache hadoop")], ["id", "text"])
+prediction = model.transform(test)
+prediction.show()
+</code></p>
+
+<p>Output:</p>
+
+<p><code>bash
++-------+---+---------------+------------------+--------------------+--------------------+----------+
+|__INDEX| id| text| words| features| probability|prediction|
++-------+---+---------------+------------------+--------------------+--------------------+----------+
+| 1.0| 12| spark i j k| [spark, i, j, k]|(20,[5,6,7],[2.0,...|[0.99999999999975...| 1.0|
+| 2.0| 13| l m n| [l, m, n]|(20,[8,9,10],[1.0...|[1.37552128844736...| 2.0|
+| 3.0| 14|mapreduce spark|[mapreduce, spark]|(20,[5,10],[1.0,1...|[0.99860290938153...| 1.0|
+| 4.0| 15| apache hadoop| [apache, hadoop]|(20,[9,14],[1.0,1...|[5.41688748236143...| 2.0|
++-------+---+---------------+------------------+--------------------+--------------------+----------+
+</code></p>
+
+<h2 id="troubleshooting-python-apis">Troubleshooting Python APIs</h2>
+
+<h4 id="unable-to-load-systemmljar-into-current-pyspark-session">Unable to load SystemML.jar into current pyspark session.</h4>
+
+<p>While using SystemML’s Python package through pyspark or notebook (SparkContext is not previously created in the session), the
+below method is not required. However, if the user wishes to use SystemML through spark-submit and has not previously invoked</p>
+
+<dl>
+ <dt><code>systemml.defmatrix.setSparkContext</code>(<em>sc</em>)</dt>
+ <dd>Before using the matrix, the user needs to invoke this function if SparkContext is not previously created in the session.
+
+ <dl>
+ <dt>sc: SparkContext</dt>
+ <dd>SparkContext</dd>
+ </dl>
+ </dd>
+</dl>
+
+<p>Example:</p>
+
+<p><code>python
+import systemml as sml
+import numpy as np
+sml.setSparkContext(sc)
+m1 = sml.matrix(np.ones((3,3)) + 2)
+m2 = sml.matrix(np.ones((3,3)) + 3)
+m2 = m1 * (m2 + m1)
+m4 = 1.0 - m2
+m4.sum(axis=1).toNumPy()
+</code></p>
+
+<p>If SystemML was not installed via pip, you may have to download SystemML.jar and provide it to pyspark via <code>--driver-class-path</code> and <code>--jars</code>.</p>
+
+<h4 id="matrix-api-is-running-slow-when-setlazyfalse-or-when-eval-is-called-often">matrix API is running slow when set_lazy(False) or when eval() is called often.</h4>
+
+<p>This is a known issue. The matrix API is slow in this scenario due to slow Py4J conversion from Java MatrixObject or Java RDD to Python NumPy or DataFrame.
+To resolve this for now, we recommend writing the matrix to FileSystemML and using <code>load</code> function.</p>
+
+<h4 id="maximum-recursion-depth-exceeded">maximum recursion depth exceeded</h4>
+
+<p>SystemML matrix is backed by lazy evaluation and uses a recursive Depth First Search (DFS).
+Python can throw <code>RuntimeError: maximum recursion depth exceeded</code> when the recursion of DFS exceeds beyond the limit
+set by Python. There are two ways to address it:</p>
+
+<ol>
+ <li>
+ <p>Increase the limit in Python:</p>
+
+ <p><code>python
+ import sys
+ some_large_number = 2000
+ sys.setrecursionlimit(some_large_number)
+</code></p>
+ </li>
+ <li>
+ <p>Evaluate the intermeditate matrix to cut-off large recursion.</p>
+ </li>
+</ol>
+
+
+ </div> <!-- /container -->
+
+
+
+ <script src="js/vendor/jquery-1.12.0.min.js"></script>
+ <script src="js/vendor/bootstrap.min.js"></script>
+ <script src="js/vendor/anchor.min.js"></script>
+ <script src="js/main.js"></script>
+
+
+
+
+
+ <!-- Analytics -->
+ <script>
+ (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
+ m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+ })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+ ga('create', 'UA-71553733-1', 'auto');
+ ga('send', 'pageview');
+ </script>
+
+
+
+ <!-- MathJax Section -->
+ <script type="text/x-mathjax-config">
+ MathJax.Hub.Config({
+ TeX: { equationNumbers: { autoNumber: "AMS" } }
+ });
+ </script>
+ <script>
+ // Note that we load MathJax this way to work with local file (file://), HTTP and HTTPS.
+ // We could use "//cdn.mathjax...", but that won't support "file://".
+ (function(d, script) {
+ script = d.createElement('script');
+ script.type = 'text/javascript';
+ script.async = true;
+ script.onload = function(){
+ MathJax.Hub.Config({
+ tex2jax: {
+ inlineMath: [ ["$", "$"], ["\\\\(","\\\\)"] ],
+ displayMath: [ ["$$","$$"], ["\\[", "\\]"] ],
+ processEscapes: true,
+ skipTags: ['script', 'noscript', 'style', 'textarea', 'pre']
+ }
+ });
+ };
+ script.src = ('https:' == document.location.protocol ? 'https://' : 'http://') +
+ 'cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML';
+ d.getElementsByTagName('head')[0].appendChild(script);
+ }(document));
+ </script>
+ </body>
+</html>