You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mahout.apache.org by bu...@apache.org on 2014/05/21 15:30:28 UTC
svn commit: r909493 - in /websites/staging/mahout/trunk/content: ./
developers/buildingmahout.html users/sparkbindings/play-with-shell.html
Author: buildbot
Date: Wed May 21 13:30:27 2014
New Revision: 909493
Log:
Staging update by buildbot for mahout
Modified:
websites/staging/mahout/trunk/content/ (props changed)
websites/staging/mahout/trunk/content/developers/buildingmahout.html
websites/staging/mahout/trunk/content/users/sparkbindings/play-with-shell.html
Propchange: websites/staging/mahout/trunk/content/
------------------------------------------------------------------------------
--- cms:source-revision (original)
+++ cms:source-revision Wed May 21 13:30:27 2014
@@ -1 +1 @@
-1595740
+1596572
Modified: websites/staging/mahout/trunk/content/developers/buildingmahout.html
==============================================================================
--- websites/staging/mahout/trunk/content/developers/buildingmahout.html (original)
+++ websites/staging/mahout/trunk/content/developers/buildingmahout.html Wed May 21 13:30:27 2014
@@ -251,18 +251,31 @@
<h2 id="compiling">Compiling</h2>
<p>Compile Mahout using standard maven commands</p>
-<div class="codehilite"><pre><span class="n">mvn</span> <span class="n">clean</span> <span class="n">install</span>
+<div class="codehilite"><pre><span class="c">#With hadoop-1.2.1 dependency</span>
+<span class="n">mvn</span> <span class="n">clean</span> <span class="n">install</span>
+
+<span class="c">#With hadoop-2.2.0 dependency</span>
+<span class="n">mvn</span> <span class="o">-</span><span class="n">Dhadoop2</span><span class="p">.</span><span class="nb">version</span><span class="p">=</span>2<span class="p">.</span>2<span class="p">.</span>0 <span class="n">clean</span> <span class="n">install</span>
</pre></div>
<p>Mahout has an extensive test suite which takes some time to run. If you just want to build Mahout, skip the tests like this</p>
-<div class="codehilite"><pre><span class="n">mvn</span> <span class="o">-</span><span class="n">DskipTests</span> <span class="n">clean</span> <span class="n">install</span>
+<div class="codehilite"><pre><span class="c">#With hadoop-1.2.1 dependency</span>
+<span class="n">mvn</span> <span class="o">-</span><span class="n">DskipTests</span> <span class="n">clean</span> <span class="n">install</span>
+
+<span class="c">#With hadoop-2.2.0 dependency</span>
+<span class="n">mvn</span> <span class="o">-</span><span class="n">Dhadoop2</span><span class="p">.</span><span class="nb">version</span><span class="p">=</span>2<span class="p">.</span>2<span class="p">.</span>0 <span class="o">-</span><span class="n">DskipTests</span> <span class="n">clean</span> <span class="n">install</span>
</pre></div>
<h2 id="packaging">Packaging</h2>
<p>In order to build the mahout jars, run</p>
-<div class="codehilite"><pre><span class="n">mvn</span> <span class="n">package</span>
+<div class="codehilite"><pre><span class="n">mvn</span> <span class="n">clean</span> <span class="n">package</span>
+</pre></div>
+
+
+<p>Or to build Mahout against to a hadoop-2 cluster (version 2.2.0, for example), run</p>
+<div class="codehilite"><pre><span class="n">mvn</span> <span class="o">-</span><span class="n">Dhadoop2</span><span class="p">.</span><span class="n">version</span><span class="p">=</span>2<span class="p">.</span>2<span class="p">.</span>0 <span class="n">clean</span> <span class="n">package</span>
</pre></div>
</div>
</div>
Modified: websites/staging/mahout/trunk/content/users/sparkbindings/play-with-shell.html
==============================================================================
--- websites/staging/mahout/trunk/content/users/sparkbindings/play-with-shell.html (original)
+++ websites/staging/mahout/trunk/content/users/sparkbindings/play-with-shell.html Wed May 21 13:30:27 2014
@@ -368,8 +368,17 @@ val drmData = drmParallelize(dense(
numPartitions = 2);
</pre></div>
-<p>Have a look at this matrix. The first four columns represent the ingredients (our features) and the last column (the rating) is the target variable for our regression. <a href="https://en.wikipedia.org/wiki/Linear_regression">Linear regression</a> assumes that the <strong>target variable y</strong> is generated by the linear combination of <strong>the feature matrix X</strong> with the <strong>parameter vector β</strong> plus the <strong>noise ε</strong>, summarized in the formula <strong>y = Xβ + ε</strong>. Our goal is to find an estimate of the parameter vector <em>β</em> that explains the data very well.</p>
-<p>As a first step, we extract <em>X</em> and <em>y</em> from our data matrix. We get <em>X</em> by slicing: we take all rows (denoted by <code>::</code>) and the first four columns, which have the ingredients in milligrams as content. Note that the result is again a DRM. The shell will not execute this code yet, it saves the history of operations and defers the execution until we really access a result. <strong>Mahout's DSL automatically optimizes and parallelizes all operations on DRMs and runs them on Apache Spark.</strong></p>
+<p>Have a look at this matrix. The first four columns represent the ingredients
+(our features) and the last column (the rating) is the target variable for
+our regression. <a href="https://en.wikipedia.org/wiki/Linear_regression">Linear regression</a>
+assumes that the <strong>target variable</strong> <code>\(\mathbf{y}\)</code> is generated by the
+linear combination of <strong>the feature matrix</strong> <code>\(\mathbf{X}\)</code> with the
+<strong>parameter vector</strong> <code>\(\boldsymbol{\beta}\)</code> plus the
+ <strong>noise</strong> <code>\(\boldsymbol{\varepsilon}\)</code>, summarized in the formula
+<code>\(\mathbf{y}=\mathbf{X}\boldsymbol{\beta}+\boldsymbol{\varepsilon}\)</code>.
+Our goal is to find an estimate of the parameter vector
+<code>\(\boldsymbol{\beta}\)</code> that explains the data very well.</p>
+<p>As a first step, we extract <code>\(\mathbf{X}\)</code> and <code>\(\mathbf{y}\)</code> from our data matrix. We get <em>X</em> by slicing: we take all rows (denoted by <code>::</code>) and the first four columns, which have the ingredients in milligrams as content. Note that the result is again a DRM. The shell will not execute this code yet, it saves the history of operations and defers the execution until we really access a result. <strong>Mahout's DSL automatically optimizes and parallelizes all operations on DRMs and runs them on Apache Spark.</strong></p>
<div class="codehilite"><pre>
val drmX = drmData(::, 0 until 4)
</pre></div>
@@ -379,17 +388,24 @@ val drmX = drmData(::, 0 until 4)
val y = drmData.collect(::, 4)
</pre></div>
-<p>Now we are ready to think about a mathematical way to estimate the parameter vector <em>β</em>. A simple textbook approach is <a href="https://en.wikipedia.org/wiki/Ordinary_least_squares">ordinary least squares (OLS)</a>, which minimizes the sum of residual squares between the true target variable and the prediction of the target variable. In OLS, there is even a closed form expression for estimating <em>Ã</em> as <strong><em>(X<sup>T</sup>X)<sup>-1</sup> X<sup>T</sup>y</em></strong>.</p>
-<p>The first thing which we compute for this is <strong><em>X<sup>T</sup>X</em></strong>. The code for doing this in Mahout's scala DSL maps directly to the mathematical formula. The operation <code>.t()</code> transposes a matrix and analogous to R <code>%*%</code> denotes matrix multiplication.</p>
+<p>Now we are ready to think about a mathematical way to estimate the parameter vector <em>β</em>. A simple textbook approach is <a href="https://en.wikipedia.org/wiki/Ordinary_least_squares">ordinary least squares (OLS)</a>, which minimizes the sum of residual squares between the true target variable and the prediction of the target variable. In OLS, there is even a closed form expression for estimating <code>\(\boldsymbol{\beta}\)</code> as
+<code>\(\left(\mathbf{X}^{\top}\mathbf{X}\right)^{-1}\mathbf{X}^{\top}\mathbf{y}\)</code>.</p>
+<p>The first thing which we compute for this is <code>\(\mathbf{X}^{\top}\mathbf{X}\)</code>. The code for doing this in Mahout's scala DSL maps directly to the mathematical formula. The operation <code>.t()</code> transposes a matrix and analogous to R <code>%*%</code> denotes matrix multiplication.</p>
<div class="codehilite"><pre>
val drmXtX = drmX.t %*% drmX
</pre></div>
-<p>The same is true for computing <strong><em>X<sup>T</sup>y</em></strong>. We can simply type the math in scala expressions into the shell. Here, <em>X</em> lives in the cluster, while is <em>y</em> in the memory of the driver, and the result is a DRM again.
+<p>The same is true for computing <code>\(\mathbf{X}^{\top}\mathbf{y}\)</code>. We can simply type the math in scala expressions into the shell. Here, <em>X</em> lives in the cluster, while is <em>y</em> in the memory of the driver, and the result is a DRM again.
<div class="codehilite"><pre>
val drmXty = drmX.t %*% y
</pre></div></p>
-<p>We're nearly done. The next step we take is to fetch <em>X<sup>T</sup>X</em> and <em>X<sup>T</sup>y</em> into the memory of our driver machine (we are targeting features matrices that are tall and skinny , so we can assume that <em>X<sup>T</sup>X</em> is small enough to fit in). Then, we provide them to an in-memory solver (Mahout provides the an analogon to R's <code>solve()</code> for that) which computes <code>beta</code>, our OLS estimate of the parameter vector <em>β</em>.</p>
+<p>We're nearly done. The next step we take is to fetch <code>\(\mathbf{X}^{\top}\mathbf{X}\)</code> and
+<code>\(\mathbf{X}^{\top}\mathbf{y}\)</code> into the memory of our driver machine (we are targeting
+features matrices that are tall and skinny ,
+so we can assume that <code>\(\mathbf{X}^{\top}\mathbf{X}\)</code> is small enough
+to fit in). Then, we provide them to an in-memory solver (Mahout provides
+the an analog to R's <code>solve()</code> for that) which computes <code>beta</code>, our
+OLS estimate of the parameter vector <code>\(\boldsymbol{\beta}\)</code>.</p>
<div class="codehilite"><pre>
val XtX = drmXtX.collect
val Xty = drmXty.collect(::, 0)
@@ -397,8 +413,16 @@ val Xty = drmXty.collect(::, 0)
val beta = solve(XtX, Xty)
</pre></div>
-<p>That's it! We have a implemented a distributed linear regression algorithm on Apache Spark. I hope you agree that we didn't have to worry a lot about parallelization and distributed systems. The goal of Mahout's linear algebra DSL is to abstract away the ugliness of programming a distributed system as much as possible, while still retaining decent performance and scalability.</p>
-<p>We can now check how well our model fits its training data. First, we multiply the feature matrix <em>X</em> by our estimate of <em>β</em>. Then, we look at the difference (via L2-norm) of the target variable <em>y</em> to the fitted target variable:</p>
+<p>That's it! We have a implemented a distributed linear regression algorithm
+on Apache Spark. I hope you agree that we didn't have to worry a lot about
+parallelization and distributed systems. The goal of Mahout's linear algebra
+DSL is to abstract away the ugliness of programming a distributed system
+as much as possible, while still retaining decent performance and
+scalability.</p>
+<p>We can now check how well our model fits its training data.
+First, we multiply the feature matrix <code>\(\mathbf{X}\)</code> by our estimate of
+<code>\(\boldsymbol{\beta}\)</code>. Then, we look at the difference (via L2-norm) of
+the target variable <code>\(\mathbf{y}\)</code> to the fitted target variable:</p>
<div class="codehilite"><pre>
val yFitted = (drmX %*% beta).collect(::, 0)
(y - yFitted).norm(2)
@@ -424,8 +448,13 @@ def goodnessOfFit(drmX: DrmLike[Int], be
}
</pre></div>
-<p>So far we have left out an important aspect of a standard linear regression model. Usually there is a constant bias term added to the model. Without that, our model always crosses through the origin and we only learn the right angle. An easy way to add such a bias term to our model is to add a column of ones to the feature matrix <em>X</em>. The corresponding weight in the parameter vector will then be the bias term.</p>
-<p>Mahout's DSL offers a <code>mapBlock()</code> method for custom modifications of a DRM. All the rows in a partition are merged to a block of the matrix which is given to custom code in a closure. For our example, we invoke <code>mapBlock</code> with <code>ncol = drmX.ncol + 1</code> to let the system know that we change the number of columns of the matrix. The input to our closure is a <code>block</code> of the DRM and an array of <code>keys</code> for the rows contained in the block. In order to add a column, we first create a new block with an additional column, then copy the data from the current block into the new block and finally set the last column to ones and return the new block.</p>
+<p>So far we have left out an important aspect of a standard linear regression
+model. Usually there is a constant bias term added to the model. Without
+that, our model always crosses through the origin and we only learn the
+right angle. An easy way to add such a bias term to our model is to add a
+column of ones to the feature matrix <code>\(\mathbf{X}\)</code>.
+The corresponding weight in the parameter vector will then be the bias term.</p>
+<p>Mahout's DSL offers a <code>mapBlock()</code> method for custom modifications of a DRM. All the rows in a partition are merged to a block of the matrix which is given to custom code in a closure. For our example, we invoke <code>mapBlock</code> with <code>ncol = drmX.ncol + 1</code> to let the system know that change the number of columns of the matrix. The input to our closure is a <code>block</code> of the DRM and an array of <code>keys</code> for the rows contained in the block. In order to add a column, we first create a new block with an additional column, then copy the data from the current block into the new block and finally set the last column to ones and return the new block.</p>
<div class="codehilite"><pre>
val drmXwithBiasColumn = drmX.mapBlock(ncol = drmX.ncol + 1) {
case(keys, block) =>
@@ -446,7 +475,7 @@ val betaWithBiasTerm = ols(drmXwithBiasC
goodnessOfFit(drmXwithBiasColumn, betaWithBiasTerm, y)
</pre></div>
-<p>As a further optimization, we can make use of the DSL's caching functionality. We use <code>drmXwithBiasColumn</code> repeatedly as input to a computation, so it might be beneficial to cache it in memory. This is achieved by calling <code>checkpoint()</code>. In the end, we remove it from the cache with <code>uncache</code>:</p>
+<p>As a further optimization, we can make use of the DSL's caching functionality. We use <code>drmXwithBiasColumn</code> repeatedly as input to a computation, so it might be beneficial to cache it in memory. This is achieved by calling <code>checkpoint()</code>. In the end, we remove it from the cache with uncache:</p>
<div class="codehilite"><pre>
val cachedDrmX = drmXwithBiasColumn.checkpoint()