You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mahout.apache.org by gi...@apache.org on 2017/11/30 00:23:39 UTC
[1/4] mahout git commit: Automatic Site Publish by Buildbot
Repository: mahout
Updated Branches:
refs/heads/asf-site f4961201e -> d9686c8ba
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/environment/out-of-core-reference.html
----------------------------------------------------------------------
diff --git a/users/environment/out-of-core-reference.html b/users/environment/out-of-core-reference.html
index 8db77d0..aed6f88 100644
--- a/users/environment/out-of-core-reference.html
+++ b/users/environment/out-of-core-reference.html
@@ -278,29 +278,29 @@
<p>The subjects of this reference are solely applicable to Mahout-Samsara’s <strong>DRM</strong> (distributed row matrix).</p>
-<p>In this reference, DRMs will be denoted as e.g. <code class="highlighter-rouge">A</code>, and in-core matrices as e.g. <code class="highlighter-rouge">inCoreA</code>.</p>
+<p>In this reference, DRMs will be denoted as e.g. <code>A</code>, and in-core matrices as e.g. <code>inCoreA</code>.</p>
<h4 id="imports">Imports</h4>
<p>The following imports are used to enable seamless in-core and distributed algebraic DSL operations:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>import org.apache.mahout.math._
+<pre><code>import org.apache.mahout.math._
import scalabindings._
import RLikeOps._
import drm._
import RLikeDRMOps._
-</code></pre></div></div>
+</code></pre>
<p>If working with mixed scala/java code:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>import collection._
+<pre><code>import collection._
import JavaConversions._
-</code></pre></div></div>
+</code></pre>
<p>If you are working with Mahout-Samsara’s Spark-specific operations e.g. for context creation:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>import org.apache.mahout.sparkbindings._
-</code></pre></div></div>
+<pre><code>import org.apache.mahout.sparkbindings._
+</code></pre>
<p>The Mahout shell does all of these imports automatically.</p>
@@ -310,38 +310,38 @@ import JavaConversions._
<p>Loading a DRM from (HD)FS:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>drmDfsRead(path = hdfsPath)
-</code></pre></div></div>
+<pre><code>drmDfsRead(path = hdfsPath)
+</code></pre>
<p>Parallelizing from an in-core matrix:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val inCoreA = (dense(1, 2, 3), (3, 4, 5))
+<pre><code>val inCoreA = (dense(1, 2, 3), (3, 4, 5))
val A = drmParallelize(inCoreA)
-</code></pre></div></div>
+</code></pre>
<p>Creating an empty DRM:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val A = drmParallelizeEmpty(100, 50)
-</code></pre></div></div>
+<pre><code>val A = drmParallelizeEmpty(100, 50)
+</code></pre>
<p>Collecting to driver’s jvm in-core:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val inCoreA = A.collect
-</code></pre></div></div>
+<pre><code>val inCoreA = A.collect
+</code></pre>
<p><strong>Warning: The collection of distributed matrices happens implicitly whenever conversion to an in-core (o.a.m.math.Matrix) type is required. E.g.:</strong></p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val inCoreA: Matrix = ...
+<pre><code>val inCoreA: Matrix = ...
val drmB: DrmLike[Int] =...
val inCoreC: Matrix = inCoreA %*%: drmB
-</code></pre></div></div>
+</code></pre>
<p><strong>implies (incoreA %*%: drmB).collect</strong></p>
<p>Collecting to (HD)FS as a Mahout’s DRM formatted file:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>A.dfsWrite(path = hdfsPath)
-</code></pre></div></div>
+<pre><code>A.dfsWrite(path = hdfsPath)
+</code></pre>
<h4 id="logical-algebraic-operators-on-drm-matrices">Logical algebraic operators on DRM matrices:</h4>
@@ -350,12 +350,12 @@ val inCoreC: Matrix = inCoreA %*%: drmB
<p>Cache a DRM and trigger an optimized physical plan:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>drmA.checkpoint(CacheHint.MEMORY_AND_DISK)
-</code></pre></div></div>
+<pre><code>drmA.checkpoint(CacheHint.MEMORY_AND_DISK)
+</code></pre>
<p>Other valid caching Instructions:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>drmA.checkpoint(CacheHint.NONE)
+<pre><code>drmA.checkpoint(CacheHint.NONE)
drmA.checkpoint(CacheHint.DISK_ONLY)
drmA.checkpoint(CacheHint.DISK_ONLY_2)
drmA.checkpoint(CacheHint.MEMORY_ONLY)
@@ -365,38 +365,38 @@ drmA.checkpoint(CacheHint.MEMORY_ONLY_SER_2)
drmA.checkpoint(CacheHint.MEMORY_AND_DISK_2)
drmA.checkpoint(CacheHint.MEMORY_AND_DISK_SER)
drmA.checkpoint(CacheHint.MEMORY_AND_DISK_SER_2)
-</code></pre></div></div>
+</code></pre>
<p><em>Note: Logical DRM operations are lazily computed. Currently the actual computations and optional caching will be triggered by dfsWrite(…), collect(…) and blockify(…).</em></p>
<p>Transposition:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>A.t
-</code></pre></div></div>
+<pre><code>A.t
+</code></pre>
<p>Elementwise addition <em>(Matrices of identical geometry and row key types)</em>:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>A + B
-</code></pre></div></div>
+<pre><code>A + B
+</code></pre>
<p>Elementwise subtraction <em>(Matrices of identical geometry and row key types)</em>:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>A - B
-</code></pre></div></div>
+<pre><code>A - B
+</code></pre>
<p>Elementwise multiplication (Hadamard) <em>(Matrices of identical geometry and row key types)</em>:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>A * B
-</code></pre></div></div>
+<pre><code>A * B
+</code></pre>
<p>Elementwise division <em>(Matrices of identical geometry and row key types)</em>:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>A / B
-</code></pre></div></div>
+<pre><code>A / B
+</code></pre>
<p><strong>Elementwise operations involving one in-core argument (int-keyed DRMs only)</strong>:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>A + inCoreB
+<pre><code>A + inCoreB
A - inCoreB
A * inCoreB
A / inCoreB
@@ -408,73 +408,73 @@ inCoreA +: B
inCoreA -: B
inCoreA *: B
inCoreA /: B
-</code></pre></div></div>
+</code></pre>
-<p>Note the Spark associativity change (e.g. <code class="highlighter-rouge">A *: inCoreB</code> means <code class="highlighter-rouge">B.leftMultiply(A</code>), same as when both arguments are in core). Whenever operator arguments include both in-core and out-of-core arguments, the operator can only be associated with the out-of-core (DRM) argument to support the distributed implementation.</p>
+<p>Note the Spark associativity change (e.g. <code>A *: inCoreB</code> means <code>B.leftMultiply(A</code>), same as when both arguments are in core). Whenever operator arguments include both in-core and out-of-core arguments, the operator can only be associated with the out-of-core (DRM) argument to support the distributed implementation.</p>
<p><strong>Matrix-matrix multiplication %*%</strong>:</p>
-<p><code class="highlighter-rouge">\(\mathbf{M}=\mathbf{AB}\)</code></p>
+<p><code>\(\mathbf{M}=\mathbf{AB}\)</code></p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>A %*% B
+<pre><code>A %*% B
A %*% inCoreB
A %*% inCoreDiagonal
A %*%: B
-</code></pre></div></div>
+</code></pre>
<p><em>Note: same as above, whenever operator arguments include both in-core and out-of-core arguments, the operator can only be associated with the out-of-core (DRM) argument to support the distributed implementation.</em></p>
<p><strong>Matrix-vector multiplication %*%</strong>
-Currently we support a right multiply product of a DRM and an in-core Vector(<code class="highlighter-rouge">\(\mathbf{Ax}\)</code>) resulting in a single column DRM, which then can be collected in front (usually the desired outcome):</p>
+Currently we support a right multiply product of a DRM and an in-core Vector(<code>\(\mathbf{Ax}\)</code>) resulting in a single column DRM, which then can be collected in front (usually the desired outcome):</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val Ax = A %*% x
+<pre><code>val Ax = A %*% x
val inCoreX = Ax.collect(::, 0)
-</code></pre></div></div>
+</code></pre>
<p><strong>Matrix-scalar +,-,*,/</strong>
Elementwise operations of every matrix element and a scalar:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>A + 5.0
+<pre><code>A + 5.0
A - 5.0
A :- 5.0
5.0 -: A
A * 5.0
A / 5.0
5.0 /: a
-</code></pre></div></div>
+</code></pre>
-<p>Note that <code class="highlighter-rouge">5.0 -: A</code> means <code class="highlighter-rouge">\(m_{ij} = 5 - a_{ij}\)</code> and <code class="highlighter-rouge">5.0 /: A</code> means <code class="highlighter-rouge">\(m_{ij} = \frac{5}{a{ij}}\)</code> for all elements of the result.</p>
+<p>Note that <code>5.0 -: A</code> means <code>\(m_{ij} = 5 - a_{ij}\)</code> and <code>5.0 /: A</code> means <code>\(m_{ij} = \frac{5}{a{ij}}\)</code> for all elements of the result.</p>
<h4 id="slicing">Slicing</h4>
<p>General slice:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>A(100 to 200, 100 to 200)
-</code></pre></div></div>
+<pre><code>A(100 to 200, 100 to 200)
+</code></pre>
<p>Horizontal Block:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>A(::, 100 to 200)
-</code></pre></div></div>
+<pre><code>A(::, 100 to 200)
+</code></pre>
<p>Vertical Block:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>A(100 to 200, ::)
-</code></pre></div></div>
+<pre><code>A(100 to 200, ::)
+</code></pre>
-<p><em>Note: if row range is not all-range (::) the the DRM must be <code class="highlighter-rouge">Int</code>-keyed. General case row slicing is not supported by DRMs with key types other than <code class="highlighter-rouge">Int</code></em>.</p>
+<p><em>Note: if row range is not all-range (::) the the DRM must be <code>Int</code>-keyed. General case row slicing is not supported by DRMs with key types other than <code>Int</code></em>.</p>
<h4 id="stitching">Stitching</h4>
<p>Stitch side by side (cbind R semantics):</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val drmAnextToB = drmA cbind drmB
-</code></pre></div></div>
+<pre><code>val drmAnextToB = drmA cbind drmB
+</code></pre>
<p>Stitch side by side (Scala):</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val drmAnextToB = drmA.cbind(drmB)
-</code></pre></div></div>
+<pre><code>val drmAnextToB = drmA.cbind(drmB)
+</code></pre>
<p>Analogously, vertical concatenation is available via <strong>rbind</strong></p>
@@ -483,126 +483,126 @@ A / 5.0
<p><strong>drm.mapBlock(…)</strong>:</p>
-<p>The DRM operator <code class="highlighter-rouge">mapBlock</code> provides transformational access to the distributed vertical blockified tuples of a matrix (Row-Keys, Vertical-Matrix-Block).</p>
+<p>The DRM operator <code>mapBlock</code> provides transformational access to the distributed vertical blockified tuples of a matrix (Row-Keys, Vertical-Matrix-Block).</p>
-<p>Using <code class="highlighter-rouge">mapBlock</code> to add 1.0 to a DRM:</p>
+<p>Using <code>mapBlock</code> to add 1.0 to a DRM:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val inCoreA = dense((1, 2, 3), (2, 3 , 4), (3, 4, 5))
+<pre><code>val inCoreA = dense((1, 2, 3), (2, 3 , 4), (3, 4, 5))
val drmA = drmParallelize(inCoreA)
val B = A.mapBlock() {
case (keys, block) => keys -> (block += 1.0)
}
-</code></pre></div></div>
+</code></pre>
<h4 id="broadcasting-vectors-and-matrices-to-closures">Broadcasting Vectors and matrices to closures</h4>
<p>Generally we can create and use one-way closure attributes to be used on the back end.</p>
<p>Scalar matrix multiplication:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val factor: Int = 15
+<pre><code>val factor: Int = 15
val drm2 = drm1.mapBlock() {
case (keys, block) => block *= factor
keys -> block
}
-</code></pre></div></div>
+</code></pre>
-<p><strong>Closure attributes must be java-serializable. Currently Mahout’s in-core Vectors and Matrices are not java-serializable, and must be broadcast to the closure using <code class="highlighter-rouge">drmBroadcast(...)</code></strong>:</p>
+<p><strong>Closure attributes must be java-serializable. Currently Mahout’s in-core Vectors and Matrices are not java-serializable, and must be broadcast to the closure using <code>drmBroadcast(...)</code></strong>:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val v: Vector ...
+<pre><code>val v: Vector ...
val bcastV = drmBroadcast(v)
val drm2 = drm1.mapBlock() {
case (keys, block) =>
for(row <- 0 until block.nrow) block(row, ::) -= bcastV
keys -> block
}
-</code></pre></div></div>
+</code></pre>
<h4 id="computations-providing-ad-hoc-summaries">Computations providing ad-hoc summaries</h4>
<p>Matrix cardinality:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>drmA.nrow
+<pre><code>drmA.nrow
drmA.ncol
-</code></pre></div></div>
+</code></pre>
-<p><em>Note: depending on the stage of optimization, these may trigger a computational action. I.e. if one calls <code class="highlighter-rouge">nrow()</code> n times, then the back end will actually recompute <code class="highlighter-rouge">nrow</code> n times.</em></p>
+<p><em>Note: depending on the stage of optimization, these may trigger a computational action. I.e. if one calls <code>nrow()</code> n times, then the back end will actually recompute <code>nrow</code> n times.</em></p>
<p>Means and sums:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>drmA.colSums
+<pre><code>drmA.colSums
drmA.colMeans
drmA.rowSums
drmA.rowMeans
-</code></pre></div></div>
+</code></pre>
-<p><em>Note: These will always trigger a computational action. I.e. if one calls <code class="highlighter-rouge">colSums()</code> n times, then the back end will actually recompute <code class="highlighter-rouge">colSums</code> n times.</em></p>
+<p><em>Note: These will always trigger a computational action. I.e. if one calls <code>colSums()</code> n times, then the back end will actually recompute <code>colSums</code> n times.</em></p>
<h4 id="distributed-matrix-decompositions">Distributed Matrix Decompositions</h4>
<p>To import the decomposition package:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>import org.apache.mahout.math._
+<pre><code>import org.apache.mahout.math._
import decompositions._
-</code></pre></div></div>
+</code></pre>
<p>Distributed thin QR:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val (drmQ, incoreR) = dqrThin(drmA)
-</code></pre></div></div>
+<pre><code>val (drmQ, incoreR) = dqrThin(drmA)
+</code></pre>
<p>Distributed SSVD:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val (drmU, drmV, s) = dssvd(drmA, k = 40, q = 1)
-</code></pre></div></div>
+<pre><code>val (drmU, drmV, s) = dssvd(drmA, k = 40, q = 1)
+</code></pre>
<p>Distributed SPCA:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val (drmU, drmV, s) = dspca(drmA, k = 30, q = 1)
-</code></pre></div></div>
+<pre><code>val (drmU, drmV, s) = dspca(drmA, k = 30, q = 1)
+</code></pre>
<p>Distributed regularized ALS:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val (drmU, drmV, i) = dals(drmA,
+<pre><code>val (drmU, drmV, i) = dals(drmA,
k = 50,
lambda = 0.0,
maxIterations = 10,
convergenceThreshold = 0.10))
-</code></pre></div></div>
+</code></pre>
<h4 id="adjusting-parallelism-of-computations">Adjusting parallelism of computations</h4>
-<p>Set the minimum parallelism to 100 for computations on <code class="highlighter-rouge">drmA</code>:</p>
+<p>Set the minimum parallelism to 100 for computations on <code>drmA</code>:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>drmA.par(min = 100)
-</code></pre></div></div>
+<pre><code>drmA.par(min = 100)
+</code></pre>
-<p>Set the exact parallelism to 100 for computations on <code class="highlighter-rouge">drmA</code>:</p>
+<p>Set the exact parallelism to 100 for computations on <code>drmA</code>:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>drmA.par(exact = 100)
-</code></pre></div></div>
+<pre><code>drmA.par(exact = 100)
+</code></pre>
-<p>Set the engine specific automatic parallelism adjustment for computations on <code class="highlighter-rouge">drmA</code>:</p>
+<p>Set the engine specific automatic parallelism adjustment for computations on <code>drmA</code>:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>drmA.par(auto = true)
-</code></pre></div></div>
+<pre><code>drmA.par(auto = true)
+</code></pre>
<h4 id="retrieving-the-engine-specific-data-structure-backing-the-drm">Retrieving the engine specific data structure backing the DRM:</h4>
<p><strong>A Spark RDD:</strong></p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val myRDD = drmA.checkpoint().rdd
-</code></pre></div></div>
+<pre><code>val myRDD = drmA.checkpoint().rdd
+</code></pre>
<p><strong>An H2O Frame and Key Vec:</strong></p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val myFrame = drmA.frame
+<pre><code>val myFrame = drmA.frame
val myKeys = drmA.keys
-</code></pre></div></div>
+</code></pre>
<p><strong>A Flink DataSet:</strong></p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val myDataSet = drmA.ds
-</code></pre></div></div>
+<pre><code>val myDataSet = drmA.ds
+</code></pre>
<p>For more information including information on Mahout-Samsara’s Algebraic Optimizer and in-core Linear algebra bindings see: <a href="http://mahout.apache.org/users/sparkbindings/ScalaSparkBindings.pdf">Mahout Scala Bindings and Mahout Spark Bindings for Linear Algebra Subroutines</a></p>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/flinkbindings/playing-with-samsara-flink.html
----------------------------------------------------------------------
diff --git a/users/flinkbindings/playing-with-samsara-flink.html b/users/flinkbindings/playing-with-samsara-flink.html
index 8e45d9e..000d052 100644
--- a/users/flinkbindings/playing-with-samsara-flink.html
+++ b/users/flinkbindings/playing-with-samsara-flink.html
@@ -276,16 +276,16 @@
<p>To get started, add the following dependency to the pom:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code><dependency>
+<pre><code><dependency>
<groupId>org.apache.mahout</groupId>
<artifactId>mahout-flink_2.10</artifactId>
<version>0.12.0</version>
</dependency>
-</code></pre></div></div>
+</code></pre>
<p>Here is how to use the Flink backend:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>import org.apache.flink.api.scala._
+<pre><code>import org.apache.flink.api.scala._
import org.apache.mahout.math.drm._
import org.apache.mahout.math.drm.RLikeDrmOps._
import org.apache.mahout.flinkbindings._
@@ -304,7 +304,7 @@ object ReadCsvExample {
}
}
-</code></pre></div></div>
+</code></pre>
<h2 id="current-status">Current Status</h2>
@@ -314,18 +314,18 @@ object ReadCsvExample {
<ul>
<li><a href="https://issues.apache.org/jira/browse/MAHOUT-1701">MAHOUT-1701</a> Mahout DSL for Flink: implement AtB ABt and AtA operators</li>
- <li><a href="https://issues.apache.org/jira/browse/MAHOUT-1702">MAHOUT-1702</a> implement element-wise operators (like <code class="highlighter-rouge">A + 2</code> or <code class="highlighter-rouge">A + B</code>)</li>
- <li><a href="https://issues.apache.org/jira/browse/MAHOUT-1703">MAHOUT-1703</a> implement <code class="highlighter-rouge">cbind</code> and <code class="highlighter-rouge">rbind</code></li>
- <li><a href="https://issues.apache.org/jira/browse/MAHOUT-1709">MAHOUT-1709</a> implement slicing (like <code class="highlighter-rouge">A(1 to 10, ::)</code>)</li>
- <li><a href="https://issues.apache.org/jira/browse/MAHOUT-1710">MAHOUT-1710</a> implement right in-core matrix multiplication (<code class="highlighter-rouge">A %*% B</code> when <code class="highlighter-rouge">B</code> is in-core)</li>
+ <li><a href="https://issues.apache.org/jira/browse/MAHOUT-1702">MAHOUT-1702</a> implement element-wise operators (like <code>A + 2</code> or <code>A + B</code>)</li>
+ <li><a href="https://issues.apache.org/jira/browse/MAHOUT-1703">MAHOUT-1703</a> implement <code>cbind</code> and <code>rbind</code></li>
+ <li><a href="https://issues.apache.org/jira/browse/MAHOUT-1709">MAHOUT-1709</a> implement slicing (like <code>A(1 to 10, ::)</code>)</li>
+ <li><a href="https://issues.apache.org/jira/browse/MAHOUT-1710">MAHOUT-1710</a> implement right in-core matrix multiplication (<code>A %*% B</code> when <code>B</code> is in-core)</li>
<li><a href="https://issues.apache.org/jira/browse/MAHOUT-1711">MAHOUT-1711</a> implement broadcasting</li>
- <li><a href="https://issues.apache.org/jira/browse/MAHOUT-1712">MAHOUT-1712</a> implement operators <code class="highlighter-rouge">At</code>, <code class="highlighter-rouge">Ax</code>, <code class="highlighter-rouge">Atx</code> - <code class="highlighter-rouge">Ax</code> and <code class="highlighter-rouge">At</code> are implemented</li>
+ <li><a href="https://issues.apache.org/jira/browse/MAHOUT-1712">MAHOUT-1712</a> implement operators <code>At</code>, <code>Ax</code>, <code>Atx</code> - <code>Ax</code> and <code>At</code> are implemented</li>
<li><a href="https://issues.apache.org/jira/browse/MAHOUT-1734">MAHOUT-1734</a> implement I/O - should be able to read results of Flink bindings</li>
- <li><a href="https://issues.apache.org/jira/browse/MAHOUT-1747">MAHOUT-1747</a> add support for different types of indexes (String, long, etc) - now supports <code class="highlighter-rouge">Int</code>, <code class="highlighter-rouge">Long</code> and <code class="highlighter-rouge">String</code></li>
+ <li><a href="https://issues.apache.org/jira/browse/MAHOUT-1747">MAHOUT-1747</a> add support for different types of indexes (String, long, etc) - now supports <code>Int</code>, <code>Long</code> and <code>String</code></li>
<li><a href="https://issues.apache.org/jira/browse/MAHOUT-1748">MAHOUT-1748</a> switch to Flink Scala API</li>
- <li><a href="https://issues.apache.org/jira/browse/MAHOUT-1749">MAHOUT-1749</a> Implement <code class="highlighter-rouge">Atx</code></li>
- <li><a href="https://issues.apache.org/jira/browse/MAHOUT-1750">MAHOUT-1750</a> Implement <code class="highlighter-rouge">ABt</code></li>
- <li><a href="https://issues.apache.org/jira/browse/MAHOUT-1751">MAHOUT-1751</a> Implement <code class="highlighter-rouge">AtA</code></li>
+ <li><a href="https://issues.apache.org/jira/browse/MAHOUT-1749">MAHOUT-1749</a> Implement <code>Atx</code></li>
+ <li><a href="https://issues.apache.org/jira/browse/MAHOUT-1750">MAHOUT-1750</a> Implement <code>ABt</code></li>
+ <li><a href="https://issues.apache.org/jira/browse/MAHOUT-1751">MAHOUT-1751</a> Implement <code>AtA</code></li>
<li><a href="https://issues.apache.org/jira/browse/MAHOUT-1755">MAHOUT-1755</a> Flush intermediate results to FS - Flink, unlike Spark, does not store intermediate results in memory.</li>
<li><a href="https://issues.apache.org/jira/browse/MAHOUT-1764">MAHOUT-1764</a> Add standard backend tests for Flink</li>
<li><a href="https://issues.apache.org/jira/browse/MAHOUT-1765">MAHOUT-1765</a> Add documentation about Flink backend</li>
@@ -355,19 +355,19 @@ object ReadCsvExample {
<p>There is a set of standard tests that all engines should pass (see <a href="https://issues.apache.org/jira/browse/MAHOUT-1764">MAHOUT-1764</a>).</p>
<ul>
- <li><code class="highlighter-rouge">DistributedDecompositionsSuite</code></li>
- <li><code class="highlighter-rouge">DrmLikeOpsSuite</code></li>
- <li><code class="highlighter-rouge">DrmLikeSuite</code></li>
- <li><code class="highlighter-rouge">RLikeDrmOpsSuite</code></li>
+ <li><code>DistributedDecompositionsSuite</code></li>
+ <li><code>DrmLikeOpsSuite</code></li>
+ <li><code>DrmLikeSuite</code></li>
+ <li><code>RLikeDrmOpsSuite</code></li>
</ul>
<p>These are Flink-backend specific tests, e.g.</p>
<ul>
- <li><code class="highlighter-rouge">DrmLikeOpsSuite</code> for operations like <code class="highlighter-rouge">norm</code>, <code class="highlighter-rouge">rowSums</code>, <code class="highlighter-rouge">rowMeans</code></li>
- <li><code class="highlighter-rouge">RLikeOpsSuite</code> for basic LA like <code class="highlighter-rouge">A.t %*% A</code>, <code class="highlighter-rouge">A.t %*% x</code>, etc</li>
- <li><code class="highlighter-rouge">LATestSuite</code> tests for specific operators like <code class="highlighter-rouge">AtB</code>, <code class="highlighter-rouge">Ax</code>, etc</li>
- <li><code class="highlighter-rouge">UseCasesSuite</code> has more complex examples, like power iteration, ridge regression, etc</li>
+ <li><code>DrmLikeOpsSuite</code> for operations like <code>norm</code>, <code>rowSums</code>, <code>rowMeans</code></li>
+ <li><code>RLikeOpsSuite</code> for basic LA like <code>A.t %*% A</code>, <code>A.t %*% x</code>, etc</li>
+ <li><code>LATestSuite</code> tests for specific operators like <code>AtB</code>, <code>Ax</code>, etc</li>
+ <li><code>UseCasesSuite</code> has more complex examples, like power iteration, ridge regression, etc</li>
</ul>
<h2 id="environment">Environment</h2>
@@ -382,9 +382,9 @@ object ReadCsvExample {
<p>When using mahout, please import the following modules:</p>
<ul>
- <li><code class="highlighter-rouge">mahout-math</code></li>
- <li><code class="highlighter-rouge">mahout-math-scala</code></li>
- <li><code class="highlighter-rouge">mahout-flink_2.10</code>
+ <li><code>mahout-math</code></li>
+ <li><code>mahout-math-scala</code></li>
+ <li><code>mahout-flink_2.10</code>
*</li>
</ul>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/misc/parallel-frequent-pattern-mining.html
----------------------------------------------------------------------
diff --git a/users/misc/parallel-frequent-pattern-mining.html b/users/misc/parallel-frequent-pattern-mining.html
index c02586a..1ec3df8 100644
--- a/users/misc/parallel-frequent-pattern-mining.html
+++ b/users/misc/parallel-frequent-pattern-mining.html
@@ -286,7 +286,7 @@ creating Iterators, Convertors and TopKPatternWritable for that particular
object. For more information please refer the package
org.apache.mahout.fpm.pfpgrowth.convertors.string</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>e.g:
+<pre><code>e.g:
FPGrowth<String> fp = new FPGrowth<String>();
Set<String> features = new HashSet<String>();
fp.generateTopKStringFrequentPatterns(
@@ -298,7 +298,7 @@ org.apache.mahout.fpm.pfpgrowth.convertors.string</p>
features,
new StringOutputConvertor(new SequenceFileOutputCollector<Text, TopKStringPatterns>(writer))
);
-</code></pre></div></div>
+</code></pre>
<ul>
<li>The first argument is the iterator of transaction in this case its
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/misc/using-mahout-with-python-via-jpype.html
----------------------------------------------------------------------
diff --git a/users/misc/using-mahout-with-python-via-jpype.html b/users/misc/using-mahout-with-python-via-jpype.html
index 5290f63..d9e72c6 100644
--- a/users/misc/using-mahout-with-python-via-jpype.html
+++ b/users/misc/using-mahout-with-python-via-jpype.html
@@ -305,14 +305,14 @@ python script. The result for me looks like the following</p>
<p>Now we can create a function to start the jvm in python using jype</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>jvm=None
+<pre><code>jvm=None
def start_jpype():
global jvm
if (jvm is None):
cpopt="-Djava.class.path={cp}".format(cp=classpath)
startJVM(jvmlib,"-ea",cpopt)
jvm="started"
-</code></pre></div></div>
+</code></pre>
<p><a name="UsingMahoutwithPythonviaJPype-WritingNamedVectorstoSequenceFilesfromPython"></a></p>
<h1 id="writing-named-vectors-to-sequence-files-from-python">Writing Named Vectors to Sequence Files from Python</h1>
@@ -320,7 +320,7 @@ jvm="started"
be used by Mahout for kmeans. The example below is a function which creates
vectors from two Gaussian distributions with unit variance.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def create_inputs(ifile,*args,**param):
+<pre><code>def create_inputs(ifile,*args,**param):
"""Create a sequence file containing some normally distributed
ifile - path to the sequence file to create
"""
@@ -383,14 +383,14 @@ vectors from two Gaussian distributions with unit variance.</p>
writer.append(wrapkey,wrapval)
writer.close()
-</code></pre></div></div>
+</code></pre>
<p><a name="UsingMahoutwithPythonviaJPype-ReadingtheKMeansClusteredPointsfromPython"></a></p>
<h1 id="reading-the-kmeans-clustered-points-from-python">Reading the KMeans Clustered Points from Python</h1>
<p>Similarly we can use JPype to easily read the clustered points outputted by
mahout.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def read_clustered_pts(ifile,*args,**param):
+<pre><code>def read_clustered_pts(ifile,*args,**param):
"""Read the clustered points
ifile - path to the sequence file containing the clustered points
"""
@@ -433,14 +433,14 @@ mahout.</p>
print "cluster={key} Name={name} x={x}y={y}".format(key=key.toString(),name=nvec.getName(),x=nvec.get(0),y=nvec.get(1))
else:
raise NotImplementedError("Vector isn't a NamedVector. Need tomodify/test the code to handle this case.")
-</code></pre></div></div>
+</code></pre>
<p><a name="UsingMahoutwithPythonviaJPype-ReadingtheKMeansCentroids"></a></p>
<h1 id="reading-the-kmeans-centroids">Reading the KMeans Centroids</h1>
<p>Finally we can create a function to print out the actual cluster centers
found by mahout,</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def getClusters(ifile,*args,**param):
+<pre><code>def getClusters(ifile,*args,**param):
"""Read the centroids from the clusters outputted by kmenas
ifile - Path to the sequence file containing the centroids
"""
@@ -479,7 +479,7 @@ found by mahout,</p>
print "id={cid}center={center}".format(cid=vecwritable.getId(),center=center.values)
pass
-</code></pre></div></div>
+</code></pre>
</div>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/recommender/intro-als-hadoop.html
----------------------------------------------------------------------
diff --git a/users/recommender/intro-als-hadoop.html b/users/recommender/intro-als-hadoop.html
index e2920c2..db3700a 100644
--- a/users/recommender/intro-als-hadoop.html
+++ b/users/recommender/intro-als-hadoop.html
@@ -350,8 +350,8 @@ for item IDs. Then after recommendations are calculated you will have to transla
<p>Assuming your <em>JAVA_HOME</em> is appropriately set and Mahout was installed properly we’re ready to configure our syntax. Enter the following command:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ mahout parallelALS --input $als_input --output $als_output --lambda 0.1 --implicitFeedback true --alpha 0.8 --numFeatures 2 --numIterations 5 --numThreadsPerSolver 1 --tempDir tmp
-</code></pre></div></div>
+<pre><code>$ mahout parallelALS --input $als_input --output $als_output --lambda 0.1 --implicitFeedback true --alpha 0.8 --numFeatures 2 --numIterations 5 --numThreadsPerSolver 1 --tempDir tmp
+</code></pre>
<p>Running the command will execute a series of jobs the final product of which will be an output file deposited to the output directory specified in the command syntax. The output directory contains 3 sub-directories: <em>M</em> stores the item to feature matrix, <em>U</em> stores the user to feature matrix and userRatings stores the user’s ratings on the items. The <em>tempDir</em> parameter specifies the directory to store the intermediate output of the job, such as the matrix output in each iteration and each item’s average rating. Using the <em>tempDir</em> will help on debugging.</p>
@@ -359,8 +359,8 @@ for item IDs. Then after recommendations are calculated you will have to transla
<p>Based on the output feature matrices from step 3, we could make recommendations for users. Enter the following command:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> $ mahout recommendfactorized --input $als_recommender_input --userFeatures $als_output/U/ --itemFeatures $als_output/M/ --numRecommendations 1 --output recommendations --maxRating 1
-</code></pre></div></div>
+<pre><code> $ mahout recommendfactorized --input $als_recommender_input --userFeatures $als_output/U/ --itemFeatures $als_output/M/ --numRecommendations 1 --output recommendations --maxRating 1
+</code></pre>
<p>The input user file is a sequence file, the sequence record key is user id and value is the user’s rated item ids which will be removed from recommendation. The output file generated in our simple example will be a text file giving the recommended item ids for each user.
Remember to translate the Mahout ids back into your application specific ids.</p>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/recommender/intro-cooccurrence-spark.html
----------------------------------------------------------------------
diff --git a/users/recommender/intro-cooccurrence-spark.html b/users/recommender/intro-cooccurrence-spark.html
index 6aaccd5..fbeb1cd 100644
--- a/users/recommender/intro-cooccurrence-spark.html
+++ b/users/recommender/intro-cooccurrence-spark.html
@@ -304,7 +304,7 @@ For instance they might say an item-view is 0.2 of an item purchase. In practice
cross-cooccurrence is a more principled way to handle this case. In effect it scrubs secondary actions with the action you want
to recommend.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>spark-itemsimilarity Mahout 1.0
+<pre><code>spark-itemsimilarity Mahout 1.0
Usage: spark-itemsimilarity [options]
Disconnected from the target VM, address: '127.0.0.1:64676', transport: 'socket'
@@ -368,7 +368,7 @@ Spark config options:
-h | --help
prints this usage text
-</code></pre></div></div>
+</code></pre>
<p>This looks daunting but defaults to simple fairly sane values to take exactly the same input as legacy code and is pretty flexible. It allows the user to point to a single text file, a directory full of files, or a tree of directories to be traversed recursively. The files included can be specified with either a regex-style pattern or filename. The schema for the file is defined by column numbers, which map to the important bits of data including IDs and values. The files can even contain filters, which allow unneeded rows to be discarded or used for cross-cooccurrence calculations.</p>
@@ -378,20 +378,20 @@ Spark config options:
<p>If all defaults are used the input can be as simple as:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>userID1,itemID1
+<pre><code>userID1,itemID1
userID2,itemID2
...
-</code></pre></div></div>
+</code></pre>
<p>With the command line:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bash$ mahout spark-itemsimilarity --input in-file --output out-dir
-</code></pre></div></div>
+<pre><code>bash$ mahout spark-itemsimilarity --input in-file --output out-dir
+</code></pre>
<p>This will use the “local” Spark context and will output the standard text version of a DRM</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>itemID1<tab>itemID2:value2<space>itemID10:value10...
-</code></pre></div></div>
+<pre><code>itemID1<tab>itemID2:value2<space>itemID10:value10...
+</code></pre>
<p>###<a name="multiple-actions">How To Use Multiple User Actions</a></p>
@@ -407,7 +407,7 @@ to calculate the cross-cooccurrence indicator matrix.</p>
<p><em>spark-itemsimilarity</em> can read separate actions from separate files or from a mixed action log by filtering certain lines. For a mixed
action log of the form:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>u1,purchase,iphone
+<pre><code>u1,purchase,iphone
u1,purchase,ipad
u2,purchase,nexus
u2,purchase,galaxy
@@ -427,13 +427,13 @@ u3,view,nexus
u4,view,iphone
u4,view,ipad
u4,view,galaxy
-</code></pre></div></div>
+</code></pre>
<p>###Command Line</p>
<p>Use the following options:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bash$ mahout spark-itemsimilarity \
+<pre><code>bash$ mahout spark-itemsimilarity \
--input in-file \ # where to look for data
--output out-path \ # root dir for output
--master masterUrl \ # URL of the Spark master server
@@ -442,35 +442,35 @@ u4,view,galaxy
--itemIDPosition 2 \ # column that has the item ID
--rowIDPosition 0 \ # column that has the user ID
--filterPosition 1 # column that has the filter word
-</code></pre></div></div>
+</code></pre>
<p>###Output</p>
<p>The output of the job will be the standard text version of two Mahout DRMs. This is a case where we are calculating
cross-cooccurrence so a primary indicator matrix and cross-cooccurrence indicator matrix will be created</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>out-path
+<pre><code>out-path
|-- similarity-matrix - TDF part files
\-- cross-similarity-matrix - TDF part-files
-</code></pre></div></div>
+</code></pre>
<p>The indicator matrix will contain the lines:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>galaxy\tnexus:1.7260924347106847
+<pre><code>galaxy\tnexus:1.7260924347106847
ipad\tiphone:1.7260924347106847
nexus\tgalaxy:1.7260924347106847
iphone\tipad:1.7260924347106847
surface
-</code></pre></div></div>
+</code></pre>
<p>The cross-cooccurrence indicator matrix will contain:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>iphone\tnexus:1.7260924347106847 iphone:1.7260924347106847 ipad:1.7260924347106847 galaxy:1.7260924347106847
+<pre><code>iphone\tnexus:1.7260924347106847 iphone:1.7260924347106847 ipad:1.7260924347106847 galaxy:1.7260924347106847
ipad\tnexus:0.6795961471815897 iphone:0.6795961471815897 ipad:0.6795961471815897 galaxy:0.6795961471815897
nexus\tnexus:0.6795961471815897 iphone:0.6795961471815897 ipad:0.6795961471815897 galaxy:0.6795961471815897
galaxy\tnexus:1.7260924347106847 iphone:1.7260924347106847 ipad:1.7260924347106847 galaxy:1.7260924347106847
surface\tsurface:4.498681156950466 nexus:0.6795961471815897
-</code></pre></div></div>
+</code></pre>
<p><strong>Note:</strong> You can run this multiple times to use more than two actions or you can use the underlying
SimilarityAnalysis.cooccurrence API, which will more efficiently calculate any number of cross-cooccurrence indicators.</p>
@@ -479,7 +479,7 @@ SimilarityAnalysis.cooccurrence API, which will more efficiently calculate any n
<p>A common method of storing data is in log files. If they are written using some delimiter they can be consumed directly by spark-itemsimilarity. For instance input of the form:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>2014-06-23 14:46:53.115\tu1\tpurchase\trandom text\tiphone
+<pre><code>2014-06-23 14:46:53.115\tu1\tpurchase\trandom text\tiphone
2014-06-23 14:46:53.115\tu1\tpurchase\trandom text\tipad
2014-06-23 14:46:53.115\tu2\tpurchase\trandom text\tnexus
2014-06-23 14:46:53.115\tu2\tpurchase\trandom text\tgalaxy
@@ -499,11 +499,11 @@ SimilarityAnalysis.cooccurrence API, which will more efficiently calculate any n
2014-06-23 14:46:53.115\tu4\tview\trandom text\tiphone
2014-06-23 14:46:53.115\tu4\tview\trandom text\tipad
2014-06-23 14:46:53.115\tu4\tview\trandom text\tgalaxy
-</code></pre></div></div>
+</code></pre>
<p>Can be parsed with the following CLI and run on the cluster producing the same output as the above example.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bash$ mahout spark-itemsimilarity \
+<pre><code>bash$ mahout spark-itemsimilarity \
--input in-file \
--output out-path \
--master spark://sparkmaster:4044 \
@@ -513,7 +513,7 @@ SimilarityAnalysis.cooccurrence API, which will more efficiently calculate any n
--itemIDPosition 4 \
--rowIDPosition 1 \
--filterPosition 2
-</code></pre></div></div>
+</code></pre>
<p>##2. spark-rowsimilarity</p>
@@ -528,7 +528,7 @@ by a list of the most similar rows.</p>
<p>The command line interface is:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>spark-rowsimilarity Mahout 1.0
+<pre><code>spark-rowsimilarity Mahout 1.0
Usage: spark-rowsimilarity [options]
Input, output options
@@ -574,7 +574,7 @@ Spark config options:
-h | --help
prints this usage text
-</code></pre></div></div>
+</code></pre>
<p>See RowSimilarityDriver.scala in Mahout’s spark module if you want to customize the code.</p>
@@ -654,32 +654,32 @@ content or metadata, not by which users interacted with them.</p>
<p>For this we need input of the form:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>itemID<tab>list-of-tags
+<pre><code>itemID<tab>list-of-tags
...
-</code></pre></div></div>
+</code></pre>
<p>The full collection will look like the tags column from a catalog DB. For our ecom example it might be:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>3459860b<tab>men long-sleeve chambray clothing casual
+<pre><code>3459860b<tab>men long-sleeve chambray clothing casual
9446577d<tab>women tops chambray clothing casual
...
-</code></pre></div></div>
+</code></pre>
<p>We’ll use <em>spark-rowimilairity</em> because we are looking for similar rows, which encode items in this case. As with the
collaborative filtering indicator and cross-cooccurrence indicator we use the –omitStrength option. The strengths created are
probabilistic log-likelihood ratios and so are used to filter unimportant similarities. Once the filtering or downsampling
is finished we no longer need the strengths. We will get an indicator matrix of the form:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>itemID<tab>list-of-item IDs
+<pre><code>itemID<tab>list-of-item IDs
...
-</code></pre></div></div>
+</code></pre>
<p>This is a content indicator since it has found other items with similar content or metadata.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>3459860b<tab>3459860b 3459860b 6749860c 5959860a 3434860a 3477860a
+<pre><code>3459860b<tab>3459860b 3459860b 6749860c 5959860a 3434860a 3477860a
9446577d<tab>9446577d 9496577d 0943577d 8346577d 9442277d 9446577e
...
-</code></pre></div></div>
+</code></pre>
<p>We now have three indicators, two collaborative filtering type and one content type.</p>
@@ -691,11 +691,11 @@ For a given user, map their history of an action or content to the correct indic
<p>We have 3 indicators, these are indexed by the search engine into 3 fields, we’ll call them “purchase”, “view”, and “tags”.
We take the user’s history that corresponds to each indicator and create a query of the form:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Query:
+<pre><code>Query:
field: purchase; q:user's-purchase-history
field: view; q:user's view-history
field: tags; q:user's-tags-associated-with-purchases
-</code></pre></div></div>
+</code></pre>
<p>The query will result in an ordered list of items recommended for purchase but skewed towards items with similar tags to
the ones the user has already purchased.</p>
@@ -707,11 +707,11 @@ by tagging items with some category of popularity (hot, warm, cold for instance)
index that as a new indicator field and include the corresponding value in a query
on the popularity field. If we use the ecom example but use the query to get “hot” recommendations it might look like this:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Query:
+<pre><code>Query:
field: purchase; q:user's-purchase-history
field: view; q:user's view-history
field: popularity; q:"hot"
-</code></pre></div></div>
+</code></pre>
<p>This will return recommendations favoring ones that have the intrinsic indicator “hot”.</p>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/recommender/intro-itembased-hadoop.html
----------------------------------------------------------------------
diff --git a/users/recommender/intro-itembased-hadoop.html b/users/recommender/intro-itembased-hadoop.html
index 887b260..bb9b618 100644
--- a/users/recommender/intro-itembased-hadoop.html
+++ b/users/recommender/intro-itembased-hadoop.html
@@ -317,8 +317,8 @@
<p>Assuming your <em>JAVA_HOME</em> is appropriately set and Mahout was installed properly we’re ready to configure our syntax. Enter the following command:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ mahout recommenditembased -s SIMILARITY_LOGLIKELIHOOD -i /path/to/input/file -o /path/to/desired/output --numRecommendations 25
-</code></pre></div></div>
+<pre><code>$ mahout recommenditembased -s SIMILARITY_LOGLIKELIHOOD -i /path/to/input/file -o /path/to/desired/output --numRecommendations 25
+</code></pre>
<p>Running the command will execute a series of jobs the final product of which will be an output file deposited to the directory specified in the command syntax. The output file will contain two columns: the <em>userID</em> and an array of <em>itemIDs</em> and scores.</p>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/recommender/matrix-factorization.html
----------------------------------------------------------------------
diff --git a/users/recommender/matrix-factorization.html b/users/recommender/matrix-factorization.html
index f15fa8d..a1c28d6 100644
--- a/users/recommender/matrix-factorization.html
+++ b/users/recommender/matrix-factorization.html
@@ -282,39 +282,39 @@ There are many different matrix decompositions, each finds use among a particula
<p>In mahout, the SVDRecommender provides an interface to build recommender based on matrix factorization.
The idea behind is to project the users and items onto a feature space and try to optimize U and M so that U * (M^t) is as close to R as possible:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> U is n * p user feature matrix,
+<pre><code> U is n * p user feature matrix,
M is m * p item feature matrix, M^t is the conjugate transpose of M,
R is n * m rating matrix,
n is the number of users,
m is the number of items,
p is the number of features
-</code></pre></div></div>
+</code></pre>
<p>We usually use RMSE to represent the deviations between predictions and atual ratings.
RMSE is defined as the squared root of the sum of squared errors at each known user item ratings.
So our matrix factorization target could be mathmatically defined as:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> find U and M, (U, M) = argmin(RMSE) = argmin(pow(SSE / K, 0.5))
+<pre><code> find U and M, (U, M) = argmin(RMSE) = argmin(pow(SSE / K, 0.5))
SSE = sum(e(u,i)^2)
e(u,i) = r(u, i) - U[u,] * (M[i,]^t) = r(u,i) - sum(U[u,f] * M[i,f]), f = 0, 1, .. p - 1
K is the number of known user item ratings.
-</code></pre></div></div>
+</code></pre>
<p><a name="MatrixFactorization-Factorizers"></a></p>
<p>Mahout has implemented matrix factorization based on</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(1) SGD(Stochastic Gradient Descent)
+<pre><code>(1) SGD(Stochastic Gradient Descent)
(2) ALSWR(Alternating-Least-Squares with Weighted-λ-Regularization).
-</code></pre></div></div>
+</code></pre>
<h2 id="sgd">SGD</h2>
<p>Stochastic gradient descent is a gradient descent optimization method for minimizing an objective function that is written as a su of differentiable functions.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> Q(w) = sum(Q_i(w)),
-</code></pre></div></div>
+<pre><code> Q(w) = sum(Q_i(w)),
+</code></pre>
<p>where w is the parameters to be estimated,
Q(w) is the objective function that could be expressed as sum of differentiable functions,
@@ -322,14 +322,14 @@ So our matrix factorization target could be mathmatically defined as:</p>
<p>In practice, w is estimated using an iterative method at each single sample until an approximate miminum is obtained,</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> w = w - alpha * (d(Q_i(w))/dw), where aplpha is the learning rate,
+<pre><code> w = w - alpha * (d(Q_i(w))/dw), where aplpha is the learning rate,
(d(Q_i(w))/dw) is the first derivative of Q_i(w) on w.
-</code></pre></div></div>
+</code></pre>
<p>In matrix factorization, the RatingSGDFactorizer class implements the SGD with w = (U, M) and objective function Q(w) = sum(Q(u,i)),</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> Q(u,i) = sum(e(u,i) * e(u,i)) / 2 + lambda * [(U[u,] * (U[u,]^t)) + (M[i,] * (M[i,]^t))] / 2
-</code></pre></div></div>
+<pre><code> Q(u,i) = sum(e(u,i) * e(u,i)) / 2 + lambda * [(U[u,] * (U[u,]^t)) + (M[i,] * (M[i,]^t))] / 2
+</code></pre>
<p>where Q(u, i) is the objecive function for user u and item i,
e(u, i) is the error between predicted rating and actual rating,
@@ -339,7 +339,7 @@ So our matrix factorization target could be mathmatically defined as:</p>
<p>The algorithm is sketched as follows:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> init U and M with randomized value between 0.0 and 1.0 with standard Gaussian distribution
+<pre><code> init U and M with randomized value between 0.0 and 1.0 with standard Gaussian distribution
for(iter = 0; iter < numIterations; iter++)
{
@@ -360,7 +360,7 @@ So our matrix factorization target could be mathmatically defined as:</p>
U[u,] = NU[u,]
}
}
-</code></pre></div></div>
+</code></pre>
<h2 id="svd">SVD++</h2>
@@ -370,18 +370,18 @@ So our matrix factorization target could be mathmatically defined as:</p>
<p>The complete model is a sum of 3 sub-models with complete prediction formula as follows:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pr(u,i) = b[u,i] + fm + nm //user u and item i
+<pre><code>pr(u,i) = b[u,i] + fm + nm //user u and item i
pr(u,i) is the predicted rating of user u on item i,
b[u,i] = U + b(u) + b(i)
fm = (q[i,]) * (p[u,] + pow(|N(u)|, -0.5) * sum(y[j,])), j is an item in N(u)
nm = pow(|R(i;u;k)|, -0.5) * sum((r[u,j0] - b[u,j0]) * w[i,j0]) + pow(|N(i;u;k)|, -0.5) * sum(c[i,j1]), j0 is an item in R(i;u;k), j1 is an item in N(i;u;k)
-</code></pre></div></div>
+</code></pre>
<p>The associated regularized squared error function to be minimized is:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{sum((r[u,i] - pr[u,i]) * (r[u,i] - pr[u,i])) - lambda * (b(u) * b(u) + b(i) * b(i) + ||q[i,]||^2 + ||p[u,]||^2 + sum(||y[j,]||^2) + sum(w[i,j0] * w[i,j0]) + sum(c[i,j1] * c[i,j1]))}
-</code></pre></div></div>
+<pre><code>{sum((r[u,i] - pr[u,i]) * (r[u,i] - pr[u,i])) - lambda * (b(u) * b(u) + b(i) * b(i) + ||q[i,]||^2 + ||p[u,]||^2 + sum(||y[j,]||^2) + sum(w[i,j0] * w[i,j0]) + sum(c[i,j1] * c[i,j1]))}
+</code></pre>
<p>b[u,i] is the baseline estimate of user u’s predicted rating on item i. U is users’ overall average rating and b(u) and b(i) indicate the observed deviations of user u and item i’s ratings from average.</p>
@@ -420,12 +420,12 @@ please refer to the paper <a href="http://research.yahoo.com/files/kdd08koren.pd
<p>The update to q, p, y in each gradient descent step is:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> err(u,i) = r[u,i] - pr[u,i]
+<pre><code> err(u,i) = r[u,i] - pr[u,i]
q[i,] = q[i,] + alpha * (err(u,i) * (p[u,] + pow(|N(u)|, -0.5) * sum(y[j,])) - lamda * q[i,])
p[u,] = p[u,] + alpha * (err(u,i) * q[i,] - lambda * p[u,])
for j that is an item in N(u):
y[j,] = y[j,] + alpha * (err(u,i) * pow(|N(u)|, -0.5) * q[i,] - lambda * y[j,])
-</code></pre></div></div>
+</code></pre>
<p>where alpha is the learning rate of gradient descent, N(u) is the items that user u has expressed preference.</p>
@@ -443,8 +443,8 @@ vanilla SGD.</p>
<p>ALSWR is an iterative algorithm to solve the low rank factorization of user feature matrix U and item feature matrix M.<br />
The loss function to be minimized is formulated as the sum of squared errors plus <a href="http://en.wikipedia.org/wiki/Tikhonov_regularization">Tikhonov regularization</a>:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> L(R, U, M) = sum(pow((R[u,i] - U[u,]* (M[i,]^t)), 2)) + lambda * (sum(n(u) * ||U[u,]||^2) + sum(n(i) * ||M[i,]||^2))
-</code></pre></div></div>
+<pre><code> L(R, U, M) = sum(pow((R[u,i] - U[u,]* (M[i,]^t)), 2)) + lambda * (sum(n(u) * ||U[u,]||^2) + sum(n(i) * ||M[i,]||^2))
+</code></pre>
<p>At the beginning of the algorithm, M is initialized with the average item ratings as its first row and random numbers for the rest row.</p>
@@ -454,14 +454,14 @@ the cost function similarly. The iteration stops until a certain stopping criter
<p>To solve the matrix U when M is given, each user’s feature vector is calculated by resolving a regularized linear least square error function
using the items the user has rated and their feature vectors:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> 1/2 * d(L(R,U,M)) / d(U[u,f]) = 0
-</code></pre></div></div>
+<pre><code> 1/2 * d(L(R,U,M)) / d(U[u,f]) = 0
+</code></pre>
<p>Similary, when M is updated, we resolve a regularized linear least square error function using feature vectors of the users that have rated the
item and their feature vectors:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> 1/2 * d(L(R,U,M)) / d(M[i,f]) = 0
-</code></pre></div></div>
+<pre><code> 1/2 * d(L(R,U,M)) / d(M[i,f]) = 0
+</code></pre>
<p>The ALSWRFactorizer class is a non-distributed implementation of ALSWR using multi-threading to dispatch the computation among several threads.
Mahout also offers a <a href="https://mahout.apache.org/users/recommender/intro-als-hadoop.html">parallel map-reduce implementation</a>.</p>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/recommender/recommender-documentation.html
----------------------------------------------------------------------
diff --git a/users/recommender/recommender-documentation.html b/users/recommender/recommender-documentation.html
index d1a4b44..55a508b 100644
--- a/users/recommender/recommender-documentation.html
+++ b/users/recommender/recommender-documentation.html
@@ -375,34 +375,34 @@ We’ll start with an example of this.</p>
on data in a file. The file should be in CSV format, with lines of the form
“userID,itemID,prefValue” (e.g. “39505,290002,3.5”):</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>DataModel model = new FileDataModel(new File("data.txt"));
-</code></pre></div></div>
+<pre><code>DataModel model = new FileDataModel(new File("data.txt"));
+</code></pre>
<p>We’ll use the <strong>PearsonCorrelationSimilarity</strong> implementation of <strong>UserSimilarity</strong>
as our user correlation algorithm, and add an optional preference inference
algorithm:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>UserSimilarity userSimilarity = new PearsonCorrelationSimilarity(model);
-</code></pre></div></div>
+<pre><code>UserSimilarity userSimilarity = new PearsonCorrelationSimilarity(model);
+</code></pre>
<p>Now we create a <strong>UserNeighborhood</strong> algorithm. Here we use nearest-3:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>UserNeighborhood neighborhood =
+<pre><code>UserNeighborhood neighborhood =
new NearestNUserNeighborhood(3, userSimilarity, model);{code}
-</code></pre></div></div>
+</code></pre>
<p>Now we can create our <strong>Recommender</strong>, and add a caching decorator:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Recommender recommender =
+<pre><code>Recommender recommender =
new GenericUserBasedRecommender(model, neighborhood, userSimilarity);
Recommender cachingRecommender = new CachingRecommender(recommender);
-</code></pre></div></div>
+</code></pre>
<p>Now we can get 10 recommendations for user ID “1234” — done!</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>List<RecommendedItem> recommendations =
+<pre><code>List<RecommendedItem> recommendations =
cachingRecommender.recommend(1234, 10);
-</code></pre></div></div>
+</code></pre>
<h2 id="item-based-recommender">Item-based Recommender</h2>
@@ -417,8 +417,8 @@ are more appropriate.</p>
<p>Let’s start over, again with a <strong>FileDataModel</strong> to start:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>DataModel model = new FileDataModel(new File("data.txt"));
-</code></pre></div></div>
+<pre><code>DataModel model = new FileDataModel(new File("data.txt"));
+</code></pre>
<p>We’ll also need an <strong>ItemSimilarity</strong>. We could use
<strong>PearsonCorrelationSimilarity</strong>, which computes item similarity in realtime,
@@ -426,22 +426,22 @@ but, this is generally too slow to be useful. Instead, in a real
application, you would feed a list of pre-computed correlations to a
<strong>GenericItemSimilarity</strong>:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>// Construct the list of pre-computed correlations
+<pre><code>// Construct the list of pre-computed correlations
Collection<GenericItemSimilarity.ItemItemSimilarity> correlations =
...;
ItemSimilarity itemSimilarity =
new GenericItemSimilarity(correlations);
-</code></pre></div></div>
+</code></pre>
<p>Then we can finish as before to produce recommendations:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Recommender recommender =
+<pre><code>Recommender recommender =
new GenericItemBasedRecommender(model, itemSimilarity);
Recommender cachingRecommender = new CachingRecommender(recommender);
...
List<RecommendedItem> recommendations =
cachingRecommender.recommend(1234, 10);
-</code></pre></div></div>
+</code></pre>
<p><a name="RecommenderDocumentation-Integrationwithyourapplication"></a></p>
<h2 id="integration-with-your-application">Integration with your application</h2>
@@ -499,7 +499,7 @@ provides a good starting point.</p>
<p>Fortunately, Mahout provides a way to evaluate the accuracy of your
Recommender on your own data, in org.apache.mahout.cf.taste.eval</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>DataModel myModel = ...;
+<pre><code>DataModel myModel = ...;
RecommenderBuilder builder = new RecommenderBuilder() {
public Recommender buildRecommender(DataModel model) {
// build and return the Recommender to evaluate here
@@ -508,7 +508,7 @@ RecommenderBuilder builder = new RecommenderBuilder() {
RecommenderEvaluator evaluator =
new AverageAbsoluteDifferenceRecommenderEvaluator();
double evaluation = evaluator.evaluate(builder, myModel, 0.9, 1.0);
-</code></pre></div></div>
+</code></pre>
<p>For “boolean” data model situations, where there are no notions of
preference value, the above evaluation based on estimated preference does
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/sparkbindings/faq.html
----------------------------------------------------------------------
diff --git a/users/sparkbindings/faq.html b/users/sparkbindings/faq.html
index 6f3627c..4665f3a 100644
--- a/users/sparkbindings/faq.html
+++ b/users/sparkbindings/faq.html
@@ -289,10 +289,10 @@ the classpath is sane and is made available to Mahout:</p>
<ol>
<li>Check Spark is of correct version (same as in Mahout’s poms), is compiled and SPARK_HOME is set.</li>
<li>Check Mahout is compiled and MAHOUT_HOME is set.</li>
- <li>Run <code class="highlighter-rouge">$SPARK_HOME/bin/compute-classpath.sh</code> and make sure it produces sane result with no errors.
+ <li>Run <code>$SPARK_HOME/bin/compute-classpath.sh</code> and make sure it produces sane result with no errors.
If it outputs something other than a straightforward classpath string, most likely Spark is not compiled/set correctly (later spark versions require
-<code class="highlighter-rouge">sbt/sbt assembly</code> to be run, simply runnig <code class="highlighter-rouge">sbt/sbt publish-local</code> is not enough any longer).</li>
- <li>Run <code class="highlighter-rouge">$MAHOUT_HOME/bin/mahout -spark classpath</code> and check that path reported in step (3) is included.</li>
+<code>sbt/sbt assembly</code> to be run, simply runnig <code>sbt/sbt publish-local</code> is not enough any longer).</li>
+ <li>Run <code>$MAHOUT_HOME/bin/mahout -spark classpath</code> and check that path reported in step (3) is included.</li>
</ol>
<p><strong>Q: I am using the command line Mahout jobs that run on Spark or am writing my own application that uses
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/sparkbindings/home.html
----------------------------------------------------------------------
diff --git a/users/sparkbindings/home.html b/users/sparkbindings/home.html
index 234f4c2..c85c2ec 100644
--- a/users/sparkbindings/home.html
+++ b/users/sparkbindings/home.html
@@ -279,14 +279,14 @@
<p>In short, Scala & Spark Bindings for Mahout is Scala DSL and algebraic optimizer of something like this (actual formula from <strong>(d)spca</strong>)</p>
-<p><code class="highlighter-rouge">\[\mathbf{G}=\mathbf{B}\mathbf{B}^{\top}-\mathbf{C}-\mathbf{C}^{\top}+\mathbf{s}_{q}\mathbf{s}_{q}^{\top}\boldsymbol{\xi}^{\top}\boldsymbol{\xi}\]</code></p>
+<p><code>\[\mathbf{G}=\mathbf{B}\mathbf{B}^{\top}-\mathbf{C}-\mathbf{C}^{\top}+\mathbf{s}_{q}\mathbf{s}_{q}^{\top}\boldsymbol{\xi}^{\top}\boldsymbol{\xi}\]</code></p>
<p>bound to in-core and distributed computations (currently, on Apache Spark).</p>
<p>Mahout Scala & Spark Bindings expression of the above:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> val g = bt.t %*% bt - c - c.t + (s_q cross s_q) * (xi dot xi)
-</code></pre></div></div>
+<pre><code> val g = bt.t %*% bt - c - c.t + (s_q cross s_q) * (xi dot xi)
+</code></pre>
<p>The main idea is that a scientist writing algebraic expressions cannot care less of distributed
operation plans and works <strong>entirely on the logical level</strong> just like he or she would do with R.</p>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/sparkbindings/play-with-shell.html
----------------------------------------------------------------------
diff --git a/users/sparkbindings/play-with-shell.html b/users/sparkbindings/play-with-shell.html
index 694e3e1..a05e764 100644
--- a/users/sparkbindings/play-with-shell.html
+++ b/users/sparkbindings/play-with-shell.html
@@ -375,15 +375,15 @@
<ol>
<li>Download <a href="http://d3kbcqa49mib13.cloudfront.net/spark-1.6.2-bin-hadoop2.6.tgz">Apache Spark 1.6.2</a> and unpack the archive file</li>
- <li>Change to the directory where you unpacked Spark and type <code class="highlighter-rouge">sbt/sbt assembly</code> to build it</li>
- <li>Create a directory for Mahout somewhere on your machine, change to there and checkout the master branch of Apache Mahout from GitHub <code class="highlighter-rouge">git clone https://github.com/apache/mahout mahout</code></li>
- <li>Change to the <code class="highlighter-rouge">mahout</code> directory and build mahout using <code class="highlighter-rouge">mvn -DskipTests clean install</code></li>
+ <li>Change to the directory where you unpacked Spark and type <code>sbt/sbt assembly</code> to build it</li>
+ <li>Create a directory for Mahout somewhere on your machine, change to there and checkout the master branch of Apache Mahout from GitHub <code>git clone https://github.com/apache/mahout mahout</code></li>
+ <li>Change to the <code>mahout</code> directory and build mahout using <code>mvn -DskipTests clean install</code></li>
</ol>
<h2 id="starting-mahouts-spark-shell">Starting Mahout’s Spark shell</h2>
<ol>
- <li>Goto the directory where you unpacked Spark and type <code class="highlighter-rouge">sbin/start-all.sh</code> to locally start Spark</li>
+ <li>Goto the directory where you unpacked Spark and type <code>sbin/start-all.sh</code> to locally start Spark</li>
<li>Open a browser, point it to <a href="http://localhost:8080/">http://localhost:8080/</a> to check whether Spark successfully started. Copy the url of the spark master at the top of the page (it starts with <strong>spark://</strong>)</li>
<li>Define the following environment variables: <pre class="codehilite">export MAHOUT_HOME=[directory into which you checked out Mahout]
export SPARK_HOME=[directory where you unpacked Spark]
@@ -391,8 +391,8 @@ export MASTER=[url of the Spark master]</li>
</ol>
<p></pre></p>
<ol>
- <li>Finally, change to the directory where you unpacked Mahout and type <code class="highlighter-rouge">bin/mahout spark-shell</code>,
-you should see the shell starting and get the prompt <code class="highlighter-rouge">mahout></code>. Check
+ <li>Finally, change to the directory where you unpacked Mahout and type <code>bin/mahout spark-shell</code>,
+you should see the shell starting and get the prompt <code>mahout></code>. Check
<a href="http://mahout.apache.org/users/sparkbindings/faq.html">FAQ</a> for further troubleshooting.</li>
</ol>
@@ -402,7 +402,7 @@ you should see the shell starting and get the prompt <code class="highlighter-ro
<p><em>Note: You can incrementally follow the example by copy-and-pasting the code into your running Mahout shell.</em></p>
-<p>Mahout’s linear algebra DSL has an abstraction called <em>DistributedRowMatrix (DRM)</em> which models a matrix that is partitioned by rows and stored in the memory of a cluster of machines. We use <code class="highlighter-rouge">dense()</code> to create a dense in-memory matrix from our toy dataset and use <code class="highlighter-rouge">drmParallelize</code> to load it into the cluster, “mimicking” a large, partitioned dataset.</p>
+<p>Mahout’s linear algebra DSL has an abstraction called <em>DistributedRowMatrix (DRM)</em> which models a matrix that is partitioned by rows and stored in the memory of a cluster of machines. We use <code>dense()</code> to create a dense in-memory matrix from our toy dataset and use <code>drmParallelize</code> to load it into the cluster, “mimicking” a large, partitioned dataset.</p>
<div class="codehilite"><pre>
val drmData = drmParallelize(dense(
@@ -421,47 +421,47 @@ val drmData = drmParallelize(dense(
<p>Have a look at this matrix. The first four columns represent the ingredients
(our features) and the last column (the rating) is the target variable for
our regression. <a href="https://en.wikipedia.org/wiki/Linear_regression">Linear regression</a>
-assumes that the <strong>target variable</strong> <code class="highlighter-rouge">\(\mathbf{y}\)</code> is generated by the
-linear combination of <strong>the feature matrix</strong> <code class="highlighter-rouge">\(\mathbf{X}\)</code> with the
-<strong>parameter vector</strong> <code class="highlighter-rouge">\(\boldsymbol{\beta}\)</code> plus the
- <strong>noise</strong> <code class="highlighter-rouge">\(\boldsymbol{\varepsilon}\)</code>, summarized in the formula
-<code class="highlighter-rouge">\(\mathbf{y}=\mathbf{X}\boldsymbol{\beta}+\boldsymbol{\varepsilon}\)</code>.
+assumes that the <strong>target variable</strong> <code>\(\mathbf{y}\)</code> is generated by the
+linear combination of <strong>the feature matrix</strong> <code>\(\mathbf{X}\)</code> with the
+<strong>parameter vector</strong> <code>\(\boldsymbol{\beta}\)</code> plus the
+ <strong>noise</strong> <code>\(\boldsymbol{\varepsilon}\)</code>, summarized in the formula
+<code>\(\mathbf{y}=\mathbf{X}\boldsymbol{\beta}+\boldsymbol{\varepsilon}\)</code>.
Our goal is to find an estimate of the parameter vector
-<code class="highlighter-rouge">\(\boldsymbol{\beta}\)</code> that explains the data very well.</p>
+<code>\(\boldsymbol{\beta}\)</code> that explains the data very well.</p>
-<p>As a first step, we extract <code class="highlighter-rouge">\(\mathbf{X}\)</code> and <code class="highlighter-rouge">\(\mathbf{y}\)</code> from our data matrix. We get <em>X</em> by slicing: we take all rows (denoted by <code class="highlighter-rouge">::</code>) and the first four columns, which have the ingredients in milligrams as content. Note that the result is again a DRM. The shell will not execute this code yet, it saves the history of operations and defers the execution until we really access a result. <strong>Mahout’s DSL automatically optimizes and parallelizes all operations on DRMs and runs them on Apache Spark.</strong></p>
+<p>As a first step, we extract <code>\(\mathbf{X}\)</code> and <code>\(\mathbf{y}\)</code> from our data matrix. We get <em>X</em> by slicing: we take all rows (denoted by <code>::</code>) and the first four columns, which have the ingredients in milligrams as content. Note that the result is again a DRM. The shell will not execute this code yet, it saves the history of operations and defers the execution until we really access a result. <strong>Mahout’s DSL automatically optimizes and parallelizes all operations on DRMs and runs them on Apache Spark.</strong></p>
<div class="codehilite"><pre>
val drmX = drmData(::, 0 until 4)
</pre></div>
-<p>Next, we extract the target variable vector <em>y</em>, the fifth column of the data matrix. We assume this one fits into our driver machine, so we fetch it into memory using <code class="highlighter-rouge">collect</code>:</p>
+<p>Next, we extract the target variable vector <em>y</em>, the fifth column of the data matrix. We assume this one fits into our driver machine, so we fetch it into memory using <code>collect</code>:</p>
<div class="codehilite"><pre>
val y = drmData.collect(::, 4)
</pre></div>
-<p>Now we are ready to think about a mathematical way to estimate the parameter vector <em>β</em>. A simple textbook approach is <a href="https://en.wikipedia.org/wiki/Ordinary_least_squares">ordinary least squares (OLS)</a>, which minimizes the sum of residual squares between the true target variable and the prediction of the target variable. In OLS, there is even a closed form expression for estimating <code class="highlighter-rouge">\(\boldsymbol{\beta}\)</code> as
-<code class="highlighter-rouge">\(\left(\mathbf{X}^{\top}\mathbf{X}\right)^{-1}\mathbf{X}^{\top}\mathbf{y}\)</code>.</p>
+<p>Now we are ready to think about a mathematical way to estimate the parameter vector <em>β</em>. A simple textbook approach is <a href="https://en.wikipedia.org/wiki/Ordinary_least_squares">ordinary least squares (OLS)</a>, which minimizes the sum of residual squares between the true target variable and the prediction of the target variable. In OLS, there is even a closed form expression for estimating <code>\(\boldsymbol{\beta}\)</code> as
+<code>\(\left(\mathbf{X}^{\top}\mathbf{X}\right)^{-1}\mathbf{X}^{\top}\mathbf{y}\)</code>.</p>
-<p>The first thing which we compute for this is <code class="highlighter-rouge">\(\mathbf{X}^{\top}\mathbf{X}\)</code>. The code for doing this in Mahout’s scala DSL maps directly to the mathematical formula. The operation <code class="highlighter-rouge">.t()</code> transposes a matrix and analogous to R <code class="highlighter-rouge">%*%</code> denotes matrix multiplication.</p>
+<p>The first thing which we compute for this is <code>\(\mathbf{X}^{\top}\mathbf{X}\)</code>. The code for doing this in Mahout’s scala DSL maps directly to the mathematical formula. The operation <code>.t()</code> transposes a matrix and analogous to R <code>%*%</code> denotes matrix multiplication.</p>
<div class="codehilite"><pre>
val drmXtX = drmX.t %*% drmX
</pre></div>
-<p>The same is true for computing <code class="highlighter-rouge">\(\mathbf{X}^{\top}\mathbf{y}\)</code>. We can simply type the math in scala expressions into the shell. Here, <em>X</em> lives in the cluster, while is <em>y</em> in the memory of the driver, and the result is a DRM again.</p>
+<p>The same is true for computing <code>\(\mathbf{X}^{\top}\mathbf{y}\)</code>. We can simply type the math in scala expressions into the shell. Here, <em>X</em> lives in the cluster, while is <em>y</em> in the memory of the driver, and the result is a DRM again.</p>
<div class="codehilite"><pre>
val drmXty = drmX.t %*% y
</pre></div>
-<p>We’re nearly done. The next step we take is to fetch <code class="highlighter-rouge">\(\mathbf{X}^{\top}\mathbf{X}\)</code> and
-<code class="highlighter-rouge">\(\mathbf{X}^{\top}\mathbf{y}\)</code> into the memory of our driver machine (we are targeting
+<p>We’re nearly done. The next step we take is to fetch <code>\(\mathbf{X}^{\top}\mathbf{X}\)</code> and
+<code>\(\mathbf{X}^{\top}\mathbf{y}\)</code> into the memory of our driver machine (we are targeting
features matrices that are tall and skinny ,
-so we can assume that <code class="highlighter-rouge">\(\mathbf{X}^{\top}\mathbf{X}\)</code> is small enough
+so we can assume that <code>\(\mathbf{X}^{\top}\mathbf{X}\)</code> is small enough
to fit in). Then, we provide them to an in-memory solver (Mahout provides
-the an analog to R’s <code class="highlighter-rouge">solve()</code> for that) which computes <code class="highlighter-rouge">beta</code>, our
-OLS estimate of the parameter vector <code class="highlighter-rouge">\(\boldsymbol{\beta}\)</code>.</p>
+the an analog to R’s <code>solve()</code> for that) which computes <code>beta</code>, our
+OLS estimate of the parameter vector <code>\(\boldsymbol{\beta}\)</code>.</p>
<div class="codehilite"><pre>
val XtX = drmXtX.collect
@@ -478,9 +478,9 @@ as much as possible, while still retaining decent performance and
scalability.</p>
<p>We can now check how well our model fits its training data.
-First, we multiply the feature matrix <code class="highlighter-rouge">\(\mathbf{X}\)</code> by our estimate of
-<code class="highlighter-rouge">\(\boldsymbol{\beta}\)</code>. Then, we look at the difference (via L2-norm) of
-the target variable <code class="highlighter-rouge">\(\mathbf{y}\)</code> to the fitted target variable:</p>
+First, we multiply the feature matrix <code>\(\mathbf{X}\)</code> by our estimate of
+<code>\(\boldsymbol{\beta}\)</code>. Then, we look at the difference (via L2-norm) of
+the target variable <code>\(\mathbf{y}\)</code> to the fitted target variable:</p>
<div class="codehilite"><pre>
val yFitted = (drmX %*% beta).collect(::, 0)
@@ -489,7 +489,7 @@ val yFitted = (drmX %*% beta).collect(::, 0)
<p>We hope that we could show that Mahout’s shell allows people to interactively and incrementally write algorithms. We have entered a lot of individual commands, one-by-one, until we got the desired results. We can now refactor a little by wrapping our statements into easy-to-use functions. The definition of functions follows standard scala syntax.</p>
-<p>We put all the commands for ordinary least squares into a function <code class="highlighter-rouge">ols</code>.</p>
+<p>We put all the commands for ordinary least squares into a function <code>ols</code>.</p>
<div class="codehilite"><pre>
def ols(drmX: DrmLike[Int], y: Vector) =
@@ -497,10 +497,10 @@ def ols(drmX: DrmLike[Int], y: Vector) =
</pre></div>
-<p>Note that DSL declares implicit <code class="highlighter-rouge">collect</code> if coersion rules require an in-core argument. Hence, we can simply
-skip explicit <code class="highlighter-rouge">collect</code>s.</p>
+<p>Note that DSL declares implicit <code>collect</code> if coersion rules require an in-core argument. Hence, we can simply
+skip explicit <code>collect</code>s.</p>
-<p>Next, we define a function <code class="highlighter-rouge">goodnessOfFit</code> that tells how well a model fits the target variable:</p>
+<p>Next, we define a function <code>goodnessOfFit</code> that tells how well a model fits the target variable:</p>
<div class="codehilite"><pre>
def goodnessOfFit(drmX: DrmLike[Int], beta: Vector, y: Vector) = {
@@ -513,7 +513,7 @@ def goodnessOfFit(drmX: DrmLike[Int], beta: Vector, y: Vector) = {
model. Usually there is a constant bias term added to the model. Without
that, our model always crosses through the origin and we only learn the
right angle. An easy way to add such a bias term to our model is to add a
-column of ones to the feature matrix <code class="highlighter-rouge">\(\mathbf{X}\)</code>.
+column of ones to the feature matrix <code>\(\mathbf{X}\)</code>.
The corresponding weight in the parameter vector will then be the bias term.</p>
<p>Here is how we add a bias column:</p>
@@ -522,14 +522,14 @@ The corresponding weight in the parameter vector will then be the bias term.</p>
val drmXwithBiasColumn = drmX cbind 1
</pre></div>
-<p>Now we can give the newly created DRM <code class="highlighter-rouge">drmXwithBiasColumn</code> to our model fitting method <code class="highlighter-rouge">ols</code> and see how well the resulting model fits the training data with <code class="highlighter-rouge">goodnessOfFit</code>. You should see a large improvement in the result.</p>
+<p>Now we can give the newly created DRM <code>drmXwithBiasColumn</code> to our model fitting method <code>ols</code> and see how well the resulting model fits the training data with <code>goodnessOfFit</code>. You should see a large improvement in the result.</p>
<div class="codehilite"><pre>
val betaWithBiasTerm = ols(drmXwithBiasColumn, y)
goodnessOfFit(drmXwithBiasColumn, betaWithBiasTerm, y)
</pre></div>
-<p>As a further optimization, we can make use of the DSL’s caching functionality. We use <code class="highlighter-rouge">drmXwithBiasColumn</code> repeatedly as input to a computation, so it might be beneficial to cache it in memory. This is achieved by calling <code class="highlighter-rouge">checkpoint()</code>. In the end, we remove it from the cache with uncache:</p>
+<p>As a further optimization, we can make use of the DSL’s caching functionality. We use <code>drmXwithBiasColumn</code> repeatedly as input to a computation, so it might be beneficial to cache it in memory. This is achieved by calling <code>checkpoint()</code>. In the end, we remove it from the cache with uncache:</p>
<div class="codehilite"><pre>
val cachedDrmX = drmXwithBiasColumn.checkpoint()
[4/4] mahout git commit: Automatic Site Publish by Buildbot
Posted by gi...@apache.org.
Automatic Site Publish by Buildbot
Project: http://git-wip-us.apache.org/repos/asf/mahout/repo
Commit: http://git-wip-us.apache.org/repos/asf/mahout/commit/d9686c8b
Tree: http://git-wip-us.apache.org/repos/asf/mahout/tree/d9686c8b
Diff: http://git-wip-us.apache.org/repos/asf/mahout/diff/d9686c8b
Branch: refs/heads/asf-site
Commit: d9686c8ba68c9126f93a0dfab0fe16e877fb493b
Parents: f496120
Author: jenkins <bu...@apache.org>
Authored: Thu Nov 30 00:23:36 2017 +0000
Committer: jenkins <bu...@apache.org>
Committed: Thu Nov 30 00:23:36 2017 +0000
----------------------------------------------------------------------
developers/buildingmahout.html | 82 +++----
developers/github.html | 54 ++---
developers/githubPRs.html | 16 +-
developers/how-to-release.html | 18 +-
developers/how-to-update-the-website.html | 4 +-
docs/latest/sitemap.xml | 2 +-
general/downloads.html | 18 +-
general/mahout-wiki.html | 4 +-
sitemap.xml | 6 +-
users/algorithms/d-als.html | 14 +-
users/algorithms/d-qr.html | 14 +-
users/algorithms/d-spca.html | 54 ++---
users/algorithms/d-ssvd.html | 60 ++---
users/algorithms/intro-cooccurrence-spark.html | 72 +++---
users/algorithms/spark-naive-bayes.html | 66 +++---
users/basics/collocations.html | 28 +--
users/basics/creating-vectors-from-text.html | 32 +--
users/basics/quickstart.html | 4 +-
users/classification/bayesian-commandline.html | 8 +-
users/classification/bayesian.html | 64 ++---
users/classification/breiman-example.html | 16 +-
users/classification/class-discovery.html | 8 +-
users/classification/hidden-markov-models.html | 20 +-
users/classification/mlp.html | 44 ++--
.../classification/partial-implementation.html | 12 +-
users/classification/twenty-newsgroups.html | 52 ++---
.../wikipedia-classifier-example.html | 18 +-
users/clustering/canopy-clustering.html | 4 +-
users/clustering/canopy-commandline.html | 8 +-
users/clustering/cluster-dumper.html | 12 +-
.../clustering-of-synthetic-control-data.html | 12 +-
users/clustering/clusteringyourdata.html | 8 +-
users/clustering/fuzzy-k-means-commandline.html | 8 +-
users/clustering/fuzzy-k-means.html | 4 +-
users/clustering/k-means-clustering.html | 8 +-
users/clustering/k-means-commandline.html | 8 +-
.../clustering/latent-dirichlet-allocation.html | 8 +-
users/clustering/lda-commandline.html | 8 +-
users/clustering/spectral-clustering.html | 34 +--
users/clustering/streaming-k-means.html | 48 ++--
users/clustering/viewing-results.html | 8 +-
.../clustering/visualizing-sample-clusters.html | 4 +-
users/dim-reduction/ssvd.html | 52 ++---
.../classify-a-doc-from-the-shell.html | 88 +++----
users/environment/h2o-internals.html | 8 +-
users/environment/how-to-build-an-app.html | 78 +++----
users/environment/in-core-reference.html | 234 +++++++++----------
users/environment/out-of-core-reference.html | 192 +++++++--------
.../playing-with-samsara-flink.html | 48 ++--
.../misc/parallel-frequent-pattern-mining.html | 4 +-
.../using-mahout-with-python-via-jpype.html | 16 +-
users/recommender/intro-als-hadoop.html | 8 +-
users/recommender/intro-cooccurrence-spark.html | 72 +++---
users/recommender/intro-itembased-hadoop.html | 4 +-
users/recommender/matrix-factorization.html | 52 ++---
.../recommender/recommender-documentation.html | 36 +--
users/sparkbindings/faq.html | 6 +-
users/sparkbindings/home.html | 6 +-
users/sparkbindings/play-with-shell.html | 68 +++---
59 files changed, 972 insertions(+), 972 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/developers/buildingmahout.html
----------------------------------------------------------------------
diff --git a/developers/buildingmahout.html b/developers/buildingmahout.html
index f7df86c..e47030a 100644
--- a/developers/buildingmahout.html
+++ b/developers/buildingmahout.html
@@ -286,10 +286,10 @@
<p>Checkout the sources from the <a href="https://github.com/apache/mahout">Mahout GitHub repository</a>
either via</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git clone git@github.com:apache/mahout.git or
+<pre><code>git clone git@github.com:apache/mahout.git or
git clone https://github.com/apache/mahout.git
-</code></pre></div></div>
+</code></pre>
<h2 id="building-from-source">Building From Source</h2>
@@ -320,49 +320,49 @@ Choose package type: Pre-Built for Hadoop 2.4</p>
<p>Install ViennaCL 1.7.0+
If running Ubuntu 16.04+</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo apt-get install libviennacl-dev
-</code></pre></div></div>
+<pre><code>sudo apt-get install libviennacl-dev
+</code></pre>
<p>Otherwise if your distribution’s package manager does not have a viennniacl-dev package >1.7.0, clone it directly into the directory which will be included in when being compiled by Mahout:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mkdir ~/tmp
+<pre><code>mkdir ~/tmp
cd ~/tmp && git clone https://github.com/viennacl/viennacl-dev.git
cp -r viennacl/ /usr/local/
cp -r CL/ /usr/local/
-</code></pre></div></div>
+</code></pre>
<p>Ensure that the OpenCL 1.2+ drivers are installed (packed with most consumer grade NVIDIA drivers). Not sure about higher end cards.</p>
-<p>Clone mahout repository into <code class="highlighter-rouge">~/apache</code>.</p>
+<p>Clone mahout repository into <code>~/apache</code>.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git clone https://github.com/apache/mahout.git
-</code></pre></div></div>
+<pre><code>git clone https://github.com/apache/mahout.git
+</code></pre>
<h6 id="configuration">Configuration</h6>
<p>When building mahout for a spark backend, we need four System Environment variables set:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> export MAHOUT_HOME=/home/<user>/apache/mahout
+<pre><code> export MAHOUT_HOME=/home/<user>/apache/mahout
export HADOOP_HOME=/home/<user>/apache/hadoop-2.4.1
export SPARK_HOME=/home/<user>/apache/spark-1.6.3-bin-hadoop2.4
export JAVA_HOME=/home/<user>/java/jdk-1.8.121
-</code></pre></div></div>
+</code></pre>
<p>Mahout on Spark regularly uses one more env variable, the IP of the Spark cluster’s master node (usually the node which one would be logged into).</p>
<p>To use 4 local cores (Spark master need not be running)</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>export MASTER=local[4]
-</code></pre></div></div>
+<pre><code>export MASTER=local[4]
+</code></pre>
<p>To use all available local cores (again, Spark master need not be running)</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>export MASTER=local[*]
-</code></pre></div></div>
+<pre><code>export MASTER=local[*]
+</code></pre>
<p>To point to a cluster with spark running:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>export MASTER=spark://master.ip.address:7077
-</code></pre></div></div>
+<pre><code>export MASTER=spark://master.ip.address:7077
+</code></pre>
<p>We then add these to the path:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> PATH=$PATH$:MAHOUT_HOME/bin:$HADOOP_HOME/bin:$SPARK_HOME/bin:$JAVA_HOME/bin
-</code></pre></div></div>
+<pre><code> PATH=$PATH$:MAHOUT_HOME/bin:$HADOOP_HOME/bin:$SPARK_HOME/bin:$JAVA_HOME/bin
+</code></pre>
<p>These should be added to the your ~/.bashrc file.</p>
@@ -371,35 +371,35 @@ cp -r CL/ /usr/local/
<p>From the $MAHOUT_HOME directory we may issue the commands to build each using mvn profiles.</p>
<p>JVM only:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mvn clean install -DskipTests
-</code></pre></div></div>
+<pre><code>mvn clean install -DskipTests
+</code></pre>
<p>JVM with native OpenMP level 2 and level 3 matrix/vector Multiplication</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mvn clean install -Pviennacl-omp -Phadoop2 -DskipTests
-</code></pre></div></div>
+<pre><code>mvn clean install -Pviennacl-omp -Phadoop2 -DskipTests
+</code></pre>
<p>JVM with native OpenMP and OpenCL for Level 2 and level 3 matrix/vector Multiplication. (GPU errors fall back to OpenMP, currently only a single GPU/node is supported).</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mvn clean install -Pviennacl -Phadoop2 -DskipTests
-</code></pre></div></div>
+<pre><code>mvn clean install -Pviennacl -Phadoop2 -DskipTests
+</code></pre>
<h3 id="changing-scala-version">Changing Scala Version</h3>
<p>To change the Scala version used it is possible to use profiles, however the resulting artifacts seem to have trouble being resolved with SBT.</p>
-<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mvn clean install <span class="nt">-Pscala-2</span>.11
-</code></pre></div></div>
+<pre><code class="language-bash">mvn clean install -Pscala-2.11
+</code></pre>
<p>Maven is able to resolve the resulting artifacts effectively, this will also work if the goal is simply to use the Mahout-Shell. However if the goal is to build with SBT, the following tool should be used</p>
-<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cd</span> <span class="nv">$MAHOUT_HOME</span>/buildtools
+<pre><code class="language-bash">cd $MAHOUT_HOME/buildtools
./change-scala-version.sh 2.11
-</code></pre></div></div>
+</code></pre>
-<p>Now go back to <code class="highlighter-rouge">$MAHOUT_HOME</code> and execute</p>
+<p>Now go back to <code>$MAHOUT_HOME</code> and execute</p>
-<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mvn clean install <span class="nt">-Pscala-2</span>.11
-</code></pre></div></div>
+<pre><code class="language-bash">mvn clean install -Pscala-2.11
+</code></pre>
-<p><strong>NOTE:</strong> you still need to pass the <code class="highlighter-rouge">-Pscala-2.11</code> profile, as this determines and propegates the minor scala version (e.g. 2.11.8)</p>
+<p><strong>NOTE:</strong> you still need to pass the <code>-Pscala-2.11</code> profile, as this determines and propegates the minor scala version (e.g. 2.11.8)</p>
<h3 id="the-distribution-profile">The Distribution Profile</h3>
@@ -426,21 +426,21 @@ cp -r CL/ /usr/local/
<li>H2O Scala-2.11</li>
</ul>
-<p>Note: * ViennaCLs are only created if the <code class="highlighter-rouge">viennacl</code> or <code class="highlighter-rouge">viennacl-omp</code> profiles are activated.</p>
+<p>Note: * ViennaCLs are only created if the <code>viennacl</code> or <code>viennacl-omp</code> profiles are activated.</p>
-<p>By default, this phase will execute the <code class="highlighter-rouge">package</code> lifecycle goal on all built “extra” varients.</p>
+<p>By default, this phase will execute the <code>package</code> lifecycle goal on all built “extra” varients.</p>
<p>E.g. if you were to run</p>
-<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mvn clean install <span class="nt">-Pdistribution</span>
-</code></pre></div></div>
+<pre><code class="language-bash">mvn clean install -Pdistribution
+</code></pre>
-<p>You will <code class="highlighter-rouge">install</code> all of the “Default Targets” but only <code class="highlighter-rouge">package</code> the “Also created”.</p>
+<p>You will <code>install</code> all of the “Default Targets” but only <code>package</code> the “Also created”.</p>
-<p>If you wish to <code class="highlighter-rouge">install</code> all of the above, you can set the <code class="highlighter-rouge">lifecycle.target</code> switch as follows:</p>
+<p>If you wish to <code>install</code> all of the above, you can set the <code>lifecycle.target</code> switch as follows:</p>
-<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mvn clean install <span class="nt">-Pdistribution</span> <span class="nt">-Dlifecycle</span>.target<span class="o">=</span>install
-</code></pre></div></div>
+<pre><code class="language-bash">mvn clean install -Pdistribution -Dlifecycle.target=install
+</code></pre>
</div>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/developers/github.html
----------------------------------------------------------------------
diff --git a/developers/github.html b/developers/github.html
index 24fe0e2..0ea9f59 100644
--- a/developers/github.html
+++ b/developers/github.html
@@ -289,29 +289,29 @@ So if you perform “git push origin master” it will go to github.</p>
<p>To attach to the apache git repo do the following:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git remote add apache https://git-wip-us.apache.org/repos/asf/mahout.git
-</code></pre></div></div>
+<pre><code>git remote add apache https://git-wip-us.apache.org/repos/asf/mahout.git
+</code></pre>
<p>To check your remote setup</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git remote -v
-</code></pre></div></div>
+<pre><code>git remote -v
+</code></pre>
<p>you should see something like this:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>origin https://github.com/your-github-id/mahout.git (fetch)
+<pre><code>origin https://github.com/your-github-id/mahout.git (fetch)
origin https://github.com/your-github-id/mahout.git (push)
apache https://git-wip-us.apache.org/repos/asf/mahout.git (fetch)
apache https://git-wip-us.apache.org/repos/asf/mahout.git (push)
-</code></pre></div></div>
+</code></pre>
<p>Now if you want to experiment with a branch everything, by default, points to your github account because ‘origin’ is default. You can work as normal using only github until you are ready to merge with the apache remote. Some conventions will integrate with Apache Jira ticket numbers.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git checkout -b mahout-xxxx #xxxx typically is a Jira ticket number
+<pre><code>git checkout -b mahout-xxxx #xxxx typically is a Jira ticket number
#do some work on the branch
git commit -a -m "doing some work"
git push origin mahout-xxxx # notice pushing to **origin** not **apache**
-</code></pre></div></div>
+</code></pre>
<p>Once you are ready to commit to the apache remote you can merge and push them directly or better yet create a PR.</p>
@@ -319,9 +319,9 @@ git push origin mahout-xxxx # notice pushing to **origin** not **apache**
<p>Push your branch to Github:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git checkout mahout-xxxx
+<pre><code>git checkout mahout-xxxx
git push origin mahout-xxxx
-</code></pre></div></div>
+</code></pre>
<p>Go to your mahout-xxxx branch on Github. Since you forked it from Github’s apache/mahout it will default
any PR to go to apache/master.</p>
@@ -362,45 +362,45 @@ same time, it is recommended to use <strong>squash commits</strong>.</p>
<p>Merging pull requests are equivalent to a “pull” of a contributor’s branch:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git checkout master # switch to local master branch
+<pre><code>git checkout master # switch to local master branch
git pull apache master # fast-forward to current remote HEAD
git pull --squash https://github.com/cuser/mahout cbranch # merge to master
-</code></pre></div></div>
+</code></pre>
<p>–squash ensures all PR history is squashed into single commit, and allows committer to use his/her own
-message. Read git help for merge or pull for more information about <code class="highlighter-rouge">--squash</code> option. In this example we
+message. Read git help for merge or pull for more information about <code>--squash</code> option. In this example we
assume that the contributor’s Github handle is “cuser” and the PR branch name is “cbranch”.
Next, resolve conflicts, if any, or ask a contributor to rebase on top of master, if PR went out of sync.</p>
<p>If you are ready to merge your own (committer’s) PR you probably only need to merge (not pull), since you have a local copy
that you’ve been working on. This is the branch that you used to create the PR.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git checkout master # switch to local master branch
+<pre><code>git checkout master # switch to local master branch
git pull apache master # fast-forward to current remote HEAD
git merge --squash mahout-xxxx
-</code></pre></div></div>
+</code></pre>
<p>Remember to run regular patch checks, build with tests enabled, and change CHANGELOG.</p>
<p>If everything is fine, you now can commit the squashed request along the lines</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git commit --author <contributor_email> -a -m "MAHOUT-XXXX description closes apache/mahout#ZZ"
-</code></pre></div></div>
+<pre><code>git commit --author <contributor_email> -a -m "MAHOUT-XXXX description closes apache/mahout#ZZ"
+</code></pre>
-<p>MAHOUT-XXXX is all caps and where <code class="highlighter-rouge">ZZ</code> is the pull request number on apache/mahout repository. Including
+<p>MAHOUT-XXXX is all caps and where <code>ZZ</code> is the pull request number on apache/mahout repository. Including
“closes apache/mahout#ZZ” will close the PR automatically. More information is found here [<a href="https://help.github.com/articles/closing-issues-via-commit-messages">3</a>].</p>
<p>Next, push to git-wip-us.a.o:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>push apache master
-</code></pre></div></div>
+<pre><code>push apache master
+</code></pre>
<p>(this will require Apache handle credentials).</p>
<p>The PR, once pushed, will get mirrored to github. To update your github version push there too:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>push origin master
-</code></pre></div></div>
+<pre><code>push origin master
+</code></pre>
<p><em>Note on squashing: Since squash discards remote branch history, repeated PRs from the same remote branch are
difficult for merging. The workflow implies that every new PR starts with a new rebased branch. This is more
@@ -412,11 +412,11 @@ would warn to begin with. Anyway, watch for dupe PRs (based on same source branc
<p>When we want to reject a PR (close without committing), we can just issue an empty commit on master’s HEAD
<em>without merging the PR</em>:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git commit --allow-empty -m "closes apache/mahout#ZZ *Won't fix*"
+<pre><code>git commit --allow-empty -m "closes apache/mahout#ZZ *Won't fix*"
git push apache master
-</code></pre></div></div>
+</code></pre>
-<p>that should close PR <code class="highlighter-rouge">ZZ</code> on github mirror without merging and any code modifications in the master repository.</p>
+<p>that should close PR <code>ZZ</code> on github mirror without merging and any code modifications in the master repository.</p>
<h2 id="apachegithub-integration-features">Apache/github integration features</h2>
@@ -424,8 +424,8 @@ git push apache master
Mahout issue handles must be in the form MAHOUT-YYYYY (all capitals). Usually it makes sense to
file a jira issue first, and then create a PR with description</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>MAHOUT-YYYY: <jira-issue-description>
-</code></pre></div></div>
+<pre><code>MAHOUT-YYYY: <jira-issue-description>
+</code></pre>
<p>In this case all subsequent comments will automatically be copied to jira without having to mention
jira issue explicitly in each comment of the PR.</p>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/developers/githubPRs.html
----------------------------------------------------------------------
diff --git a/developers/githubPRs.html b/developers/githubPRs.html
index 9627917..e083236 100644
--- a/developers/githubPRs.html
+++ b/developers/githubPRs.html
@@ -291,17 +291,17 @@ same time, it is recommended to use <strong>squash commits</strong>.</p>
<p>Read [<a href="https://help.github.com/articles/merging-a-pull-request#merging-locally">2</a>] (merging locally). Merging pull requests are equivalent to merging contributor’s branch:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git checkout master # switch to local master branch
+<pre><code>git checkout master # switch to local master branch
git pull apache master # fast-forward to current remote HEAD
git pull --squash https://github.com/cuser/mahout cbranch # merge to master
-</code></pre></div></div>
+</code></pre>
<p>In this example we assume that contributor Github handle is “cuser” and the PR branch name is “cbranch” there. We also
assume that <em>apache</em> remote is configured as</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>apache https://git-wip-us.apache.org/repos/asf/mahout.git (fetch)
+<pre><code>apache https://git-wip-us.apache.org/repos/asf/mahout.git (fetch)
apache https://git-wip-us.apache.org/repos/asf/mahout.git (push)
-</code></pre></div></div>
+</code></pre>
<p>Squash pull ensures all PR history is squashed into single commit. Also, it is not yet committed, even if
fast forward is possible, so you get chance to change things before committing.</p>
@@ -312,8 +312,8 @@ fast forward is possible, so you get chance to change things before committing.<
<p>Suppose everything is fine, you now can commit the squashed request</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git commit -a
-</code></pre></div></div>
+<pre><code>git commit -a
+</code></pre>
<p>edit message to contain “MAHOUT-YYYY description <strong>closes #ZZ</strong>”, where ZZ is the pull request number.
Including “closes #ZZ” will close PR automatically. More information [<a href="https://help.github.com/articles/closing-issues-via-commit-messages">3</a>].</p>
@@ -332,9 +332,9 @@ would warn to begin with. Anyway, watch for dupe PRs (based on same source branc
<p>When we want to reject a PR (close without committing), just do the following commit on master’s HEAD
<em>without merging the PR</em>:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git commit --allow-empty -m "closes #ZZ *Won't fix*"
+<pre><code>git commit --allow-empty -m "closes #ZZ *Won't fix*"
git push apache master
-</code></pre></div></div>
+</code></pre>
<p>that should close PR without merging and any code modifications in the master repository.</p>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/developers/how-to-release.html
----------------------------------------------------------------------
diff --git a/developers/how-to-release.html b/developers/how-to-release.html
index f25809b..2a6322c 100644
--- a/developers/how-to-release.html
+++ b/developers/how-to-release.html
@@ -312,13 +312,13 @@ warnings)</li>
<p><a name="HowToRelease-Beforebuildingrelease"></a></p>
<h2 id="before-building-release">Before building release</h2>
<ol>
- <li>Check that all tests pass after a clean compile: <code class="highlighter-rouge">mvn clean test</code></li>
+ <li>Check that all tests pass after a clean compile: <code>mvn clean test</code></li>
<li>Check that there are no remaining unresolved Jira issues with the upcoming version number listed as the “Fix” version</li>
<li>Publish any previously unpublished third-party dependenciess: <a href="thirdparty-dependencies.html">Thirdparty Dependencies</a></li>
<li>Build and preview resulting artifacts:
- <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nb">cd </span>buildtools
+ <pre><code class="language-bash"> cd buildtools
./build-all-release-jars.sh
-</code></pre></div> </div>
+</code></pre>
</li>
<li>Make sure packages will come out looking right</li>
</ol>
@@ -333,10 +333,10 @@ warnings)</li>
<li>Ensure you have set up standard Apache committer settings in
~/.m2/settings.xml as per <a href="http://maven.apache.org/developers/committer-settings.html">this page</a>
.</li>
- <li>Add a profile to your <code class="highlighter-rouge">~/.m2/settings.xml</code> in the <code class="highlighter-rouge"><profiles></code> section with:</li>
+ <li>Add a profile to your <code>~/.m2/settings.xml</code> in the <code><profiles></code> section with:</li>
</ul>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code><profiles>
+<pre><code><profiles>
<profile>
<id>mahout_release</id>
<properties>
@@ -348,10 +348,10 @@ warnings)</li>
</properties>
</profile>
</profiles>
-</code></pre></div></div>
+</code></pre>
<ul>
- <li>You may also need to add the following to the <code class="highlighter-rouge"><servers></code> section in <code class="highlighter-rouge">~/.m2/settings.xml</code> in order to upload artifacts (as the <code class="highlighter-rouge">-Dusername=</code> <code class="highlighter-rouge">-Dpassword=</code> didn’t work for gsingers for 0.8, but this did; n.b. it didn’t work for akm for the 0.13 release):
+ <li>You may also need to add the following to the <code><servers></code> section in <code>~/.m2/settings.xml</code> in order to upload artifacts (as the <code>-Dusername=</code> <code>-Dpassword=</code> didn’t work for gsingers for 0.8, but this did; n.b. it didn’t work for akm for the 0.13 release):
```</li>
</ul>
<server>
@@ -359,7 +359,7 @@ warnings)</li>
<username>USERNAME</username>
<password>PASSWORD</password>
</server>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
+<pre><code>
* *Clarify which env var is better or choose one* Set environment variable `MAVEN_OPTS` to `-Xmx1024m` to ensure the tests can run: `export JAVA_OPTIONS="-Xmx1g"`
* If you are outside the US, then svn.apache.org may not resolve to the main US-based Subversion servers. (Compare the IP address you get for svn.apache.org with svn.us.apache.org to see if they are different.) This will cause problems during the release since it will create a revision and then immediately access, but, there is a replication lag of perhaps a minute to the non-US servers. To temporarily force using the US-based server, edit your equivalent of /etc/hosts and map the IP address of svn.us.apache.org to svn.apache.org.
* Create the release candidate: `mvn -Pmahout-release,apache-release release:prepare release:perform`
@@ -378,7 +378,7 @@ release:perform target
* Call a VOTE on dev@mahout.apache.org. Votes require 3 days before passing. See Apache [release policy|http://www.apache.org/foundation/voting.html#ReleaseVotes] for more info.
* If there's a problem, you need to unwind the release and start all over.
-</code></pre></div></div>
+</code></pre>
<p>mvn -Pmahout-release,apache-release versions:set -DnewVersion=PREVIOUS_SNAPSHOT
mvn -Pmahout-release,apache-release versions:commit
git commit
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/developers/how-to-update-the-website.html
----------------------------------------------------------------------
diff --git a/developers/how-to-update-the-website.html b/developers/how-to-update-the-website.html
index 42c38f0..d316f06 100644
--- a/developers/how-to-update-the-website.html
+++ b/developers/how-to-update-the-website.html
@@ -276,11 +276,11 @@
<p>Website updates are handled by updating code in the trunk.</p>
-<p>You will find markdown pages in <code class="highlighter-rouge">mahout/website</code>.</p>
+<p>You will find markdown pages in <code>mahout/website</code>.</p>
<p>Jenkins rebuilds and publishes the website whenever a change is detected in master.</p>
-<p><code class="highlighter-rouge">mahout/website/build_site.sh</code> contains the script that is used to do this.</p>
+<p><code>mahout/website/build_site.sh</code> contains the script that is used to do this.</p>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/docs/latest/sitemap.xml
----------------------------------------------------------------------
diff --git a/docs/latest/sitemap.xml b/docs/latest/sitemap.xml
index 5d9c772..35ce323 100644
--- a/docs/latest/sitemap.xml
+++ b/docs/latest/sitemap.xml
@@ -299,6 +299,6 @@
</url>
<url>
<loc>/distributed/spark-bindings/ScalaSparkBindings.pdf</loc>
-<lastmod>2017-11-30T00:12:37+00:00</lastmod>
+<lastmod>2017-11-30T00:23:17+00:00</lastmod>
</url>
</urlset>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/general/downloads.html
----------------------------------------------------------------------
diff --git a/general/downloads.html b/general/downloads.html
index 4a29fe9..0a59a9e 100644
--- a/general/downloads.html
+++ b/general/downloads.html
@@ -286,19 +286,19 @@ the Apache mirrors. The latest Mahout release is available for download at:</p>
<p>Apache Mahout is mirrored to <a href="https://github.com/apache/mahout">Github</a>. To get all source:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git clone https://github.com/apache/mahout.git mahout
-</code></pre></div></div>
+<pre><code>git clone https://github.com/apache/mahout.git mahout
+</code></pre>
<h1 id="environment">Environment</h1>
<p>Whether you are using Mahout’s Shell, running command line jobs or using it as a library to build your own apps
you’ll need to setup several environment variables.
-Edit your environment in <code class="highlighter-rouge">~/.bash_profile</code> for Mac or <code class="highlighter-rouge">~/.bashrc</code> for many linux distributions. Add the following</p>
+Edit your environment in <code>~/.bash_profile</code> for Mac or <code>~/.bashrc</code> for many linux distributions. Add the following</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>export MAHOUT_HOME=/path/to/mahout
+<pre><code>export MAHOUT_HOME=/path/to/mahout
export MAHOUT_LOCAL=true # for running standalone on your dev machine,
# unset MAHOUT_LOCAL for running on a cluster
-</code></pre></div></div>
+</code></pre>
<p>If you are running on Spark you will also need $SPARK_HOME</p>
@@ -311,21 +311,21 @@ Then add the appropriate setting to your pom.xml or build.sbt following the temp
<p>If you only need the math part of Mahout:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code><dependency>
+<pre><code><dependency>
<groupId>org.apache.mahout</groupId>
<artifactId>mahout-math</artifactId>
<version>${mahout.version}</version>
</dependency>
-</code></pre></div></div>
+</code></pre>
<p>In case you would like to use some of our integration tooling (e.g. for generating vectors from Lucene):</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code><dependency>
+<pre><code><dependency>
<groupId>org.apache.mahout</groupId>
<artifactId>mahout-hdfs</artifactId>
<version>${mahout.version}</version>
</dependency>
-</code></pre></div></div>
+</code></pre>
<p>In case you are using Ivy, Gradle, Buildr, Grape or SBT you might want to directly head over to the official <a href="http://mvnrepository.com/artifact/org.apache.mahout/mahout-core">Maven Repository search</a>.</p>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/general/mahout-wiki.html
----------------------------------------------------------------------
diff --git a/general/mahout-wiki.html b/general/mahout-wiki.html
index 7c9f32a..67a1bef 100644
--- a/general/mahout-wiki.html
+++ b/general/mahout-wiki.html
@@ -474,9 +474,9 @@ and picking a username and password.</li>
<p>There are some conventions used on the Mahout wiki:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>* {noformat}+*TODO:*+{noformat} (+*TODO:*+ ) is used to denote sections that definitely need to be cleaned up.
+<pre><code>* {noformat}+*TODO:*+{noformat} (+*TODO:*+ ) is used to denote sections that definitely need to be cleaned up.
* {noformat}+*Mahout_(version)*+{noformat} (+*Mahout_0.2*+) is used to draw attention to which version of Mahout a feature was (or will be) added to Mahout.
-</code></pre></div></div>
+</code></pre>
</div>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/sitemap.xml
----------------------------------------------------------------------
diff --git a/sitemap.xml b/sitemap.xml
index ea32436..f4fee43 100644
--- a/sitemap.xml
+++ b/sitemap.xml
@@ -347,14 +347,14 @@
</url>
<url>
<loc>/release-notes/Apache-Mahout-0.10.0-Release-Notes.pdf</loc>
-<lastmod>2017-11-30T00:12:37+00:00</lastmod>
+<lastmod>2017-11-30T00:23:17+00:00</lastmod>
</url>
<url>
<loc>/users/dim-reduction/ssvd.page/SSVD-CLI.pdf</loc>
-<lastmod>2017-11-30T00:12:37+00:00</lastmod>
+<lastmod>2017-11-30T00:23:17+00:00</lastmod>
</url>
<url>
<loc>/users/sparkbindings/ScalaSparkBindings.pdf</loc>
-<lastmod>2017-11-30T00:12:37+00:00</lastmod>
+<lastmod>2017-11-30T00:23:17+00:00</lastmod>
</url>
</urlset>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/algorithms/d-als.html
----------------------------------------------------------------------
diff --git a/users/algorithms/d-als.html b/users/algorithms/d-als.html
index 2677918..0a98e03 100644
--- a/users/algorithms/d-als.html
+++ b/users/algorithms/d-als.html
@@ -280,13 +280,13 @@
<h2 id="algorithm">Algorithm</h2>
-<p>For the classic QR decomposition of the form <code class="highlighter-rouge">\(\mathbf{A}=\mathbf{QR},\mathbf{A}\in\mathbb{R}^{m\times n}\)</code> a distributed version is fairly easily achieved if <code class="highlighter-rouge">\(\mathbf{A}\)</code> is tall and thin such that <code class="highlighter-rouge">\(\mathbf{A}^{\top}\mathbf{A}\)</code> fits in memory, i.e. <em>m</em> is large but <em>n</em> < ~5000 Under such circumstances, only <code class="highlighter-rouge">\(\mathbf{A}\)</code> and <code class="highlighter-rouge">\(\mathbf{Q}\)</code> are distributed matricies and <code class="highlighter-rouge">\(\mathbf{A^{\top}A}\)</code> and <code class="highlighter-rouge">\(\mathbf{R}\)</code> are in-core products. We just compute the in-core version of the Cholesky decomposition in the form of <code class="highlighter-rouge">\(\mathbf{LL}^{\top}= \mathbf{A}^{\top}\mathbf{A}\)</code>. After that we take <code class="highlighter-rouge">\(\mathbf{R}= \mathbf{L}^{\top}\)</co
de> and <code class="highlighter-rouge">\(\mathbf{Q}=\mathbf{A}\left(\mathbf{L}^{\top}\right)^{-1}\)</code>. The latter is easily achieved by multiplying each verticle block of <code class="highlighter-rouge">\(\mathbf{A}\)</code> by <code class="highlighter-rouge">\(\left(\mathbf{L}^{\top}\right)^{-1}\)</code>. (There is no actual matrix inversion happening).</p>
+<p>For the classic QR decomposition of the form <code>\(\mathbf{A}=\mathbf{QR},\mathbf{A}\in\mathbb{R}^{m\times n}\)</code> a distributed version is fairly easily achieved if <code>\(\mathbf{A}\)</code> is tall and thin such that <code>\(\mathbf{A}^{\top}\mathbf{A}\)</code> fits in memory, i.e. <em>m</em> is large but <em>n</em> < ~5000 Under such circumstances, only <code>\(\mathbf{A}\)</code> and <code>\(\mathbf{Q}\)</code> are distributed matricies and <code>\(\mathbf{A^{\top}A}\)</code> and <code>\(\mathbf{R}\)</code> are in-core products. We just compute the in-core version of the Cholesky decomposition in the form of <code>\(\mathbf{LL}^{\top}= \mathbf{A}^{\top}\mathbf{A}\)</code>. After that we take <code>\(\mathbf{R}= \mathbf{L}^{\top}\)</code> and <code>\(\mathbf{Q}=\mathbf{A}\left(\mathbf{L}^{\top}\right)^{-1}\)</code>. The latter is easily achieved by multiplying each verticle block of <code>\(\mathbf{A}\)</code> by <code>\(\left(\mathbf{L}^{\top}\right)^{-1}\)</code
>. (There is no actual matrix inversion happening).</p>
<h2 id="implementation">Implementation</h2>
-<p>Mahout <code class="highlighter-rouge">dqrThin(...)</code> is implemented in the mahout <code class="highlighter-rouge">math-scala</code> algebraic optimizer which translates Mahout’s R-like linear algebra operators into a physical plan for both Spark and H2O distributed engines.</p>
+<p>Mahout <code>dqrThin(...)</code> is implemented in the mahout <code>math-scala</code> algebraic optimizer which translates Mahout’s R-like linear algebra operators into a physical plan for both Spark and H2O distributed engines.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def dqrThin[K: ClassTag](A: DrmLike[K], checkRankDeficiency: Boolean = true): (DrmLike[K], Matrix) = {
+<pre><code>def dqrThin[K: ClassTag](A: DrmLike[K], checkRankDeficiency: Boolean = true): (DrmLike[K], Matrix) = {
if (drmA.ncol > 5000)
log.warn("A is too fat. A'A must fit in memory and easily broadcasted.")
implicit val ctx = drmA.context
@@ -302,18 +302,18 @@
}
Q -> inCoreR
}
-</code></pre></div></div>
+</code></pre>
<h2 id="usage">Usage</h2>
-<p>The scala <code class="highlighter-rouge">dqrThin(...)</code> method can easily be called in any Spark or H2O application built with the <code class="highlighter-rouge">math-scala</code> library and the corresponding <code class="highlighter-rouge">Spark</code> or <code class="highlighter-rouge">H2O</code> engine module as follows:</p>
+<p>The scala <code>dqrThin(...)</code> method can easily be called in any Spark or H2O application built with the <code>math-scala</code> library and the corresponding <code>Spark</code> or <code>H2O</code> engine module as follows:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>import org.apache.mahout.math._
+<pre><code>import org.apache.mahout.math._
import decompositions._
import drm._
val(drmQ, inCoreR) = dqrThin(drma)
-</code></pre></div></div>
+</code></pre>
<h2 id="references">References</h2>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/algorithms/d-qr.html
----------------------------------------------------------------------
diff --git a/users/algorithms/d-qr.html b/users/algorithms/d-qr.html
index add7d10..f1aec35 100644
--- a/users/algorithms/d-qr.html
+++ b/users/algorithms/d-qr.html
@@ -280,13 +280,13 @@
<h2 id="algorithm">Algorithm</h2>
-<p>For the classic QR decomposition of the form <code class="highlighter-rouge">\(\mathbf{A}=\mathbf{QR},\mathbf{A}\in\mathbb{R}^{m\times n}\)</code> a distributed version is fairly easily achieved if <code class="highlighter-rouge">\(\mathbf{A}\)</code> is tall and thin such that <code class="highlighter-rouge">\(\mathbf{A}^{\top}\mathbf{A}\)</code> fits in memory, i.e. <em>m</em> is large but <em>n</em> < ~5000 Under such circumstances, only <code class="highlighter-rouge">\(\mathbf{A}\)</code> and <code class="highlighter-rouge">\(\mathbf{Q}\)</code> are distributed matrices and <code class="highlighter-rouge">\(\mathbf{A^{\top}A}\)</code> and <code class="highlighter-rouge">\(\mathbf{R}\)</code> are in-core products. We just compute the in-core version of the Cholesky decomposition in the form of <code class="highlighter-rouge">\(\mathbf{LL}^{\top}= \mathbf{A}^{\top}\mathbf{A}\)</code>. After that we take <code class="highlighter-rouge">\(\mathbf{R}= \mathbf{L}^{\top}\)</cod
e> and <code class="highlighter-rouge">\(\mathbf{Q}=\mathbf{A}\left(\mathbf{L}^{\top}\right)^{-1}\)</code>. The latter is easily achieved by multiplying each vertical block of <code class="highlighter-rouge">\(\mathbf{A}\)</code> by <code class="highlighter-rouge">\(\left(\mathbf{L}^{\top}\right)^{-1}\)</code>. (There is no actual matrix inversion happening).</p>
+<p>For the classic QR decomposition of the form <code>\(\mathbf{A}=\mathbf{QR},\mathbf{A}\in\mathbb{R}^{m\times n}\)</code> a distributed version is fairly easily achieved if <code>\(\mathbf{A}\)</code> is tall and thin such that <code>\(\mathbf{A}^{\top}\mathbf{A}\)</code> fits in memory, i.e. <em>m</em> is large but <em>n</em> < ~5000 Under such circumstances, only <code>\(\mathbf{A}\)</code> and <code>\(\mathbf{Q}\)</code> are distributed matrices and <code>\(\mathbf{A^{\top}A}\)</code> and <code>\(\mathbf{R}\)</code> are in-core products. We just compute the in-core version of the Cholesky decomposition in the form of <code>\(\mathbf{LL}^{\top}= \mathbf{A}^{\top}\mathbf{A}\)</code>. After that we take <code>\(\mathbf{R}= \mathbf{L}^{\top}\)</code> and <code>\(\mathbf{Q}=\mathbf{A}\left(\mathbf{L}^{\top}\right)^{-1}\)</code>. The latter is easily achieved by multiplying each vertical block of <code>\(\mathbf{A}\)</code> by <code>\(\left(\mathbf{L}^{\top}\right)^{-1}\)</code>
. (There is no actual matrix inversion happening).</p>
<h2 id="implementation">Implementation</h2>
-<p>Mahout <code class="highlighter-rouge">dqrThin(...)</code> is implemented in the mahout <code class="highlighter-rouge">math-scala</code> algebraic optimizer which translates Mahout’s R-like linear algebra operators into a physical plan for both Spark and H2O distributed engines.</p>
+<p>Mahout <code>dqrThin(...)</code> is implemented in the mahout <code>math-scala</code> algebraic optimizer which translates Mahout’s R-like linear algebra operators into a physical plan for both Spark and H2O distributed engines.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def dqrThin[K: ClassTag](A: DrmLike[K], checkRankDeficiency: Boolean = true): (DrmLike[K], Matrix) = {
+<pre><code>def dqrThin[K: ClassTag](A: DrmLike[K], checkRankDeficiency: Boolean = true): (DrmLike[K], Matrix) = {
if (drmA.ncol > 5000)
log.warn("A is too fat. A'A must fit in memory and easily broadcasted.")
implicit val ctx = drmA.context
@@ -302,18 +302,18 @@
}
Q -> inCoreR
}
-</code></pre></div></div>
+</code></pre>
<h2 id="usage">Usage</h2>
-<p>The scala <code class="highlighter-rouge">dqrThin(...)</code> method can easily be called in any Spark or H2O application built with the <code class="highlighter-rouge">math-scala</code> library and the corresponding <code class="highlighter-rouge">Spark</code> or <code class="highlighter-rouge">H2O</code> engine module as follows:</p>
+<p>The scala <code>dqrThin(...)</code> method can easily be called in any Spark or H2O application built with the <code>math-scala</code> library and the corresponding <code>Spark</code> or <code>H2O</code> engine module as follows:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>import org.apache.mahout.math._
+<pre><code>import org.apache.mahout.math._
import decompositions._
import drm._
val(drmQ, inCoreR) = dqrThin(drma)
-</code></pre></div></div>
+</code></pre>
<h2 id="references">References</h2>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/algorithms/d-spca.html
----------------------------------------------------------------------
diff --git a/users/algorithms/d-spca.html b/users/algorithms/d-spca.html
index d0feb7c..2c58f6b 100644
--- a/users/algorithms/d-spca.html
+++ b/users/algorithms/d-spca.html
@@ -276,43 +276,43 @@
<h2 id="intro">Intro</h2>
-<p>Mahout has a distributed implementation of Stochastic PCA<a href="Lyubimov and Palumbo, ["Apache Mahout: Beyond MapReduce; Distributed Algorithm Design"](https://www.amazon.com/Apache-Mahout-MapReduce-Dmitriy-Lyubimov/dp/1523775785)">1</a>. This algorithm computes the exact equivalent of Mahout’s dssvd(<code class="highlighter-rouge">\(\mathbf{A-1\mu^\top}\)</code>) by modifying the <code class="highlighter-rouge">dssvd</code> algorithm so as to avoid forming <code class="highlighter-rouge">\(\mathbf{A-1\mu^\top}\)</code>, which would densify a sparse input. Thus, it is suitable for work with both dense and sparse inputs.</p>
+<p>Mahout has a distributed implementation of Stochastic PCA<a href="Lyubimov and Palumbo, ["Apache Mahout: Beyond MapReduce; Distributed Algorithm Design"](https://www.amazon.com/Apache-Mahout-MapReduce-Dmitriy-Lyubimov/dp/1523775785)">1</a>. This algorithm computes the exact equivalent of Mahout’s dssvd(<code>\(\mathbf{A-1\mu^\top}\)</code>) by modifying the <code>dssvd</code> algorithm so as to avoid forming <code>\(\mathbf{A-1\mu^\top}\)</code>, which would densify a sparse input. Thus, it is suitable for work with both dense and sparse inputs.</p>
<h2 id="algorithm">Algorithm</h2>
-<p>Given an <em>m</em> <code class="highlighter-rouge">\(\times\)</code> <em>n</em> matrix <code class="highlighter-rouge">\(\mathbf{A}\)</code>, a target rank <em>k</em>, and an oversampling parameter <em>p</em>, this procedure computes a <em>k</em>-rank PCA by finding the unknowns in <code class="highlighter-rouge">\(\mathbf{A−1\mu^\top \approx U\Sigma V^\top}\)</code>:</p>
+<p>Given an <em>m</em> <code>\(\times\)</code> <em>n</em> matrix <code>\(\mathbf{A}\)</code>, a target rank <em>k</em>, and an oversampling parameter <em>p</em>, this procedure computes a <em>k</em>-rank PCA by finding the unknowns in <code>\(\mathbf{A−1\mu^\top \approx U\Sigma V^\top}\)</code>:</p>
<ol>
- <li>Create seed for random <em>n</em> <code class="highlighter-rouge">\(\times\)</code> <em>(k+p)</em> matrix <code class="highlighter-rouge">\(\Omega\)</code>.</li>
- <li><code class="highlighter-rouge">\(\mathbf{s_\Omega \leftarrow \Omega^\top \mu}\)</code>.</li>
- <li><code class="highlighter-rouge">\(\mathbf{Y_0 \leftarrow A\Omega − 1 {s_\Omega}^\top, Y \in \mathbb{R}^{m\times(k+p)}}\)</code>.</li>
- <li>Column-orthonormalize <code class="highlighter-rouge">\(\mathbf{Y_0} \rightarrow \mathbf{Q}\)</code> by computing thin decomposition <code class="highlighter-rouge">\(\mathbf{Y_0} = \mathbf{QR}\)</code>. Also, <code class="highlighter-rouge">\(\mathbf{Q}\in\mathbb{R}^{m\times(k+p)}, \mathbf{R}\in\mathbb{R}^{(k+p)\times(k+p)}\)</code>.</li>
- <li><code class="highlighter-rouge">\(\mathbf{s_Q \leftarrow Q^\top 1}\)</code>.</li>
- <li><code class="highlighter-rouge">\(\mathbf{B_0 \leftarrow Q^\top A: B \in \mathbb{R}^{(k+p)\times n}}\)</code>.</li>
- <li><code class="highlighter-rouge">\(\mathbf{s_B \leftarrow {B_0}^\top \mu}\)</code>.</li>
+ <li>Create seed for random <em>n</em> <code>\(\times\)</code> <em>(k+p)</em> matrix <code>\(\Omega\)</code>.</li>
+ <li><code>\(\mathbf{s_\Omega \leftarrow \Omega^\top \mu}\)</code>.</li>
+ <li><code>\(\mathbf{Y_0 \leftarrow A\Omega − 1 {s_\Omega}^\top, Y \in \mathbb{R}^{m\times(k+p)}}\)</code>.</li>
+ <li>Column-orthonormalize <code>\(\mathbf{Y_0} \rightarrow \mathbf{Q}\)</code> by computing thin decomposition <code>\(\mathbf{Y_0} = \mathbf{QR}\)</code>. Also, <code>\(\mathbf{Q}\in\mathbb{R}^{m\times(k+p)}, \mathbf{R}\in\mathbb{R}^{(k+p)\times(k+p)}\)</code>.</li>
+ <li><code>\(\mathbf{s_Q \leftarrow Q^\top 1}\)</code>.</li>
+ <li><code>\(\mathbf{B_0 \leftarrow Q^\top A: B \in \mathbb{R}^{(k+p)\times n}}\)</code>.</li>
+ <li><code>\(\mathbf{s_B \leftarrow {B_0}^\top \mu}\)</code>.</li>
<li>For <em>i</em> in 1..<em>q</em> repeat (power iterations):
<ul>
- <li>For <em>j</em> in 1..<em>n</em> apply <code class="highlighter-rouge">\(\mathbf{(B_{i−1})_{∗j} \leftarrow (B_{i−1})_{∗j}−\mu_j s_Q}\)</code>.</li>
- <li><code class="highlighter-rouge">\(\mathbf{Y_i \leftarrow A{B_{i−1}}^\top−1(s_B−\mu^\top \mu s_Q)^\top}\)</code>.</li>
- <li>Column-orthonormalize <code class="highlighter-rouge">\(\mathbf{Y_i} \rightarrow \mathbf{Q}\)</code> by computing thin decomposition <code class="highlighter-rouge">\(\mathbf{Y_i = QR}\)</code>.</li>
- <li><code class="highlighter-rouge">\(\mathbf{s_Q \leftarrow Q^\top 1}\)</code>.</li>
- <li><code class="highlighter-rouge">\(\mathbf{B_i \leftarrow Q^\top A}\)</code>.</li>
- <li><code class="highlighter-rouge">\(\mathbf{s_B \leftarrow {B_i}^\top \mu}\)</code>.</li>
+ <li>For <em>j</em> in 1..<em>n</em> apply <code>\(\mathbf{(B_{i−1})_{∗j} \leftarrow (B_{i−1})_{∗j}−\mu_j s_Q}\)</code>.</li>
+ <li><code>\(\mathbf{Y_i \leftarrow A{B_{i−1}}^\top−1(s_B−\mu^\top \mu s_Q)^\top}\)</code>.</li>
+ <li>Column-orthonormalize <code>\(\mathbf{Y_i} \rightarrow \mathbf{Q}\)</code> by computing thin decomposition <code>\(\mathbf{Y_i = QR}\)</code>.</li>
+ <li><code>\(\mathbf{s_Q \leftarrow Q^\top 1}\)</code>.</li>
+ <li><code>\(\mathbf{B_i \leftarrow Q^\top A}\)</code>.</li>
+ <li><code>\(\mathbf{s_B \leftarrow {B_i}^\top \mu}\)</code>.</li>
</ul>
</li>
- <li>Let <code class="highlighter-rouge">\(\mathbf{C \triangleq s_Q {s_B}^\top}\)</code>. <code class="highlighter-rouge">\(\mathbf{M \leftarrow B_q {B_q}^\top − C − C^\top + \mu^\top \mu s_Q {s_Q}^\top}\)</code>.</li>
- <li>Compute an eigensolution of the small symmetric <code class="highlighter-rouge">\(\mathbf{M = \hat{U} \Lambda \hat{U}^\top: M \in \mathbb{R}^{(k+p)\times(k+p)}}\)</code>.</li>
- <li>The singular values <code class="highlighter-rouge">\(\Sigma = \Lambda^{\circ 0.5}\)</code>, or, in other words, <code class="highlighter-rouge">\(\mathbf{\sigma_i= \sqrt{\lambda_i}}\)</code>.</li>
- <li>If needed, compute <code class="highlighter-rouge">\(\mathbf{U = Q\hat{U}}\)</code>.</li>
- <li>If needed, compute <code class="highlighter-rouge">\(\mathbf{V = B^\top \hat{U} \Sigma^{−1}}\)</code>.</li>
- <li>If needed, items converted to the PCA space can be computed as <code class="highlighter-rouge">\(\mathbf{U\Sigma}\)</code>.</li>
+ <li>Let <code>\(\mathbf{C \triangleq s_Q {s_B}^\top}\)</code>. <code>\(\mathbf{M \leftarrow B_q {B_q}^\top − C − C^\top + \mu^\top \mu s_Q {s_Q}^\top}\)</code>.</li>
+ <li>Compute an eigensolution of the small symmetric <code>\(\mathbf{M = \hat{U} \Lambda \hat{U}^\top: M \in \mathbb{R}^{(k+p)\times(k+p)}}\)</code>.</li>
+ <li>The singular values <code>\(\Sigma = \Lambda^{\circ 0.5}\)</code>, or, in other words, <code>\(\mathbf{\sigma_i= \sqrt{\lambda_i}}\)</code>.</li>
+ <li>If needed, compute <code>\(\mathbf{U = Q\hat{U}}\)</code>.</li>
+ <li>If needed, compute <code>\(\mathbf{V = B^\top \hat{U} \Sigma^{−1}}\)</code>.</li>
+ <li>If needed, items converted to the PCA space can be computed as <code>\(\mathbf{U\Sigma}\)</code>.</li>
</ol>
<h2 id="implementation">Implementation</h2>
-<p>Mahout <code class="highlighter-rouge">dspca(...)</code> is implemented in the mahout <code class="highlighter-rouge">math-scala</code> algebraic optimizer which translates Mahout’s R-like linear algebra operators into a physical plan for both Spark and H2O distributed engines.</p>
+<p>Mahout <code>dspca(...)</code> is implemented in the mahout <code>math-scala</code> algebraic optimizer which translates Mahout’s R-like linear algebra operators into a physical plan for both Spark and H2O distributed engines.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def dspca[K](drmA: DrmLike[K], k: Int, p: Int = 15, q: Int = 0):
+<pre><code>def dspca[K](drmA: DrmLike[K], k: Int, p: Int = 15, q: Int = 0):
(DrmLike[K], DrmLike[Int], Vector) = {
// Some mapBlock() calls need it
@@ -429,18 +429,18 @@
(drmU(::, 0 until k), drmV(::, 0 until k), s(0 until k))
}
-</code></pre></div></div>
+</code></pre>
<h2 id="usage">Usage</h2>
-<p>The scala <code class="highlighter-rouge">dspca(...)</code> method can easily be called in any Spark, Flink, or H2O application built with the <code class="highlighter-rouge">math-scala</code> library and the corresponding <code class="highlighter-rouge">Spark</code>, <code class="highlighter-rouge">Flink</code>, or <code class="highlighter-rouge">H2O</code> engine module as follows:</p>
+<p>The scala <code>dspca(...)</code> method can easily be called in any Spark, Flink, or H2O application built with the <code>math-scala</code> library and the corresponding <code>Spark</code>, <code>Flink</code>, or <code>H2O</code> engine module as follows:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>import org.apache.mahout.math._
+<pre><code>import org.apache.mahout.math._
import decompositions._
import drm._
val (drmU, drmV, s) = dspca(drmA, k=200, q=1)
-</code></pre></div></div>
+</code></pre>
<p>Note the parameter is optional and its default value is zero.</p>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/algorithms/d-ssvd.html
----------------------------------------------------------------------
diff --git a/users/algorithms/d-ssvd.html b/users/algorithms/d-ssvd.html
index e3775d9..5165c60 100644
--- a/users/algorithms/d-ssvd.html
+++ b/users/algorithms/d-ssvd.html
@@ -280,58 +280,58 @@
<h2 id="modified-ssvd-algorithm">Modified SSVD Algorithm</h2>
-<p>Given an <code class="highlighter-rouge">\(m\times n\)</code>
-matrix <code class="highlighter-rouge">\(\mathbf{A}\)</code>, a target rank <code class="highlighter-rouge">\(k\in\mathbb{N}_{1}\)</code>
-, an oversampling parameter <code class="highlighter-rouge">\(p\in\mathbb{N}_{1}\)</code>,
-and the number of additional power iterations <code class="highlighter-rouge">\(q\in\mathbb{N}_{0}\)</code>,
-this procedure computes an <code class="highlighter-rouge">\(m\times\left(k+p\right)\)</code>
-SVD <code class="highlighter-rouge">\(\mathbf{A\approx U}\boldsymbol{\Sigma}\mathbf{V}^{\top}\)</code>:</p>
+<p>Given an <code>\(m\times n\)</code>
+matrix <code>\(\mathbf{A}\)</code>, a target rank <code>\(k\in\mathbb{N}_{1}\)</code>
+, an oversampling parameter <code>\(p\in\mathbb{N}_{1}\)</code>,
+and the number of additional power iterations <code>\(q\in\mathbb{N}_{0}\)</code>,
+this procedure computes an <code>\(m\times\left(k+p\right)\)</code>
+SVD <code>\(\mathbf{A\approx U}\boldsymbol{\Sigma}\mathbf{V}^{\top}\)</code>:</p>
<ol>
<li>
- <p>Create seed for random <code class="highlighter-rouge">\(n\times\left(k+p\right)\)</code>
- matrix <code class="highlighter-rouge">\(\boldsymbol{\Omega}\)</code>. The seed defines matrix <code class="highlighter-rouge">\(\mathbf{\Omega}\)</code>
+ <p>Create seed for random <code>\(n\times\left(k+p\right)\)</code>
+ matrix <code>\(\boldsymbol{\Omega}\)</code>. The seed defines matrix <code>\(\mathbf{\Omega}\)</code>
using Gaussian unit vectors per one of suggestions in [Halko, Martinsson, Tropp].</p>
</li>
<li>
- <p><code class="highlighter-rouge">\(\mathbf{Y=A\boldsymbol{\Omega}},\,\mathbf{Y}\in\mathbb{R}^{m\times\left(k+p\right)}\)</code></p>
+ <p><code>\(\mathbf{Y=A\boldsymbol{\Omega}},\,\mathbf{Y}\in\mathbb{R}^{m\times\left(k+p\right)}\)</code></p>
</li>
<li>
- <p>Column-orthonormalize <code class="highlighter-rouge">\(\mathbf{Y}\rightarrow\mathbf{Q}\)</code>
- by computing thin decomposition <code class="highlighter-rouge">\(\mathbf{Y}=\mathbf{Q}\mathbf{R}\)</code>.
- Also, <code class="highlighter-rouge">\(\mathbf{Q}\in\mathbb{R}^{m\times\left(k+p\right)},\,\mathbf{R}\in\mathbb{R}^{\left(k+p\right)\times\left(k+p\right)}\)</code>; denoted as <code class="highlighter-rouge">\(\mathbf{Q}=\mbox{qr}\left(\mathbf{Y}\right).\mathbf{Q}\)</code></p>
+ <p>Column-orthonormalize <code>\(\mathbf{Y}\rightarrow\mathbf{Q}\)</code>
+ by computing thin decomposition <code>\(\mathbf{Y}=\mathbf{Q}\mathbf{R}\)</code>.
+ Also, <code>\(\mathbf{Q}\in\mathbb{R}^{m\times\left(k+p\right)},\,\mathbf{R}\in\mathbb{R}^{\left(k+p\right)\times\left(k+p\right)}\)</code>; denoted as <code>\(\mathbf{Q}=\mbox{qr}\left(\mathbf{Y}\right).\mathbf{Q}\)</code></p>
</li>
<li>
- <p><code class="highlighter-rouge">\(\mathbf{B}_{0}=\mathbf{Q}^{\top}\mathbf{A}:\,\,\mathbf{B}\in\mathbb{R}^{\left(k+p\right)\times n}\)</code>.</p>
+ <p><code>\(\mathbf{B}_{0}=\mathbf{Q}^{\top}\mathbf{A}:\,\,\mathbf{B}\in\mathbb{R}^{\left(k+p\right)\times n}\)</code>.</p>
</li>
<li>
- <p>If <code class="highlighter-rouge">\(q>0\)</code>
- repeat: for <code class="highlighter-rouge">\(i=1..q\)</code>:
- <code class="highlighter-rouge">\(\mathbf{B}_{i}^{\top}=\mathbf{A}^{\top}\mbox{qr}\left(\mathbf{A}\mathbf{B}_{i-1}^{\top}\right).\mathbf{Q}\)</code>
+ <p>If <code>\(q>0\)</code>
+ repeat: for <code>\(i=1..q\)</code>:
+ <code>\(\mathbf{B}_{i}^{\top}=\mathbf{A}^{\top}\mbox{qr}\left(\mathbf{A}\mathbf{B}_{i-1}^{\top}\right).\mathbf{Q}\)</code>
(power iterations step).</p>
</li>
<li>
- <p>Compute Eigensolution of a small Hermitian <code class="highlighter-rouge">\(\mathbf{B}_{q}\mathbf{B}_{q}^{\top}=\mathbf{\hat{U}}\boldsymbol{\Lambda}\mathbf{\hat{U}}^{\top}\)</code>,
- <code class="highlighter-rouge">\(\mathbf{B}_{q}\mathbf{B}_{q}^{\top}\in\mathbb{R}^{\left(k+p\right)\times\left(k+p\right)}\)</code>.</p>
+ <p>Compute Eigensolution of a small Hermitian <code>\(\mathbf{B}_{q}\mathbf{B}_{q}^{\top}=\mathbf{\hat{U}}\boldsymbol{\Lambda}\mathbf{\hat{U}}^{\top}\)</code>,
+ <code>\(\mathbf{B}_{q}\mathbf{B}_{q}^{\top}\in\mathbb{R}^{\left(k+p\right)\times\left(k+p\right)}\)</code>.</p>
</li>
<li>
- <p>Singular values <code class="highlighter-rouge">\(\mathbf{\boldsymbol{\Sigma}}=\boldsymbol{\Lambda}^{0.5}\)</code>,
- or, in other words, <code class="highlighter-rouge">\(s_{i}=\sqrt{\sigma_{i}}\)</code>.</p>
+ <p>Singular values <code>\(\mathbf{\boldsymbol{\Sigma}}=\boldsymbol{\Lambda}^{0.5}\)</code>,
+ or, in other words, <code>\(s_{i}=\sqrt{\sigma_{i}}\)</code>.</p>
</li>
<li>
- <p>If needed, compute <code class="highlighter-rouge">\(\mathbf{U}=\mathbf{Q}\hat{\mathbf{U}}\)</code>.</p>
+ <p>If needed, compute <code>\(\mathbf{U}=\mathbf{Q}\hat{\mathbf{U}}\)</code>.</p>
</li>
<li>
- <p>If needed, compute <code class="highlighter-rouge">\(\mathbf{V}=\mathbf{B}_{q}^{\top}\hat{\mathbf{U}}\boldsymbol{\Sigma}^{-1}\)</code>.
-Another way is <code class="highlighter-rouge">\(\mathbf{V}=\mathbf{A}^{\top}\mathbf{U}\boldsymbol{\Sigma}^{-1}\)</code>.</p>
+ <p>If needed, compute <code>\(\mathbf{V}=\mathbf{B}_{q}^{\top}\hat{\mathbf{U}}\boldsymbol{\Sigma}^{-1}\)</code>.
+Another way is <code>\(\mathbf{V}=\mathbf{A}^{\top}\mathbf{U}\boldsymbol{\Sigma}^{-1}\)</code>.</p>
</li>
</ol>
<h2 id="implementation">Implementation</h2>
-<p>Mahout <code class="highlighter-rouge">dssvd(...)</code> is implemented in the mahout <code class="highlighter-rouge">math-scala</code> algebraic optimizer which translates Mahout’s R-like linear algebra operators into a physical plan for both Spark and H2O distributed engines.</p>
+<p>Mahout <code>dssvd(...)</code> is implemented in the mahout <code>math-scala</code> algebraic optimizer which translates Mahout’s R-like linear algebra operators into a physical plan for both Spark and H2O distributed engines.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def dssvd[K: ClassTag](drmA: DrmLike[K], k: Int, p: Int = 15, q: Int = 0):
+<pre><code>def dssvd[K: ClassTag](drmA: DrmLike[K], k: Int, p: Int = 15, q: Int = 0):
(DrmLike[K], DrmLike[Int], Vector) = {
val drmAcp = drmA.checkpoint()
@@ -387,21 +387,21 @@ Another way is <code class="highlighter-rouge">\(\mathbf{V}=\mathbf{A}^{\top}\ma
(drmU(::, 0 until k), drmV(::, 0 until k), s(0 until k))
}
-</code></pre></div></div>
+</code></pre>
-<p>Note: As a side effect of checkpointing, U and V values are returned as logical operators (i.e. they are neither checkpointed nor computed). Therefore there is no physical work actually done to compute <code class="highlighter-rouge">\(\mathbf{U}\)</code> or <code class="highlighter-rouge">\(\mathbf{V}\)</code> until they are used in a subsequent expression.</p>
+<p>Note: As a side effect of checkpointing, U and V values are returned as logical operators (i.e. they are neither checkpointed nor computed). Therefore there is no physical work actually done to compute <code>\(\mathbf{U}\)</code> or <code>\(\mathbf{V}\)</code> until they are used in a subsequent expression.</p>
<h2 id="usage">Usage</h2>
-<p>The scala <code class="highlighter-rouge">dssvd(...)</code> method can easily be called in any Spark or H2O application built with the <code class="highlighter-rouge">math-scala</code> library and the corresponding <code class="highlighter-rouge">Spark</code> or <code class="highlighter-rouge">H2O</code> engine module as follows:</p>
+<p>The scala <code>dssvd(...)</code> method can easily be called in any Spark or H2O application built with the <code>math-scala</code> library and the corresponding <code>Spark</code> or <code>H2O</code> engine module as follows:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>import org.apache.mahout.math._
+<pre><code>import org.apache.mahout.math._
import decompositions._
import drm._
val(drmU, drmV, s) = dssvd(drma, k = 40, q = 1)
-</code></pre></div></div>
+</code></pre>
<h2 id="references">References</h2>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/algorithms/intro-cooccurrence-spark.html
----------------------------------------------------------------------
diff --git a/users/algorithms/intro-cooccurrence-spark.html b/users/algorithms/intro-cooccurrence-spark.html
index 517157a..a84b90b 100644
--- a/users/algorithms/intro-cooccurrence-spark.html
+++ b/users/algorithms/intro-cooccurrence-spark.html
@@ -312,7 +312,7 @@ For instance they might say an item-view is 0.2 of an item purchase. In practice
cross-cooccurrence is a more principled way to handle this case. In effect it scrubs secondary actions with the action you want
to recommend.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>spark-itemsimilarity Mahout 1.0
+<pre><code>spark-itemsimilarity Mahout 1.0
Usage: spark-itemsimilarity [options]
Disconnected from the target VM, address: '127.0.0.1:64676', transport: 'socket'
@@ -376,7 +376,7 @@ Spark config options:
-h | --help
prints this usage text
-</code></pre></div></div>
+</code></pre>
<p>This looks daunting but defaults to simple fairly sane values to take exactly the same input as legacy code and is pretty flexible. It allows the user to point to a single text file, a directory full of files, or a tree of directories to be traversed recursively. The files included can be specified with either a regex-style pattern or filename. The schema for the file is defined by column numbers, which map to the important bits of data including IDs and values. The files can even contain filters, which allow unneeded rows to be discarded or used for cross-cooccurrence calculations.</p>
@@ -386,20 +386,20 @@ Spark config options:
<p>If all defaults are used the input can be as simple as:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>userID1,itemID1
+<pre><code>userID1,itemID1
userID2,itemID2
...
-</code></pre></div></div>
+</code></pre>
<p>With the command line:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bash$ mahout spark-itemsimilarity --input in-file --output out-dir
-</code></pre></div></div>
+<pre><code>bash$ mahout spark-itemsimilarity --input in-file --output out-dir
+</code></pre>
<p>This will use the “local” Spark context and will output the standard text version of a DRM</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>itemID1<tab>itemID2:value2<space>itemID10:value10...
-</code></pre></div></div>
+<pre><code>itemID1<tab>itemID2:value2<space>itemID10:value10...
+</code></pre>
<p>###<a name="multiple-actions">How To Use Multiple User Actions</a></p>
@@ -415,7 +415,7 @@ to calculate the cross-cooccurrence indicator matrix.</p>
<p><em>spark-itemsimilarity</em> can read separate actions from separate files or from a mixed action log by filtering certain lines. For a mixed
action log of the form:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>u1,purchase,iphone
+<pre><code>u1,purchase,iphone
u1,purchase,ipad
u2,purchase,nexus
u2,purchase,galaxy
@@ -435,13 +435,13 @@ u3,view,nexus
u4,view,iphone
u4,view,ipad
u4,view,galaxy
-</code></pre></div></div>
+</code></pre>
<p>###Command Line</p>
<p>Use the following options:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bash$ mahout spark-itemsimilarity \
+<pre><code>bash$ mahout spark-itemsimilarity \
--input in-file \ # where to look for data
--output out-path \ # root dir for output
--master masterUrl \ # URL of the Spark master server
@@ -450,35 +450,35 @@ u4,view,galaxy
--itemIDPosition 2 \ # column that has the item ID
--rowIDPosition 0 \ # column that has the user ID
--filterPosition 1 # column that has the filter word
-</code></pre></div></div>
+</code></pre>
<p>###Output</p>
<p>The output of the job will be the standard text version of two Mahout DRMs. This is a case where we are calculating
cross-cooccurrence so a primary indicator matrix and cross-cooccurrence indicator matrix will be created</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>out-path
+<pre><code>out-path
|-- similarity-matrix - TDF part files
\-- cross-similarity-matrix - TDF part-files
-</code></pre></div></div>
+</code></pre>
<p>The similarity-matrix will contain the lines:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>galaxy\tnexus:1.7260924347106847
+<pre><code>galaxy\tnexus:1.7260924347106847
ipad\tiphone:1.7260924347106847
nexus\tgalaxy:1.7260924347106847
iphone\tipad:1.7260924347106847
surface
-</code></pre></div></div>
+</code></pre>
<p>The cross-similarity-matrix will contain:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>iphone\tnexus:1.7260924347106847 iphone:1.7260924347106847 ipad:1.7260924347106847 galaxy:1.7260924347106847
+<pre><code>iphone\tnexus:1.7260924347106847 iphone:1.7260924347106847 ipad:1.7260924347106847 galaxy:1.7260924347106847
ipad\tnexus:0.6795961471815897 iphone:0.6795961471815897 ipad:0.6795961471815897 galaxy:0.6795961471815897
nexus\tnexus:0.6795961471815897 iphone:0.6795961471815897 ipad:0.6795961471815897 galaxy:0.6795961471815897
galaxy\tnexus:1.7260924347106847 iphone:1.7260924347106847 ipad:1.7260924347106847 galaxy:1.7260924347106847
surface\tsurface:4.498681156950466 nexus:0.6795961471815897
-</code></pre></div></div>
+</code></pre>
<p><strong>Note:</strong> You can run this multiple times to use more than two actions or you can use the underlying
SimilarityAnalysis.cooccurrence API, which will more efficiently calculate any number of cross-cooccurrence indicators.</p>
@@ -487,7 +487,7 @@ SimilarityAnalysis.cooccurrence API, which will more efficiently calculate any n
<p>A common method of storing data is in log files. If they are written using some delimiter they can be consumed directly by spark-itemsimilarity. For instance input of the form:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>2014-06-23 14:46:53.115\tu1\tpurchase\trandom text\tiphone
+<pre><code>2014-06-23 14:46:53.115\tu1\tpurchase\trandom text\tiphone
2014-06-23 14:46:53.115\tu1\tpurchase\trandom text\tipad
2014-06-23 14:46:53.115\tu2\tpurchase\trandom text\tnexus
2014-06-23 14:46:53.115\tu2\tpurchase\trandom text\tgalaxy
@@ -507,11 +507,11 @@ SimilarityAnalysis.cooccurrence API, which will more efficiently calculate any n
2014-06-23 14:46:53.115\tu4\tview\trandom text\tiphone
2014-06-23 14:46:53.115\tu4\tview\trandom text\tipad
2014-06-23 14:46:53.115\tu4\tview\trandom text\tgalaxy
-</code></pre></div></div>
+</code></pre>
<p>Can be parsed with the following CLI and run on the cluster producing the same output as the above example.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bash$ mahout spark-itemsimilarity \
+<pre><code>bash$ mahout spark-itemsimilarity \
--input in-file \
--output out-path \
--master spark://sparkmaster:4044 \
@@ -521,7 +521,7 @@ SimilarityAnalysis.cooccurrence API, which will more efficiently calculate any n
--itemIDPosition 4 \
--rowIDPosition 1 \
--filterPosition 2
-</code></pre></div></div>
+</code></pre>
<p>##2. spark-rowsimilarity</p>
@@ -536,7 +536,7 @@ by a list of the most similar rows.</p>
<p>The command line interface is:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>spark-rowsimilarity Mahout 1.0
+<pre><code>spark-rowsimilarity Mahout 1.0
Usage: spark-rowsimilarity [options]
Input, output options
@@ -582,7 +582,7 @@ Spark config options:
-h | --help
prints this usage text
-</code></pre></div></div>
+</code></pre>
<p>See RowSimilarityDriver.scala in Mahout’s spark module if you want to customize the code.</p>
@@ -664,32 +664,32 @@ content or metadata, not by which users interacted with them.</p>
<p>For this we need input of the form:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>itemID<tab>list-of-tags
+<pre><code>itemID<tab>list-of-tags
...
-</code></pre></div></div>
+</code></pre>
<p>The full collection will look like the tags column from a catalog DB. For our ecom example it might be:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>3459860b<tab>men long-sleeve chambray clothing casual
+<pre><code>3459860b<tab>men long-sleeve chambray clothing casual
9446577d<tab>women tops chambray clothing casual
...
-</code></pre></div></div>
+</code></pre>
<p>We’ll use <em>spark-rowimilairity</em> because we are looking for similar rows, which encode items in this case. As with the
collaborative filtering indicators we use the –omitStrength option. The strengths created are
probabilistic log-likelihood ratios and so are used to filter unimportant similarities. Once the filtering or downsampling
is finished we no longer need the strengths. We will get an indicator matrix of the form:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>itemID<tab>list-of-item IDs
+<pre><code>itemID<tab>list-of-item IDs
...
-</code></pre></div></div>
+</code></pre>
<p>This is a content indicator since it has found other items with similar content or metadata.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>3459860b<tab>3459860b 3459860b 6749860c 5959860a 3434860a 3477860a
+<pre><code>3459860b<tab>3459860b 3459860b 6749860c 5959860a 3434860a 3477860a
9446577d<tab>9446577d 9496577d 0943577d 8346577d 9442277d 9446577e
...
-</code></pre></div></div>
+</code></pre>
<p>We now have three indicators, two collaborative filtering type and one content type.</p>
@@ -700,11 +700,11 @@ is finished we no longer need the strengths. We will get an indicator matrix of
<p>We have 3 indicators, these are indexed by the search engine into 3 fields, we’ll call them “purchase”, “view”, and “tags”.
We take the user’s history that corresponds to each indicator and create a query of the form:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Query:
+<pre><code>Query:
field: purchase; q:user's-purchase-history
field: view; q:user's view-history
field: tags; q:user's-tags-associated-with-purchases
-</code></pre></div></div>
+</code></pre>
<p>The query will result in an ordered list of items recommended for purchase but skewed towards items with similar tags to
the ones the user has already purchased.</p>
@@ -716,11 +716,11 @@ by tagging items with some category of popularity (hot, warm, cold for instance)
index that as a new indicator field and include the corresponding value in a query
on the popularity field. If we use the ecom example but use the query to get “hot” recommendations it might look like this:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Query:
+<pre><code>Query:
field: purchase; q:user's-purchase-history
field: view; q:user's view-history
field: popularity; q:"hot"
-</code></pre></div></div>
+</code></pre>
<p>This will return recommendations favoring ones that have the intrinsic indicator “hot”.</p>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/algorithms/spark-naive-bayes.html
----------------------------------------------------------------------
diff --git a/users/algorithms/spark-naive-bayes.html b/users/algorithms/spark-naive-bayes.html
index 46c9e06..ea8d2d3 100644
--- a/users/algorithms/spark-naive-bayes.html
+++ b/users/algorithms/spark-naive-bayes.html
@@ -281,45 +281,45 @@
<p>Where Bayes has long been a standard in text classification, CBayes is an extension of Bayes that performs particularly well on datasets with skewed classes and has been shown to be competitive with algorithms of higher complexity such as Support Vector Machines.</p>
<h2 id="implementations">Implementations</h2>
-<p>The mahout <code class="highlighter-rouge">math-scala</code> library has an implemetation of both Bayes and CBayes which is further optimized in the <code class="highlighter-rouge">spark</code> module. Currently the Spark optimized version provides CLI drivers for training and testing. Mahout Spark-Naive-Bayes models can also be trained, tested and saved to the filesystem from the Mahout Spark Shell.</p>
+<p>The mahout <code>math-scala</code> library has an implemetation of both Bayes and CBayes which is further optimized in the <code>spark</code> module. Currently the Spark optimized version provides CLI drivers for training and testing. Mahout Spark-Naive-Bayes models can also be trained, tested and saved to the filesystem from the Mahout Spark Shell.</p>
<h2 id="preprocessing-and-algorithm">Preprocessing and Algorithm</h2>
<p>As described in <a href="http://people.csail.mit.edu/jrennie/papers/icml03-nb.pdf">[1]</a> Mahout Naive Bayes is broken down into the following steps (assignments are over all possible index values):</p>
<ul>
- <li>Let <code class="highlighter-rouge">\(\vec{d}=(\vec{d_1},...,\vec{d_n})\)</code> be a set of documents; <code class="highlighter-rouge">\(d_{ij}\)</code> is the count of word <code class="highlighter-rouge">\(i\)</code> in document <code class="highlighter-rouge">\(j\)</code>.</li>
- <li>Let <code class="highlighter-rouge">\(\vec{y}=(y_1,...,y_n)\)</code> be their labels.</li>
- <li>Let <code class="highlighter-rouge">\(\alpha_i\)</code> be a smoothing parameter for all words in the vocabulary; let <code class="highlighter-rouge">\(\alpha=\sum_i{\alpha_i}\)</code>.</li>
- <li><strong>Preprocessing</strong>(via seq2Sparse) TF-IDF transformation and L2 length normalization of <code class="highlighter-rouge">\(\vec{d}\)</code>
+ <li>Let <code>\(\vec{d}=(\vec{d_1},...,\vec{d_n})\)</code> be a set of documents; <code>\(d_{ij}\)</code> is the count of word <code>\(i\)</code> in document <code>\(j\)</code>.</li>
+ <li>Let <code>\(\vec{y}=(y_1,...,y_n)\)</code> be their labels.</li>
+ <li>Let <code>\(\alpha_i\)</code> be a smoothing parameter for all words in the vocabulary; let <code>\(\alpha=\sum_i{\alpha_i}\)</code>.</li>
+ <li><strong>Preprocessing</strong>(via seq2Sparse) TF-IDF transformation and L2 length normalization of <code>\(\vec{d}\)</code>
<ol>
- <li><code class="highlighter-rouge">\(d_{ij} = \sqrt{d_{ij}}\)</code></li>
- <li><code class="highlighter-rouge">\(d_{ij} = d_{ij}\left(\log{\frac{\sum_k1}{\sum_k\delta_{ik}+1}}+1\right)\)</code></li>
- <li><code class="highlighter-rouge">\(d_{ij} =\frac{d_{ij}}{\sqrt{\sum_k{d_{kj}^2}}}\)</code></li>
+ <li><code>\(d_{ij} = \sqrt{d_{ij}}\)</code></li>
+ <li><code>\(d_{ij} = d_{ij}\left(\log{\frac{\sum_k1}{\sum_k\delta_{ik}+1}}+1\right)\)</code></li>
+ <li><code>\(d_{ij} =\frac{d_{ij}}{\sqrt{\sum_k{d_{kj}^2}}}\)</code></li>
</ol>
</li>
- <li><strong>Training: Bayes</strong><code class="highlighter-rouge">\((\vec{d},\vec{y})\)</code> calculate term weights <code class="highlighter-rouge">\(w_{ci}\)</code> as:
+ <li><strong>Training: Bayes</strong><code>\((\vec{d},\vec{y})\)</code> calculate term weights <code>\(w_{ci}\)</code> as:
<ol>
- <li><code class="highlighter-rouge">\(\hat\theta_{ci}=\frac{d_{ic}+\alpha_i}{\sum_k{d_{kc}}+\alpha}\)</code></li>
- <li><code class="highlighter-rouge">\(w_{ci}=\log{\hat\theta_{ci}}\)</code></li>
+ <li><code>\(\hat\theta_{ci}=\frac{d_{ic}+\alpha_i}{\sum_k{d_{kc}}+\alpha}\)</code></li>
+ <li><code>\(w_{ci}=\log{\hat\theta_{ci}}\)</code></li>
</ol>
</li>
- <li><strong>Training: CBayes</strong><code class="highlighter-rouge">\((\vec{d},\vec{y})\)</code> calculate term weights <code class="highlighter-rouge">\(w_{ci}\)</code> as:
+ <li><strong>Training: CBayes</strong><code>\((\vec{d},\vec{y})\)</code> calculate term weights <code>\(w_{ci}\)</code> as:
<ol>
- <li><code class="highlighter-rouge">\(\hat\theta_{ci} = \frac{\sum_{j:y_j\neq c}d_{ij}+\alpha_i}{\sum_{j:y_j\neq c}{\sum_k{d_{kj}}}+\alpha}\)</code></li>
- <li><code class="highlighter-rouge">\(w_{ci}=-\log{\hat\theta_{ci}}\)</code></li>
- <li><code class="highlighter-rouge">\(w_{ci}=\frac{w_{ci}}{\sum_i \lvert w_{ci}\rvert}\)</code></li>
+ <li><code>\(\hat\theta_{ci} = \frac{\sum_{j:y_j\neq c}d_{ij}+\alpha_i}{\sum_{j:y_j\neq c}{\sum_k{d_{kj}}}+\alpha}\)</code></li>
+ <li><code>\(w_{ci}=-\log{\hat\theta_{ci}}\)</code></li>
+ <li><code>\(w_{ci}=\frac{w_{ci}}{\sum_i \lvert w_{ci}\rvert}\)</code></li>
</ol>
</li>
<li><strong>Label Assignment/Testing:</strong>
<ol>
- <li>Let <code class="highlighter-rouge">\(\vec{t}= (t_1,...,t_n)\)</code> be a test document; let <code class="highlighter-rouge">\(t_i\)</code> be the count of the word <code class="highlighter-rouge">\(t\)</code>.</li>
- <li>Label the document according to <code class="highlighter-rouge">\(l(t)=\arg\max_c \sum\limits_{i} t_i w_{ci}\)</code></li>
+ <li>Let <code>\(\vec{t}= (t_1,...,t_n)\)</code> be a test document; let <code>\(t_i\)</code> be the count of the word <code>\(t\)</code>.</li>
+ <li>Label the document according to <code>\(l(t)=\arg\max_c \sum\limits_{i} t_i w_{ci}\)</code></li>
</ol>
</li>
</ul>
-<p>As we can see, the main difference between Bayes and CBayes is the weight calculation step. Where Bayes weighs terms more heavily based on the likelihood that they belong to class <code class="highlighter-rouge">\(c\)</code>, CBayes seeks to maximize term weights on the likelihood that they do not belong to any other class.</p>
+<p>As we can see, the main difference between Bayes and CBayes is the weight calculation step. Where Bayes weighs terms more heavily based on the likelihood that they belong to class <code>\(c\)</code>, CBayes seeks to maximize term weights on the likelihood that they do not belong to any other class.</p>
<h2 id="running-from-the-command-line">Running from the command line</h2>
@@ -330,34 +330,34 @@
<p><strong>Preprocessing:</strong>
For a set of Sequence File Formatted documents in PATH_TO_SEQUENCE_FILES the <a href="https://mahout.apache.org/users/basics/creating-vectors-from-text.html">mahout seq2sparse</a> command performs the TF-IDF transformations (-wt tfidf option) and L2 length normalization (-n 2 option) as follows:</p>
- <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> $ mahout seq2sparse
+ <pre><code> $ mahout seq2sparse
-i ${PATH_TO_SEQUENCE_FILES}
-o ${PATH_TO_TFIDF_VECTORS}
-nv
-n 2
-wt tfidf
-</code></pre></div> </div>
+</code></pre>
</li>
<li>
<p><strong>Training:</strong>
-The model is then trained using <code class="highlighter-rouge">mahout spark-trainnb</code>. The default is to train a Bayes model. The -c option is given to train a CBayes model:</p>
+The model is then trained using <code>mahout spark-trainnb</code>. The default is to train a Bayes model. The -c option is given to train a CBayes model:</p>
- <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> $ mahout spark-trainnb
+ <pre><code> $ mahout spark-trainnb
-i ${PATH_TO_TFIDF_VECTORS}
-o ${PATH_TO_MODEL}
-ow
-c
-</code></pre></div> </div>
+</code></pre>
</li>
<li>
<p><strong>Label Assignment/Testing:</strong>
-Classification and testing on a holdout set can then be performed via <code class="highlighter-rouge">mahout spark-testnb</code>. Again, the -c option indicates that the model is CBayes:</p>
+Classification and testing on a holdout set can then be performed via <code>mahout spark-testnb</code>. Again, the -c option indicates that the model is CBayes:</p>
- <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> $ mahout spark-testnb
+ <pre><code> $ mahout spark-testnb
-i ${PATH_TO_TFIDF_TEST_VECTORS}
-m ${PATH_TO_MODEL}
-c
-</code></pre></div> </div>
+</code></pre>
</li>
</ul>
@@ -367,9 +367,9 @@ Classification and testing on a holdout set can then be performed via <code clas
<li>
<p><strong>Preprocessing:</strong> <em>note: still reliant on MapReduce seq2sparse</em></p>
- <p>Only relevant parameters used for Bayes/CBayes as detailed above are shown. Several other transformations can be performed by <code class="highlighter-rouge">mahout seq2sparse</code> and used as input to Bayes/CBayes. For a full list of <code class="highlighter-rouge">mahout seq2Sparse</code> options see the <a href="https://mahout.apache.org/users/basics/creating-vectors-from-text.html">Creating vectors from text</a> page.</p>
+ <p>Only relevant parameters used for Bayes/CBayes as detailed above are shown. Several other transformations can be performed by <code>mahout seq2sparse</code> and used as input to Bayes/CBayes. For a full list of <code>mahout seq2Sparse</code> options see the <a href="https://mahout.apache.org/users/basics/creating-vectors-from-text.html">Creating vectors from text</a> page.</p>
- <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> $ mahout seq2sparse
+ <pre><code> $ mahout seq2sparse
--output (-o) output The directory pathname for output.
--input (-i) input Path to job input directory.
--weight (-wt) weight The kind of weight to use. Currently TF
@@ -384,12 +384,12 @@ Classification and testing on a holdout set can then be performed via <code clas
else false
--namedVector (-nv) (Optional) Whether output vectors should
be NamedVectors. If set true else false
-</code></pre></div> </div>
+</code></pre>
</li>
<li>
<p><strong>Training:</strong></p>
- <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> $ mahout spark-trainnb
+ <pre><code> $ mahout spark-trainnb
--input (-i) input Path to job input directory.
--output (-o) output The directory pathname for output.
--trainComplementary (-c) Train complementary? Default is false.
@@ -398,12 +398,12 @@ Classification and testing on a holdout set can then be performed via <code clas
cores to get a performance improvement,
for example "local[4]"
--help (-h) Print out help
-</code></pre></div> </div>
+</code></pre>
</li>
<li>
<p><strong>Testing:</strong></p>
- <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> $ mahout spark-testnb
+ <pre><code> $ mahout spark-testnb
--input (-i) input Path to job input directory.
--model (-m) model The path to the model built during training.
--testComplementary (-c) Test complementary? Default is false.
@@ -412,7 +412,7 @@ Classification and testing on a holdout set can then be performed via <code clas
cores to get a performance improvement,
for example "local[4]"
--help (-h) Print out help
-</code></pre></div> </div>
+</code></pre>
</li>
</ul>
[3/4] mahout git commit: Automatic Site Publish by Buildbot
Posted by gi...@apache.org.
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/basics/collocations.html
----------------------------------------------------------------------
diff --git a/users/basics/collocations.html b/users/basics/collocations.html
index 5c6cd24..875b720 100644
--- a/users/basics/collocations.html
+++ b/users/basics/collocations.html
@@ -369,7 +369,7 @@ specified LLR score from being emitted, and the –minSupport argument can
be used to filter out collocations that appear below a certain number of
times.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bin/mahout seq2sparse
+<pre><code>bin/mahout seq2sparse
Usage:
[--minSupport <minSupport> --analyzerName <analyzerName> --chunkSize <chunkSize>
@@ -418,12 +418,12 @@ Options
--sequentialAccessVector (-seq) (Optional) Whether output vectors should
be SequentialAccessVectors If set true
else false
-</code></pre></div></div>
+</code></pre>
<p><a name="Collocations-CollocDriver"></a></p>
<h3 id="collocdriver">CollocDriver</h3>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bin/mahout org.apache.mahout.vectorizer.collocations.llr.CollocDriver
+<pre><code>bin/mahout org.apache.mahout.vectorizer.collocations.llr.CollocDriver
Usage:
[--input <input> --output <output> --maxNGramSize <ngramSize> --overwrite
@@ -462,7 +462,7 @@ Options
final output alongside collocations
--help (-h) Print out help
-</code></pre></div></div>
+</code></pre>
<p><a name="Collocations-Algorithmdetails"></a></p>
<h2 id="algorithm-details">Algorithm details</h2>
@@ -494,14 +494,14 @@ frequencies are collected across the entire document.</p>
<p>Once this is done, ngrams are split into head and tail portions. A key of type GramKey is generated which is used later to join ngrams with their heads and tails in the reducer phase. The GramKey is a composite key made up of a string n-gram fragement as the primary key and a secondary key used for grouping and sorting in the reduce phase. The secondary key will either be EMPTY in the case where we are collecting either the head or tail of an ngram as the value or it will contain the byte<a href=".html"></a>
form of the ngram when collecting an ngram as the value.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>head_key(EMPTY) -> (head subgram, head frequency)
+<pre><code>head_key(EMPTY) -> (head subgram, head frequency)
head_key(ngram) -> (ngram, ngram frequency)
tail_key(EMPTY) -> (tail subgram, tail frequency)
tail_key(ngram) -> (ngram, ngram frequency)
-</code></pre></div></div>
+</code></pre>
<p>subgram and ngram values are packaged in Gram objects.</p>
@@ -543,7 +543,7 @@ or (subgram_key, ngram) tuple; one from each map task executed in which the
particular subgram was found.
The input will be traversed in the following order:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(head subgram, frequency 1)
+<pre><code>(head subgram, frequency 1)
(head subgram, frequency 2)
...
(head subgram, frequency N)
@@ -560,7 +560,7 @@ The input will be traversed in the following order:</p>
(ngram N, frequency 2)
...
(ngram N, frequency N)
-</code></pre></div></div>
+</code></pre>
<p>Where all of the ngrams above share the same head. Data is presented in the
same manner for the tail subgrams.</p>
@@ -574,18 +574,18 @@ be incremented.</p>
<p>Pairs are passed to the collector in the following format:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ngram, ngram frequency -> subgram subgram frequency
-</code></pre></div></div>
+<pre><code>ngram, ngram frequency -> subgram subgram frequency
+</code></pre>
<p>In this manner, the output becomes an unsorted version of the following:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ngram 1, frequency -> ngram 1 head, head frequency
+<pre><code>ngram 1, frequency -> ngram 1 head, head frequency
ngram 1, frequency -> ngram 1 tail, tail frequency
ngram 2, frequency -> ngram 2 head, head frequency
ngram 2, frequency -> ngram 2 tail, tail frequency
ngram N, frequency -> ngram N head, head frequency
ngram N, frequency -> ngram N tail, tail frequency
-</code></pre></div></div>
+</code></pre>
<p>Output is in the format k:Gram (ngram, frequency), v:Gram (subgram,
frequency)</p>
@@ -610,11 +610,11 @@ the work for llr calculation is done in the reduce phase.</p>
<p>This phase receives the head and tail subgrams and their frequencies for
each ngram (with frequency) produced for the input:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ngram 1, frequency -> ngram 1 head, frequency; ngram 1 tail, frequency
+<pre><code>ngram 1, frequency -> ngram 1 head, frequency; ngram 1 tail, frequency
ngram 2, frequency -> ngram 2 head, frequency; ngram 2 tail, frequency
...
ngram 1, frequency -> ngram N head, frequency; ngram N tail, frequency
-</code></pre></div></div>
+</code></pre>
<p>It also reads the full ngram count obtained from the first pass, passed in
as a configuration option. The parameters to the llr calculation are
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/basics/creating-vectors-from-text.html
----------------------------------------------------------------------
diff --git a/users/basics/creating-vectors-from-text.html b/users/basics/creating-vectors-from-text.html
index ecd9b1e..1dfb217 100644
--- a/users/basics/creating-vectors-from-text.html
+++ b/users/basics/creating-vectors-from-text.html
@@ -310,7 +310,7 @@ option. Examples of running the driver are included below:</p>
<p><a name="CreatingVectorsfromText-GeneratinganoutputfilefromaLuceneIndex"></a></p>
<h4 id="generating-an-output-file-from-a-lucene-index">Generating an output file from a Lucene Index</h4>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$MAHOUT_HOME/bin/mahout lucene.vector
+<pre><code>$MAHOUT_HOME/bin/mahout lucene.vector
--dir (-d) dir The Lucene directory
--idField idField The field in the index
containing the index. If
@@ -362,17 +362,17 @@ option. Examples of running the driver are included below:</p>
percentage is expressed
as a value between 0 and
1. The default is 0.
-</code></pre></div></div>
+</code></pre>
<h4 id="example-create-50-vectors-from-an-index">Example: Create 50 Vectors from an Index</h4>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$MAHOUT_HOME/bin/mahout lucene.vector
+<pre><code>$MAHOUT_HOME/bin/mahout lucene.vector
--dir $WORK_DIR/wikipedia/solr/data/index
--field body
--dictOut $WORK_DIR/solr/wikipedia/dict.txt
--output $WORK_DIR/solr/wikipedia/out.txt
--max 50
-</code></pre></div></div>
+</code></pre>
<p>This uses the index specified by –dir and the body field in it and writes
out the info to the output dir and the dictionary to dict.txt. It only
@@ -382,14 +382,14 @@ the index are output.</p>
<p><a name="CreatingVectorsfromText-50VectorsFromLuceneL2Norm"></a></p>
<h4 id="example-creating-50-normalized-vectors-from-a-lucene-index-using-the-l_2-norm">Example: Creating 50 Normalized Vectors from a Lucene Index using the <a href="http://en.wikipedia.org/wiki/Lp_space">L_2 Norm</a></h4>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$MAHOUT_HOME/bin/mahout lucene.vector
+<pre><code>$MAHOUT_HOME/bin/mahout lucene.vector
--dir $WORK_DIR/wikipedia/solr/data/index
--field body
--dictOut $WORK_DIR/solr/wikipedia/dict.txt
--output $WORK_DIR/solr/wikipedia/out.txt
--max 50
--norm 2
-</code></pre></div></div>
+</code></pre>
<p><a name="CreatingVectorsfromText-FromDirectoryofTextdocuments"></a></p>
<h2 id="from-a-directory-of-text-documents">From A Directory of Text documents</h2>
@@ -408,7 +408,7 @@ binary documents to text.</p>
<p>Mahout has a nifty utility which reads a directory path including its
sub-directories and creates the SequenceFile in a chunked manner for us.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$MAHOUT_HOME/bin/mahout seqdirectory
+<pre><code>$MAHOUT_HOME/bin/mahout seqdirectory
--input (-i) input Path to job input directory.
--output (-o) output The directory pathname for
output.
@@ -438,7 +438,7 @@ sub-directories and creates the SequenceFile in a chunked manner for us.</p>
--tempDir tempDir Intermediate output directory
--startPhase startPhase First phase to run
--endPhase endPhase Last phase to run
-</code></pre></div></div>
+</code></pre>
<p>The output of seqDirectory will be a Sequence file < Text, Text > of all documents (/sub-directory-path/documentFileName, documentText).</p>
@@ -448,7 +448,7 @@ sub-directories and creates the SequenceFile in a chunked manner for us.</p>
<p>From the sequence file generated from the above step run the following to
generate vectors.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$MAHOUT_HOME/bin/mahout seq2sparse
+<pre><code>$MAHOUT_HOME/bin/mahout seq2sparse
--minSupport (-s) minSupport (Optional) Minimum Support. Default
Value: 2
--analyzerName (-a) analyzerName The class name of the analyzer
@@ -497,7 +497,7 @@ generate vectors.</p>
be NamedVectors. If set true else false
--logNormalize (-lnorm) (Optional) Whether output vectors should
be logNormalize. If set true else false
-</code></pre></div></div>
+</code></pre>
<p>This will create SequenceFiles of tokenized documents < Text, StringTuple > (docID, tokenizedDoc) and vectorized documents < Text, VectorWritable > (docID, TF-IDF Vector).</p>
@@ -510,17 +510,17 @@ generate vectors.</p>
<h4 id="example-creating-normalized-tf-idf-vectors-from-a-directory-of-text-documents-using-trigrams-and-the-l_2-norm">Example: Creating Normalized <a href="http://en.wikipedia.org/wiki/Tf%E2%80%93idf">TF-IDF</a> Vectors from a directory of text documents using <a href="http://en.wikipedia.org/wiki/N-gram">trigrams</a> and the <a href="http://en.wikipedia.org/wiki/Lp_space">L_2 Norm</a></h4>
<p>Create sequence files from the directory of text documents:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$MAHOUT_HOME/bin/mahout seqdirectory
+<pre><code>$MAHOUT_HOME/bin/mahout seqdirectory
-i $WORK_DIR/reuters
-o $WORK_DIR/reuters-seqdir
-c UTF-8
-chunk 64
-xm sequential
-</code></pre></div></div>
+</code></pre>
<p>Vectorize the documents using trigrams, L_2 length normalization and a maximum document frequency cutoff of 85%.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$MAHOUT_HOME/bin/mahout seq2sparse
+<pre><code>$MAHOUT_HOME/bin/mahout seq2sparse
-i $WORK_DIR/reuters-out-seqdir/
-o $WORK_DIR/reuters-out-seqdir-sparse-kmeans
--namedVec
@@ -528,7 +528,7 @@ generate vectors.</p>
-ng 3
-n 2
--maxDFPercent 85
-</code></pre></div></div>
+</code></pre>
<p>The sequence file in the $WORK_DIR/reuters-out-seqdir-sparse-kmeans/tfidf-vectors directory can now be used as input to the Mahout <a href="http://mahout.apache.org/users/clustering/k-means-clustering.html">k-Means</a> clustering algorithm.</p>
@@ -549,14 +549,14 @@ format. Probably the easiest way to go would be to implement your own
Iterable<Vector> (called VectorIterable in the example below) and then
reuse the existing VectorWriter classes:</Vector></p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>VectorWriter vectorWriter = SequenceFile.createWriter(filesystem,
+<pre><code>VectorWriter vectorWriter = SequenceFile.createWriter(filesystem,
configuration,
outfile,
LongWritable.class,
SparseVector.class);
long numDocs = vectorWriter.write(new VectorIterable(), Long.MAX_VALUE);
-</code></pre></div></div>
+</code></pre>
</div>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/basics/quickstart.html
----------------------------------------------------------------------
diff --git a/users/basics/quickstart.html b/users/basics/quickstart.html
index 6d8a4c0..b6f689d 100644
--- a/users/basics/quickstart.html
+++ b/users/basics/quickstart.html
@@ -287,12 +287,12 @@
<p>Mahout is also available via a <a href="http://mvnrepository.com/artifact/org.apache.mahout">maven repository</a> under the group id <em>org.apache.mahout</em>.
If you would like to import the latest release of mahout into a java project, add the following dependency in your <em>pom.xml</em>:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code><dependency>
+<pre><code><dependency>
<groupId>org.apache.mahout</groupId>
<artifactId>mahout-mr</artifactId>
<version>0.10.0</version>
</dependency>
-</code></pre></div></div>
+</code></pre>
<h2 id="features">Features</h2>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/classification/bayesian-commandline.html
----------------------------------------------------------------------
diff --git a/users/classification/bayesian-commandline.html b/users/classification/bayesian-commandline.html
index 6039cfd..ffeea8b 100644
--- a/users/classification/bayesian-commandline.html
+++ b/users/classification/bayesian-commandline.html
@@ -288,14 +288,14 @@ complementary naive bayesian classification algorithms on a Hadoop cluster.</p>
<p>In the examples directory type:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mvn -q exec:java
+<pre><code>mvn -q exec:java
-Dexec.mainClass="org.apache.mahout.classifier.bayes.mapreduce.bayes.<JOB>"
-Dexec.args="<OPTIONS>"
mvn -q exec:java
-Dexec.mainClass="org.apache.mahout.classifier.bayes.mapreduce.cbayes.<JOB>"
-Dexec.args="<OPTIONS>"
-</code></pre></div></div>
+</code></pre>
<p><a name="bayesian-commandline-Runningitonthecluster"></a></p>
<h3 id="running-it-on-the-cluster">Running it on the cluster</h3>
@@ -328,7 +328,7 @@ to view all outputs.</p>
<p><a name="bayesian-commandline-Commandlineoptions"></a></p>
<h2 id="command-line-options">Command line options</h2>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>BayesDriver, BayesThetaNormalizerDriver, CBayesNormalizedWeightDriver, CBayesDriver, CBayesThetaDriver, CBayesThetaNormalizerDriver, BayesWeightSummerDriver, BayesFeatureDriver, BayesTfIdfDriver Usage:
+<pre><code>BayesDriver, BayesThetaNormalizerDriver, CBayesNormalizedWeightDriver, CBayesDriver, CBayesThetaDriver, CBayesThetaNormalizerDriver, BayesWeightSummerDriver, BayesFeatureDriver, BayesTfIdfDriver Usage:
[--input <input> --output <output> --help]
Options
@@ -336,7 +336,7 @@ Options
--input (-i) input The Path for input Vectors. Must be a SequenceFile of Writable, Vector.
--output (-o) output The directory pathname for output points.
--help (-h) Print out help.
-</code></pre></div></div>
+</code></pre>
</div>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/classification/bayesian.html
----------------------------------------------------------------------
diff --git a/users/classification/bayesian.html b/users/classification/bayesian.html
index 128e658..22c48df 100644
--- a/users/classification/bayesian.html
+++ b/users/classification/bayesian.html
@@ -288,38 +288,38 @@
<p>As described in <a href="http://people.csail.mit.edu/jrennie/papers/icml03-nb.pdf">[1]</a> Mahout Naive Bayes is broken down into the following steps (assignments are over all possible index values):</p>
<ul>
- <li>Let <code class="highlighter-rouge">\(\vec{d}=(\vec{d_1},...,\vec{d_n})\)</code> be a set of documents; <code class="highlighter-rouge">\(d_{ij}\)</code> is the count of word <code class="highlighter-rouge">\(i\)</code> in document <code class="highlighter-rouge">\(j\)</code>.</li>
- <li>Let <code class="highlighter-rouge">\(\vec{y}=(y_1,...,y_n)\)</code> be their labels.</li>
- <li>Let <code class="highlighter-rouge">\(\alpha_i\)</code> be a smoothing parameter for all words in the vocabulary; let <code class="highlighter-rouge">\(\alpha=\sum_i{\alpha_i}\)</code>.</li>
- <li><strong>Preprocessing</strong>(via seq2Sparse) TF-IDF transformation and L2 length normalization of <code class="highlighter-rouge">\(\vec{d}\)</code>
+ <li>Let <code>\(\vec{d}=(\vec{d_1},...,\vec{d_n})\)</code> be a set of documents; <code>\(d_{ij}\)</code> is the count of word <code>\(i\)</code> in document <code>\(j\)</code>.</li>
+ <li>Let <code>\(\vec{y}=(y_1,...,y_n)\)</code> be their labels.</li>
+ <li>Let <code>\(\alpha_i\)</code> be a smoothing parameter for all words in the vocabulary; let <code>\(\alpha=\sum_i{\alpha_i}\)</code>.</li>
+ <li><strong>Preprocessing</strong>(via seq2Sparse) TF-IDF transformation and L2 length normalization of <code>\(\vec{d}\)</code>
<ol>
- <li><code class="highlighter-rouge">\(d_{ij} = \sqrt{d_{ij}}\)</code></li>
- <li><code class="highlighter-rouge">\(d_{ij} = d_{ij}\left(\log{\frac{\sum_k1}{\sum_k\delta_{ik}+1}}+1\right)\)</code></li>
- <li><code class="highlighter-rouge">\(d_{ij} =\frac{d_{ij}}{\sqrt{\sum_k{d_{kj}^2}}}\)</code></li>
+ <li><code>\(d_{ij} = \sqrt{d_{ij}}\)</code></li>
+ <li><code>\(d_{ij} = d_{ij}\left(\log{\frac{\sum_k1}{\sum_k\delta_{ik}+1}}+1\right)\)</code></li>
+ <li><code>\(d_{ij} =\frac{d_{ij}}{\sqrt{\sum_k{d_{kj}^2}}}\)</code></li>
</ol>
</li>
- <li><strong>Training: Bayes</strong><code class="highlighter-rouge">\((\vec{d},\vec{y})\)</code> calculate term weights <code class="highlighter-rouge">\(w_{ci}\)</code> as:
+ <li><strong>Training: Bayes</strong><code>\((\vec{d},\vec{y})\)</code> calculate term weights <code>\(w_{ci}\)</code> as:
<ol>
- <li><code class="highlighter-rouge">\(\hat\theta_{ci}=\frac{d_{ic}+\alpha_i}{\sum_k{d_{kc}}+\alpha}\)</code></li>
- <li><code class="highlighter-rouge">\(w_{ci}=\log{\hat\theta_{ci}}\)</code></li>
+ <li><code>\(\hat\theta_{ci}=\frac{d_{ic}+\alpha_i}{\sum_k{d_{kc}}+\alpha}\)</code></li>
+ <li><code>\(w_{ci}=\log{\hat\theta_{ci}}\)</code></li>
</ol>
</li>
- <li><strong>Training: CBayes</strong><code class="highlighter-rouge">\((\vec{d},\vec{y})\)</code> calculate term weights <code class="highlighter-rouge">\(w_{ci}\)</code> as:
+ <li><strong>Training: CBayes</strong><code>\((\vec{d},\vec{y})\)</code> calculate term weights <code>\(w_{ci}\)</code> as:
<ol>
- <li><code class="highlighter-rouge">\(\hat\theta_{ci} = \frac{\sum_{j:y_j\neq c}d_{ij}+\alpha_i}{\sum_{j:y_j\neq c}{\sum_k{d_{kj}}}+\alpha}\)</code></li>
- <li><code class="highlighter-rouge">\(w_{ci}=-\log{\hat\theta_{ci}}\)</code></li>
- <li><code class="highlighter-rouge">\(w_{ci}=\frac{w_{ci}}{\sum_i \lvert w_{ci}\rvert}\)</code></li>
+ <li><code>\(\hat\theta_{ci} = \frac{\sum_{j:y_j\neq c}d_{ij}+\alpha_i}{\sum_{j:y_j\neq c}{\sum_k{d_{kj}}}+\alpha}\)</code></li>
+ <li><code>\(w_{ci}=-\log{\hat\theta_{ci}}\)</code></li>
+ <li><code>\(w_{ci}=\frac{w_{ci}}{\sum_i \lvert w_{ci}\rvert}\)</code></li>
</ol>
</li>
<li><strong>Label Assignment/Testing:</strong>
<ol>
- <li>Let <code class="highlighter-rouge">\(\vec{t}= (t_1,...,t_n)\)</code> be a test document; let <code class="highlighter-rouge">\(t_i\)</code> be the count of the word <code class="highlighter-rouge">\(t\)</code>.</li>
- <li>Label the document according to <code class="highlighter-rouge">\(l(t)=\arg\max_c \sum\limits_{i} t_i w_{ci}\)</code></li>
+ <li>Let <code>\(\vec{t}= (t_1,...,t_n)\)</code> be a test document; let <code>\(t_i\)</code> be the count of the word <code>\(t\)</code>.</li>
+ <li>Label the document according to <code>\(l(t)=\arg\max_c \sum\limits_{i} t_i w_{ci}\)</code></li>
</ol>
</li>
</ul>
-<p>As we can see, the main difference between Bayes and CBayes is the weight calculation step. Where Bayes weighs terms more heavily based on the likelihood that they belong to class <code class="highlighter-rouge">\(c\)</code>, CBayes seeks to maximize term weights on the likelihood that they do not belong to any other class.</p>
+<p>As we can see, the main difference between Bayes and CBayes is the weight calculation step. Where Bayes weighs terms more heavily based on the likelihood that they belong to class <code>\(c\)</code>, CBayes seeks to maximize term weights on the likelihood that they do not belong to any other class.</p>
<h2 id="running-from-the-command-line">Running from the command line</h2>
@@ -330,31 +330,31 @@
<p><strong>Preprocessing:</strong>
For a set of Sequence File Formatted documents in PATH_TO_SEQUENCE_FILES the <a href="https://mahout.apache.org/users/basics/creating-vectors-from-text.html">mahout seq2sparse</a> command performs the TF-IDF transformations (-wt tfidf option) and L2 length normalization (-n 2 option) as follows:</p>
- <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> mahout seq2sparse
+ <pre><code> mahout seq2sparse
-i ${PATH_TO_SEQUENCE_FILES}
-o ${PATH_TO_TFIDF_VECTORS}
-nv
-n 2
-wt tfidf
-</code></pre></div> </div>
+</code></pre>
</li>
<li>
<p><strong>Training:</strong>
-The model is then trained using <code class="highlighter-rouge">mahout trainnb</code> . The default is to train a Bayes model. The -c option is given to train a CBayes model:</p>
+The model is then trained using <code>mahout trainnb</code> . The default is to train a Bayes model. The -c option is given to train a CBayes model:</p>
- <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> mahout trainnb
+ <pre><code> mahout trainnb
-i ${PATH_TO_TFIDF_VECTORS}
-o ${PATH_TO_MODEL}/model
-li ${PATH_TO_MODEL}/labelindex
-ow
-c
-</code></pre></div> </div>
+</code></pre>
</li>
<li>
<p><strong>Label Assignment/Testing:</strong>
-Classification and testing on a holdout set can then be performed via <code class="highlighter-rouge">mahout testnb</code>. Again, the -c option indicates that the model is CBayes. The -seq option tells <code class="highlighter-rouge">mahout testnb</code> to run sequentially:</p>
+Classification and testing on a holdout set can then be performed via <code>mahout testnb</code>. Again, the -c option indicates that the model is CBayes. The -seq option tells <code>mahout testnb</code> to run sequentially:</p>
- <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> mahout testnb
+ <pre><code> mahout testnb
-i ${PATH_TO_TFIDF_TEST_VECTORS}
-m ${PATH_TO_MODEL}/model
-l ${PATH_TO_MODEL}/labelindex
@@ -362,7 +362,7 @@ Classification and testing on a holdout set can then be performed via <code clas
-o ${PATH_TO_OUTPUT}
-c
-seq
-</code></pre></div> </div>
+</code></pre>
</li>
</ul>
@@ -372,9 +372,9 @@ Classification and testing on a holdout set can then be performed via <code clas
<li>
<p><strong>Preprocessing:</strong></p>
- <p>Only relevant parameters used for Bayes/CBayes as detailed above are shown. Several other transformations can be performed by <code class="highlighter-rouge">mahout seq2sparse</code> and used as input to Bayes/CBayes. For a full list of <code class="highlighter-rouge">mahout seq2Sparse</code> options see the <a href="https://mahout.apache.org/users/basics/creating-vectors-from-text.html">Creating vectors from text</a> page.</p>
+ <p>Only relevant parameters used for Bayes/CBayes as detailed above are shown. Several other transformations can be performed by <code>mahout seq2sparse</code> and used as input to Bayes/CBayes. For a full list of <code>mahout seq2Sparse</code> options see the <a href="https://mahout.apache.org/users/basics/creating-vectors-from-text.html">Creating vectors from text</a> page.</p>
- <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> mahout seq2sparse
+ <pre><code> mahout seq2sparse
--output (-o) output The directory pathname for output.
--input (-i) input Path to job input directory.
--weight (-wt) weight The kind of weight to use. Currently TF
@@ -389,12 +389,12 @@ Classification and testing on a holdout set can then be performed via <code clas
else false
--namedVector (-nv) (Optional) Whether output vectors should
be NamedVectors. If set true else false
-</code></pre></div> </div>
+</code></pre>
</li>
<li>
<p><strong>Training:</strong></p>
- <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> mahout trainnb
+ <pre><code> mahout trainnb
--input (-i) input Path to job input directory.
--output (-o) output The directory pathname for output.
--alphaI (-a) alphaI Smoothing parameter. Default is 1.0
@@ -406,12 +406,12 @@ Classification and testing on a holdout set can then be performed via <code clas
--tempDir tempDir Intermediate output directory
--startPhase startPhase First phase to run
--endPhase endPhase Last phase to run
-</code></pre></div> </div>
+</code></pre>
</li>
<li>
<p><strong>Testing:</strong></p>
- <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> mahout testnb
+ <pre><code> mahout testnb
--input (-i) input Path to job input directory.
--output (-o) output The directory pathname for output.
--overwrite (-ow) If present, overwrite the output directory
@@ -426,7 +426,7 @@ Classification and testing on a holdout set can then be performed via <code clas
--tempDir tempDir Intermediate output directory
--startPhase startPhase First phase to run
--endPhase endPhase Last phase to run
-</code></pre></div> </div>
+</code></pre>
</li>
</ul>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/classification/breiman-example.html
----------------------------------------------------------------------
diff --git a/users/classification/breiman-example.html b/users/classification/breiman-example.html
index 8d1a60f..c239bd7 100644
--- a/users/classification/breiman-example.html
+++ b/users/classification/breiman-example.html
@@ -300,8 +300,8 @@ results to greater values of <em>m</em></li>
<p>First, we deal with <a href="http://archive.ics.uci.edu/ml/datasets/Glass+Identification">Glass Identification</a>: download the <a href="http://archive.ics.uci.edu/ml/machine-learning-databases/glass/glass.data">dataset</a> file called <strong>glass.data</strong> and store it onto your local machine. Next, we must generate the descriptor file <strong>glass.info</strong> for this dataset with the following command:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bin/mahout org.apache.mahout.classifier.df.tools.Describe -p /path/to/glass.data -f /path/to/glass.info -d I 9 N L
-</code></pre></div></div>
+<pre><code>bin/mahout org.apache.mahout.classifier.df.tools.Describe -p /path/to/glass.data -f /path/to/glass.info -d I 9 N L
+</code></pre>
<p>Substitute <em>/path/to/</em> with the folder where you downloaded the dataset, the argument “I 9 N L” indicates the nature of the variables. Here it means 1
ignored (I) attribute, followed by 9 numerical(N) attributes, followed by
@@ -309,8 +309,8 @@ the label (L).</p>
<p>Finally, we build and evaluate our random forest classifier as follows:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bin/mahout org.apache.mahout.classifier.df.BreimanExample -d /path/to/glass.data -ds /path/to/glass.info -i 10 -t 100 which builds 100 trees (-t argument) and repeats the test 10 iterations (-i argument)
-</code></pre></div></div>
+<pre><code>bin/mahout org.apache.mahout.classifier.df.BreimanExample -d /path/to/glass.data -ds /path/to/glass.info -i 10 -t 100 which builds 100 trees (-t argument) and repeats the test 10 iterations (-i argument)
+</code></pre>
<p>The example outputs the following results:</p>
@@ -327,13 +327,13 @@ iterations</li>
<p>We can repeat this for a <a href="http://archive.ics.uci.edu/ml/datasets/Connectionist+Bench+%28Sonar,+Mines+vs.+Rocks%29">Sonar</a> usecase: download the <a href="http://archive.ics.uci.edu/ml/machine-learning-databases/undocumented/connectionist-bench/sonar/sonar.all-data">dataset</a> file called <strong>sonar.all-data</strong> and store it onto your local machine. Generate the descriptor file <strong>sonar.info</strong> for this dataset with the following command:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bin/mahout org.apache.mahout.classifier.df.tools.Describe -p /path/to/sonar.all-data -f /path/to/sonar.info -d 60 N L
-</code></pre></div></div>
+<pre><code>bin/mahout org.apache.mahout.classifier.df.tools.Describe -p /path/to/sonar.all-data -f /path/to/sonar.info -d 60 N L
+</code></pre>
<p>The argument “60 N L” means 60 numerical(N) attributes, followed by the label (L). Analogous to the previous case, we run the evaluation as follows:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bin/mahout org.apache.mahout.classifier.df.BreimanExample -d /path/to/sonar.all-data -ds /path/to/sonar.info -i 10 -t 100
-</code></pre></div></div>
+<pre><code>bin/mahout org.apache.mahout.classifier.df.BreimanExample -d /path/to/sonar.all-data -ds /path/to/sonar.info -i 10 -t 100
+</code></pre>
</div>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/classification/class-discovery.html
----------------------------------------------------------------------
diff --git a/users/classification/class-discovery.html b/users/classification/class-discovery.html
index 9dcfe83..20f30fc 100644
--- a/users/classification/class-discovery.html
+++ b/users/classification/class-discovery.html
@@ -304,13 +304,13 @@ A classification rule can be represented as follows:</p>
<p>For a given <em>target</em> class and a weight <em>threshold</em>, the classification
rule can be read :</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>for each row of the dataset
+<pre><code>for each row of the dataset
if (rule.w1 < threshold || (rule.w1 >= threshold && row.value1 rule.op1 rule.value1)) &&
(rule.w2 < threshold || (rule.w2 >= threshold && row.value2 rule.op2 rule.value2)) &&
...
(rule.wN < threshold || (rule.wN >= threshold && row.valueN rule.opN rule.valueN)) then
row is part of the target class
-</code></pre></div></div>
+</code></pre>
<p><em>Important:</em> The label attribute is not evaluated by the rule.</p>
@@ -344,11 +344,11 @@ and the following parameters: threshold = 1 and target = 0 (brown).
<p>This rule can be read as follows:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>for each row of the dataset
+<pre><code>for each row of the dataset
if (0 < 1 || (0 >= 1 && row.value1 < 20)) &&
(1 < 1 || (1 >= 1 && row.value2 != light)) then
row is part of the "brown Eye Color" class
-</code></pre></div></div>
+</code></pre>
<p>Please note how the rule skipped the label attribute (Eye Color), and how
the first condition is ignored because its weight is < threshold.</p>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/classification/hidden-markov-models.html
----------------------------------------------------------------------
diff --git a/users/classification/hidden-markov-models.html b/users/classification/hidden-markov-models.html
index 1a84234..6f4fe33 100644
--- a/users/classification/hidden-markov-models.html
+++ b/users/classification/hidden-markov-models.html
@@ -330,18 +330,18 @@ can be efficiently solved using the Baum-Welch algorithm.</li>
<p>Create an input file to train the model. Here we have a sequence drawn from the set of states 0, 1, 2, and 3, separated by space characters.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ echo "0 1 2 2 2 1 1 0 0 3 3 3 2 1 2 1 1 1 1 2 2 2 0 0 0 0 0 0 2 2 2 0 0 0 0 0 0 2 2 2 3 3 3 3 3 3 2 3 2 3 2 3 2 1 3 0 0 0 1 0 1 0 2 1 2 1 2 1 2 3 3 3 3 2 2 3 2 1 1 0" > hmm-input
-</code></pre></div></div>
+<pre><code>$ echo "0 1 2 2 2 1 1 0 0 3 3 3 2 1 2 1 1 1 1 2 2 2 0 0 0 0 0 0 2 2 2 0 0 0 0 0 0 2 2 2 3 3 3 3 3 3 2 3 2 3 2 3 2 1 3 0 0 0 1 0 1 0 2 1 2 1 2 1 2 3 3 3 3 2 2 3 2 1 1 0" > hmm-input
+</code></pre>
<p>Now run the baumwelch job to train your model, after first setting MAHOUT_LOCAL to true, to use your local file system.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ export MAHOUT_LOCAL=true
+<pre><code>$ export MAHOUT_LOCAL=true
$ $MAHOUT_HOME/bin/mahout baumwelch -i hmm-input -o hmm-model -nh 3 -no 4 -e .0001 -m 1000
-</code></pre></div></div>
+</code></pre>
<p>Output like the following should appear in the console.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Initial probabilities:
+<pre><code>Initial probabilities:
0 1 2
1.0 0.0 3.5659361683006626E-251
Transition matrix:
@@ -355,18 +355,18 @@ Emission matrix:
1 7.495656581383351E-34 0.2241269055449904 0.4510889999455847 0.32478409450942497
2 0.815051477991782 0.18494852200821799 8.465660634827592E-33 2.8603899591778015E-36
14/03/22 09:52:21 INFO driver.MahoutDriver: Program took 180 ms (Minutes: 0.003)
-</code></pre></div></div>
+</code></pre>
<p>The model trained with the input set now is in the file ‘hmm-model’, which we can use to build a predicted sequence.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ $MAHOUT_HOME/bin/mahout hmmpredict -m hmm-model -o hmm-predictions -l 10
-</code></pre></div></div>
+<pre><code>$ $MAHOUT_HOME/bin/mahout hmmpredict -m hmm-model -o hmm-predictions -l 10
+</code></pre>
<p>To see the predictions:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ cat hmm-predictions
+<pre><code>$ cat hmm-predictions
0 1 3 3 2 2 2 2 1 2
-</code></pre></div></div>
+</code></pre>
<p><a name="HiddenMarkovModels-Resources"></a></p>
<h2 id="resources">Resources</h2>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/classification/mlp.html
----------------------------------------------------------------------
diff --git a/users/classification/mlp.html b/users/classification/mlp.html
index 5283911..4983775 100644
--- a/users/classification/mlp.html
+++ b/users/classification/mlp.html
@@ -285,9 +285,9 @@ can be used for classification and regression tasks in a supervised learning app
can be used with the following commands:</p>
<h1 id="model-training">model training</h1>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ bin/mahout org.apache.mahout.classifier.mlp.TrainMultilayerPerceptron # model usage
+<pre><code>$ bin/mahout org.apache.mahout.classifier.mlp.TrainMultilayerPerceptron # model usage
$ bin/mahout org.apache.mahout.classifier.mlp.RunMultilayerPerceptron
-</code></pre></div></div>
+</code></pre>
<p>To train and use the model, a number of parameters can be specified. Parameters without default values have to be specified by the user. Consider that not all parameters can be used both for training and running the model. We give an example of the usage below.</p>
@@ -336,7 +336,7 @@ $ bin/mahout org.apache.mahout.classifier.mlp.RunMultilayerPerceptron
<tr>
<td style="text-align: left">–layerSize -ls</td>
<td style="text-align: right"> </td>
- <td style="text-align: left">Number of units per layer, including input, hidden and ouput layers. This parameter specifies the topology of the network (see <a href="mlperceptron_structure.png" title="Architecture of a three-layer MLP">this image</a> for an example specified by <code class="highlighter-rouge">-ls 4 8 3</code>).</td>
+ <td style="text-align: left">Number of units per layer, including input, hidden and ouput layers. This parameter specifies the topology of the network (see <a href="mlperceptron_structure.png" title="Architecture of a three-layer MLP">this image</a> for an example specified by <code>-ls 4 8 3</code>).</td>
<td style="text-align: left">training</td>
</tr>
<tr>
@@ -372,7 +372,7 @@ $ bin/mahout org.apache.mahout.classifier.mlp.RunMultilayerPerceptron
<tr>
<td style="text-align: left">–columnRange -cr</td>
<td style="text-align: right"> </td>
- <td style="text-align: left">Range of the columns to use from the input file, starting with 0 (i.e. <code class="highlighter-rouge">-cr 0 5</code> for including the first six columns only)</td>
+ <td style="text-align: left">Range of the columns to use from the input file, starting with 0 (i.e. <code>-cr 0 5</code> for including the first six columns only)</td>
<td style="text-align: left">testing</td>
</tr>
<tr>
@@ -393,23 +393,23 @@ The dimensions of the data set are given through some flower parameters (sepal l
<p>To train our multilayer perceptron model from the command line, we call the following command</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ bin/mahout org.apache.mahout.classifier.mlp.TrainMultilayerPerceptron \
+<pre><code>$ bin/mahout org.apache.mahout.classifier.mlp.TrainMultilayerPerceptron \
-i ./mrlegacy/src/test/resources/iris.csv -sh \
-labels setosa versicolor virginica \
-mo /tmp/model.model -ls 4 8 3 -l 0.2 -m 0.35 -r 0.0001
-</code></pre></div></div>
+</code></pre>
<p>The individual parameters are explained in the following.</p>
<ul>
- <li><code class="highlighter-rouge">-i ./mrlegacy/src/test/resources/iris.csv</code> use the iris data set as input data</li>
- <li><code class="highlighter-rouge">-sh</code> since the file <code class="highlighter-rouge">iris.csv</code> contains a header row, this row needs to be skipped</li>
- <li><code class="highlighter-rouge">-labels setosa versicolor virginica</code> we specify, which class labels should be learnt (which are the flower species in this case)</li>
- <li><code class="highlighter-rouge">-mo /tmp/model.model</code> specify where to store the model file</li>
- <li><code class="highlighter-rouge">-ls 4 8 3</code> we specify the structure and depth of our layers. The actual network structure can be seen in the figure below.</li>
- <li><code class="highlighter-rouge">-l 0.2</code> we set the learning rate to <code class="highlighter-rouge">0.2</code></li>
- <li><code class="highlighter-rouge">-m 0.35</code> momemtum weight is set to <code class="highlighter-rouge">0.35</code></li>
- <li><code class="highlighter-rouge">-r 0.0001</code> regularization weight is set to <code class="highlighter-rouge">0.0001</code></li>
+ <li><code>-i ./mrlegacy/src/test/resources/iris.csv</code> use the iris data set as input data</li>
+ <li><code>-sh</code> since the file <code>iris.csv</code> contains a header row, this row needs to be skipped</li>
+ <li><code>-labels setosa versicolor virginica</code> we specify, which class labels should be learnt (which are the flower species in this case)</li>
+ <li><code>-mo /tmp/model.model</code> specify where to store the model file</li>
+ <li><code>-ls 4 8 3</code> we specify the structure and depth of our layers. The actual network structure can be seen in the figure below.</li>
+ <li><code>-l 0.2</code> we set the learning rate to <code>0.2</code></li>
+ <li><code>-m 0.35</code> momemtum weight is set to <code>0.35</code></li>
+ <li><code>-r 0.0001</code> regularization weight is set to <code>0.0001</code></li>
</ul>
<table>
@@ -431,19 +431,19 @@ The dimensions of the data set are given through some flower parameters (sepal l
<p>To test / run the multilayer perceptron classification on the trained model, we can use the following command</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ bin/mahout org.apache.mahout.classifier.mlp.RunMultilayerPerceptron \
+<pre><code>$ bin/mahout org.apache.mahout.classifier.mlp.RunMultilayerPerceptron \
-i ./mrlegacy/src/test/resources/iris.csv -sh -cr 0 3 \
-mo /tmp/model.model -o /tmp/labelResult.txt
-</code></pre></div></div>
+</code></pre>
<p>The individual parameters are explained in the following.</p>
<ul>
- <li><code class="highlighter-rouge">-i ./mrlegacy/src/test/resources/iris.csv</code> use the iris data set as input data</li>
- <li><code class="highlighter-rouge">-sh</code> since the file <code class="highlighter-rouge">iris.csv</code> contains a header row, this row needs to be skipped</li>
- <li><code class="highlighter-rouge">-cr 0 3</code> we specify the column range of the input file</li>
- <li><code class="highlighter-rouge">-mo /tmp/model.model</code> specify where the model file is stored</li>
- <li><code class="highlighter-rouge">-o /tmp/labelResult.txt</code> specify where the labeled output file will be stored</li>
+ <li><code>-i ./mrlegacy/src/test/resources/iris.csv</code> use the iris data set as input data</li>
+ <li><code>-sh</code> since the file <code>iris.csv</code> contains a header row, this row needs to be skipped</li>
+ <li><code>-cr 0 3</code> we specify the column range of the input file</li>
+ <li><code>-mo /tmp/model.model</code> specify where the model file is stored</li>
+ <li><code>-o /tmp/labelResult.txt</code> specify where the labeled output file will be stored</li>
</ul>
<h2 id="implementation">Implementation</h2>
@@ -460,7 +460,7 @@ Currently, the logistic sigmoid is used as a squashing function in every hidden
<p>The command line version <strong>does not perform iterations</strong> which leads to bad results on small datasets. Another restriction is, that the CLI version of the MLP only supports classification, since the labels have to be given explicitly when executing on the command line.</p>
-<p>A learned model can be stored and updated with new training instanced using the <code class="highlighter-rouge">--update</code> flag. Output of classification reults is saved as a .txt-file and only consists of the assigned labels. Apart from the command-line interface, it is possible to construct and compile more specialized neural networks using the API and interfaces in the mrlegacy package.</p>
+<p>A learned model can be stored and updated with new training instanced using the <code>--update</code> flag. Output of classification reults is saved as a .txt-file and only consists of the assigned labels. Apart from the command-line interface, it is possible to construct and compile more specialized neural networks using the API and interfaces in the mrlegacy package.</p>
<h2 id="theoretical-background">Theoretical Background</h2>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/classification/partial-implementation.html
----------------------------------------------------------------------
diff --git a/users/classification/partial-implementation.html b/users/classification/partial-implementation.html
index 5028896..6310eca 100644
--- a/users/classification/partial-implementation.html
+++ b/users/classification/partial-implementation.html
@@ -316,8 +316,8 @@ $HADOOP_HOME/bin/hadoop fs -put <PATH TO="" DATA=""> testdata{code}</PATH></li>
<h2 id="generate-a-file-descriptor-for-the-dataset">Generate a file descriptor for the dataset:</h2>
<p>run the following command:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$HADOOP_HOME/bin/hadoop jar $MAHOUT_HOME/core/target/mahout-core-<VERSION>-job.jar org.apache.mahout.classifier.df.tools.Describe -p testdata/KDDTrain+.arff -f testdata/KDDTrain+.info -d N 3 C 2 N C 4 N C 8 N 2 C 19 N L
-</code></pre></div></div>
+<pre><code>$HADOOP_HOME/bin/hadoop jar $MAHOUT_HOME/core/target/mahout-core-<VERSION>-job.jar org.apache.mahout.classifier.df.tools.Describe -p testdata/KDDTrain+.arff -f testdata/KDDTrain+.info -d N 3 C 2 N C 4 N C 8 N 2 C 19 N L
+</code></pre>
<p>The “N 3 C 2 N C 4 N C 8 N 2 C 19 N L” string describes all the attributes
of the data. In this cases, it means 1 numerical(N) attribute, followed by
@@ -327,8 +327,8 @@ to ignore some attributes</p>
<p><a name="PartialImplementation-Runtheexample"></a></p>
<h2 id="run-the-example">Run the example</h2>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$HADOOP_HOME/bin/hadoop jar $MAHOUT_HOME/examples/target/mahout-examples-<version>-job.jar org.apache.mahout.classifier.df.mapreduce.BuildForest -Dmapred.max.split.size=1874231 -d testdata/KDDTrain+.arff -ds testdata/KDDTrain+.info -sl 5 -p -t 100 -o nsl-forest
-</code></pre></div></div>
+<pre><code>$HADOOP_HOME/bin/hadoop jar $MAHOUT_HOME/examples/target/mahout-examples-<version>-job.jar org.apache.mahout.classifier.df.mapreduce.BuildForest -Dmapred.max.split.size=1874231 -d testdata/KDDTrain+.arff -ds testdata/KDDTrain+.info -sl 5 -p -t 100 -o nsl-forest
+</code></pre>
<p>which builds 100 trees (-t argument) using the partial implementation (-p).
Each tree is built using 5 random selected attribute per node (-sl
@@ -356,8 +356,8 @@ nsl-forest/forest.seq</p>
<h2 id="using-the-decision-forest-to-classify-new-data">Using the Decision Forest to Classify new data</h2>
<p>run the following command:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$HADOOP_HOME/bin/hadoop jar $MAHOUT_HOME/examples/target/mahout-examples-<version>-job.jar org.apache.mahout.classifier.df.mapreduce.TestForest -i nsl-kdd/KDDTest+.arff -ds nsl-kdd/KDDTrain+.info -m nsl-forest -a -mr -o predictions
-</code></pre></div></div>
+<pre><code>$HADOOP_HOME/bin/hadoop jar $MAHOUT_HOME/examples/target/mahout-examples-<version>-job.jar org.apache.mahout.classifier.df.mapreduce.TestForest -i nsl-kdd/KDDTest+.arff -ds nsl-kdd/KDDTrain+.info -m nsl-forest -a -mr -o predictions
+</code></pre>
<p>This will compute the predictions of “KDDTest+.arff” dataset (-i argument)
using the same data descriptor generated for the training dataset (-ds) and
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/classification/twenty-newsgroups.html
----------------------------------------------------------------------
diff --git a/users/classification/twenty-newsgroups.html b/users/classification/twenty-newsgroups.html
index 291719f..c671aab 100644
--- a/users/classification/twenty-newsgroups.html
+++ b/users/classification/twenty-newsgroups.html
@@ -307,35 +307,35 @@ the 20 newsgroups.</p>
<li>
<p>If running Hadoop in cluster mode, start the hadoop daemons by executing the following commands:</p>
- <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> $ cd $HADOOP_HOME/bin
+ <pre><code> $ cd $HADOOP_HOME/bin
$ ./start-all.sh
-</code></pre></div> </div>
+</code></pre>
<p>Otherwise:</p>
- <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> $ export MAHOUT_LOCAL=true
-</code></pre></div> </div>
+ <pre><code> $ export MAHOUT_LOCAL=true
+</code></pre>
</li>
<li>
<p>In the trunk directory of Mahout, compile and install Mahout:</p>
- <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> $ cd $MAHOUT_HOME
+ <pre><code> $ cd $MAHOUT_HOME
$ mvn -DskipTests clean install
-</code></pre></div> </div>
+</code></pre>
</li>
<li>
<p>Run the <a href="https://github.com/apache/mahout/blob/master/examples/bin/classify-20newsgroups.sh">20 newsgroups example script</a> by executing:</p>
- <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> $ ./examples/bin/classify-20newsgroups.sh
-</code></pre></div> </div>
+ <pre><code> $ ./examples/bin/classify-20newsgroups.sh
+</code></pre>
</li>
<li>
<p>You will be prompted to select a classification method algorithm:</p>
- <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> 1. Complement Naive Bayes
+ <pre><code> 1. Complement Naive Bayes
2. Naive Bayes
3. Stochastic Gradient Descent
-</code></pre></div> </div>
+</code></pre>
</li>
</ol>
@@ -353,7 +353,7 @@ the 20 newsgroups.</p>
<p>Output should look something like:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>=======================================================
+<pre><code>=======================================================
Confusion Matrix
-------------------------------------------------------
a b c d e f g h i j k l m n o p q r s t <--Classified as
@@ -384,7 +384,7 @@ Kappa 0.8808
Accuracy 90.8596%
Reliability 86.3632%
Reliability (standard deviation) 0.2131
-</code></pre></div></div>
+</code></pre>
<p><a name="TwentyNewsgroups-ComplementaryNaiveBayes"></a></p>
<h2 id="end-to-end-commands-to-build-a-cbayes-model-for-20-newsgroups">End to end commands to build a CBayes model for 20 newsgroups</h2>
@@ -396,14 +396,14 @@ Reliability (standard deviation) 0.2131
<li>
<p>Create a working directory for the dataset and all input/output.</p>
- <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> $ export WORK_DIR=/tmp/mahout-work-${USER}
+ <pre><code> $ export WORK_DIR=/tmp/mahout-work-${USER}
$ mkdir -p ${WORK_DIR}
-</code></pre></div> </div>
+</code></pre>
</li>
<li>
<p>Download and extract the <em>20news-bydate.tar.gz</em> from the <a href="http://people.csail.mit.edu/jrennie/20Newsgroups/20news-bydate.tar.gz">20newsgroups dataset</a> to the working directory.</p>
- <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> $ curl http://people.csail.mit.edu/jrennie/20Newsgroups/20news-bydate.tar.gz
+ <pre><code> $ curl http://people.csail.mit.edu/jrennie/20Newsgroups/20news-bydate.tar.gz
-o ${WORK_DIR}/20news-bydate.tar.gz
$ mkdir -p ${WORK_DIR}/20news-bydate
$ cd ${WORK_DIR}/20news-bydate && tar xzf ../20news-bydate.tar.gz && cd .. && cd ..
@@ -411,62 +411,62 @@ Reliability (standard deviation) 0.2131
$ cp -R ${WORK_DIR}/20news-bydate/*/* ${WORK_DIR}/20news-all * If you're running on a Hadoop cluster:
$ hadoop dfs -put ${WORK_DIR}/20news-all ${WORK_DIR}/20news-all
-</code></pre></div> </div>
+</code></pre>
</li>
<li>
<p>Convert the full 20 newsgroups dataset into a < Text, Text > SequenceFile.</p>
- <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> $ mahout seqdirectory
+ <pre><code> $ mahout seqdirectory
-i ${WORK_DIR}/20news-all
-o ${WORK_DIR}/20news-seq
-ow
-</code></pre></div> </div>
+</code></pre>
</li>
<li>
<p>Convert and preprocesses the dataset into a < Text, VectorWritable > SequenceFile containing term frequencies for each document.</p>
- <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> $ mahout seq2sparse
+ <pre><code> $ mahout seq2sparse
-i ${WORK_DIR}/20news-seq
-o ${WORK_DIR}/20news-vectors
-lnorm
-nv
-wt tfidf If we wanted to use different parsing methods or transformations on the term frequency vectors we could supply different options here e.g.: -ng 2 for bigrams or -n 2 for L2 length normalization. See the [Creating vectors from text](http://mahout.apache.org/users/basics/creating-vectors-from-text.html) page for a list of all seq2sparse options.
-</code></pre></div> </div>
+</code></pre>
</li>
<li>
<p>Split the preprocessed dataset into training and testing sets.</p>
- <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> $ mahout split
+ <pre><code> $ mahout split
-i ${WORK_DIR}/20news-vectors/tfidf-vectors
--trainingOutput ${WORK_DIR}/20news-train-vectors
--testOutput ${WORK_DIR}/20news-test-vectors
--randomSelectionPct 40
--overwrite --sequenceFiles -xm sequential
-</code></pre></div> </div>
+</code></pre>
</li>
<li>
<p>Train the classifier.</p>
- <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> $ mahout trainnb
+ <pre><code> $ mahout trainnb
-i ${WORK_DIR}/20news-train-vectors
-el
-o ${WORK_DIR}/model
-li ${WORK_DIR}/labelindex
-ow
-c
-</code></pre></div> </div>
+</code></pre>
</li>
<li>
<p>Test the classifier.</p>
- <div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> $ mahout testnb
+ <pre><code> $ mahout testnb
-i ${WORK_DIR}/20news-test-vectors
-m ${WORK_DIR}/model
-l ${WORK_DIR}/labelindex
-ow
-o ${WORK_DIR}/20news-testing
-c
-</code></pre></div> </div>
+</code></pre>
</li>
</ol>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/classification/wikipedia-classifier-example.html
----------------------------------------------------------------------
diff --git a/users/classification/wikipedia-classifier-example.html b/users/classification/wikipedia-classifier-example.html
index bda386c..0d10dd1 100644
--- a/users/classification/wikipedia-classifier-example.html
+++ b/users/classification/wikipedia-classifier-example.html
@@ -281,32 +281,32 @@
<h2 id="oververview">Oververview</h2>
-<p>Tou run the example simply execute the <code class="highlighter-rouge">$MAHOUT_HOME/examples/bin/classify-wikipedia.sh</code> script.</p>
+<p>Tou run the example simply execute the <code>$MAHOUT_HOME/examples/bin/classify-wikipedia.sh</code> script.</p>
<p>By defult the script is set to run on a medium sized Wikipedia XML dump. To run on the full set (the entire english Wikipedia) you can change the download by commenting out line 78, and uncommenting line 80 of <a href="https://github.com/apache/mahout/blob/master/examples/bin/classify-wikipedia.sh">classify-wikipedia.sh</a> [1]. However this is not recommended unless you have the resources to do so. <em>Be sure to clean your work directory when changing datasets- option (3).</em></p>
-<p>The step by step process for Creating a Naive Bayes Classifier for the Wikipedia XML dump is very similar to that for <a href="http://mahout.apache.org/users/classification/twenty-newsgroups.html">creating a 20 Newsgroups Classifier</a> [4]. The only difference being that instead of running <code class="highlighter-rouge">$mahout seqdirectory</code> on the unzipped 20 Newsgroups file, you’ll run <code class="highlighter-rouge">$mahout seqwiki</code> on the unzipped Wikipedia xml dump.</p>
+<p>The step by step process for Creating a Naive Bayes Classifier for the Wikipedia XML dump is very similar to that for <a href="http://mahout.apache.org/users/classification/twenty-newsgroups.html">creating a 20 Newsgroups Classifier</a> [4]. The only difference being that instead of running <code>$mahout seqdirectory</code> on the unzipped 20 Newsgroups file, you’ll run <code>$mahout seqwiki</code> on the unzipped Wikipedia xml dump.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ mahout seqwiki
-</code></pre></div></div>
+<pre><code>$ mahout seqwiki
+</code></pre>
-<p>The above command launches <code class="highlighter-rouge">WikipediaToSequenceFile.java</code> which accepts a text file of categories [3] and starts an MR job to parse the each document in the XML file. This process will seek to extract documents with a wikipedia category tag which (exactly, if the <code class="highlighter-rouge">-exactMatchOnly</code> option is set) matches a line in the category file. If no match is found and the <code class="highlighter-rouge">-all</code> option is set, the document will be dumped into an “unknown” category. The documents will then be written out as a <code class="highlighter-rouge"><Text,Text></code> sequence file of the form (K:/category/document_title , V: document).</p>
+<p>The above command launches <code>WikipediaToSequenceFile.java</code> which accepts a text file of categories [3] and starts an MR job to parse the each document in the XML file. This process will seek to extract documents with a wikipedia category tag which (exactly, if the <code>-exactMatchOnly</code> option is set) matches a line in the category file. If no match is found and the <code>-all</code> option is set, the document will be dumped into an “unknown” category. The documents will then be written out as a <code><Text,Text></code> sequence file of the form (K:/category/document_title , V: document).</p>
<p>There are 3 different example category files available to in the /examples/src/test/resources
directory: country.txt, country10.txt and country2.txt. You can edit these categories to extract a different corpus from the Wikipedia dataset.</p>
-<p>The CLI options for <code class="highlighter-rouge">seqwiki</code> are as follows:</p>
+<p>The CLI options for <code>seqwiki</code> are as follows:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>--input (-i) input pathname String
+<pre><code>--input (-i) input pathname String
--output (-o) the output pathname String
--categories (-c) the file containing the Wikipedia categories
--exactMatchOnly (-e) if set, then the Wikipedia category must match
exactly instead of simply containing the category string
--all (-all) if set select all categories
--removeLabels (-rl) if set, remove [[Category:labels]] from document text after extracting label.
-</code></pre></div></div>
+</code></pre>
-<p>After <code class="highlighter-rouge">seqwiki</code>, the script runs <code class="highlighter-rouge">seq2sparse</code>, <code class="highlighter-rouge">split</code>, <code class="highlighter-rouge">trainnb</code> and <code class="highlighter-rouge">testnb</code> as in the <a href="http://mahout.apache.org/users/classification/twenty-newsgroups.html">step by step 20newsgroups example</a>. When all of the jobs have finished, a confusion matrix will be displayed.</p>
+<p>After <code>seqwiki</code>, the script runs <code>seq2sparse</code>, <code>split</code>, <code>trainnb</code> and <code>testnb</code> as in the <a href="http://mahout.apache.org/users/classification/twenty-newsgroups.html">step by step 20newsgroups example</a>. When all of the jobs have finished, a confusion matrix will be displayed.</p>
<p>#Resourcese</p>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/clustering/canopy-clustering.html
----------------------------------------------------------------------
diff --git a/users/clustering/canopy-clustering.html b/users/clustering/canopy-clustering.html
index 1b17ff2..06d0a13 100644
--- a/users/clustering/canopy-clustering.html
+++ b/users/clustering/canopy-clustering.html
@@ -361,7 +361,7 @@ Both require several arguments:</p>
<p>Invocation using the command line takes the form:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bin/mahout canopy \
+<pre><code>bin/mahout canopy \
-i <input vectors directory> \
-o <output working directory> \
-dm <DistanceMeasure> \
@@ -373,7 +373,7 @@ Both require several arguments:</p>
-ow <overwrite output directory if present>
-cl <run input vector clustering after computing Canopies>
-xm <execution method: sequential or mapreduce>
-</code></pre></div></div>
+</code></pre>
<p>Invocation using Java involves supplying the following arguments:</p>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/clustering/canopy-commandline.html
----------------------------------------------------------------------
diff --git a/users/clustering/canopy-commandline.html b/users/clustering/canopy-commandline.html
index e878275..fb7f2eb 100644
--- a/users/clustering/canopy-commandline.html
+++ b/users/clustering/canopy-commandline.html
@@ -282,8 +282,8 @@ an operating Hadoop cluster on the target machine then the invocation will
run Canopy on that cluster. If either of the environment variables are
missing then the stand-alone Hadoop configuration will be invoked instead.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./bin/mahout canopy <OPTIONS>
-</code></pre></div></div>
+<pre><code>./bin/mahout canopy <OPTIONS>
+</code></pre>
<ul>
<li>In $MAHOUT_HOME/, build the jar containing the job (mvn install) The job
@@ -326,7 +326,7 @@ to view all outputs.</li>
<p><a name="canopy-commandline-Commandlineoptions"></a></p>
<h1 id="command-line-options">Command line options</h1>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> --input (-i) input Path to job input directory.Must
+<pre><code> --input (-i) input Path to job input directory.Must
be a SequenceFile of
VectorWritable
--output (-o) output The directory pathname for output.
@@ -340,7 +340,7 @@ to view all outputs.</li>
--clustering (-cl) If present, run clustering after
the iterations have taken place
--help (-h) Print out help
-</code></pre></div></div>
+</code></pre>
</div>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/clustering/cluster-dumper.html
----------------------------------------------------------------------
diff --git a/users/clustering/cluster-dumper.html b/users/clustering/cluster-dumper.html
index ba4e841..2fe2421 100644
--- a/users/clustering/cluster-dumper.html
+++ b/users/clustering/cluster-dumper.html
@@ -295,15 +295,15 @@ you can run clusterdumper in 2 modes:</p>
<h3 id="hadoop-environment">Hadoop Environment</h3>
<p>If you have setup your HADOOP_HOME environment variable, you can use the
-command line utility <code class="highlighter-rouge">mahout</code> to execute the ClusterDumper on Hadoop. In
+command line utility <code>mahout</code> to execute the ClusterDumper on Hadoop. In
this case we wont need to get the output clusters to our local machines.
The utility will read the output clusters present in HDFS and output the
human-readable cluster values into our local file system. Say you’ve just
executed the <a href="clustering-of-synthetic-control-data.html">synthetic control example </a>
- and want to analyze the output, you can execute the <code class="highlighter-rouge">mahout clusterdumper</code> utility from the command line.</p>
+ and want to analyze the output, you can execute the <code>mahout clusterdumper</code> utility from the command line.</p>
<h4 id="cli-options">CLI options:</h4>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>--help Print out help
+<pre><code>--help Print out help
--input (-i) input The directory containing Sequence
Files for the Clusters
--output (-o) output The output file. If not specified,
@@ -329,7 +329,7 @@ executed the <a href="clustering-of-synthetic-control-data.html">synthetic contr
--evaluate (-e) Run ClusterEvaluator and CDbwEvaluator over the
input. The output will be appended to the rest of
the output at the end.
-</code></pre></div></div>
+</code></pre>
<h3 id="standalone-java-program">Standalone Java Program</h3>
@@ -350,11 +350,11 @@ executed the <a href="clustering-of-synthetic-control-data.html">synthetic contr
<p>In the arguments tab, specify the below arguments</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>--seqFileDir <MAHOUT_HOME>/examples/output/clusters-10
+<pre><code>--seqFileDir <MAHOUT_HOME>/examples/output/clusters-10
--pointsDir <MAHOUT_HOME>/examples/output/clusteredPoints
--output <MAHOUT_HOME>/examples/output/clusteranalyze.txt
replace <MAHOUT_HOME> with the actual path of your $MAHOUT_HOME
-</code></pre></div></div>
+</code></pre>
<ul>
<li>Hit run to execute the ClusterDumper using Eclipse. Setting breakpoints etc should just work fine.</li>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/clustering/clustering-of-synthetic-control-data.html
----------------------------------------------------------------------
diff --git a/users/clustering/clustering-of-synthetic-control-data.html b/users/clustering/clustering-of-synthetic-control-data.html
index 2441536..ec32638 100644
--- a/users/clustering/clustering-of-synthetic-control-data.html
+++ b/users/clustering/clustering-of-synthetic-control-data.html
@@ -312,22 +312,22 @@
<li><a href="/users/clustering/canopy-clustering.html">Canopy Clustering</a></li>
</ul>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bin/mahout org.apache.mahout.clustering.syntheticcontrol.canopy.Job
-</code></pre></div></div>
+<pre><code>bin/mahout org.apache.mahout.clustering.syntheticcontrol.canopy.Job
+</code></pre>
<ul>
<li><a href="/users/clustering/k-means-clustering.html">k-Means Clustering</a></li>
</ul>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bin/mahout org.apache.mahout.clustering.syntheticcontrol.kmeans.Job
-</code></pre></div></div>
+<pre><code>bin/mahout org.apache.mahout.clustering.syntheticcontrol.kmeans.Job
+</code></pre>
<ul>
<li><a href="/users/clustering/fuzzy-k-means.html">Fuzzy k-Means Clustering</a></li>
</ul>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bin/mahout org.apache.mahout.clustering.syntheticcontrol.fuzzykmeans.Job
-</code></pre></div></div>
+<pre><code>bin/mahout org.apache.mahout.clustering.syntheticcontrol.fuzzykmeans.Job
+</code></pre>
<p>The clustering output will be produced in the <em>output</em> directory. The output data points are in vector format. In order to read/analyze the output, you can use the <a href="/users/clustering/cluster-dumper.html">clusterdump</a> utility provided by Mahout.</p>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/clustering/clusteringyourdata.html
----------------------------------------------------------------------
diff --git a/users/clustering/clusteringyourdata.html b/users/clustering/clusteringyourdata.html
index 695ed10..6dbe65c 100644
--- a/users/clustering/clusteringyourdata.html
+++ b/users/clustering/clusteringyourdata.html
@@ -315,13 +315,13 @@ In particular for text preparation check out <a href="../basics/creating-vectors
<p>Mahout has a cluster dumper utility that can be used to retrieve and evaluate your clustering data.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./bin/mahout clusterdump <OPTIONS>
-</code></pre></div></div>
+<pre><code>./bin/mahout clusterdump <OPTIONS>
+</code></pre>
<p><a name="ClusteringYourData-Theclusterdumperoptionsare:"></a></p>
<h2 id="the-cluster-dumper-options-are">The cluster dumper options are:</h2>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> --help (-h) Print out help
+<pre><code> --help (-h) Print out help
--input (-i) input The directory containing Sequence
Files for the Clusters
@@ -359,7 +359,7 @@ In particular for text preparation check out <a href="../basics/creating-vectors
--evaluate (-e) Run ClusterEvaluator and CDbwEvaluator over the
input. The output will be appended to the rest of
the output at the end.
-</code></pre></div></div>
+</code></pre>
<p>More information on using clusterdump utility can be found <a href="cluster-dumper.html">here</a></p>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/clustering/fuzzy-k-means-commandline.html
----------------------------------------------------------------------
diff --git a/users/clustering/fuzzy-k-means-commandline.html b/users/clustering/fuzzy-k-means-commandline.html
index 4b8cb3d..7be184e 100644
--- a/users/clustering/fuzzy-k-means-commandline.html
+++ b/users/clustering/fuzzy-k-means-commandline.html
@@ -282,8 +282,8 @@ an operating Hadoop cluster on the target machine then the invocation will
run FuzzyK on that cluster. If either of the environment variables are
missing then the stand-alone Hadoop configuration will be invoked instead.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./bin/mahout fkmeans <OPTIONS>
-</code></pre></div></div>
+<pre><code>./bin/mahout fkmeans <OPTIONS>
+</code></pre>
<ul>
<li>In $MAHOUT_HOME/, build the jar containing the job (mvn install) The job
@@ -324,7 +324,7 @@ to view all outputs.</li>
<p><a name="fuzzy-k-means-commandline-Commandlineoptions"></a></p>
<h1 id="command-line-options">Command line options</h1>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> --input (-i) input Path to job input directory.
+<pre><code> --input (-i) input Path to job input directory.
Must be a SequenceFile of
VectorWritable
--clusters (-c) clusters The input centroids, as Vectors.
@@ -366,7 +366,7 @@ to view all outputs.</li>
is 0
--clustering (-cl) If present, run clustering after
the iterations have taken place
-</code></pre></div></div>
+</code></pre>
</div>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/clustering/fuzzy-k-means.html
----------------------------------------------------------------------
diff --git a/users/clustering/fuzzy-k-means.html b/users/clustering/fuzzy-k-means.html
index 44f5c14..648c188 100644
--- a/users/clustering/fuzzy-k-means.html
+++ b/users/clustering/fuzzy-k-means.html
@@ -351,7 +351,7 @@ FuzzyKMeansDriver.run().</p>
<p>Invocation using the command line takes the form:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bin/mahout fkmeans \
+<pre><code>bin/mahout fkmeans \
-i <input vectors directory> \
-c <input clusters directory> \
-o <output working directory> \
@@ -365,7 +365,7 @@ FuzzyKMeansDriver.run().</p>
-e <emit vectors to most likely cluster during clustering>
-t <threshold to use for clustering if -e is false>
-xm <execution method: sequential or mapreduce>
-</code></pre></div></div>
+</code></pre>
<p><em>Note:</em> if the -k argument is supplied, any clusters in the -c directory
will be overwritten and -k random points will be sampled from the input
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/clustering/k-means-clustering.html
----------------------------------------------------------------------
diff --git a/users/clustering/k-means-clustering.html b/users/clustering/k-means-clustering.html
index 21f9e2f..431aaa7 100644
--- a/users/clustering/k-means-clustering.html
+++ b/users/clustering/k-means-clustering.html
@@ -331,14 +331,14 @@ clustering and convergence values.</p>
<p>Canopy clustering can be used to compute the initial clusters for k-KMeans:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>// run the CanopyDriver job
+<pre><code>// run the CanopyDriver job
CanopyDriver.runJob("testdata", "output"
ManhattanDistanceMeasure.class.getName(), (float) 3.1, (float) 2.1, false);
// now run the KMeansDriver job
KMeansDriver.runJob("testdata", "output/clusters-0", "output",
EuclideanDistanceMeasure.class.getName(), "0.001", "10", true);
-</code></pre></div></div>
+</code></pre>
<p>In the above example, the input data points are stored in ‘testdata’ and
the CanopyDriver is configured to output to the ‘output/clusters-0’
@@ -359,7 +359,7 @@ on KMeansDriver.main or by making a Java call to KMeansDriver.runJob().</p>
<p>Invocation using the command line takes the form:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bin/mahout kmeans \
+<pre><code>bin/mahout kmeans \
-i <input vectors directory> \
-c <input clusters directory> \
-o <output working directory> \
@@ -370,7 +370,7 @@ on KMeansDriver.main or by making a Java call to KMeansDriver.runJob().</p>
-ow <overwrite output directory if present>
-cl <run input vector clustering after computing Canopies>
-xm <execution method: sequential or mapreduce>
-</code></pre></div></div>
+</code></pre>
<p>Note: if the -k argument is supplied, any clusters in the -c directory
will be overwritten and -k random points will be sampled from the input
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/clustering/k-means-commandline.html
----------------------------------------------------------------------
diff --git a/users/clustering/k-means-commandline.html b/users/clustering/k-means-commandline.html
index 318b847..cf7de7a 100644
--- a/users/clustering/k-means-commandline.html
+++ b/users/clustering/k-means-commandline.html
@@ -289,8 +289,8 @@ an operating Hadoop cluster on the target machine then the invocation will
run k-Means on that cluster. If either of the environment variables are
missing then the stand-alone Hadoop configuration will be invoked instead.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./bin/mahout kmeans <OPTIONS>
-</code></pre></div></div>
+<pre><code>./bin/mahout kmeans <OPTIONS>
+</code></pre>
<p>In $MAHOUT_HOME/, build the jar containing the job (mvn install) The job
will be generated in $MAHOUT_HOME/core/target/ and it’s name will contain
@@ -331,7 +331,7 @@ to view all outputs.</li>
<p><a name="k-means-commandline-Commandlineoptions"></a></p>
<h1 id="command-line-options">Command line options</h1>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> --input (-i) input Path to job input directory.
+<pre><code> --input (-i) input Path to job input directory.
Must be a SequenceFile of
VectorWritable
--clusters (-c) clusters The input centroids, as Vectors.
@@ -362,7 +362,7 @@ to view all outputs.</li>
--help (-h) Print out help
--clustering (-cl) If present, run clustering after
the iterations have taken place
-</code></pre></div></div>
+</code></pre>
</div>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/clustering/latent-dirichlet-allocation.html
----------------------------------------------------------------------
diff --git a/users/clustering/latent-dirichlet-allocation.html b/users/clustering/latent-dirichlet-allocation.html
index 78a8e4f..e857424 100644
--- a/users/clustering/latent-dirichlet-allocation.html
+++ b/users/clustering/latent-dirichlet-allocation.html
@@ -343,7 +343,7 @@ vectors, it’s recommended that you follow the instructions in <a href="../basi
<p>Invocation takes the form:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bin/mahout cvb \
+<pre><code>bin/mahout cvb \
-i <input path for document vectors> \
-dict <path to term-dictionary file(s) , glob expression supported> \
-o <output path for topic-term distributions>
@@ -358,7 +358,7 @@ vectors, it’s recommended that you follow the instructions in <a href="../basi
-seed <random seed> \
-tf <fraction of data to hold for testing> \
-block <number of iterations per perplexity check, ignored unless test_set_percentage>0> \
-</code></pre></div></div>
+</code></pre>
<p>Topic smoothing should generally be about 50/K, where K is the number of
topics. The number of words in the vocabulary can be an upper bound, though
@@ -370,14 +370,14 @@ recommended that you try several values.</p>
<p>After running LDA you can obtain an output of the computed topics using the
LDAPrintTopics utility:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bin/mahout ldatopics \
+<pre><code>bin/mahout ldatopics \
-i <input vectors directory> \
-d <input dictionary file> \
-w <optional number of words to print> \
-o <optional output working directory. Default is to console> \
-h <print out help> \
-dt <optional dictionary type (text|sequencefile). Default is text>
-</code></pre></div></div>
+</code></pre>
<p><a name="LatentDirichletAllocation-Example"></a></p>
<h1 id="example">Example</h1>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/clustering/lda-commandline.html
----------------------------------------------------------------------
diff --git a/users/clustering/lda-commandline.html b/users/clustering/lda-commandline.html
index d3f4c67..729c061 100644
--- a/users/clustering/lda-commandline.html
+++ b/users/clustering/lda-commandline.html
@@ -285,8 +285,8 @@ Hadoop cluster on the target machine then the invocation will run the LDA
algorithm on that cluster. If either of the environment variables are
missing then the stand-alone Hadoop configuration will be invoked instead.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./bin/mahout cvb <OPTIONS>
-</code></pre></div></div>
+<pre><code>./bin/mahout cvb <OPTIONS>
+</code></pre>
<ul>
<li>In $MAHOUT_HOME/, build the jar containing the job (mvn install) The job
@@ -327,7 +327,7 @@ to view all outputs.</li>
<p><a name="lda-commandline-CommandlineoptionsfromMahoutcvbversion0.8"></a></p>
<h1 id="command-line-options-from-mahout-cvb-version-08">Command line options from Mahout cvb version 0.8</h1>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mahout cvb -h
+<pre><code>mahout cvb -h
--input (-i) input Path to job input directory.
--output (-o) output The directory pathname for output.
--maxIter (-x) maxIter The maximum number of iterations.
@@ -352,7 +352,7 @@ to view all outputs.</li>
--tempDir tempDir Intermediate output directory
--startPhase startPhase First phase to run
--endPhase endPhase Last phase to run
-</code></pre></div></div>
+</code></pre>
</div>
[2/4] mahout git commit: Automatic Site Publish by Buildbot
Posted by gi...@apache.org.
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/clustering/spectral-clustering.html
----------------------------------------------------------------------
diff --git a/users/clustering/spectral-clustering.html b/users/clustering/spectral-clustering.html
index e8b6e9a..093cb3c 100644
--- a/users/clustering/spectral-clustering.html
+++ b/users/clustering/spectral-clustering.html
@@ -280,16 +280,16 @@
<ol>
<li>
- <p>Computing a similarity (or <em>affinity</em>) matrix <code class="highlighter-rouge">\(\mathbf{A}\)</code> from the data. This involves determining a pairwise distance function <code class="highlighter-rouge">\(f\)</code> that takes a pair of data points and returns a scalar.</p>
+ <p>Computing a similarity (or <em>affinity</em>) matrix <code>\(\mathbf{A}\)</code> from the data. This involves determining a pairwise distance function <code>\(f\)</code> that takes a pair of data points and returns a scalar.</p>
</li>
<li>
- <p>Computing a graph Laplacian <code class="highlighter-rouge">\(\mathbf{L}\)</code> from the affinity matrix. There are several types of graph Laplacians; which is used will often depends on the situation.</p>
+ <p>Computing a graph Laplacian <code>\(\mathbf{L}\)</code> from the affinity matrix. There are several types of graph Laplacians; which is used will often depends on the situation.</p>
</li>
<li>
- <p>Computing the eigenvectors and eigenvalues of <code class="highlighter-rouge">\(\mathbf{L}\)</code>. The degree of this decomposition is often modulated by <code class="highlighter-rouge">\(k\)</code>, or the number of clusters. Put another way, <code class="highlighter-rouge">\(k\)</code> eigenvectors and eigenvalues are computed.</p>
+ <p>Computing the eigenvectors and eigenvalues of <code>\(\mathbf{L}\)</code>. The degree of this decomposition is often modulated by <code>\(k\)</code>, or the number of clusters. Put another way, <code>\(k\)</code> eigenvectors and eigenvalues are computed.</p>
</li>
<li>
- <p>The <code class="highlighter-rouge">\(k\)</code> eigenvectors are used as “proxy” data for the original dataset, and fed into k-means clustering. The resulting cluster assignments are transparently passed back to the original data.</p>
+ <p>The <code>\(k\)</code> eigenvectors are used as “proxy” data for the original dataset, and fed into k-means clustering. The resulting cluster assignments are transparently passed back to the original data.</p>
</li>
</ol>
@@ -303,13 +303,13 @@
<h2 id="input">Input</h2>
-<p>The input format for the algorithm currently takes the form of a Hadoop-backed affinity matrix in the form of text files. Each line of the text file specifies a single element of the affinity matrix: the row index <code class="highlighter-rouge">\(i\)</code>, the column index <code class="highlighter-rouge">\(j\)</code>, and the value:</p>
+<p>The input format for the algorithm currently takes the form of a Hadoop-backed affinity matrix in the form of text files. Each line of the text file specifies a single element of the affinity matrix: the row index <code>\(i\)</code>, the column index <code>\(j\)</code>, and the value:</p>
-<p><code class="highlighter-rouge">i, j, value</code></p>
+<p><code>i, j, value</code></p>
-<p>The affinity matrix is symmetric, and any unspecified <code class="highlighter-rouge">\(i, j\)</code> pairs are assumed to be 0 for sparsity. The row and column indices are 0-indexed. Thus, only the non-zero entries of either the upper or lower triangular need be specified.</p>
+<p>The affinity matrix is symmetric, and any unspecified <code>\(i, j\)</code> pairs are assumed to be 0 for sparsity. The row and column indices are 0-indexed. Thus, only the non-zero entries of either the upper or lower triangular need be specified.</p>
-<p>The matrix elements specified in the text files are collected into a Mahout <code class="highlighter-rouge">DistributedRowMatrix</code>.</p>
+<p>The matrix elements specified in the text files are collected into a Mahout <code>DistributedRowMatrix</code>.</p>
<p><strong>(<a href="https://issues.apache.org/jira/browse/MAHOUT-1539">MAHOUT-1539</a> will allow for the creation of the affinity matrix to occur as part of the core spectral clustering algorithm, as opposed to the current requirement that the user create this matrix themselves and provide it, rather than the original data, to the algorithm)</strong></p>
@@ -319,21 +319,21 @@
<p>Spectral clustering can be invoked with the following arguments.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bin/mahout spectralkmeans \
+<pre><code>bin/mahout spectralkmeans \
-i <affinity matrix directory> \
-o <output working directory> \
-d <number of data points> \
-k <number of clusters AND number of top eigenvectors to use> \
-x <maximum number of k-means iterations>
-</code></pre></div></div>
+</code></pre>
-<p>The affinity matrix can be contained in a single text file (using the aforementioned one-line-per-entry format) or span many text files <a href="https://issues.apache.org/jira/browse/MAHOUT-978">per (MAHOUT-978</a>, do not prefix text files with a leading underscore ‘_’ or period ‘.’). The <code class="highlighter-rouge">-d</code> flag is required for the algorithm to know the dimensions of the affinity matrix. <code class="highlighter-rouge">-k</code> is the number of top eigenvectors from the normalized graph Laplacian in the SSVD step, and also the number of clusters given to k-means after the SSVD step.</p>
+<p>The affinity matrix can be contained in a single text file (using the aforementioned one-line-per-entry format) or span many text files <a href="https://issues.apache.org/jira/browse/MAHOUT-978">per (MAHOUT-978</a>, do not prefix text files with a leading underscore ‘_’ or period ‘.’). The <code>-d</code> flag is required for the algorithm to know the dimensions of the affinity matrix. <code>-k</code> is the number of top eigenvectors from the normalized graph Laplacian in the SSVD step, and also the number of clusters given to k-means after the SSVD step.</p>
<h2 id="example">Example</h2>
-<p>To provide a simple example, take the following affinity matrix, contained in a text file called <code class="highlighter-rouge">affinity.txt</code>:</p>
+<p>To provide a simple example, take the following affinity matrix, contained in a text file called <code>affinity.txt</code>:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>0, 0, 0
+<pre><code>0, 0, 0
0, 1, 0.8
0, 2, 0.5
1, 0, 0.8
@@ -342,21 +342,21 @@
2, 0, 0.5
2, 1, 0.9
2, 2, 0
-</code></pre></div></div>
+</code></pre>
-<p>With this 3-by-3 matrix, <code class="highlighter-rouge">-d</code> would be <code class="highlighter-rouge">3</code>. Furthermore, since all affinity matrices are assumed to be symmetric, the entries specifying both <code class="highlighter-rouge">1, 2, 0.9</code> and <code class="highlighter-rouge">2, 1, 0.9</code> are redundant; only one of these is needed. Additionally, any entries that are 0, such as those along the diagonal, also need not be specified at all. They are provided here for completeness.</p>
+<p>With this 3-by-3 matrix, <code>-d</code> would be <code>3</code>. Furthermore, since all affinity matrices are assumed to be symmetric, the entries specifying both <code>1, 2, 0.9</code> and <code>2, 1, 0.9</code> are redundant; only one of these is needed. Additionally, any entries that are 0, such as those along the diagonal, also need not be specified at all. They are provided here for completeness.</p>
<p>In general, larger values indicate a stronger “connectedness”, whereas smaller values indicate a weaker connectedness. This will vary somewhat depending on the distance function used, though a common one is the <a href="http://en.wikipedia.org/wiki/RBF_kernel">RBF kernel</a> (used in the above example) which returns values in the range [0, 1], where 0 indicates completely disconnected (or completely dissimilar) and 1 is fully connected (or identical).</p>
<p>The call signature with this matrix could be as follows:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bin/mahout spectralkmeans \
+<pre><code>bin/mahout spectralkmeans \
-i s3://mahout-example/input/ \
-o s3://mahout-example/output/ \
-d 3 \
-k 2 \
-x 10
-</code></pre></div></div>
+</code></pre>
<p>There are many other optional arguments, in particular for tweaking the SSVD process (block size, number of power iterations, etc) and the k-means clustering step (distance measure, convergence delta, etc).</p>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/clustering/streaming-k-means.html
----------------------------------------------------------------------
diff --git a/users/clustering/streaming-k-means.html b/users/clustering/streaming-k-means.html
index 1056c5d..bd51e2f 100644
--- a/users/clustering/streaming-k-means.html
+++ b/users/clustering/streaming-k-means.html
@@ -397,7 +397,7 @@ The algorithm can be instructed to take multiple independent runs (using the <em
<p>##Usage of <em>StreamingKMeans</em></p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> bin/mahout streamingkmeans
+<pre><code> bin/mahout streamingkmeans
-i <input>
-o <output>
-ow
@@ -420,33 +420,33 @@ The algorithm can be instructed to take multiple independent runs (using the <em
--tempDir <tempDir>
--startPhase <startPhase>
--endPhase <endPhase>
-</code></pre></div></div>
+</code></pre>
<p>###Details on Job-Specific Options:</p>
<ul>
- <li><code class="highlighter-rouge">--input (-i) <input></code>: Path to job input directory.</li>
- <li><code class="highlighter-rouge">--output (-o) <output></code>: The directory pathname for output.</li>
- <li><code class="highlighter-rouge">--overwrite (-ow)</code>: If present, overwrite the output directory before running job.</li>
- <li><code class="highlighter-rouge">--numClusters (-k) <k></code>: The k in k-Means. Approximately this many clusters will be generated.</li>
- <li><code class="highlighter-rouge">--estimatedNumMapClusters (-km) <estimatedNumMapClusters></code>: The estimated number of clusters to use for the Map phase of the job when running StreamingKMeans. This should be around k * log(n), where k is the final number of clusters and n is the total number of data points to cluster.</li>
- <li><code class="highlighter-rouge">--estimatedDistanceCutoff (-e) <estimatedDistanceCutoff></code>: The initial estimated distance cutoff between two points for forming new clusters. If no value is given, it’s estimated from the data set</li>
- <li><code class="highlighter-rouge">--maxNumIterations (-mi) <maxNumIterations></code>: The maximum number of iterations to run for the BallKMeans algorithm used by the reducer. If no value is given, defaults to 10.</li>
- <li><code class="highlighter-rouge">--trimFraction (-tf) <trimFraction></code>: The ‘ball’ aspect of ball k-means means that only the closest points to the centroid will actually be used for updating. The fraction of the points to be used is those points whose distance to the center is within trimFraction * distance to the closest other center. If no value is given, defaults to 0.9.</li>
- <li><code class="highlighter-rouge">--randomInit</code> (<code class="highlighter-rouge">-ri</code>) Whether to use k-means++ initialization or random initialization of the seed centroids. Essentially, k-means++ provides better clusters, but takes longer, whereas random initialization takes less time, but produces worse clusters, and tends to fail more often and needs multiple runs to compare to k-means++. If set, uses the random initialization.</li>
- <li><code class="highlighter-rouge">--ignoreWeights (-iw)</code>: Whether to correct the weights of the centroids after the clustering is done. The weights end up being wrong because of the trimFraction and possible train/test splits. In some cases, especially in a pipeline, having an accurate count of the weights is useful. If set, ignores the final weights.</li>
- <li><code class="highlighter-rouge">--testProbability (-testp) <testProbability></code>: A double value between 0 and 1 that represents the percentage of points to be used for ‘testing’ different clustering runs in the final BallKMeans step. If no value is given, defaults to 0.1</li>
- <li><code class="highlighter-rouge">--numBallKMeansRuns (-nbkm) <numBallKMeansRuns></code>: Number of BallKMeans runs to use at the end to try to cluster the points. If no value is given, defaults to 4</li>
- <li><code class="highlighter-rouge">--distanceMeasure (-dm) <distanceMeasure></code>: The classname of the DistanceMeasure. Default is SquaredEuclidean.</li>
- <li><code class="highlighter-rouge">--searcherClass (-sc) <searcherClass></code>: The type of searcher to be used when performing nearest neighbor searches. Defaults to ProjectionSearch.</li>
- <li><code class="highlighter-rouge">--numProjections (-np) <numProjections></code>: The number of projections considered in estimating the distances between vectors. Only used when the distance measure requested is either ProjectionSearch or FastProjectionSearch. If no value is given, defaults to 3.</li>
- <li><code class="highlighter-rouge">--searchSize (-s) <searchSize></code>: In more efficient searches (non BruteSearch), not all distances are calculated for determining the nearest neighbors. The number of elements whose distances from the query vector is actually computer is proportional to searchSize. If no value is given, defaults to 1.</li>
- <li><code class="highlighter-rouge">--reduceStreamingKMeans (-rskm)</code>: There might be too many intermediate clusters from the mapper to fit into memory, so the reducer can run another pass of StreamingKMeans to collapse them down to a fewer clusters.</li>
- <li><code class="highlighter-rouge">--method (-xm)</code> method The execution method to use: sequential or mapreduce. Default is mapreduce.</li>
- <li><code class="highlighter-rouge">-- help (-h)</code>: Print out help</li>
- <li><code class="highlighter-rouge">--tempDir <tempDir></code>: Intermediate output directory.</li>
- <li><code class="highlighter-rouge">--startPhase <startPhase></code> First phase to run.</li>
- <li><code class="highlighter-rouge">--endPhase <endPhase></code> Last phase to run.</li>
+ <li><code>--input (-i) <input></code>: Path to job input directory.</li>
+ <li><code>--output (-o) <output></code>: The directory pathname for output.</li>
+ <li><code>--overwrite (-ow)</code>: If present, overwrite the output directory before running job.</li>
+ <li><code>--numClusters (-k) <k></code>: The k in k-Means. Approximately this many clusters will be generated.</li>
+ <li><code>--estimatedNumMapClusters (-km) <estimatedNumMapClusters></code>: The estimated number of clusters to use for the Map phase of the job when running StreamingKMeans. This should be around k * log(n), where k is the final number of clusters and n is the total number of data points to cluster.</li>
+ <li><code>--estimatedDistanceCutoff (-e) <estimatedDistanceCutoff></code>: The initial estimated distance cutoff between two points for forming new clusters. If no value is given, it’s estimated from the data set</li>
+ <li><code>--maxNumIterations (-mi) <maxNumIterations></code>: The maximum number of iterations to run for the BallKMeans algorithm used by the reducer. If no value is given, defaults to 10.</li>
+ <li><code>--trimFraction (-tf) <trimFraction></code>: The ‘ball’ aspect of ball k-means means that only the closest points to the centroid will actually be used for updating. The fraction of the points to be used is those points whose distance to the center is within trimFraction * distance to the closest other center. If no value is given, defaults to 0.9.</li>
+ <li><code>--randomInit</code> (<code>-ri</code>) Whether to use k-means++ initialization or random initialization of the seed centroids. Essentially, k-means++ provides better clusters, but takes longer, whereas random initialization takes less time, but produces worse clusters, and tends to fail more often and needs multiple runs to compare to k-means++. If set, uses the random initialization.</li>
+ <li><code>--ignoreWeights (-iw)</code>: Whether to correct the weights of the centroids after the clustering is done. The weights end up being wrong because of the trimFraction and possible train/test splits. In some cases, especially in a pipeline, having an accurate count of the weights is useful. If set, ignores the final weights.</li>
+ <li><code>--testProbability (-testp) <testProbability></code>: A double value between 0 and 1 that represents the percentage of points to be used for ‘testing’ different clustering runs in the final BallKMeans step. If no value is given, defaults to 0.1</li>
+ <li><code>--numBallKMeansRuns (-nbkm) <numBallKMeansRuns></code>: Number of BallKMeans runs to use at the end to try to cluster the points. If no value is given, defaults to 4</li>
+ <li><code>--distanceMeasure (-dm) <distanceMeasure></code>: The classname of the DistanceMeasure. Default is SquaredEuclidean.</li>
+ <li><code>--searcherClass (-sc) <searcherClass></code>: The type of searcher to be used when performing nearest neighbor searches. Defaults to ProjectionSearch.</li>
+ <li><code>--numProjections (-np) <numProjections></code>: The number of projections considered in estimating the distances between vectors. Only used when the distance measure requested is either ProjectionSearch or FastProjectionSearch. If no value is given, defaults to 3.</li>
+ <li><code>--searchSize (-s) <searchSize></code>: In more efficient searches (non BruteSearch), not all distances are calculated for determining the nearest neighbors. The number of elements whose distances from the query vector is actually computer is proportional to searchSize. If no value is given, defaults to 1.</li>
+ <li><code>--reduceStreamingKMeans (-rskm)</code>: There might be too many intermediate clusters from the mapper to fit into memory, so the reducer can run another pass of StreamingKMeans to collapse them down to a fewer clusters.</li>
+ <li><code>--method (-xm)</code> method The execution method to use: sequential or mapreduce. Default is mapreduce.</li>
+ <li><code>-- help (-h)</code>: Print out help</li>
+ <li><code>--tempDir <tempDir></code>: Intermediate output directory.</li>
+ <li><code>--startPhase <startPhase></code> First phase to run.</li>
+ <li><code>--endPhase <endPhase></code> Last phase to run.</li>
</ul>
<p>##References</p>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/clustering/viewing-results.html
----------------------------------------------------------------------
diff --git a/users/clustering/viewing-results.html b/users/clustering/viewing-results.html
index a462caf..c36024a 100644
--- a/users/clustering/viewing-results.html
+++ b/users/clustering/viewing-results.html
@@ -294,16 +294,16 @@ demonstrate the various ways one might inspect the outcome of various jobs.
<p>Run the following to print out all options:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>java -cp "*" org.apache.mahout.utils.clustering.ClusterDumper --help
-</code></pre></div></div>
+<pre><code>java -cp "*" org.apache.mahout.utils.clustering.ClusterDumper --help
+</code></pre>
<p><a name="ViewingResults-Example"></a></p>
<h3 id="example">Example</h3>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>java -cp "*" org.apache.mahout.utils.clustering.ClusterDumper --seqFileDir ./solr-clust-n2/out/clusters-2
+<pre><code>java -cp "*" org.apache.mahout.utils.clustering.ClusterDumper --seqFileDir ./solr-clust-n2/out/clusters-2
--dictionary ./solr-clust-n2/dictionary.txt
--substring 100 --pointsDir ./solr-clust-n2/out/points/
-</code></pre></div></div>
+</code></pre>
<p><a name="ViewingResults-ClusterLabels(MAHOUT-163)"></a></p>
<h2 id="cluster-labels-mahout-163">Cluster Labels (MAHOUT-163)</h2>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/clustering/visualizing-sample-clusters.html
----------------------------------------------------------------------
diff --git a/users/clustering/visualizing-sample-clusters.html b/users/clustering/visualizing-sample-clusters.html
index eefae35..a10ee0b 100644
--- a/users/clustering/visualizing-sample-clusters.html
+++ b/users/clustering/visualizing-sample-clusters.html
@@ -306,9 +306,9 @@ programs.</li>
<p>If you are using Eclipse, just right-click on each of the classes mentioned above and choose “Run As -Java Application”. To run these directly from the command line:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cd $MAHOUT_HOME/examples
+<pre><code>cd $MAHOUT_HOME/examples
mvn -q exec:java -Dexec.mainClass=org.apache.mahout.clustering.display.DisplayClustering
-</code></pre></div></div>
+</code></pre>
<p>You can substitute other names above for <em>DisplayClustering</em>.</p>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/dim-reduction/ssvd.html
----------------------------------------------------------------------
diff --git a/users/dim-reduction/ssvd.html b/users/dim-reduction/ssvd.html
index b06f0dc..ebe184d 100644
--- a/users/dim-reduction/ssvd.html
+++ b/users/dim-reduction/ssvd.html
@@ -325,7 +325,7 @@ approximations of matrices” contains comprehensive definition of parallelizati
<p><strong>tests.R</strong></p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>n<-1000
+<pre><code>n<-1000
m<-2000
k<-10
@@ -339,7 +339,7 @@ vsim<- qr.Q(qr( matrix(rnorm(n*k,mean=5), nrow=n,ncol=k)))
x<- usim %*% svalsim %*% t(vsim)
-</code></pre></div></div>
+</code></pre>
<p>and try to compare ssvd.svd(x) and stock svd(x) performance for the same rank k, notice the difference in the running time. Also play with power iterations (qIter) and compare accuracies of standard svd and SSVD.</p>
@@ -347,51 +347,51 @@ x<- usim %*% svalsim %*% t(vsim)
<h4 id="modified-ssvd-algorithm">Modified SSVD Algorithm.</h4>
-<p>Given an <code class="highlighter-rouge">\(m\times n\)</code>
-matrix <code class="highlighter-rouge">\(\mathbf{A}\)</code>, a target rank <code class="highlighter-rouge">\(k\in\mathbb{N}_{1}\)</code>
-, an oversampling parameter <code class="highlighter-rouge">\(p\in\mathbb{N}_{1}\)</code>,
-and the number of additional power iterations <code class="highlighter-rouge">\(q\in\mathbb{N}_{0}\)</code>,
-this procedure computes an <code class="highlighter-rouge">\(m\times\left(k+p\right)\)</code>
-SVD <code class="highlighter-rouge">\(\mathbf{A\approx U}\boldsymbol{\Sigma}\mathbf{V}^{\top}\)</code>:</p>
+<p>Given an <code>\(m\times n\)</code>
+matrix <code>\(\mathbf{A}\)</code>, a target rank <code>\(k\in\mathbb{N}_{1}\)</code>
+, an oversampling parameter <code>\(p\in\mathbb{N}_{1}\)</code>,
+and the number of additional power iterations <code>\(q\in\mathbb{N}_{0}\)</code>,
+this procedure computes an <code>\(m\times\left(k+p\right)\)</code>
+SVD <code>\(\mathbf{A\approx U}\boldsymbol{\Sigma}\mathbf{V}^{\top}\)</code>:</p>
<ol>
<li>
- <p>Create seed for random <code class="highlighter-rouge">\(n\times\left(k+p\right)\)</code>
- matrix <code class="highlighter-rouge">\(\boldsymbol{\Omega}\)</code>. The seed defines matrix <code class="highlighter-rouge">\(\mathbf{\Omega}\)</code>
+ <p>Create seed for random <code>\(n\times\left(k+p\right)\)</code>
+ matrix <code>\(\boldsymbol{\Omega}\)</code>. The seed defines matrix <code>\(\mathbf{\Omega}\)</code>
using Gaussian unit vectors per one of suggestions in <a href="http://arxiv.org/abs/0909.4061">Halko, Martinsson, Tropp</a>.</p>
</li>
<li>
- <p><code class="highlighter-rouge">\(\mathbf{Y=A\boldsymbol{\Omega}},\,\mathbf{Y}\in\mathbb{R}^{m\times\left(k+p\right)}\)</code></p>
+ <p><code>\(\mathbf{Y=A\boldsymbol{\Omega}},\,\mathbf{Y}\in\mathbb{R}^{m\times\left(k+p\right)}\)</code></p>
</li>
<li>
- <p>Column-orthonormalize <code class="highlighter-rouge">\(\mathbf{Y}\rightarrow\mathbf{Q}\)</code>
- by computing thin decomposition <code class="highlighter-rouge">\(\mathbf{Y}=\mathbf{Q}\mathbf{R}\)</code>.
- Also, <code class="highlighter-rouge">\(\mathbf{Q}\in\mathbb{R}^{m\times\left(k+p\right)},\,\mathbf{R}\in\mathbb{R}^{\left(k+p\right)\times\left(k+p\right)}\)</code>.
- I denote this as <code class="highlighter-rouge">\(\mathbf{Q}=\mbox{qr}\left(\mathbf{Y}\right).\mathbf{Q}\)</code></p>
+ <p>Column-orthonormalize <code>\(\mathbf{Y}\rightarrow\mathbf{Q}\)</code>
+ by computing thin decomposition <code>\(\mathbf{Y}=\mathbf{Q}\mathbf{R}\)</code>.
+ Also, <code>\(\mathbf{Q}\in\mathbb{R}^{m\times\left(k+p\right)},\,\mathbf{R}\in\mathbb{R}^{\left(k+p\right)\times\left(k+p\right)}\)</code>.
+ I denote this as <code>\(\mathbf{Q}=\mbox{qr}\left(\mathbf{Y}\right).\mathbf{Q}\)</code></p>
</li>
<li>
- <p><code class="highlighter-rouge">\(\mathbf{B}_{0}=\mathbf{Q}^{\top}\mathbf{A}:\,\,\mathbf{B}\in\mathbb{R}^{\left(k+p\right)\times n}\)</code>.</p>
+ <p><code>\(\mathbf{B}_{0}=\mathbf{Q}^{\top}\mathbf{A}:\,\,\mathbf{B}\in\mathbb{R}^{\left(k+p\right)\times n}\)</code>.</p>
</li>
<li>
- <p>If <code class="highlighter-rouge">\(q>0\)</code>
- repeat: for <code class="highlighter-rouge">\(i=1..q\)</code>:
- <code class="highlighter-rouge">\(\mathbf{B}_{i}^{\top}=\mathbf{A}^{\top}\mbox{qr}\left(\mathbf{A}\mathbf{B}_{i-1}^{\top}\right).\mathbf{Q}\)</code>
+ <p>If <code>\(q>0\)</code>
+ repeat: for <code>\(i=1..q\)</code>:
+ <code>\(\mathbf{B}_{i}^{\top}=\mathbf{A}^{\top}\mbox{qr}\left(\mathbf{A}\mathbf{B}_{i-1}^{\top}\right).\mathbf{Q}\)</code>
(power iterations step).</p>
</li>
<li>
- <p>Compute Eigensolution of a small Hermitian <code class="highlighter-rouge">\(\mathbf{B}_{q}\mathbf{B}_{q}^{\top}=\mathbf{\hat{U}}\boldsymbol{\Lambda}\mathbf{\hat{U}}^{\top}\)</code>,
- <code class="highlighter-rouge">\(\mathbf{B}_{q}\mathbf{B}_{q}^{\top}\in\mathbb{R}^{\left(k+p\right)\times\left(k+p\right)}\)</code>.</p>
+ <p>Compute Eigensolution of a small Hermitian <code>\(\mathbf{B}_{q}\mathbf{B}_{q}^{\top}=\mathbf{\hat{U}}\boldsymbol{\Lambda}\mathbf{\hat{U}}^{\top}\)</code>,
+ <code>\(\mathbf{B}_{q}\mathbf{B}_{q}^{\top}\in\mathbb{R}^{\left(k+p\right)\times\left(k+p\right)}\)</code>.</p>
</li>
<li>
- <p>Singular values <code class="highlighter-rouge">\(\mathbf{\boldsymbol{\Sigma}}=\boldsymbol{\Lambda}^{0.5}\)</code>,
- or, in other words, <code class="highlighter-rouge">\(s_{i}=\sqrt{\sigma_{i}}\)</code>.</p>
+ <p>Singular values <code>\(\mathbf{\boldsymbol{\Sigma}}=\boldsymbol{\Lambda}^{0.5}\)</code>,
+ or, in other words, <code>\(s_{i}=\sqrt{\sigma_{i}}\)</code>.</p>
</li>
<li>
- <p>If needed, compute <code class="highlighter-rouge">\(\mathbf{U}=\mathbf{Q}\hat{\mathbf{U}}\)</code>.</p>
+ <p>If needed, compute <code>\(\mathbf{U}=\mathbf{Q}\hat{\mathbf{U}}\)</code>.</p>
</li>
<li>
- <p>If needed, compute <code class="highlighter-rouge">\(\mathbf{V}=\mathbf{B}_{q}^{\top}\hat{\mathbf{U}}\boldsymbol{\Sigma}^{-1}\)</code>.
-Another way is <code class="highlighter-rouge">\(\mathbf{V}=\mathbf{A}^{\top}\mathbf{U}\boldsymbol{\Sigma}^{-1}\)</code>.</p>
+ <p>If needed, compute <code>\(\mathbf{V}=\mathbf{B}_{q}^{\top}\hat{\mathbf{U}}\boldsymbol{\Sigma}^{-1}\)</code>.
+Another way is <code>\(\mathbf{V}=\mathbf{A}^{\top}\mathbf{U}\boldsymbol{\Sigma}^{-1}\)</code>.</p>
</li>
</ol>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/environment/classify-a-doc-from-the-shell.html
----------------------------------------------------------------------
diff --git a/users/environment/classify-a-doc-from-the-shell.html b/users/environment/classify-a-doc-from-the-shell.html
index 6427c25..85c9728 100644
--- a/users/environment/classify-a-doc-from-the-shell.html
+++ b/users/environment/classify-a-doc-from-the-shell.html
@@ -274,72 +274,72 @@
<p>#Building a text classifier in Mahout’s Spark Shell</p>
-<p>This tutorial will take you through the steps used to train a Multinomial Naive Bayes model and create a text classifier based on that model using the <code class="highlighter-rouge">mahout spark-shell</code>.</p>
+<p>This tutorial will take you through the steps used to train a Multinomial Naive Bayes model and create a text classifier based on that model using the <code>mahout spark-shell</code>.</p>
<h2 id="prerequisites">Prerequisites</h2>
-<p>This tutorial assumes that you have your Spark environment variables set for the <code class="highlighter-rouge">mahout spark-shell</code> see: <a href="http://mahout.apache.org/users/sparkbindings/play-with-shell.html">Playing with Mahout’s Shell</a>. As well we assume that Mahout is running in cluster mode (i.e. with the <code class="highlighter-rouge">MAHOUT_LOCAL</code> environment variable <strong>unset</strong>) as we’ll be reading and writing to HDFS.</p>
+<p>This tutorial assumes that you have your Spark environment variables set for the <code>mahout spark-shell</code> see: <a href="http://mahout.apache.org/users/sparkbindings/play-with-shell.html">Playing with Mahout’s Shell</a>. As well we assume that Mahout is running in cluster mode (i.e. with the <code>MAHOUT_LOCAL</code> environment variable <strong>unset</strong>) as we’ll be reading and writing to HDFS.</p>
<h2 id="downloading-and-vectorizing-the-wikipedia-dataset">Downloading and Vectorizing the Wikipedia dataset</h2>
-<p><em>As of Mahout v. 0.10.0, we are still reliant on the MapReduce versions of <code class="highlighter-rouge">mahout seqwiki</code> and <code class="highlighter-rouge">mahout seq2sparse</code> to extract and vectorize our text. A</em> <a href="https://issues.apache.org/jira/browse/MAHOUT-1663"><em>Spark implementation of seq2sparse</em></a> <em>is in the works for Mahout v. 0.11.</em> However, to download the Wikipedia dataset, extract the bodies of the documentation, label each document and vectorize the text into TF-IDF vectors, we can simpmly run the <a href="https://github.com/apache/mahout/blob/master/examples/bin/classify-wikipedia.sh">wikipedia-classifier.sh</a> example.</p>
+<p><em>As of Mahout v. 0.10.0, we are still reliant on the MapReduce versions of <code>mahout seqwiki</code> and <code>mahout seq2sparse</code> to extract and vectorize our text. A</em> <a href="https://issues.apache.org/jira/browse/MAHOUT-1663"><em>Spark implementation of seq2sparse</em></a> <em>is in the works for Mahout v. 0.11.</em> However, to download the Wikipedia dataset, extract the bodies of the documentation, label each document and vectorize the text into TF-IDF vectors, we can simpmly run the <a href="https://github.com/apache/mahout/blob/master/examples/bin/classify-wikipedia.sh">wikipedia-classifier.sh</a> example.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Please select a number to choose the corresponding task to run
+<pre><code>Please select a number to choose the corresponding task to run
1. CBayes (may require increased heap space on yarn)
2. BinaryCBayes
3. clean -- cleans up the work area in /tmp/mahout-work-wiki
Enter your choice :
-</code></pre></div></div>
+</code></pre>
-<p>Enter (2). This will download a large recent XML dump of the Wikipedia database, into a <code class="highlighter-rouge">/tmp/mahout-work-wiki</code> directory, unzip it and place it into HDFS. It will run a <a href="http://mahout.apache.org/users/classification/wikipedia-classifier-example.html">MapReduce job to parse the wikipedia set</a>, extracting and labeling only pages with category tags for [United States] and [United Kingdom] (~11600 documents). It will then run <code class="highlighter-rouge">mahout seq2sparse</code> to convert the documents into TF-IDF vectors. The script will also a build and test a <a href="http://mahout.apache.org/users/classification/bayesian.html">Naive Bayes model using MapReduce</a>. When it is completed, you should see a confusion matrix on your screen. For this tutorial, we will ignore the MapReduce model, and build a new model using Spark based on the vectorized text output by <code class="highlighter-rouge">seq2sparse</code>.</p>
+<p>Enter (2). This will download a large recent XML dump of the Wikipedia database, into a <code>/tmp/mahout-work-wiki</code> directory, unzip it and place it into HDFS. It will run a <a href="http://mahout.apache.org/users/classification/wikipedia-classifier-example.html">MapReduce job to parse the wikipedia set</a>, extracting and labeling only pages with category tags for [United States] and [United Kingdom] (~11600 documents). It will then run <code>mahout seq2sparse</code> to convert the documents into TF-IDF vectors. The script will also a build and test a <a href="http://mahout.apache.org/users/classification/bayesian.html">Naive Bayes model using MapReduce</a>. When it is completed, you should see a confusion matrix on your screen. For this tutorial, we will ignore the MapReduce model, and build a new model using Spark based on the vectorized text output by <code>seq2sparse</code>.</p>
<h2 id="getting-started">Getting Started</h2>
-<p>Launch the <code class="highlighter-rouge">mahout spark-shell</code>. There is an example script: <code class="highlighter-rouge">spark-document-classifier.mscala</code> (.mscala denotes a Mahout-Scala script which can be run similarly to an R script). We will be walking through this script for this tutorial but if you wanted to simply run the script, you could just issue the command:</p>
+<p>Launch the <code>mahout spark-shell</code>. There is an example script: <code>spark-document-classifier.mscala</code> (.mscala denotes a Mahout-Scala script which can be run similarly to an R script). We will be walking through this script for this tutorial but if you wanted to simply run the script, you could just issue the command:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mahout> :load /path/to/mahout/examples/bin/spark-document-classifier.mscala
-</code></pre></div></div>
+<pre><code>mahout> :load /path/to/mahout/examples/bin/spark-document-classifier.mscala
+</code></pre>
-<p>For now, lets take the script apart piece by piece. You can cut and paste the following code blocks into the <code class="highlighter-rouge">mahout spark-shell</code>.</p>
+<p>For now, lets take the script apart piece by piece. You can cut and paste the following code blocks into the <code>mahout spark-shell</code>.</p>
<h2 id="imports">Imports</h2>
<p>Our Mahout Naive Bayes imports:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>import org.apache.mahout.classifier.naivebayes._
+<pre><code>import org.apache.mahout.classifier.naivebayes._
import org.apache.mahout.classifier.stats._
import org.apache.mahout.nlp.tfidf._
-</code></pre></div></div>
+</code></pre>
<p>Hadoop imports needed to read our dictionary:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>import org.apache.hadoop.io.Text
+<pre><code>import org.apache.hadoop.io.Text
import org.apache.hadoop.io.IntWritable
import org.apache.hadoop.io.LongWritable
-</code></pre></div></div>
+</code></pre>
<h2 id="read-in-our-full-set-from-hdfs-as-vectorized-by-seq2sparse-in-classify-wikipediash">Read in our full set from HDFS as vectorized by seq2sparse in classify-wikipedia.sh</h2>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val pathToData = "/tmp/mahout-work-wiki/"
+<pre><code>val pathToData = "/tmp/mahout-work-wiki/"
val fullData = drmDfsRead(pathToData + "wikipediaVecs/tfidf-vectors")
-</code></pre></div></div>
+</code></pre>
<h2 id="extract-the-category-of-each-observation-and-aggregate-those-observations-by-category">Extract the category of each observation and aggregate those observations by category</h2>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val (labelIndex, aggregatedObservations) = SparkNaiveBayes.extractLabelsAndAggregateObservations(
+<pre><code>val (labelIndex, aggregatedObservations) = SparkNaiveBayes.extractLabelsAndAggregateObservations(
fullData)
-</code></pre></div></div>
+</code></pre>
<h2 id="build-a-muitinomial-naive-bayes-model-and-self-test-on-the-training-set">Build a Muitinomial Naive Bayes model and self test on the training set</h2>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val model = SparkNaiveBayes.train(aggregatedObservations, labelIndex, false)
+<pre><code>val model = SparkNaiveBayes.train(aggregatedObservations, labelIndex, false)
val resAnalyzer = SparkNaiveBayes.test(model, fullData, false)
println(resAnalyzer)
-</code></pre></div></div>
+</code></pre>
-<p>printing the <code class="highlighter-rouge">ResultAnalyzer</code> will display the confusion matrix.</p>
+<p>printing the <code>ResultAnalyzer</code> will display the confusion matrix.</p>
<h2 id="read-in-the-dictionary-and-document-frequency-count-from-hdfs">Read in the dictionary and document frequency count from HDFS</h2>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val dictionary = sdc.sequenceFile(pathToData + "wikipediaVecs/dictionary.file-0",
+<pre><code>val dictionary = sdc.sequenceFile(pathToData + "wikipediaVecs/dictionary.file-0",
classOf[Text],
classOf[IntWritable])
val documentFrequencyCount = sdc.sequenceFile(pathToData + "wikipediaVecs/df-count",
@@ -359,13 +359,13 @@ val documentFrequencyCountRDD = documentFrequencyCount.map {
val dictionaryMap = dictionaryRDD.collect.map(x => x._1.toString -> x._2.toInt).toMap
val dfCountMap = documentFrequencyCountRDD.collect.map(x => x._1.toInt -> x._2.toLong).toMap
-</code></pre></div></div>
+</code></pre>
<h2 id="define-a-function-to-tokenize-and-vectorize-new-text-using-our-current-dictionary">Define a function to tokenize and vectorize new text using our current dictionary</h2>
-<p>For this simple example, our function <code class="highlighter-rouge">vectorizeDocument(...)</code> will tokenize a new document into unigrams using native Java String methods and vectorize using our dictionary and document frequencies. You could also use a <a href="https://lucene.apache.org/core/">Lucene</a> analyzer for bigrams, trigrams, etc., and integrate Apache <a href="https://tika.apache.org/">Tika</a> to extract text from different document types (PDF, PPT, XLS, etc.). Here, however we will keep it simple, stripping and tokenizing our text using regexs and native String methods.</p>
+<p>For this simple example, our function <code>vectorizeDocument(...)</code> will tokenize a new document into unigrams using native Java String methods and vectorize using our dictionary and document frequencies. You could also use a <a href="https://lucene.apache.org/core/">Lucene</a> analyzer for bigrams, trigrams, etc., and integrate Apache <a href="https://tika.apache.org/">Tika</a> to extract text from different document types (PDF, PPT, XLS, etc.). Here, however we will keep it simple, stripping and tokenizing our text using regexs and native String methods.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def vectorizeDocument(document: String,
+<pre><code>def vectorizeDocument(document: String,
dictionaryMap: Map[String,Int],
dfMap: Map[Int,Long]): Vector = {
val wordCounts = document.replaceAll("[^\\p{L}\\p{Nd}]+", " ")
@@ -392,11 +392,11 @@ val dfCountMap = documentFrequencyCountRDD.collect.map(x => x._1.toInt ->
}
vec
}
-</code></pre></div></div>
+</code></pre>
<h2 id="setup-our-classifier">Setup our classifier</h2>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val labelMap = model.labelIndex
+<pre><code>val labelMap = model.labelIndex
val numLabels = model.numLabels
val reverseLabelMap = labelMap.map(x => x._2 -> x._1)
@@ -405,13 +405,13 @@ val classifier = model.isComplementary match {
case true => new ComplementaryNBClassifier(model)
case _ => new StandardNBClassifier(model)
}
-</code></pre></div></div>
+</code></pre>
<h2 id="define-an-argmax-function">Define an argmax function</h2>
<p>The label with the highest score wins the classification for a given document.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def argmax(v: Vector): (Int, Double) = {
+<pre><code>def argmax(v: Vector): (Int, Double) = {
var bestIdx: Int = Integer.MIN_VALUE
var bestScore: Double = Integer.MIN_VALUE.asInstanceOf[Int].toDouble
for(i <- 0 until v.size) {
@@ -422,20 +422,20 @@ val classifier = model.isComplementary match {
}
(bestIdx, bestScore)
}
-</code></pre></div></div>
+</code></pre>
<h2 id="define-our-tf-idf-vector-classifier">Define our TF(-IDF) vector classifier</h2>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def classifyDocument(clvec: Vector) : String = {
+<pre><code>def classifyDocument(clvec: Vector) : String = {
val cvec = classifier.classifyFull(clvec)
val (bestIdx, bestScore) = argmax(cvec)
reverseLabelMap(bestIdx)
}
-</code></pre></div></div>
+</code></pre>
<h2 id="two-sample-news-articles-united-states-football-and-united-kingdom-football">Two sample news articles: United States Football and United Kingdom Football</h2>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>// A random United States football article
+<pre><code>// A random United States football article
// http://www.reuters.com/article/2015/01/28/us-nfl-superbowl-security-idUSKBN0L12JR20150128
val UStextToClassify = new String("(Reuters) - Super Bowl security officials acknowledge" +
" the NFL championship game represents a high profile target on a world stage but are" +
@@ -500,11 +500,11 @@ val UKtextToClassify = new String("(Reuters) - Manchester United have signed a s
" Premier League last season and missing out on a place in the lucrative Champions League." +
" ($1 = 0.8910 Swiss francs) (Writing by Neil Maidment, additional reporting by Jemima" +
" Kelly; editing by Keith Weir)")
-</code></pre></div></div>
+</code></pre>
<h2 id="vectorize-and-classify-our-documents">Vectorize and classify our documents</h2>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val usVec = vectorizeDocument(UStextToClassify, dictionaryMap, dfCountMap)
+<pre><code>val usVec = vectorizeDocument(UStextToClassify, dictionaryMap, dfCountMap)
val ukVec = vectorizeDocument(UKtextToClassify, dictionaryMap, dfCountMap)
println("Classifying the news article about superbowl security (united states)")
@@ -512,33 +512,33 @@ classifyDocument(usVec)
println("Classifying the news article about Manchester United (united kingdom)")
classifyDocument(ukVec)
-</code></pre></div></div>
+</code></pre>
<h2 id="tie-everything-together-in-a-new-method-to-classify-text">Tie everything together in a new method to classify text</h2>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def classifyText(txt: String): String = {
+<pre><code>def classifyText(txt: String): String = {
val v = vectorizeDocument(txt, dictionaryMap, dfCountMap)
classifyDocument(v)
}
-</code></pre></div></div>
+</code></pre>
<h2 id="now-we-can-simply-call-our-classifytext-method-on-any-string">Now we can simply call our classifyText(…) method on any String</h2>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>classifyText("Hello world from Queens")
+<pre><code>classifyText("Hello world from Queens")
classifyText("Hello world from London")
-</code></pre></div></div>
+</code></pre>
<h2 id="model-persistance">Model persistance</h2>
<p>You can save the model to HDFS:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>model.dfsWrite("/path/to/model")
-</code></pre></div></div>
+<pre><code>model.dfsWrite("/path/to/model")
+</code></pre>
<p>And retrieve it with:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val model = NBModel.dfsRead("/path/to/model")
-</code></pre></div></div>
+<pre><code>val model = NBModel.dfsRead("/path/to/model")
+</code></pre>
<p>The trained model can now be embedded in an external application.</p>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/environment/h2o-internals.html
----------------------------------------------------------------------
diff --git a/users/environment/h2o-internals.html b/users/environment/h2o-internals.html
index 9abe6c2..a0e6ed0 100644
--- a/users/environment/h2o-internals.html
+++ b/users/environment/h2o-internals.html
@@ -274,7 +274,7 @@
<h1 id="introduction">Introduction</h1>
-<p>This document provides an overview of how the Mahout Samsara environment is implemented over the H2O backend engine. The document is aimed at Mahout developers, to give a high level description of the design so that one can explore the code inside <code class="highlighter-rouge">h2o/</code> with some context.</p>
+<p>This document provides an overview of how the Mahout Samsara environment is implemented over the H2O backend engine. The document is aimed at Mahout developers, to give a high level description of the design so that one can explore the code inside <code>h2o/</code> with some context.</p>
<h2 id="h2o-overview">H2O Overview</h2>
@@ -290,13 +290,13 @@
<h2 id="h2o-environment-engine">H2O Environment Engine</h2>
-<p>The H2O backend implements the abstract DRM as an H2O Frame. Each logical column in the DRM is an H2O Vector. All elements of a logical DRM row are guaranteed to be homed on the same server. A set of rows stored on a server are presented as a read-only virtual in-core Matrix (i.e BlockMatrix) for the closure method in the <code class="highlighter-rouge">mapBlock(...)</code> API.</p>
+<p>The H2O backend implements the abstract DRM as an H2O Frame. Each logical column in the DRM is an H2O Vector. All elements of a logical DRM row are guaranteed to be homed on the same server. A set of rows stored on a server are presented as a read-only virtual in-core Matrix (i.e BlockMatrix) for the closure method in the <code>mapBlock(...)</code> API.</p>
-<p>H2O provides a flexible execution framework called <code class="highlighter-rouge">MRTask</code>. The <code class="highlighter-rouge">MRTask</code> framework typically executes over a Frame (or even a Vector), supports various types of map() methods, can optionally modify the Frame or Vector (though this never happens in the Mahout integration), and optionally create a new Vector or set of Vectors (to combine them into a new Frame, and consequently a new DRM).</p>
+<p>H2O provides a flexible execution framework called <code>MRTask</code>. The <code>MRTask</code> framework typically executes over a Frame (or even a Vector), supports various types of map() methods, can optionally modify the Frame or Vector (though this never happens in the Mahout integration), and optionally create a new Vector or set of Vectors (to combine them into a new Frame, and consequently a new DRM).</p>
<h2 id="source-layout">Source Layout</h2>
-<p>Within mahout.git, the top level directory, <code class="highlighter-rouge">h2o/</code> holds all the source code related to the H2O backend engine. Part of the code (that interfaces with the rest of the Mahout componenets) is in Scala, and part of the code (that interfaces with h2o-core and implements algebraic operators) is in Java. Here is a brief overview of what functionality can be found where within <code class="highlighter-rouge">h2o/</code>.</p>
+<p>Within mahout.git, the top level directory, <code>h2o/</code> holds all the source code related to the H2O backend engine. Part of the code (that interfaces with the rest of the Mahout componenets) is in Scala, and part of the code (that interfaces with h2o-core and implements algebraic operators) is in Java. Here is a brief overview of what functionality can be found where within <code>h2o/</code>.</p>
<p>h2o/ - top level directory containing all H2O related code</p>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/environment/how-to-build-an-app.html
----------------------------------------------------------------------
diff --git a/users/environment/how-to-build-an-app.html b/users/environment/how-to-build-an-app.html
index 76d3b91..8195509 100644
--- a/users/environment/how-to-build-an-app.html
+++ b/users/environment/how-to-build-an-app.html
@@ -293,38 +293,38 @@ In order to build and run the CooccurrenceDriver you need to install the follow
<p>Spark requires a set of jars on the classpath for the client side part of an app and another set of jars must be passed to the Spark Context for running distributed code. The example should discover all the neccessary classes automatically.</p>
<p>##Application
-Using Mahout as a library in an application will require a little Scala code. Scala has an App trait so we’ll create an object, which inherits from <code class="highlighter-rouge">App</code></p>
+Using Mahout as a library in an application will require a little Scala code. Scala has an App trait so we’ll create an object, which inherits from <code>App</code></p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>object CooccurrenceDriver extends App {
+<pre><code>object CooccurrenceDriver extends App {
}
-</code></pre></div></div>
+</code></pre>
-<p>This will look a little different than Java since <code class="highlighter-rouge">App</code> does delayed initialization, which causes the body to be executed when the App is launched, just as in Java you would create a main method.</p>
+<p>This will look a little different than Java since <code>App</code> does delayed initialization, which causes the body to be executed when the App is launched, just as in Java you would create a main method.</p>
<p>Before we can execute something on Spark we’ll need to create a context. We could use raw Spark calls here but default values are setup for a Mahout context by using the Mahout helper function.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>implicit val mc = mahoutSparkContext(masterUrl = "local",
+<pre><code>implicit val mc = mahoutSparkContext(masterUrl = "local",
appName = "CooccurrenceDriver")
-</code></pre></div></div>
+</code></pre>
<p>We need to read in three files containing different interaction types. The files will each be read into a Mahout IndexedDataset. This allows us to preserve application-specific user and item IDs throughout the calculations.</p>
<p>For example, here is data/purchase.csv:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>u1,iphone
+<pre><code>u1,iphone
u1,ipad
u2,nexus
u2,galaxy
u3,surface
u4,iphone
u4,galaxy
-</code></pre></div></div>
+</code></pre>
<p>Mahout has a helper function that reads the text delimited files SparkEngine.indexedDatasetDFSReadElements. The function reads single element tuples (user-id,item-id) in a distributed way to create the IndexedDataset. Distributed Row Matrices (DRM) and Vectors are important data types supplied by Mahout and IndexedDataset is like a very lightweight Dataframe in R, it wraps a DRM with HashBiMaps for row and column IDs.</p>
<p>One important thing to note about this example is that we read in all datasets before we adjust the number of rows in them to match the total number of users in the data. This is so the math works out <a href="http://mahout.apache.org/users/algorithms/intro-cooccurrence-spark.html">(A’A, A’B, A’C)</a> even if some users took one action but not another there must be the same number of rows in all matrices.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/**
+<pre><code>/**
* Read files of element tuples and create IndexedDatasets one per action. These
* share a userID BiMap but have their own itemID BiMaps
*/
@@ -358,25 +358,25 @@ def readActions(actionInput: Array[(String, String)]): Array[(String, IndexedDat
}
resizedNameActionPairs // return the array of Tuples
}
-</code></pre></div></div>
+</code></pre>
<p>Now that we have the data read in we can perform the cooccurrence calculation.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>// actions.map creates an array of just the IndeedDatasets
+<pre><code>// actions.map creates an array of just the IndeedDatasets
val indicatorMatrices = SimilarityAnalysis.cooccurrencesIDSs(
actions.map(a => a._2))
-</code></pre></div></div>
+</code></pre>
<p>All we need to do now is write the indicators.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>// zip a pair of arrays into an array of pairs, reattaching the action names
+<pre><code>// zip a pair of arrays into an array of pairs, reattaching the action names
val indicatorDescriptions = actions.map(a => a._1).zip(indicatorMatrices)
writeIndicators(indicatorDescriptions)
-</code></pre></div></div>
+</code></pre>
-<p>The <code class="highlighter-rouge">writeIndicators</code> method uses the default write function <code class="highlighter-rouge">dfsWrite</code>.</p>
+<p>The <code>writeIndicators</code> method uses the default write function <code>dfsWrite</code>.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/**
+<pre><code>/**
* Write indicatorMatrices to the output dir in the default format
* for indexing by a search engine.
*/
@@ -391,11 +391,11 @@ def writeIndicators( indicators: Array[(String, IndexedDataset)]) = {
IndexedDatasetWriteBooleanSchema)
}
}
-</code></pre></div></div>
+</code></pre>
<p>See the Github project for the full source. Now we create a build.sbt to build the example.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>name := "cooccurrence-driver"
+<pre><code>name := "cooccurrence-driver"
organization := "com.finderbots"
@@ -424,25 +424,25 @@ packSettings
packMain := Map(
"cooc" -> "CooccurrenceDriver")
-</code></pre></div></div>
+</code></pre>
<p>##Build
Building the examples from project’s root folder:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ sbt pack
-</code></pre></div></div>
+<pre><code>$ sbt pack
+</code></pre>
<p>This will automatically set up some launcher scripts for the driver. To run execute</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ target/pack/bin/cooc
-</code></pre></div></div>
+<pre><code>$ target/pack/bin/cooc
+</code></pre>
<p>The driver will execute in Spark standalone mode and put the data in /path/to/3-input-cooc/data/indicators/<em>indicator-type</em></p>
<p>##Using a Debugger
To build and run this example in a debugger like IntelliJ IDEA. Install from the IntelliJ site and add the Scala plugin.</p>
-<p>Open IDEA and go to the menu File->New->Project from existing sources->SBT->/path/to/3-input-cooc. This will create an IDEA project from <code class="highlighter-rouge">build.sbt</code> in the root directory.</p>
+<p>Open IDEA and go to the menu File->New->Project from existing sources->SBT->/path/to/3-input-cooc. This will create an IDEA project from <code>build.sbt</code> in the root directory.</p>
<p>At this point you may create a “Debug Configuration” to run. In the menu choose Run->Edit Configurations. Under “Default” choose “Application”. In the dialog hit the elipsis button “…” to the right of “Environment Variables” and fill in your versions of JAVA_HOME, SPARK_HOME, and MAHOUT_HOME. In configuration editor under “Use classpath from” choose root-3-input-cooc module.</p>
@@ -463,24 +463,24 @@ To build and run this example in a debugger like IntelliJ IDEA. Install from the
<ul>
<li>You won’t need the context, since it is created when the shell is launched, comment that line out.</li>
<li>Replace the logger.info lines with println</li>
- <li>Remove the package info since it’s not needed, this will produce the file in <code class="highlighter-rouge">path/to/3-input-cooc/bin/CooccurrenceDriver.mscala</code>.</li>
+ <li>Remove the package info since it’s not needed, this will produce the file in <code>path/to/3-input-cooc/bin/CooccurrenceDriver.mscala</code>.</li>
</ul>
-<p>Note the extension <code class="highlighter-rouge">.mscala</code> to indicate we are using Mahout’s scala extensions for math, otherwise known as <a href="http://mahout.apache.org/users/environment/out-of-core-reference.html">Mahout-Samsara</a></p>
+<p>Note the extension <code>.mscala</code> to indicate we are using Mahout’s scala extensions for math, otherwise known as <a href="http://mahout.apache.org/users/environment/out-of-core-reference.html">Mahout-Samsara</a></p>
<p>To run the code make sure the output does not exist already</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ rm -r /path/to/3-input-cooc/data/indicators
-</code></pre></div></div>
+<pre><code>$ rm -r /path/to/3-input-cooc/data/indicators
+</code></pre>
<p>Launch the Mahout + Spark shell:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ mahout spark-shell
-</code></pre></div></div>
+<pre><code>$ mahout spark-shell
+</code></pre>
<p>You’ll see the Mahout splash:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>MAHOUT_LOCAL is set, so we don't add HADOOP_CONF_DIR to classpath.
+<pre><code>MAHOUT_LOCAL is set, so we don't add HADOOP_CONF_DIR to classpath.
_ _
_ __ ___ __ _| |__ ___ _ _| |_
@@ -496,11 +496,11 @@ Type :help for more information.
Created spark context..
Mahout distributed context is available as "implicit val sdc".
mahout>
-</code></pre></div></div>
+</code></pre>
<p>To load the driver type:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mahout> :load /path/to/3-input-cooc/bin/CooccurrenceDriver.mscala
+<pre><code>mahout> :load /path/to/3-input-cooc/bin/CooccurrenceDriver.mscala
Loading ./bin/CooccurrenceDriver.mscala...
import com.google.common.collect.{HashBiMap, BiMap}
import org.apache.log4j.Logger
@@ -510,16 +510,16 @@ import org.apache.mahout.sparkbindings._
import scala.collection.immutable.HashMap
defined module CooccurrenceDriver
mahout>
-</code></pre></div></div>
+</code></pre>
<p>To run the driver type:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mahout> CooccurrenceDriver.main(args = Array(""))
-</code></pre></div></div>
+<pre><code>mahout> CooccurrenceDriver.main(args = Array(""))
+</code></pre>
<p>You’ll get some stats printed:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Total number of users for all actions = 5
+<pre><code>Total number of users for all actions = 5
purchase indicator matrix:
Number of rows for matrix = 4
Number of columns for matrix = 5
@@ -532,9 +532,9 @@ category indicator matrix:
Number of rows for matrix = 5
Number of columns for matrix = 7
Number of rows after resize = 5
-</code></pre></div></div>
+</code></pre>
-<p>If you look in <code class="highlighter-rouge">path/to/3-input-cooc/data/indicators</code> you should find folders containing the indicator matrices.</p>
+<p>If you look in <code>path/to/3-input-cooc/data/indicators</code> you should find folders containing the indicator matrices.</p>
</div>
</div>
http://git-wip-us.apache.org/repos/asf/mahout/blob/d9686c8b/users/environment/in-core-reference.html
----------------------------------------------------------------------
diff --git a/users/environment/in-core-reference.html b/users/environment/in-core-reference.html
index 313fc63..0c7a62c 100644
--- a/users/environment/in-core-reference.html
+++ b/users/environment/in-core-reference.html
@@ -278,218 +278,218 @@
<p>The following imports are used to enable Mahout-Samsara’s Scala DSL bindings for in-core Linear Algebra:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>import org.apache.mahout.math._
+<pre><code>import org.apache.mahout.math._
import scalabindings._
import RLikeOps._
-</code></pre></div></div>
+</code></pre>
<h4 id="inline-initalization">Inline initalization</h4>
<p>Dense vectors:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val densVec1: Vector = (1.0, 1.1, 1.2)
+<pre><code>val densVec1: Vector = (1.0, 1.1, 1.2)
val denseVec2 = dvec(1, 0, 1,1 ,1,2)
-</code></pre></div></div>
+</code></pre>
<p>Sparse vectors:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val sparseVec1: Vector = (5 -> 1.0) :: (10 -> 2.0) :: Nil
+<pre><code>val sparseVec1: Vector = (5 -> 1.0) :: (10 -> 2.0) :: Nil
val sparseVec1 = svec((5 -> 1.0) :: (10 -> 2.0) :: Nil)
// to create a vector with specific cardinality
val sparseVec1 = svec((5 -> 1.0) :: (10 -> 2.0) :: Nil, cardinality = 20)
-</code></pre></div></div>
+</code></pre>
<p>Inline matrix initialization, either sparse or dense, is always done row wise.</p>
<p>Dense matrices:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val A = dense((1, 2, 3), (3, 4, 5))
-</code></pre></div></div>
+<pre><code>val A = dense((1, 2, 3), (3, 4, 5))
+</code></pre>
<p>Sparse matrices:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val A = sparse(
+<pre><code>val A = sparse(
(1, 3) :: Nil,
(0, 2) :: (1, 2.5) :: Nil
)
-</code></pre></div></div>
+</code></pre>
<p>Diagonal matrix with constant diagonal elements:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>diag(3.5, 10)
-</code></pre></div></div>
+<pre><code>diag(3.5, 10)
+</code></pre>
<p>Diagonal matrix with main diagonal backed by a vector:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>diagv((1, 2, 3, 4, 5))
-</code></pre></div></div>
+<pre><code>diagv((1, 2, 3, 4, 5))
+</code></pre>
<p>Identity matrix:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>eye(10)
-</code></pre></div></div>
+<pre><code>eye(10)
+</code></pre>
<p>####Slicing and Assigning</p>
<p>Getting a vector element:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val d = vec(5)
-</code></pre></div></div>
+<pre><code>val d = vec(5)
+</code></pre>
<p>Setting a vector element:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>vec(5) = 3.0
-</code></pre></div></div>
+<pre><code>vec(5) = 3.0
+</code></pre>
<p>Getting a matrix element:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val d = m(3,5)
-</code></pre></div></div>
+<pre><code>val d = m(3,5)
+</code></pre>
<p>Setting a matrix element:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>M(3,5) = 3.0
-</code></pre></div></div>
+<pre><code>M(3,5) = 3.0
+</code></pre>
<p>Getting a matrix row or column:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val rowVec = M(3, ::)
+<pre><code>val rowVec = M(3, ::)
val colVec = M(::, 3)
-</code></pre></div></div>
+</code></pre>
<p>Setting a matrix row or column via vector assignment:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>M(3, ::) := (1, 2, 3)
+<pre><code>M(3, ::) := (1, 2, 3)
M(::, 3) := (1, 2, 3)
-</code></pre></div></div>
+</code></pre>
<p>Setting a subslices of a matrix row or column:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>a(0, 0 to 1) = (3, 5)
-</code></pre></div></div>
+<pre><code>a(0, 0 to 1) = (3, 5)
+</code></pre>
<p>Setting a subslices of a matrix row or column via vector assignment:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>a(0, 0 to 1) := (3, 5)
-</code></pre></div></div>
+<pre><code>a(0, 0 to 1) := (3, 5)
+</code></pre>
<p>Getting a matrix as from matrix contiguous block:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val B = A(2 to 3, 3 to 4)
-</code></pre></div></div>
+<pre><code>val B = A(2 to 3, 3 to 4)
+</code></pre>
<p>Assigning a contiguous block to a matrix:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>A(0 to 1, 1 to 2) = dense((3, 2), (3 ,3))
-</code></pre></div></div>
+<pre><code>A(0 to 1, 1 to 2) = dense((3, 2), (3 ,3))
+</code></pre>
<p>Assigning a contiguous block to a matrix using the matrix assignment operator:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>A(o to 1, 1 to 2) := dense((3, 2), (3, 3))
-</code></pre></div></div>
+<pre><code>A(o to 1, 1 to 2) := dense((3, 2), (3, 3))
+</code></pre>
<p>Assignment operator used for copying between vectors or matrices:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>vec1 := vec2
+<pre><code>vec1 := vec2
M1 := M2
-</code></pre></div></div>
+</code></pre>
<p>Assignment operator using assignment through a functional literal for a matrix:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>M := ((row, col, x) => if (row == col) 1 else 0
-</code></pre></div></div>
+<pre><code>M := ((row, col, x) => if (row == col) 1 else 0
+</code></pre>
<p>Assignment operator using assignment through a functional literal for a vector:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>vec := ((index, x) => sqrt(x)
-</code></pre></div></div>
+<pre><code>vec := ((index, x) => sqrt(x)
+</code></pre>
<h4 id="blas-like-operations">BLAS-like operations</h4>
<p>Plus/minus either vector or numeric with assignment or not:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>a + b
+<pre><code>a + b
a - b
a + 5.0
a - 5.0
-</code></pre></div></div>
+</code></pre>
<p>Hadamard (elementwise) product, either vector or matrix or numeric operands:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>a * b
+<pre><code>a * b
a * 0.5
-</code></pre></div></div>
+</code></pre>
<p>Operations with assignment:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>a += b
+<pre><code>a += b
a -= b
a += 5.0
a -= 5.0
a *= b
a *= 5
-</code></pre></div></div>
+</code></pre>
<p><em>Some nuanced rules</em>:</p>
<p>1/x in R (where x is a vector or a matrix) is elementwise inverse. In scala it would be expressed as:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val xInv = 1 /: x
-</code></pre></div></div>
+<pre><code>val xInv = 1 /: x
+</code></pre>
<p>and R’s 5.0 - x would be:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val x1 = 5.0 -: x
-</code></pre></div></div>
+<pre><code>val x1 = 5.0 -: x
+</code></pre>
<p><em>note: All assignment operations, including :=, return the assignee just like in C++</em>:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>a -= b
-</code></pre></div></div>
+<pre><code>a -= b
+</code></pre>
<p>assigns <strong>a - b</strong> to <strong>b</strong> (in-place) and returns <strong>b</strong>. Similarly for <strong>a /=: b</strong> or <strong>1 /=: v</strong></p>
<p>Dot product:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>a dot b
-</code></pre></div></div>
+<pre><code>a dot b
+</code></pre>
<p>Matrix and vector equivalency (or non-equivalency). <strong>Dangerous, exact equivalence is rarely useful, better to use norm comparisons with an allowance of small errors.</strong></p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>a === b
+<pre><code>a === b
a !== b
-</code></pre></div></div>
+</code></pre>
<p>Matrix multiply:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>a %*% b
-</code></pre></div></div>
+<pre><code>a %*% b
+</code></pre>
<p>Optimized Right Multiply with a diagonal matrix:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>diag(5, 5) :%*% b
-</code></pre></div></div>
+<pre><code>diag(5, 5) :%*% b
+</code></pre>
<p>Optimized Left Multiply with a diagonal matrix:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>A %*%: diag(5, 5)
-</code></pre></div></div>
+<pre><code>A %*%: diag(5, 5)
+</code></pre>
<p>Second norm, of a vector or matrix:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>a.norm
-</code></pre></div></div>
+<pre><code>a.norm
+</code></pre>
<p>Transpose:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val Mt = M.t
-</code></pre></div></div>
+<pre><code>val Mt = M.t
+</code></pre>
-<p><em>note: Transposition is currently handled via view, i.e. updating a transposed matrix will be updating the original.</em> Also computing something like <code class="highlighter-rouge">\(\mathbf{X^\top}\mathbf{X}\)</code>:</p>
+<p><em>note: Transposition is currently handled via view, i.e. updating a transposed matrix will be updating the original.</em> Also computing something like <code>\(\mathbf{X^\top}\mathbf{X}\)</code>:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val XtX = X.t %*% X
-</code></pre></div></div>
+<pre><code>val XtX = X.t %*% X
+</code></pre>
<p>will not therefore incur any additional data copying.</p>
@@ -497,115 +497,115 @@ a !== b
<p>Matrix decompositions require an additional import:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>import org.apache.mahout.math.decompositions._
-</code></pre></div></div>
+<pre><code>import org.apache.mahout.math.decompositions._
+</code></pre>
<p>All arguments in the following are matricies.</p>
<p><strong>Cholesky decomposition</strong></p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val ch = chol(M)
-</code></pre></div></div>
+<pre><code>val ch = chol(M)
+</code></pre>
<p><strong>SVD</strong></p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val (U, V, s) = svd(M)
-</code></pre></div></div>
+<pre><code>val (U, V, s) = svd(M)
+</code></pre>
<p><strong>EigenDecomposition</strong></p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val (V, d) = eigen(M)
-</code></pre></div></div>
+<pre><code>val (V, d) = eigen(M)
+</code></pre>
<p><strong>QR decomposition</strong></p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val (Q, R) = qr(M)
-</code></pre></div></div>
+<pre><code>val (Q, R) = qr(M)
+</code></pre>
<p><strong>Rank</strong>: Check for rank deficiency (runs rank-revealing QR)</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>M.isFullRank
-</code></pre></div></div>
+<pre><code>M.isFullRank
+</code></pre>
<p><strong>In-core SSVD</strong></p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Val (U, V, s) = ssvd(A, k = 50, p = 15, q = 1)
-</code></pre></div></div>
+<pre><code>Val (U, V, s) = ssvd(A, k = 50, p = 15, q = 1)
+</code></pre>
<p><strong>Solving linear equation systems and matrix inversion:</strong> fully similar to R semantics; there are three forms of invocation:</p>
-<p>Solve <code class="highlighter-rouge">\(\mathbf{AX}=\mathbf{B}\)</code>:</p>
+<p>Solve <code>\(\mathbf{AX}=\mathbf{B}\)</code>:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>solve(A, B)
-</code></pre></div></div>
+<pre><code>solve(A, B)
+</code></pre>
-<p>Solve <code class="highlighter-rouge">\(\mathbf{Ax}=\mathbf{b}\)</code>:</p>
+<p>Solve <code>\(\mathbf{Ax}=\mathbf{b}\)</code>:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>solve(A, b)
-</code></pre></div></div>
+<pre><code>solve(A, b)
+</code></pre>
-<p>Compute <code class="highlighter-rouge">\(\mathbf{A^{-1}}\)</code>:</p>
+<p>Compute <code>\(\mathbf{A^{-1}}\)</code>:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>solve(A)
-</code></pre></div></div>
+<pre><code>solve(A)
+</code></pre>
<h4 id="misc">Misc</h4>
<p>Vector cardinality:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>a.length
-</code></pre></div></div>
+<pre><code>a.length
+</code></pre>
<p>Matrix cardinality:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>m.nrow
+<pre><code>m.nrow
m.ncol
-</code></pre></div></div>
+</code></pre>
<p>Means and sums:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>m.colSums
+<pre><code>m.colSums
m.colMeans
m.rowSums
m.rowMeans
-</code></pre></div></div>
+</code></pre>
<p>Copy-By-Value:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val b = a cloned
-</code></pre></div></div>
+<pre><code>val b = a cloned
+</code></pre>
<h4 id="random-matrices">Random Matrices</h4>
-<p><code class="highlighter-rouge">\(\mathcal{U}\)</code>(0,1) random matrix view:</p>
+<p><code>\(\mathcal{U}\)</code>(0,1) random matrix view:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val incCoreA = Matrices.uniformView(m, n, seed)
-</code></pre></div></div>
+<pre><code>val incCoreA = Matrices.uniformView(m, n, seed)
+</code></pre>
-<p><code class="highlighter-rouge">\(\mathcal{U}\)</code>(-1,1) random matrix view:</p>
+<p><code>\(\mathcal{U}\)</code>(-1,1) random matrix view:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val incCoreA = Matrices.symmetricUniformView(m, n, seed)
-</code></pre></div></div>
+<pre><code>val incCoreA = Matrices.symmetricUniformView(m, n, seed)
+</code></pre>
-<p><code class="highlighter-rouge">\(\mathcal{N}\)</code>(-1,1) random matrix view:</p>
+<p><code>\(\mathcal{N}\)</code>(-1,1) random matrix view:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>val incCoreA = Matrices.gaussianView(m, n, seed)
-</code></pre></div></div>
+<pre><code>val incCoreA = Matrices.gaussianView(m, n, seed)
+</code></pre>
<h4 id="iterators">Iterators</h4>
<p>Mahout-Math already exposes a number of iterators. Scala code just needs the following imports to enable implicit conversions to scala iterators.</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>import collection._
+<pre><code>import collection._
import JavaConversions._
-</code></pre></div></div>
+</code></pre>
<p>Iterating over rows in a Matrix:</p>
-<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>for (row <- m) {
+<pre><code>for (row <- m) {
... do something with row
}
-</code></pre></div></div>
+</code></pre>
<!--Iterating over non-zero and all elements of a vector:
*Note that Vector.Element also has some implicit syntatic sugar, e.g to add 5.0 to every non-zero element of a matrix, the following code may be used:*