You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mahout.apache.org by bu...@apache.org on 2013/11/21 12:41:27 UTC
svn commit: r887514 - in /websites/staging/mahout/trunk/content: ./
users/emr/mahout-on-elastic-mapreduce.html
Author: buildbot
Date: Thu Nov 21 11:41:27 2013
New Revision: 887514
Log:
Staging update by buildbot for mahout
Modified:
websites/staging/mahout/trunk/content/ (props changed)
websites/staging/mahout/trunk/content/users/emr/mahout-on-elastic-mapreduce.html
Propchange: websites/staging/mahout/trunk/content/
------------------------------------------------------------------------------
--- cms:source-revision (original)
+++ cms:source-revision Thu Nov 21 11:41:27 2013
@@ -1 +1 @@
-1544133
+1544135
Modified: websites/staging/mahout/trunk/content/users/emr/mahout-on-elastic-mapreduce.html
==============================================================================
--- websites/staging/mahout/trunk/content/users/emr/mahout-on-elastic-mapreduce.html (original)
+++ websites/staging/mahout/trunk/content/users/emr/mahout-on-elastic-mapreduce.html Thu Nov 21 11:41:27 2013
@@ -381,15 +381,16 @@
<div id="content-wrap" class="clearfix">
<div id="main">
- <p><a name="MahoutonElasticMapReduce-Introduction"></a></p>
-<h1 id="introduction">Introduction</h1>
+ <h1 id="mahout-on-emr-elastic-mapreduce">Mahout on EMR (Elastic MapReduce)</h1>
+<p><a name="MahoutonElasticMapReduce-Introduction"></a></p>
+<h2 id="introduction">Introduction</h2>
<p>This page details the set of steps that was necessary to get an example of
k-Means clustering running on Amazon's <a href="http://aws.amazon.com/elasticmapreduce/">Elastic MapReduce</a>
- (EMR). </p>
+(EMR). </p>
<p>Note: Some of this work is due in part to credits donated by Amazon Web
Services Apache Projects Testing Program.</p>
<p><a name="MahoutonElasticMapReduce-GettingStarted"></a></p>
-<h1 id="getting-started">Getting Started</h1>
+<h2 id="getting-started">Getting Started</h2>
<ul>
<li>Get yourself an EMR account. If you're already using EC2, then you
can do this from <a href="https://console.aws.amazon.com/">Amazon's AWS Managment Console</a>
@@ -423,13 +424,14 @@ Mahout output will be going, you should
input for the clustering (I used the synthetic control data, you can find
it easily from the <a href="quickstart.html">Quickstart</a>
page), and the output of the clustering. </p>
-<p>You will need to upload:
-1. The Mahout Job jar. For the example here, we are using
-<em>mahout-core-0.4-SNAPSHOT.job</em>
-1. The data. In this example, we uploaded two files: dictionary.txt and
+<p>You will need to upload:</p>
+<ol>
+<li>The Mahout Job jar. For the example here, we are using <em>mahout-core-0.4-SNAPSHOT.job</em></li>
+<li>The data. In this example, we uploaded two files: dictionary.txt and
part-out.vec. The latter is the main vector file and the former is the
dictionary that maps words to columns. It was created by converting a
-Lucene index to Mahout vectors.</p>
+Lucene index to Mahout vectors.</li>
+</ol>
<p><a name="MahoutonElasticMapReduce-Runningk-meansClustering"></a></p>
<h1 id="running-k-means-clustering">Running k-means Clustering</h1>
<p>EMR offers two modes for running MapReduce jobs. The first is a
@@ -453,14 +455,14 @@ sure that the given file exists, which i
then provide the arguments for the job just as you would on the command
line. The arguments for the k-means job that were as follows:</p>
<div class="codehilite"><pre><span class="n">org</span><span class="p">.</span><span class="n">apache</span><span class="p">.</span><span class="n">mahout</span><span class="p">.</span><span class="n">clustering</span><span class="p">.</span><span class="n">kmeans</span><span class="p">.</span><span class="n">KMeansDriver</span> <span class="o">--</span><span class="n">input</span>
+ <span class="n">s3n</span><span class="p">:</span><span class="o">//</span><span class="n">news</span><span class="o">-</span><span class="n">vecs</span><span class="o">/</span><span class="n">part</span><span class="o">-</span><span class="n">out</span><span class="p">.</span><span class="n">vec</span> <span class="o">--</span><span class="n">clusters</span>
+ <span class="n">s3n</span><span class="p">:</span><span class="o">//</span><span class="n">news</span><span class="o">-</span><span class="n">vecs</span><span class="o">/</span><span class="n">kmeans</span><span class="o">/</span><span class="n">clusters</span><span class="o">-</span>9<span class="o">-</span>11<span class="o">/</span> <span class="o">-</span><span class="n">k</span> 10 <span class="o">--</span><span class="n">output</span>
+ <span class="n">s3n</span><span class="p">:</span><span class="o">//</span><span class="n">news</span><span class="o">-</span><span class="n">vecs</span><span class="o">/</span><span class="n">out</span><span class="o">-</span>9<span class="o">-</span>11<span class="o">/</span> <span class="o">--</span><span class="n">distanceMeasure</span>
+ <span class="n">org</span><span class="p">.</span><span class="n">apache</span><span class="p">.</span><span class="n">mahout</span><span class="p">.</span><span class="n">common</span><span class="p">.</span><span class="n">distance</span><span class="p">.</span><span class="n">CosineDistanceMeasure</span> <span class="o">--</span><span class="n">convergenceDelta</span>
+ 0<span class="p">.</span>001 <span class="o">--</span><span class="n">overwrite</span> <span class="o">--</span><span class="n">maxIter</span> 50 <span class="o">--</span><span class="n">clustering</span>
</pre></div>
-<p>s3n://news-vecs/part-out.vec --clusters
-s3n://news-vecs/kmeans/clusters-9-11/ -k 10 --output
-s3n://news-vecs/out-9-11/ --distanceMeasure
-org.apache.mahout.common.distance.CosineDistanceMeasure --convergenceDelta
-0.001 --overwrite --maxIter 50 --clustering</p>
<p>TODO: Screenshot</p>
<p>The main failing with the GUI mode is that you can only specify a single
job to run, and you can't run another job in the same set of instances.
@@ -495,13 +497,13 @@ calling:</p>
<p>Let's list our job flows:</p>
-<div class="codehilite"><pre><span class="p">[</span><span class="n">stgreen</span><span class="p">@</span><span class="n">dhcp</span><span class="o">-</span><span class="n">ubur02</span><span class="o">-</span>74<span class="o">-</span>153 14<span class="p">:</span>16<span class="p">:</span>15 <span class="n">emr</span><span class="p">]</span>
+<div class="codehilite"><pre><span class="k">[stgreen@dhcp-ubur02-74-153 14:16:15 emr]</span>
+ <span class="err">$</span> <span class="err">./elastic-mapreduce</span> <span class="err">--list</span>
+ <span class="err">j-3JB4UF7CQQ025</span> <span class="err">WAITING</span>
+ <span class="err">ec2-174-129-90-97.compute-1.amazonaws.com</span> <span class="err">kmeans</span>
</pre></div>
-<p>$ ./elastic-mapreduce --list
- j-3JB4UF7CQQ025 WAITING <br />
-ec2-174-129-90-97.compute-1.amazonaws.com kmeans</p>
<p>At this point, everything's started up, and it's waiting for us to add a
step to the job. When we started the job flow, we specified a key pair
that we created earlier so that we can log into the master while the job
@@ -512,17 +514,17 @@ flow is running:</p>
<p>Let's add a step to run a job:</p>
<div class="codehilite"><pre> <span class="n">elastic</span><span class="o">-</span><span class="n">mapreduce</span> <span class="o">-</span><span class="nb">j</span> <span class="nb">j</span><span class="o">-</span>3<span class="n">JB4UF7CQQ025</span> <span class="o">--</span><span class="n">jar</span>
+ <span class="n">s3n</span><span class="p">:</span><span class="o">//</span><span class="n">PATH</span><span class="o">/</span><span class="n">mahout</span><span class="o">-</span><span class="n">core</span><span class="o">-</span>0<span class="p">.</span>4<span class="o">-</span><span class="n">SNAPSHOT</span><span class="o">-</span><span class="n">job</span><span class="p">.</span><span class="n">jar</span> <span class="o">--</span><span class="n">main</span><span class="o">-</span><span class="n">class</span>
+ <span class="n">org</span><span class="p">.</span><span class="n">apache</span><span class="p">.</span><span class="n">mahout</span><span class="p">.</span><span class="n">clustering</span><span class="p">.</span><span class="n">kmeans</span><span class="p">.</span><span class="n">KMeansDriver</span> <span class="o">--</span><span class="n">arg</span> <span class="o">--</span><span class="n">input</span> <span class="o">--</span><span class="n">arg</span>
+ <span class="n">s3n</span><span class="p">:</span><span class="o">//</span><span class="n">PATH</span><span class="o">/</span><span class="n">part</span><span class="o">-</span><span class="n">out</span><span class="p">.</span><span class="n">vec</span> <span class="o">--</span><span class="n">arg</span> <span class="o">--</span><span class="n">clusters</span> <span class="o">--</span><span class="n">arg</span> <span class="n">s3n</span><span class="p">:</span><span class="o">//</span><span class="n">PATH</span><span class="o">/</span><span class="n">kmeans</span><span class="o">/</span><span class="n">clusters</span><span class="o">/</span>
+ <span class="o">--</span><span class="n">arg</span> <span class="o">-</span><span class="n">k</span> <span class="o">--</span><span class="n">arg</span> 10 <span class="o">--</span><span class="n">arg</span> <span class="o">--</span><span class="n">output</span> <span class="o">--</span><span class="n">arg</span> <span class="n">s3n</span><span class="p">:</span><span class="o">//</span><span class="n">PATH</span><span class="o">/</span><span class="n">out</span><span class="o">-</span>9<span class="o">-</span>11<span class="o">/</span> <span class="o">--</span><span class="n">arg</span>
+ <span class="o">--</span><span class="n">distanceMeasure</span> <span class="o">--</span><span class="n">arg</span>
+ <span class="n">org</span><span class="p">.</span><span class="n">apache</span><span class="p">.</span><span class="n">mahout</span><span class="p">.</span><span class="n">common</span><span class="p">.</span><span class="n">distance</span><span class="p">.</span><span class="n">CosineDistanceMeasure</span> <span class="o">--</span><span class="n">arg</span>
+ <span class="o">--</span><span class="n">convergenceDelta</span> <span class="o">--</span><span class="n">arg</span> 0<span class="p">.</span>001 <span class="o">--</span><span class="n">arg</span> <span class="o">--</span><span class="n">overwrite</span> <span class="o">--</span><span class="n">arg</span> <span class="o">--</span><span class="n">maxIter</span> <span class="o">--</span><span class="n">arg</span> 50
+ <span class="o">--</span><span class="n">arg</span> <span class="o">--</span><span class="n">clustering</span>
</pre></div>
-<p>s3n://PATH/mahout-core-0.4-SNAPSHOT-job.jar --main-class
-org.apache.mahout.clustering.kmeans.KMeansDriver --arg --input --arg
-s3n://PATH/part-out.vec --arg --clusters --arg s3n://PATH/kmeans/clusters/
---arg -k --arg 10 --arg --output --arg s3n://PATH/out-9-11/ --arg
---distanceMeasure --arg
-org.apache.mahout.common.distance.CosineDistanceMeasure --arg
---convergenceDelta --arg 0.001 --arg --overwrite --arg --maxIter --arg 50
---arg --clustering</p>
<p>When you do this, the job flow goes into the <em>RUNNING</em> state for a while
and then returns to <em>WAITING</em> once the step has finished. You can use
the CLI or the GUI to monitor the step while it runs. Once you've finished
@@ -680,24 +682,22 @@ bucket:</p>
<h4 id="5-vectorize">5. Vectorize</h4>
<p>Schedule a jobflow step to vectorize (1-grams only) using Mahout's
seq2sparse job:</p>
-<div class="codehilite"><pre><span class="n">elastic</span><span class="o">-</span><span class="n">mapreduce</span> <span class="o">--</span><span class="n">jar</span>
+<div class="codehilite"><pre><span class="n">elastic</span><span class="o">-</span><span class="n">mapreduce</span> <span class="o">--</span><span class="n">jar</span> <span class="n">s3</span><span class="p">:</span><span class="o">//</span><span class="n">asf</span><span class="o">-</span><span class="n">mail</span><span class="o">-</span><span class="n">archives</span><span class="o">/</span><span class="n">mahout</span><span class="o">-</span>0<span class="p">.</span>4<span class="o">/</span><span class="n">mahout</span><span class="o">-</span><span class="n">examples</span><span class="o">-</span>0<span class="p">.</span>4<span class="o">-</span><span class="n">job</span><span class="o">-</span><span class="n">ext</span><span class="p">.</span><span class="n">jar</span> <span class="o">\</span>
+ <span class="o">--</span><span class="n">main</span><span class="o">-</span><span class="n">class</span> <span class="n">org</span><span class="p">.</span><span class="n">apache</span><span class="p">.</span><span class="n">mahout</span><span class="p">.</span><span class="n">driver</span><span class="p">.</span><span class="n">MahoutDriver</span> <span class="o">\</span>
+ <span class="o">--</span><span class="n">arg</span> <span class="n">seq2sparse</span> <span class="o">\</span>
+ <span class="o">--</span><span class="n">arg</span> <span class="o">-</span><span class="nb">i</span> <span class="o">--</span><span class="n">arg</span> <span class="n">s3n</span><span class="p">:</span><span class="o">//</span><span class="n">asf</span><span class="o">-</span><span class="n">mail</span><span class="o">-</span><span class="n">archives</span><span class="o">/</span><span class="n">mahout</span><span class="o">-</span>0<span class="p">.</span>4<span class="o">/</span><span class="n">sequence</span><span class="o">-</span><span class="n">files</span><span class="o">/</span> <span class="o">\</span>
+ <span class="o">--</span><span class="n">arg</span> <span class="o">-</span><span class="n">o</span> <span class="o">--</span><span class="n">arg</span> <span class="o">/</span><span class="n">asf</span><span class="o">-</span><span class="n">mail</span><span class="o">-</span><span class="n">archives</span><span class="o">/</span><span class="n">mahout</span><span class="o">-</span>0<span class="p">.</span>4<span class="o">/</span><span class="n">vectors</span><span class="o">/</span> <span class="o">\</span>
+ <span class="o">--</span><span class="n">arg</span> <span class="o">--</span><span class="n">weight</span> <span class="o">--</span><span class="n">arg</span> <span class="n">tfidf</span> <span class="o">\</span>
+ <span class="o">--</span><span class="n">arg</span> <span class="o">--</span><span class="n">minSupport</span> <span class="o">--</span><span class="n">arg</span> 500 <span class="o">\</span>
+ <span class="o">--</span><span class="n">arg</span> <span class="o">--</span><span class="n">maxDFPercent</span> <span class="o">--</span><span class="n">arg</span> 70 <span class="o">\</span>
+ <span class="o">--</span><span class="n">arg</span> <span class="o">--</span><span class="n">norm</span> <span class="o">--</span><span class="n">arg</span> 2 <span class="o">\</span>
+ <span class="o">--</span><span class="n">arg</span> <span class="o">--</span><span class="n">numReducers</span> <span class="o">--</span><span class="n">arg</span> # <span class="o">\</span>
+ <span class="o">--</span><span class="n">arg</span> <span class="o">--</span><span class="n">analyzerName</span> <span class="o">--</span><span class="n">arg</span> <span class="n">org</span><span class="p">.</span><span class="n">apache</span><span class="p">.</span><span class="n">mahout</span><span class="p">.</span><span class="n">text</span><span class="p">.</span><span class="n">MailArchivesClusteringAnalyzer</span> <span class="o">\</span>
+ <span class="o">--</span><span class="n">arg</span> <span class="o">--</span><span class="n">maxNGramSize</span> <span class="o">--</span><span class="n">arg</span> 1 <span class="o">\</span>
+ <span class="o">-</span><span class="nb">j</span> <span class="n">JOB_ID</span>
</pre></div>
-<p>s3://asf-mail-archives/mahout-0.4/mahout-examples-0.4-job-ext.jar \
- --main-class org.apache.mahout.driver.MahoutDriver \
- --arg seq2sparse \
- --arg -i --arg s3n://asf-mail-archives/mahout-0.4/sequence-files/ \
- --arg -o --arg /asf-mail-archives/mahout-0.4/vectors/ \
- --arg --weight --arg tfidf \
- --arg --minSupport --arg 500 \
- --arg --maxDFPercent --arg 70 \
- --arg --norm --arg 2 \
- --arg --numReducers --arg # \
- --arg --analyzerName --arg
-org.apache.mahout.text.MailArchivesClusteringAnalyzer \
- --arg --maxNGramSize --arg 1 \
- -j JOB_ID</p>
<p>You need to determine the correct number of reducers based on the EC2
instance type and size of your cluster. For xlarge nodes, set the number of
reducers to 3 x N (where N is the size of your EMR cluster not counting the
@@ -738,57 +738,50 @@ the EMR cluster. There are two approache
<p>Once logged in, do:</p>
-<div class="codehilite"><pre><span class="n">hadoop</span> <span class="n">distcp</span> <span class="o">/</span><span class="n">asf</span><span class="o">-</span><span class="n">mail</span><span class="o">-</span><span class="n">archives</span><span class="o">/</span><span class="n">mahout</span><span class="o">-</span>0<span class="p">.</span>4<span class="o">/</span><span class="n">vectors</span><span class="o">/</span>
+<div class="codehilite"><pre><span class="n">hadoop</span> <span class="n">distcp</span> <span class="o">/</span><span class="n">asf</span><span class="o">-</span><span class="n">mail</span><span class="o">-</span><span class="n">archives</span><span class="o">/</span><span class="n">mahout</span><span class="o">-</span>0<span class="p">.</span>4<span class="o">/</span><span class="n">vectors</span><span class="o">/</span> <span class="n">s3n</span><span class="p">:</span><span class="o">//</span><span class="n">ACCESS_KEY</span><span class="p">:</span><span class="n">SECRET_KEY</span><span class="p">@</span><span class="n">BUCKET</span><span class="o">/</span><span class="n">asf</span><span class="o">-</span><span class="n">mail</span><span class="o">-</span><span class="n">archives</span><span class="o">/</span><span class="n">mahout</span><span class="o">-</span>0<span class="p">.</span>4<span class="o">/</span><span class="n">vectors</span><span class="o">/</span> <span class="o
">&</span>
</pre></div>
-<p>s3n://ACCESS_KEY:SECRET_KEY@BUCKET/asf-mail-archives/mahout-0.4/vectors/ &</p>
<p>Or, you can just add another job flow step to do it:</p>
<div class="codehilite"><pre><span class="n">elastic</span><span class="o">-</span><span class="n">mapreduce</span> <span class="o">--</span><span class="n">jar</span> <span class="n">s3</span><span class="p">:</span><span class="o">//</span><span class="n">elasticmapreduce</span><span class="o">/</span><span class="n">samples</span><span class="o">/</span><span class="n">distcp</span><span class="o">/</span><span class="n">distcp</span><span class="p">.</span><span class="n">jar</span> <span class="o">\</span>
<span class="o">--</span><span class="n">arg</span> <span class="n">hdfs</span><span class="p">:</span><span class="o">///</span><span class="n">asf</span><span class="o">-</span><span class="n">mail</span><span class="o">-</span><span class="n">archives</span><span class="o">/</span><span class="n">mahout</span><span class="o">-</span>0<span class="p">.</span>4<span class="o">/</span><span class="n">vectors</span><span class="o">/</span> <span class="o">\</span>
- <span class="o">--</span><span class="n">arg</span>
+ <span class="o">--</span><span class="n">arg</span> <span class="n">s3n</span><span class="p">:</span><span class="o">//</span><span class="n">ACCESS_KEY</span><span class="p">:</span><span class="n">SECRET_KEY</span><span class="p">@</span><span class="n">BUCKET</span><span class="o">/</span><span class="n">asf</span><span class="o">-</span><span class="n">mail</span><span class="o">-</span><span class="n">archives</span><span class="o">/</span><span class="n">mahout</span><span class="o">-</span>0<span class="p">.</span>4<span class="o">/</span><span class="n">vectors</span><span class="o">/</span> <span class="o">\</span>
+ <span class="o">-</span><span class="nb">j</span> <span class="n">JOB_ID</span>
</pre></div>
-<p>s3n://ACCESS_KEY:SECRET_KEY@BUCKET/asf-mail-archives/mahout-0.4/vectors/ \
- -j JOB_ID</p>
<p><em>Note: You will need all the output from the vectorize step in order to run
Mahout's clusterdump.</em></p>
<p>Once copied, if you would like to share your results with the Mahout
community, make the vectors public in S3 using the Amazon console or s3cmd:</p>
-<div class="codehilite"><pre><span class="n">s3cmd</span> <span class="n">setacl</span> <span class="o">--</span><span class="n">acl</span><span class="o">-</span><span class="n">public</span> <span class="o">--</span><span class="n">recursive</span>
+<div class="codehilite"><pre><span class="n">s3cmd</span> <span class="n">setacl</span> <span class="o">--</span><span class="n">acl</span><span class="o">-</span><span class="n">public</span> <span class="o">--</span><span class="n">recursive</span> <span class="n">s3</span><span class="p">:</span><span class="o">//</span><span class="n">BUCKET</span><span class="o">/</span><span class="n">asf</span><span class="o">-</span><span class="n">mail</span><span class="o">-</span><span class="n">archives</span><span class="o">/</span><span class="n">mahout</span><span class="o">-</span>0<span class="p">.</span>4<span class="o">/</span><span class="n">vectors</span><span class="o">/</span>
</pre></div>
-<p>s3://BUCKET/asf-mail-archives/mahout-0.4/vectors/</p>
<p>Dump out the size of the vectors:</p>
-<div class="codehilite"><pre><span class="n">bin</span><span class="o">/</span><span class="n">mahout</span> <span class="n">vectordump</span> <span class="o">--</span><span class="n">seqFile</span>
+<div class="codehilite"><pre><span class="n">bin</span><span class="o">/</span><span class="n">mahout</span> <span class="n">vectordump</span> <span class="o">--</span><span class="n">seqFile</span> <span class="n">s3n</span><span class="p">:</span><span class="o">//</span><span class="n">ACCESS_KEY</span><span class="p">:</span><span class="n">SECRET_KEY</span><span class="p">@</span><span class="n">BUCKET</span><span class="o">/</span><span class="n">asf</span><span class="o">-</span><span class="n">mail</span><span class="o">-</span><span class="n">archives</span><span class="o">/</span><span class="n">mahout</span><span class="o">-</span>0<span class="p">.</span>4<span class="o">/</span><span class="n">vectors</span><span class="o">/</span><span class="n">tfidf</span><span class="o">-</span><span class="n">vectors</span><span class="o">/</span><span class="n">part</span><span class="o">-</span><span class="n">r</span><span class="o">-</span>00000 <span class="o">--</span><span c
lass="n">sizeOnly</span> <span class="o">|</span> <span class="n">more</span>
</pre></div>
-<p>s3n://ACCESS_KEY:SECRET_KEY@BUCKET/asf-mail-archives/mahout-0.4/vectors/tfidf-vectors/part-r-00000
---sizeOnly | more</p>
<p><a name="MahoutonElasticMapReduce-7.k-MeansClustering"></a></p>
<h4 id="7-k-means-clustering">7. k-Means Clustering</h4>
<p>Now that you have vectors, you can do some clustering! The following
command will create a new jobflow step to run the k-Means job using the
TFIDF vectors produced by seq2sparse:</p>
-<div class="codehilite"><pre><span class="n">elastic</span><span class="o">-</span><span class="n">mapreduce</span> <span class="o">--</span><span class="n">jar</span>
+<div class="codehilite"><pre><span class="n">elastic</span><span class="o">-</span><span class="n">mapreduce</span> <span class="o">--</span><span class="n">jar</span> <span class="n">s3</span><span class="p">:</span><span class="o">//</span><span class="n">asf</span><span class="o">-</span><span class="n">mail</span><span class="o">-</span><span class="n">archives</span><span class="o">/</span><span class="n">mahout</span><span class="o">-</span>0<span class="p">.</span>4<span class="o">/</span><span class="n">mahout</span><span class="o">-</span><span class="n">examples</span><span class="o">-</span>0<span class="p">.</span>4<span class="o">-</span><span class="n">job</span><span class="o">-</span><span class="n">ext</span><span class="p">.</span><span class="n">jar</span> <span class="o">\</span>
+ <span class="o">--</span><span class="n">main</span><span class="o">-</span><span class="n">class</span> <span class="n">org</span><span class="p">.</span><span class="n">apache</span><span class="p">.</span><span class="n">mahout</span><span class="p">.</span><span class="n">driver</span><span class="p">.</span><span class="n">MahoutDriver</span> <span class="o">\</span>
+ <span class="o">--</span><span class="n">arg</span> <span class="n">kmeans</span> <span class="o">\</span>
+ <span class="o">--</span><span class="n">arg</span> <span class="o">-</span><span class="nb">i</span> <span class="o">--</span><span class="n">arg</span> <span class="o">/</span><span class="n">asf</span><span class="o">-</span><span class="n">mail</span><span class="o">-</span><span class="n">archives</span><span class="o">/</span><span class="n">mahout</span><span class="o">-</span>0<span class="p">.</span>4<span class="o">/</span><span class="n">vectors</span><span class="o">/</span><span class="n">tfidf</span><span class="o">-</span><span class="n">vectors</span><span class="o">/</span> <span class="o">\</span>
+ <span class="o">--</span><span class="n">arg</span> <span class="o">-</span><span class="n">c</span> <span class="o">--</span><span class="n">arg</span> <span class="o">/</span><span class="n">asf</span><span class="o">-</span><span class="n">mail</span><span class="o">-</span><span class="n">archives</span><span class="o">/</span><span class="n">mahout</span><span class="o">-</span>0<span class="p">.</span>4<span class="o">/</span><span class="n">initial</span><span class="o">-</span><span class="n">clusters</span><span class="o">/</span> <span class="o">\</span>
+ <span class="o">--</span><span class="n">arg</span> <span class="o">-</span><span class="n">o</span> <span class="o">--</span><span class="n">arg</span> <span class="o">/</span><span class="n">asf</span><span class="o">-</span><span class="n">mail</span><span class="o">-</span><span class="n">archives</span><span class="o">/</span><span class="n">mahout</span><span class="o">-</span>0<span class="p">.</span>4<span class="o">/</span><span class="n">kmeans</span><span class="o">-</span><span class="n">clusters</span> <span class="o">\</span>
+ <span class="o">--</span><span class="n">arg</span> <span class="o">-</span><span class="n">x</span> <span class="o">--</span><span class="n">arg</span> 10 <span class="o">\</span>
+ <span class="o">--</span><span class="n">arg</span> <span class="o">-</span><span class="n">cd</span> <span class="o">--</span><span class="n">arg</span> 0<span class="p">.</span>01 <span class="o">\</span>
+ <span class="o">--</span><span class="n">arg</span> <span class="o">-</span><span class="n">k</span> <span class="o">--</span><span class="n">arg</span> 60 <span class="o">\</span>
+ <span class="o">--</span><span class="n">arg</span> <span class="o">--</span><span class="n">distanceMeasure</span> <span class="o">--</span><span class="n">arg</span> <span class="n">org</span><span class="p">.</span><span class="n">apache</span><span class="p">.</span><span class="n">mahout</span><span class="p">.</span><span class="n">common</span><span class="p">.</span><span class="n">distance</span><span class="p">.</span><span class="n">CosineDistanceMeasure</span> <span class="o">\</span>
+ <span class="o">-</span><span class="nb">j</span> <span class="n">JOB_ID</span>
</pre></div>
-<p>s3://asf-mail-archives/mahout-0.4/mahout-examples-0.4-job-ext.jar \
- --main-class org.apache.mahout.driver.MahoutDriver \
- --arg kmeans \
- --arg -i --arg /asf-mail-archives/mahout-0.4/vectors/tfidf-vectors/ \
- --arg -c --arg /asf-mail-archives/mahout-0.4/initial-clusters/ \
- --arg -o --arg /asf-mail-archives/mahout-0.4/kmeans-clusters \
- --arg -x --arg 10 \
- --arg -cd --arg 0.01 \
- --arg -k --arg 60 \
- --arg --distanceMeasure --arg
-org.apache.mahout.common.distance.CosineDistanceMeasure \
- -j JOB_ID</p>
<p>Depending on the EC2 instance type and size of your cluster, the k-Means
job can take a couple of hours to complete. The input is the HDFS location
of the vectors created by the seq2sparse job. If you copied the vectors to