You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mahout.apache.org by bu...@apache.org on 2013/11/20 21:30:10 UTC
svn commit: r887404 - in /websites/staging/mahout/trunk/content: ./
users/clustering/clusteringyourdata.html
Author: buildbot
Date: Wed Nov 20 20:30:10 2013
New Revision: 887404
Log:
Staging update by buildbot for mahout
Modified:
websites/staging/mahout/trunk/content/ (props changed)
websites/staging/mahout/trunk/content/users/clustering/clusteringyourdata.html
Propchange: websites/staging/mahout/trunk/content/
------------------------------------------------------------------------------
--- cms:source-revision (original)
+++ cms:source-revision Wed Nov 20 20:30:10 2013
@@ -1 +1 @@
-1543934
+1543937
Modified: websites/staging/mahout/trunk/content/users/clustering/clusteringyourdata.html
==============================================================================
--- websites/staging/mahout/trunk/content/users/clustering/clusteringyourdata.html (original)
+++ websites/staging/mahout/trunk/content/users/clustering/clusteringyourdata.html Wed Nov 20 20:30:10 2013
@@ -382,101 +382,88 @@
<div id="content-wrap" class="clearfix">
<div id="main">
<p>+<em>Mahout_0.8</em>+</p>
-<p>After you've done the <a href="quickstart.html">Quickstart</a>
- and are familiar with the basics of Mahout, it is time to cluster your own
-data. </p>
+<p>After you've done the <a href="quickstart.html">Quickstart</a> and are familiar with the basics of Mahout, it is time to cluster your own
+data. See also <a href="en.wikipedia.org/wiki/Cluster_analysis">Wikipedia on cluster analysis</a> for more background.</p>
<p>The following pieces <em>may</em> be useful for in getting started:</p>
<p><a name="ClusteringYourData-Input"></a></p>
<h1 id="input">Input</h1>
-<p>For starters, you will need your data in an appropriate Vector format
-(which has changed since Mahout 0.1)</p>
-<ul>
-<li>See <a href="creating-vectors.html">Creating Vectors</a></li>
-</ul>
-<p><a name="ClusteringYourData-TextPreparation"></a></p>
-<h2 id="text-preparation">Text Preparation</h2>
-<ul>
-<li>See <a href="creating-vectors-from-text.html">Creating Vectors from Text</a>
-*
-http://www.lucidimagination.com/search/document/4a0e528982b2dac3/document_clustering</li>
-</ul>
+<p>For starters, you will need your data in an appropriate Vector format, see <a href="../basics/creating-vectors.html">Creating Vectors</a>.
+In particular for text preparation check out <a href="../basics/creating-vectors-from-text.html">Creating Vectors from Text</a>.</p>
<p><a name="ClusteringYourData-RunningtheProcess"></a></p>
<h1 id="running-the-process">Running the Process</h1>
-<p><a name="ClusteringYourData-Canopy"></a></p>
-<h2 id="canopy">Canopy</h2>
-<p>Background: <a href="-canopy-clustering.html">canopy </a></p>
-<p>Documentation of running canopy from the command line: <a href="canopy-commandline.html">canopy-commandline</a></p>
-<p><a name="ClusteringYourData-kMeans"></a></p>
-<h2 id="kmeans">kMeans</h2>
-<p>Background: <a href="k-means-clustering.html">K-Means Clustering</a></p>
-<p>Documentation of running kMeans from the command line: <a href="k-means-commandline.html">k-means-commandline</a></p>
-<p>Documentation of running fuzzy kMeans from the command line: <a href="fuzzy-k-means-commandline.html">fuzzy-k-means-commandline</a></p>
-<p><a name="ClusteringYourData-Dirichlet"></a></p>
-<h2 id="dirichlet">Dirichlet</h2>
-<p>Background: <a href="-dirichlet-process-clustering.html">dirichlet </a></p>
-<p>Documentation of running dirichlet from the command line: <a href="dirichlet-commandline.html">dirichlet-commandline</a></p>
-<p><a name="ClusteringYourData-Mean-shift"></a></p>
-<h2 id="mean-shift">Mean-shift</h2>
-<p>Background: <a href="-mean-shift-clustering.html">meanshift </a></p>
-<p>Documentation of running mean shift from the command line: <a href="mean-shift-commandline.html">mean-shift-commandline</a></p>
-<p><a name="ClusteringYourData-LatentDirichletAllocation"></a></p>
-<h2 id="latent-dirichlet-allocation">Latent Dirichlet Allocation</h2>
-<p>Background and documentation: <a href="-latent-dirichlet-allocation.html">LDA</a></p>
-<p>Documentation of running LDA from the command line: <a href="lda-commandline.html">lda-commandline</a></p>
+<ul>
+<li>
+<p><a href="canopy-clustering.html">Canopy background</a> and <a href="canopy-commandline.html">canopy-commandline</a>.</p>
+</li>
+<li>
+<p><a href="k-means-clustering.html">K-Means background</a>, <a href="k-means-commandline.html">k-means-commandline</a>, and
+<a href="fuzzy-k-means-commandline.html">fuzzy-k-means-commandline</a>.</p>
+</li>
+<li>
+<p><a href="dirichlet-process-clustering.html">Dirichlet background</a> and <a href="dirichlet-commandline.html">dirichlet-commandline</a>.</p>
+</li>
+<li>
+<p><a href="mean-shift-clustering.html">Meanshift background</a> and <a href="mean-shift-commandline.html">mean-shift-commandline</a>.</p>
+</li>
+<li>
+<p><a href="-latent-dirichlet-allocation.html">LDA (Latent Dirichlet Allocation) background</a> and <a href="lda-commandline.html">lda-commandline</a>.</p>
+</li>
+</ul>
<p><a name="ClusteringYourData-RetrievingtheOutput"></a></p>
<h1 id="retrieving-the-output">Retrieving the Output</h1>
-<p>Mahout has a cluster dumper utility that can be used to retrieve and
-evaluate your clustering data.</p>
+<p>Mahout has a cluster dumper utility that can be used to retrieve and evaluate your clustering data.</p>
<div class="codehilite"><pre><span class="o">./</span><span class="n">bin</span><span class="o">/</span><span class="n">mahout</span> <span class="n">clusterdump</span> <span class="o"><</span><span class="n">OPTIONS</span><span class="o">></span>
</pre></div>
<p><a name="ClusteringYourData-Theclusterdumperoptionsare:"></a></p>
<h2 id="the-cluster-dumper-options-are">The cluster dumper options are:</h2>
-<div class="codehilite"><pre> <span class="o">--</span><span class="n">help</span> <span class="p">(</span><span class="o">-</span><span class="n">h</span><span class="p">)</span> <span class="n">Print</span> <span class="n">out</span> <span class="n">help</span>
- <span class="o">--</span><span class="n">input</span> <span class="p">(</span><span class="o">-</span><span class="nb">i</span><span class="p">)</span> <span class="n">input</span> <span class="n">The</span> <span class="n">directory</span> <span class="n">containing</span>
+<div class="codehilite"><pre> <span class="o">--</span><span class="n">help</span> <span class="p">(</span><span class="o">-</span><span class="n">h</span><span class="p">)</span> <span class="n">Print</span> <span class="n">out</span> <span class="n">help</span>
+
+ <span class="o">--</span><span class="n">input</span> <span class="p">(</span><span class="o">-</span><span class="nb">i</span><span class="p">)</span> <span class="n">input</span> <span class="n">The</span> <span class="n">directory</span> <span class="n">containing</span> <span class="n">Sequence</span>
+ <span class="n">Files</span> <span class="k">for</span> <span class="n">the</span> <span class="n">Clusters</span>
+
+ <span class="o">--</span><span class="n">output</span> <span class="p">(</span><span class="o">-</span><span class="n">o</span><span class="p">)</span> <span class="n">output</span> <span class="n">The</span> <span class="n">output</span> <span class="n">file</span><span class="p">.</span> <span class="n">If</span> <span class="n">not</span> <span class="n">specified</span><span class="p">,</span>
+ <span class="n">dumps</span> <span class="n">to</span> <span class="n">the</span> <span class="n">console</span><span class="p">.</span>
+
+ <span class="o">--</span><span class="n">outputFormat</span> <span class="p">(</span><span class="o">-</span><span class="n">of</span><span class="p">)</span> <span class="n">outputFormat</span> <span class="n">The</span> <span class="n">optional</span> <span class="n">output</span> <span class="n">format</span> <span class="n">to</span> <span class="n">write</span>
+ <span class="n">the</span> <span class="n">results</span> <span class="n">as</span><span class="p">.</span> <span class="n">Options</span><span class="p">:</span> <span class="n">TEXT</span><span class="p">,</span> <span class="n">CSV</span><span class="p">,</span> <span class="n">or</span> <span class="n">GRAPH_ML</span>
+
+ <span class="o">--</span><span class="n">substring</span> <span class="p">(</span><span class="o">-</span><span class="n">b</span><span class="p">)</span> <span class="n">substring</span> <span class="n">The</span> <span class="n">number</span> <span class="n">of</span> <span class="n">chars</span> <span class="n">of</span> <span class="n">the</span>
+ <span class="n">asFormatString</span><span class="p">()</span> <span class="n">to</span> <span class="n">print</span>
+
+ <span class="o">--</span><span class="n">pointsDir</span> <span class="p">(</span><span class="o">-</span><span class="n">p</span><span class="p">)</span> <span class="n">pointsDir</span> <span class="n">The</span> <span class="n">directory</span> <span class="n">containing</span> <span class="n">points</span>
+ <span class="n">sequence</span> <span class="n">files</span> <span class="n">mapping</span> <span class="n">input</span> <span class="n">vectors</span> <span class="n">to</span> <span class="n">their</span> <span class="n">cluster</span><span class="p">.</span> <span class="n">If</span> <span class="n">specified</span><span class="p">,</span>
+ <span class="n">then</span> <span class="n">the</span> <span class="n">program</span> <span class="n">will</span> <span class="n">output</span> <span class="n">the</span>
+ <span class="n">points</span> <span class="n">associated</span> <span class="n">with</span> <span class="n">a</span> <span class="n">cluster</span>
+
+ <span class="o">--</span><span class="n">dictionary</span> <span class="p">(</span><span class="o">-</span><span class="n">d</span><span class="p">)</span> <span class="n">dictionary</span> <span class="n">The</span> <span class="n">dictionary</span> <span class="n">file</span><span class="p">.</span>
+
+ <span class="o">--</span><span class="n">dictionaryType</span> <span class="p">(</span><span class="o">-</span><span class="n">dt</span><span class="p">)</span> <span class="n">dictionaryType</span> <span class="n">The</span> <span class="n">dictionary</span> <span class="n">file</span> <span class="n">type</span>
+ <span class="p">(</span><span class="n">text</span><span class="o">|</span><span class="n">sequencefile</span><span class="p">)</span>
+
+ <span class="o">--</span><span class="n">distanceMeasure</span> <span class="p">(</span><span class="o">-</span><span class="n">dm</span><span class="p">)</span> <span class="n">distanceMeasure</span> <span class="n">The</span> <span class="n">classname</span> <span class="n">of</span> <span class="n">the</span> <span class="n">DistanceMeasure</span><span class="p">.</span>
+ <span class="n">Default</span> <span class="n">is</span> <span class="n">SquaredEuclidean</span><span class="p">.</span>
+
+ <span class="o">--</span><span class="n">numWords</span> <span class="p">(</span><span class="o">-</span><span class="n">n</span><span class="p">)</span> <span class="n">numWords</span> <span class="n">The</span> <span class="n">number</span> <span class="n">of</span> <span class="n">top</span> <span class="n">terms</span> <span class="n">to</span> <span class="n">print</span>
+
+ <span class="o">--</span><span class="n">tempDir</span> <span class="n">tempDir</span> <span class="n">Intermediate</span> <span class="n">output</span> <span class="n">directory</span>
+
+ <span class="o">--</span><span class="n">startPhase</span> <span class="n">startPhase</span> <span class="n">First</span> <span class="n">phase</span> <span class="n">to</span> <span class="n">run</span>
+
+ <span class="o">--</span><span class="n">endPhase</span> <span class="n">endPhase</span> <span class="n">Last</span> <span class="n">phase</span> <span class="n">to</span> <span class="n">run</span>
+
+ <span class="o">--</span><span class="n">evaluate</span> <span class="p">(</span><span class="o">-</span><span class="n">e</span><span class="p">)</span> <span class="n">Run</span> <span class="n">ClusterEvaluator</span> <span class="n">and</span> <span class="n">CDbwEvaluator</span> <span class="n">over</span> <span class="n">the</span>
+ <span class="n">input</span><span class="p">.</span> <span class="n">The</span> <span class="n">output</span> <span class="n">will</span> <span class="n">be</span> <span class="n">appended</span> <span class="n">to</span> <span class="n">the</span> <span class="n">rest</span> <span class="n">of</span>
+ <span class="n">the</span> <span class="n">output</span> <span class="n">at</span> <span class="n">the</span> <span class="k">end</span><span class="p">.</span>
</pre></div>
-<p>Sequence <br />
- Files for the Clusters <br />
- --output (-o) output The output file. If not
-specified,<br />
- dumps to the console.
- --outputFormat (-of) outputFormat The optional output format to
-write
- the results as. Options: TEXT,
-CSV, or GRAPH_ML <br />
- --substring (-b) substring The number of chars of the <br />
- asFormatString() to print <br />
- --pointsDir (-p) pointsDir The directory containing points<br />
- sequence files mapping input
-vectors
- to their cluster. If specified,
- then the program will output the
- points associated with a cluster
- --dictionary (-d) dictionary The dictionary file. <br />
- --dictionaryType (-dt) dictionaryType The dictionary file type <br />
- (text|sequencefile)
- --distanceMeasure (-dm) distanceMeasure The classname of the
-DistanceMeasure.
- Default is SquaredEuclidean. <br />
- --numWords (-n) numWords The number of top terms to print
- --tempDir tempDir Intermediate output directory
- --startPhase startPhase First phase to run
- --endPhase endPhase Last phase to run
- --evaluate (-e) Run ClusterEvaluator and
-CDbwEvaluator over the
- input. The output will be
-appended to the rest of
- the output at the end. </p>
<p>More information on using clusterdump utility can be found <a href="cluster-dumper.html">here</a></p>
<p><a name="ClusteringYourData-ValidatingtheOutput"></a></p>
<h1 id="validating-the-output">Validating the Output</h1>
-<p>From Ted Dunning's response on See
-http://www.lucidimagination.com/search/document/dab8c1f3c3addcfe/validating_clustering_output
-{quote}
-A principled approach to cluster evaluation is to measure how well the
+<p>{quote}
+Ted Dunning: A principled approach to cluster evaluation is to measure how well the
cluster membership captures the structure of unseen data. A natural
measure for this is to measure how much of the entropy of the data is
captured by cluster membership. For k-means and its natural L_2 metric,
@@ -503,11 +490,6 @@ Unfortunately, it is pretty easy to kid
is working using this kind of inspection. The problem is that we are too
good at seeing (making up) patterns.
{quote}</p>
-<p><a name="ClusteringYourData-References"></a></p>
-<h1 id="references">References</h1>
-<ul>
-<li><a href="http://www.lucidimagination.com/search/p:mahout?q=clustering">Mahout archive references</a></li>
-</ul>
</div>
</div>
</div>