You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mahout.apache.org by bu...@apache.org on 2014/05/25 13:40:52 UTC
svn commit: r909887 - in /websites/staging/mahout/trunk/content: ./
users/basics/creating-vectors-from-text.html
Author: buildbot
Date: Sun May 25 11:40:52 2014
New Revision: 909887
Log:
Staging update by buildbot for mahout
Modified:
websites/staging/mahout/trunk/content/ (props changed)
websites/staging/mahout/trunk/content/users/basics/creating-vectors-from-text.html
Propchange: websites/staging/mahout/trunk/content/
------------------------------------------------------------------------------
--- cms:source-revision (original)
+++ cms:source-revision Sun May 25 11:40:52 2014
@@ -1 +1 @@
-1596770
+1597416
Modified: websites/staging/mahout/trunk/content/users/basics/creating-vectors-from-text.html
==============================================================================
--- websites/staging/mahout/trunk/content/users/basics/creating-vectors-from-text.html (original)
+++ websites/staging/mahout/trunk/content/users/basics/creating-vectors-from-text.html Sun May 25 11:40:52 2014
@@ -241,8 +241,7 @@
<p><a name="CreatingVectorsfromText-Introduction"></a></p>
<h1 id="introduction">Introduction</h1>
<p>For clustering and classifying documents it is usually necessary to convert the raw text
-into vectors that can then be consumed by the clustering <a href="algorithms.html">Algorithms</a>
-. These approaches are described below.</p>
+into vectors that can then be consumed by the clustering <a href="algorithms.html">Algorithms</a>. These approaches are described below.</p>
<p><a name="CreatingVectorsfromText-FromLucene"></a></p>
<h1 id="from-lucene">From Lucene</h1>
<p><em>NOTE: Your Lucene index must be created with the same version of Lucene
@@ -260,13 +259,13 @@ index. For those wanting to use just Lu
McCandless.</p>
<p>To get started, make sure you get a fresh copy of Mahout from <a href="http://mahout.apache.org/developers/buildingmahout.html">SVN</a>
and are comfortable building it. It defines interfaces and implementations
-for efficiently iterating over a Data Source (it only supports Lucene
+for efficiently iterating over a data source (it only supports Lucene
currently, but should be extensible to databases, Solr, etc.) and produces
a Mahout Vector file and term dictionary which can then be used for
-clustering. The main code for driving this is the Driver program located
-in the org.apache.mahout.utils.vectors package. The Driver program offers
+clustering. The main code for driving this is the driver program located
+in the org.apache.mahout.utils.vectors package. The driver program offers
several input options, which can be displayed by specifying the --help
-option. Examples of running the Driver are included below:</p>
+option. Examples of running the driver are included below:</p>
<p><a name="CreatingVectorsfromText-GeneratinganoutputfilefromaLuceneIndex"></a></p>
<h4 id="generating-an-output-file-from-a-lucene-index">Generating an output file from a Lucene Index</h4>
<div class="codehilite"><pre>$<span class="n">MAHOUT_HOME</span><span class="o">/</span><span class="n">bin</span><span class="o">/</span><span class="n">mahout</span> <span class="n">lucene</span><span class="p">.</span><span class="n">vector</span>
@@ -324,12 +323,12 @@ option. Examples of running the Driver
</pre></div>
-<h4 id="create-50-vectors-from-an-index">Create 50 Vectors from an Index</h4>
+<h4 id="example-create-50-vectors-from-an-index">Example: Create 50 Vectors from an Index</h4>
<div class="codehilite"><pre>$<span class="n">MAHOUT_HOME</span><span class="o">/</span><span class="n">bin</span><span class="o">/</span><span class="n">mahout</span> <span class="n">lucene</span><span class="p">.</span><span class="n">vector</span>
- <span class="o">--</span><span class="n">dir</span> <span class="o"><</span><span class="n">PATH</span><span class="o">>/</span><span class="n">wikipedia</span><span class="o">/</span><span class="n">solr</span><span class="o">/</span><span class="n">data</span><span class="o">/</span><span class="n">index</span>
+ <span class="o">--</span><span class="n">dir</span> $<span class="n">WORK_DIR</span><span class="o">/</span><span class="n">wikipedia</span><span class="o">/</span><span class="n">solr</span><span class="o">/</span><span class="n">data</span><span class="o">/</span><span class="n">index</span>
<span class="o">--</span><span class="n">field</span> <span class="n">body</span>
- <span class="o">--</span><span class="n">dictOut</span> <span class="o"><</span><span class="n">PATH</span><span class="o">>/</span><span class="n">solr</span><span class="o">/</span><span class="n">wikipedia</span><span class="o">/</span><span class="n">dict</span><span class="p">.</span><span class="n">txt</span>
- <span class="o">--</span><span class="n">output</span> <span class="o"><</span><span class="n">PATH</span><span class="o">>/</span><span class="n">solr</span><span class="o">/</span><span class="n">wikipedia</span><span class="o">/</span><span class="n">out</span><span class="p">.</span><span class="n">txt</span>
+ <span class="o">--</span><span class="n">dictOut</span> $<span class="n">WORK_DIR</span><span class="o">/</span><span class="n">solr</span><span class="o">/</span><span class="n">wikipedia</span><span class="o">/</span><span class="n">dict</span><span class="p">.</span><span class="n">txt</span>
+ <span class="o">--</span><span class="n">output</span> $<span class="n">WORK_DIR</span><span class="o">/</span><span class="n">solr</span><span class="o">/</span><span class="n">wikipedia</span><span class="o">/</span><span class="n">out</span><span class="p">.</span><span class="n">txt</span>
<span class="o">--</span><span class="n">max</span> 50
</pre></div>
@@ -339,12 +338,12 @@ out the info to the output dir and the d
outputs 50 vectors. If you don't specify --max, then all the documents in
the index are output.</p>
<p><a name="CreatingVectorsfromText-50VectorsFromLuceneL2Norm"></a></p>
-<h4 id="creating-50-normalized-vectors-from-a-lucene-index-using-the-l_2-norm">Creating 50 Normalized Vectors from a Lucene Index using the <a href="http://en.wikipedia.org/wiki/Lp_space">L_2 Norm</a></h4>
+<h4 id="example-creating-50-normalized-vectors-from-a-lucene-index-using-the-l_2-norm">Example: Creating 50 Normalized Vectors from a Lucene Index using the <a href="http://en.wikipedia.org/wiki/Lp_space">L_2 Norm</a></h4>
<div class="codehilite"><pre>$<span class="n">MAHOUT_HOME</span><span class="o">/</span><span class="n">bin</span><span class="o">/</span><span class="n">mahout</span> <span class="n">lucene</span><span class="p">.</span><span class="n">vector</span>
- <span class="o">--</span><span class="n">dir</span> <span class="o"><</span><span class="n">PATH</span><span class="o">>/</span><span class="n">wikipedia</span><span class="o">/</span><span class="n">solr</span><span class="o">/</span><span class="n">data</span><span class="o">/</span><span class="n">index</span>
+ <span class="o">--</span><span class="n">dir</span> $<span class="n">WORK_DIR</span><span class="o">/</span><span class="n">wikipedia</span><span class="o">/</span><span class="n">solr</span><span class="o">/</span><span class="n">data</span><span class="o">/</span><span class="n">index</span>
<span class="o">--</span><span class="n">field</span> <span class="n">body</span>
- <span class="o">--</span><span class="n">dictOut</span> <span class="o"><</span><span class="n">PATH</span><span class="o">>/</span><span class="n">solr</span><span class="o">/</span><span class="n">wikipedia</span><span class="o">/</span><span class="n">dict</span><span class="p">.</span><span class="n">txt</span>
- <span class="o">--</span><span class="n">output</span> <span class="o"><</span><span class="n">PATH</span><span class="o">>/</span><span class="n">solr</span><span class="o">/</span><span class="n">wikipedia</span><span class="o">/</span><span class="n">out</span><span class="p">.</span><span class="n">txt</span>
+ <span class="o">--</span><span class="n">dictOut</span> $<span class="n">WORK_DIR</span><span class="o">/</span><span class="n">solr</span><span class="o">/</span><span class="n">wikipedia</span><span class="o">/</span><span class="n">dict</span><span class="p">.</span><span class="n">txt</span>
+ <span class="o">--</span><span class="n">output</span> $<span class="n">WORK_DIR</span><span class="o">/</span><span class="n">solr</span><span class="o">/</span><span class="n">wikipedia</span><span class="o">/</span><span class="n">out</span><span class="p">.</span><span class="n">txt</span>
<span class="o">--</span><span class="n">max</span> 50
<span class="o">--</span><span class="n">norm</span> 2
</pre></div>
@@ -355,7 +354,7 @@ the index are output.</p>
<p>Mahout has utilities to generate Vectors from a directory of text
documents. Before creating the vectors, you need to convert the documents
to SequenceFile format. SequenceFile is a hadoop class which allows us to
-write arbitary key,value pairs into it. The DocumentVectorizer requires the
+write arbitary (key, value) pairs into it. The DocumentVectorizer requires the
key to be a Text with a unique document id, and value to be the Text
content in UTF-8 format.</p>
<p>You may find <a href="http://tika.apache.org/">Tika</a> helpful in converting
@@ -363,9 +362,7 @@ binary documents to text.</p>
<p><a name="CreatingVectorsfromText-ConvertingdirectoryofdocumentstoSequenceFileformat"></a></p>
<h4 id="converting-directory-of-documents-to-sequencefile-format">Converting directory of documents to SequenceFile format</h4>
<p>Mahout has a nifty utility which reads a directory path including its
-sub-directories and creates the SequenceFile in a chunked manner for us.
-the document id generated is <PREFIX><RELATIVE PATH FROM
-PARENT>/document.txt</p>
+sub-directories and creates the SequenceFile in a chunked manner for us.</p>
<div class="codehilite"><pre>$<span class="n">MAHOUT_HOME</span><span class="o">/</span><span class="n">bin</span><span class="o">/</span><span class="n">mahout</span> <span class="n">seqdirectory</span>
<span class="o">--</span><span class="n">input</span> <span class="p">(</span><span class="o">-</span><span class="nb">i</span><span class="p">)</span> <span class="n">input</span> <span class="n">Path</span> <span class="n">to</span> <span class="n">job</span> <span class="n">input</span> <span class="n">directory</span><span class="p">.</span>
<span class="o">--</span><span class="n">output</span> <span class="p">(</span><span class="o">-</span><span class="n">o</span><span class="p">)</span> <span class="n">output</span> <span class="n">The</span> <span class="n">directory</span> <span class="n">pathname</span> <span class="k">for</span>
@@ -395,10 +392,11 @@ PARENT>/document.txt</p>
<span class="o">--</span><span class="n">help</span> <span class="p">(</span><span class="o">-</span><span class="n">h</span><span class="p">)</span> <span class="n">Print</span> <span class="n">out</span> <span class="n">help</span>
<span class="o">--</span><span class="n">tempDir</span> <span class="n">tempDir</span> <span class="n">Intermediate</span> <span class="n">output</span> <span class="n">directory</span>
<span class="o">--</span><span class="n">startPhase</span> <span class="n">startPhase</span> <span class="n">First</span> <span class="n">phase</span> <span class="n">to</span> <span class="n">run</span>
- <span class="o">--</span><span class="n">endPhase</span> <span class="n">endPhase</span> <span class="n">Last</span> <span class="n">phase</span> <span class="n">to</span> <span class="n">run</span> <span class="o">></span>
+ <span class="o">--</span><span class="n">endPhase</span> <span class="n">endPhase</span> <span class="n">Last</span> <span class="n">phase</span> <span class="n">to</span> <span class="n">run</span>
</pre></div>
+<p>The output of seqDirectory will be a Sequence file < Text, Text > of all documents (/sub-directory-path/documentFileName, documentText).</p>
<p><a name="CreatingVectorsfromText-CreatingVectorsfromSequenceFile"></a></p>
<h4 id="creating-vectors-from-sequencefile">Creating Vectors from SequenceFile</h4>
<p>From the sequence file generated from the above step run the following to
@@ -455,8 +453,35 @@ generate vectors. </p>
</pre></div>
-<p>--minSupport is the min frequency for the word to be considered as a feature. --minDF is the min number of documents the word needs to be in --maxDFPercent is the max value of the expression (document frequency of a word/total number of document) to be considered as good feature to be in the document. These options are helpful in removing high frequency features like stop words.
-<a name="CreatingVectorsfromText-Background"></a></p>
+<p>This will create SequenceFiles of tokenized documents < Text, StringTuple > (docID, tokenizedDoc) and vectorized documents < Text, VectorWritable > (docID, TF-IDF Vector). </p>
+<p>As well, seq2sparse will create SequenceFiles for: a dictionary (wordIndex, word), a word frequency count (wordIndex, count) and a document frequency count (wordIndex, DFCount) in the output directory. </p>
+<p>The --minSupport option is the min frequency for the word to be considered as a feature; --minDF is the min number of documents the word needs to be in; --maxDFPercent is the max value of the expression (document frequency of a word/total number of document) to be considered as good feature to be in the document. These options are helpful in removing high frequency features like stop words.</p>
+<p>The vectorized documents can then be used as input to many of Mahout's classification and clustering algorithms.</p>
+<h4 id="example-creating-normalized-tf-idf-vectors-from-a-directory-of-text-documents-using-trigrams-and-the-l_2-norm">Example: Creating Normalized <a href="http://en.wikipedia.org/wiki/Tf%E2%80%93idf">TF-IDF</a> Vectors from a directory of text documents using <a href="http://en.wikipedia.org/wiki/N-gram">trigrams</a> and the <a href="http://en.wikipedia.org/wiki/Lp_space">L_2 Norm</a></h4>
+<p>Create sequence files from the directory of text documents:</p>
+<div class="codehilite"><pre>$<span class="n">MAHOUT_HOME</span><span class="o">/</span><span class="n">bin</span><span class="o">/</span><span class="n">mahout</span> <span class="n">seqdirectory</span>
+ <span class="o">-</span><span class="nb">i</span> $<span class="n">WORK_DIR</span><span class="o">/</span><span class="n">reuters</span>
+ <span class="o">-</span><span class="n">o</span> $<span class="n">WORK_DIR</span><span class="o">/</span><span class="n">reuters</span><span class="o">-</span><span class="n">seqdir</span>
+ <span class="o">-</span><span class="n">c</span> <span class="n">UTF</span><span class="o">-</span>8
+ <span class="o">-</span><span class="n">chunk</span> 64
+ <span class="o">-</span><span class="n">xm</span> <span class="n">sequential</span>
+</pre></div>
+
+
+<p>Vectorize the documents using trigrams, L_2 length normalization and a maximum document frequency cutoff of 85%.</p>
+<div class="codehilite"><pre>$<span class="n">MAHOUT_HOME</span><span class="o">/</span><span class="n">bin</span><span class="o">/</span><span class="n">mahout</span> <span class="n">seq2sparse</span>
+ <span class="o">-</span><span class="nb">i</span> $<span class="n">WORK_DIR</span><span class="o">/</span><span class="n">reuters</span><span class="o">-</span><span class="n">out</span><span class="o">-</span><span class="n">seqdir</span><span class="o">/</span>
+ <span class="o">-</span><span class="n">o</span> $<span class="n">WORK_DIR</span><span class="o">/</span><span class="n">reuters</span><span class="o">-</span><span class="n">out</span><span class="o">-</span><span class="n">seqdir</span><span class="o">-</span><span class="n">sparse</span><span class="o">-</span><span class="n">kmeans</span>
+ <span class="o">--</span><span class="n">namedVec</span>
+ <span class="o">-</span><span class="n">wt</span> <span class="n">tfidf</span>
+ <span class="o">-</span><span class="n">ng</span> 3
+ <span class="o">-</span><span class="n">n</span> 2
+ <span class="o">--</span><span class="n">maxDFPercent</span> 85
+</pre></div>
+
+
+<p>The sequence file in the $WORK_DIR/reuters-out-seqdir-sparse-kmeans/tfidf-vectors directory can now be used as input to the Mahout <a href="http://mahout.apache.org/users/clustering/k-means-clustering.html">k-Means</a> clustering algorithm.</p>
+<p><a name="CreatingVectorsfromText-Background"></a></p>
<h2 id="background">Background</h2>
<ul>
<li><a href="http://markmail.org/thread/l5zi3yk446goll3o">Discussion on centroid calculations with sparse vectors</a></li>
@@ -469,7 +494,11 @@ question arises of how to convert the ve
format. Probably the easiest way to go would be to implement your own
Iterable<Vector> (called VectorIterable in the example below) and then
reuse the existing VectorWriter classes:</p>
-<div class="codehilite"><pre><span class="n">VectorWriter</span> <span class="n">vectorWriter</span> <span class="p">=</span> <span class="n">SequenceFile</span><span class="p">.</span><span class="n">createWriter</span><span class="p">(</span><span class="n">filesystem</span><span class="p">,</span> <span class="n">configuration</span><span class="p">,</span> <span class="n">outfile</span><span class="p">,</span> <span class="n">LongWritable</span><span class="p">.</span><span class="n">class</span><span class="p">,</span> <span class="n">SparseVector</span><span class="p">.</span><span class="n">class</span><span class="p">);</span>
+<div class="codehilite"><pre><span class="n">VectorWriter</span> <span class="n">vectorWriter</span> <span class="p">=</span> <span class="n">SequenceFile</span><span class="p">.</span><span class="n">createWriter</span><span class="p">(</span><span class="n">filesystem</span><span class="p">,</span>
+ <span class="n">configuration</span><span class="p">,</span>
+ <span class="n">outfile</span><span class="p">,</span>
+ <span class="n">LongWritable</span><span class="p">.</span><span class="n">class</span><span class="p">,</span>
+ <span class="n">SparseVector</span><span class="p">.</span><span class="n">class</span><span class="p">);</span>
<span class="n">long</span> <span class="n">numDocs</span> <span class="p">=</span> <span class="n">vectorWriter</span><span class="p">.</span><span class="n">write</span><span class="p">(</span><span class="n">new</span> <span class="n">VectorIterable</span><span class="p">(),</span> <span class="n">Long</span><span class="p">.</span><span class="n">MAX_VALUE</span><span class="p">);</span>
</pre></div>