You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mahout.apache.org by bu...@apache.org on 2013/11/20 17:06:05 UTC
svn commit: r887368 - in /websites/staging/mahout/trunk/content: ./
users/basics/collocations.html
Author: buildbot
Date: Wed Nov 20 16:06:04 2013
New Revision: 887368
Log:
Staging update by buildbot for mahout
Modified:
websites/staging/mahout/trunk/content/ (props changed)
websites/staging/mahout/trunk/content/users/basics/collocations.html
Propchange: websites/staging/mahout/trunk/content/
------------------------------------------------------------------------------
--- cms:source-revision (original)
+++ cms:source-revision Wed Nov 20 16:06:04 2013
@@ -1 +1 @@
-1543851
+1543854
Modified: websites/staging/mahout/trunk/content/users/basics/collocations.html
==============================================================================
--- websites/staging/mahout/trunk/content/users/basics/collocations.html (original)
+++ websites/staging/mahout/trunk/content/users/basics/collocations.html Wed Nov 20 16:06:04 2013
@@ -387,17 +387,17 @@
more often than would be expected by chance. Statistically relevant
combinations of terms identify additional lexical units which can be
treated as features in a vector-based representation of a text. A detailed
-discussion of collocations can be found on wikipedia <a href="http://en.wikipedia.org/wiki/Collocation">1</a>
-.</p>
+discussion of collocations can be found on <a href="http://en.wikipedia.org/wiki/Collocation">Wikipedia</a>.</p>
+<p>See there for a more detailed discussion of collocations in the <a href="http://comments.gmane.org/gmane.comp.apache.mahout.user/5685">Reuters example</a>.</p>
<p><a name="Collocations-Log-LikelihoodbasedCollocationIdentification"></a></p>
-<h2 id="log-likelihood-based-collocation-identification">Log-Likelihood based Collocation Identification</h2>
+<h2 id="theory-behind-implementation-log-likelihood-based-collocation-identification">Theory behind implementation: Log-Likelihood based Collocation Identification</h2>
<p>Mahout provides an implementation of a collocation identification algorithm
which scores collocations using log-likelihood ratio. The log-likelihood
score indicates the relative usefulness of a collocation with regards other
term combinations in the text. Collocations with the highest scores in a
particular corpus will generally be more useful as features.</p>
<p>Calculating the LLR is very straightforward and is described concisely in
-Ted Dunning's blog post <a href="http://tdunning.blogspot.com/2008/03/surprise-and-coincidence.html">2</a>
+<a href="http://tdunning.blogspot.com/2008/03/surprise-and-coincidence.html">Ted Dunning's blog post</a>
. Ted describes the series of counts reqired to calculate the LLR for two
events A and B in order to determine if they co-occur more often than pure
chance. These counts include the number of times the events co-occur (k11),
@@ -464,100 +464,99 @@ times. </p>
<div class="codehilite"><pre><span class="n">bin</span><span class="o">/</span><span class="n">mahout</span> <span class="n">seq2sparse</span>
<span class="n">Usage</span><span class="p">:</span>
- <span class="p">[</span><span class="o">--</span><span class="n">minSupport</span> <span class="o"><</span><span class="n">minSupport</span><span class="o">></span> <span class="o">--</span><span class="n">analyzerName</span> <span class="o"><</span><span class="n">analyzerName</span><span class="o">></span> <span class="o">--</span><span class="n">chunkSize</span>
-<span class="o"><</span><span class="n">chunkSize</span><span class="o">></span> <span class="o">--</span><span class="n">output</span> <span class="o"><</span><span class="n">output</span><span class="o">></span> <span class="o">--</span><span class="n">input</span> <span class="o"><</span><span class="n">input</span><span class="o">></span> <span class="o">--</span><span class="n">minDF</span> <span class="o"><</span><span class="n">minDF</span><span class="o">></span>
+ <span class="p">[</span><span class="o">--</span><span class="n">minSupport</span> <span class="o"><</span><span class="n">minSupport</span><span class="o">></span> <span class="o">--</span><span class="n">analyzerName</span> <span class="o"><</span><span class="n">analyzerName</span><span class="o">></span> <span class="o">--</span><span class="n">chunkSize</span> <span class="o"><</span><span class="n">chunkSize</span><span class="o">></span>
+ <span class="o">--</span><span class="n">output</span> <span class="o"><</span><span class="n">output</span><span class="o">></span> <span class="o">--</span><span class="n">input</span> <span class="o"><</span><span class="n">input</span><span class="o">></span> <span class="o">--</span><span class="n">minDF</span> <span class="o"><</span><span class="n">minDF</span><span class="o">></span>
+ <span class="o">--</span><span class="n">maxDFPercent</span><span class="o"><</span><span class="n">maxDFPercent</span><span class="o">></span> <span class="o">--</span><span class="n">weight</span> <span class="o"><</span><span class="n">weight</span><span class="o">></span> <span class="o">--</span><span class="n">norm</span> <span class="o"><</span><span class="n">norm</span><span class="o">></span> <span class="o">--</span><span class="n">minLLR</span> <span class="o"><</span><span class="n">minLLR</span><span class="o">></span>
+ <span class="o">--</span><span class="n">numReducers</span> <span class="o"><</span><span class="n">numReducers</span><span class="o">></span> <span class="o">--</span><span class="n">maxNGramSize</span> <span class="o"><</span><span class="n">ngramSize</span><span class="o">></span> <span class="o">--</span><span class="n">overwrite</span> <span class="o">--</span><span class="n">help</span>
+ <span class="o">--</span><span class="n">sequentialAccessVector</span><span class="p">]</span>
+<span class="n">Options</span>
+
+ <span class="o">--</span><span class="n">minSupport</span> <span class="p">(</span><span class="o">-</span><span class="n">s</span><span class="p">)</span> <span class="n">minSupport</span> <span class="p">(</span><span class="n">Optional</span><span class="p">)</span> <span class="n">Minimum</span> <span class="n">Support</span><span class="p">.</span> <span class="n">Default</span> <span class="n">Value</span><span class="p">:</span> 2
+
+ <span class="o">--</span><span class="n">analyzerName</span> <span class="p">(</span><span class="o">-</span><span class="n">a</span><span class="p">)</span> <span class="n">analyzerName</span> <span class="n">The</span> <span class="n">class</span> <span class="n">name</span> <span class="n">of</span> <span class="n">the</span> <span class="n">analyzer</span>
+
+ <span class="o">--</span><span class="n">chunkSize</span> <span class="p">(</span><span class="o">-</span><span class="n">chunk</span><span class="p">)</span> <span class="n">chunkSize</span> <span class="n">The</span> <span class="n">chunkSize</span> <span class="n">in</span> <span class="n">MegaBytes</span><span class="p">.</span> 100<span class="o">-</span>10000<span class="n">MB</span>
+
+ <span class="o">--</span><span class="n">output</span> <span class="p">(</span><span class="o">-</span><span class="n">o</span><span class="p">)</span> <span class="n">output</span> <span class="n">The</span> <span class="n">output</span> <span class="n">directory</span>
+
+ <span class="o">--</span><span class="n">input</span> <span class="p">(</span><span class="o">-</span><span class="nb">i</span><span class="p">)</span> <span class="n">input</span> <span class="n">Input</span> <span class="n">dir</span> <span class="n">containing</span> <span class="n">the</span> <span class="n">documents</span> <span class="n">in</span> <span class="n">sequence</span> <span class="n">file</span> <span class="n">format</span>
+
+ <span class="o">--</span><span class="n">minDF</span> <span class="p">(</span><span class="o">-</span><span class="n">md</span><span class="p">)</span> <span class="n">minDF</span> <span class="n">The</span> <span class="n">minimum</span> <span class="n">document</span> <span class="n">frequency</span><span class="p">.</span> <span class="n">Default</span> <span class="n">is</span> 1
+
+ <span class="o">--</span><span class="n">maxDFPercent</span> <span class="p">(</span><span class="o">-</span><span class="n">x</span><span class="p">)</span> <span class="n">maxDFPercent</span> <span class="n">The</span> <span class="n">max</span> <span class="n">percentage</span> <span class="n">of</span> <span class="n">docs</span> <span class="k">for</span> <span class="n">the</span> <span class="n">DF</span><span class="p">.</span> <span class="n">Can</span> <span class="n">be</span> <span class="n">used</span> <span class="n">to</span> <span class="n">remove</span>
+ <span class="n">really</span> <span class="n">high</span> <span class="n">frequency</span> <span class="n">terms</span><span class="p">.</span> <span class="n">Expressed</span> <span class="n">as</span> <span class="n">an</span>
+ <span class="n">integer</span> <span class="n">between</span> 0 <span class="n">and</span> 100<span class="p">.</span> <span class="n">Default</span> <span class="n">is</span> 99<span class="p">.</span>
+
+ <span class="o">--</span><span class="n">weight</span> <span class="p">(</span><span class="o">-</span><span class="n">wt</span><span class="p">)</span> <span class="n">weight</span> <span class="n">The</span> <span class="n">kind</span> <span class="n">of</span> <span class="n">weight</span> <span class="n">to</span> <span class="n">use</span><span class="p">.</span> <span class="n">Currently</span> <span class="n">TF</span>
+ <span class="n">or</span> <span class="n">TFIDF</span>
+
+ <span class="o">--</span><span class="n">norm</span> <span class="p">(</span><span class="o">-</span><span class="n">n</span><span class="p">)</span> <span class="n">norm</span> <span class="n">The</span> <span class="n">norm</span> <span class="n">to</span> <span class="n">use</span><span class="p">,</span> <span class="n">expressed</span> <span class="n">as</span> <span class="n">either</span> <span class="n">a</span>
+ <span class="n">float</span> <span class="n">or</span> "<span class="n">INF</span>" <span class="k">if</span> <span class="n">you</span> <span class="n">want</span> <span class="n">to</span> <span class="n">use</span> <span class="n">the</span>
+ <span class="n">Infinite</span> <span class="n">norm</span><span class="p">.</span> <span class="n">Must</span> <span class="n">be</span> <span class="n">greater</span> <span class="n">orequal</span>
+ <span class="n">to</span> 0<span class="p">.</span> <span class="n">The</span> <span class="n">default</span> <span class="n">is</span> <span class="n">not</span> <span class="n">to</span> <span class="n">normalize</span>
+
+ <span class="o">--</span><span class="n">minLLR</span> <span class="p">(</span><span class="o">-</span><span class="n">ml</span><span class="p">)</span> <span class="n">minLLR</span> <span class="p">(</span><span class="n">Optional</span><span class="p">)</span><span class="n">The</span> <span class="n">minimum</span> <span class="n">Log</span> <span class="n">Likelihood</span>
+ <span class="n">Ratio</span><span class="p">(</span><span class="n">Float</span><span class="p">)</span> <span class="n">Default</span> <span class="n">is</span> 1<span class="p">.</span>0
+
+ <span class="o">--</span><span class="n">numReducers</span> <span class="p">(</span><span class="o">-</span><span class="n">nr</span><span class="p">)</span> <span class="n">numReducers</span> <span class="p">(</span><span class="n">Optional</span><span class="p">)</span> <span class="n">Number</span> <span class="n">of</span> <span class="n">reduce</span> <span class="n">tasks</span><span class="p">.</span>
+ <span class="n">Default</span> <span class="n">Value</span><span class="p">:</span> 1
+
+ <span class="o">--</span><span class="n">maxNGramSize</span> <span class="p">(</span><span class="o">-</span><span class="n">ng</span><span class="p">)</span> <span class="n">ngramSize</span> <span class="p">(</span><span class="n">Optional</span><span class="p">)</span> <span class="n">The</span> <span class="n">maximum</span> <span class="nb">size</span> <span class="n">of</span> <span class="n">ngrams</span> <span class="n">to</span>
+ <span class="n">create</span> <span class="p">(</span>2 <span class="p">=</span> <span class="n">bigrams</span><span class="p">,</span> 3 <span class="p">=</span> <span class="n">trigrams</span><span class="p">,</span> <span class="n">etc</span><span class="p">)</span>
+ <span class="n">Default</span> <span class="n">Value</span><span class="p">:</span>2
+
+ <span class="o">--</span><span class="n">overwrite</span> <span class="p">(</span><span class="o">-</span><span class="n">w</span><span class="p">)</span> <span class="n">If</span> <span class="n">set</span><span class="p">,</span> <span class="n">overwrite</span> <span class="n">the</span> <span class="n">output</span> <span class="n">directory</span>
+ <span class="o">--</span><span class="n">help</span> <span class="p">(</span><span class="o">-</span><span class="n">h</span><span class="p">)</span> <span class="n">Print</span> <span class="n">out</span> <span class="n">help</span>
+ <span class="o">--</span><span class="n">sequentialAccessVector</span> <span class="p">(</span><span class="o">-</span><span class="n">seq</span><span class="p">)</span> <span class="p">(</span><span class="n">Optional</span><span class="p">)</span> <span class="n">Whether</span> <span class="n">output</span> <span class="n">vectors</span> <span class="n">should</span>
+ <span class="n">be</span> <span class="n">SequentialAccessVectors</span> <span class="n">If</span> <span class="n">set</span> <span class="n">true</span>
+ <span class="k">else</span> <span class="n">false</span>
</pre></div>
-<p>--maxDFPercent <br />
- <maxDFPercent> --weight <weight> --norm <norm> --minLLR <minLLR>
---numReducers<br />
- <numReducers> --maxNGramSize <ngramSize> --overwrite --help <br />
- --sequentialAccessVector]
- Options <br />
- --minSupport (-s) minSupport (Optional) Minimum Support. Default <br />
- Value: 2 <br />
- --analyzerName (-a) analyzerName The class name of the analyzer <br />
- --chunkSize (-chunk) chunkSize The chunkSize in MegaBytes. 100-10000
-MB<br />
- --output (-o) output The output directory <br />
- --input (-i) input input dir containing the documents in
- sequence file format <br />
- --minDF (-md) minDF The minimum document frequency.
-Default<br />
- is 1 <br />
- --maxDFPercent (-x) maxDFPercent The max percentage of docs for the
-DF. <br />
- Can be used to remove really high <br />
- frequency terms. Expressed as an
-integer<br />
- between 0 and 100. Default is 99. <br />
- --weight (-wt) weight The kind of weight to use. Currently
-TF <br />
- or TFIDF <br />
- --norm (-n) norm The norm to use, expressed as either
-a <br />
- float or "INF" if you want to use the
- Infinite norm. Must be greater or
-equal<br />
- to 0. The default is not to
-normalize <br />
- --minLLR (-ml) minLLR (Optional)The minimum Log Likelihood<br />
- Ratio(Float) Default is 1.0 <br />
- --numReducers (-nr) numReducers (Optional) Number of reduce tasks. <br />
- Default Value: 1 <br />
- --maxNGramSize (-ng) ngramSize (Optional) The maximum size of ngrams
-to<br />
- create (2 = bigrams, 3 = trigrams,
-etc) <br />
- Default Value:2 <br />
- --overwrite (-w) If set, overwrite the output
-directory <br />
- --help (-h) Print out help <br />
- --sequentialAccessVector (-seq) (Optional) Whether output vectors
-should<br />
- be SequentialAccessVectors If set
-true <br />
- else false </p>
<p><a name="Collocations-CollocDriver"></a></p>
<h3 id="collocdriver">CollocDriver</h3>
-<p><em>TODO</em></p>
<div class="codehilite"><pre><span class="n">bin</span><span class="o">/</span><span class="n">mahout</span> <span class="n">org</span><span class="p">.</span><span class="n">apache</span><span class="p">.</span><span class="n">mahout</span><span class="p">.</span><span class="n">vectorizer</span><span class="p">.</span><span class="n">collocations</span><span class="p">.</span><span class="n">llr</span><span class="p">.</span><span class="n">CollocDriver</span>
<span class="n">Usage</span><span class="p">:</span>
<span class="p">[</span><span class="o">--</span><span class="n">input</span> <span class="o"><</span><span class="n">input</span><span class="o">></span> <span class="o">--</span><span class="n">output</span> <span class="o"><</span><span class="n">output</span><span class="o">></span> <span class="o">--</span><span class="n">maxNGramSize</span> <span class="o"><</span><span class="n">ngramSize</span><span class="o">></span> <span class="o">--</span><span class="n">overwrite</span>
<span class="o">--</span><span class="n">minSupport</span> <span class="o"><</span><span class="n">minSupport</span><span class="o">></span> <span class="o">--</span><span class="n">minLLR</span> <span class="o"><</span><span class="n">minLLR</span><span class="o">></span> <span class="o">--</span><span class="n">numReducers</span> <span class="o"><</span><span class="n">numReducers</span><span class="o">></span>
<span class="o">--</span><span class="n">analyzerName</span> <span class="o"><</span><span class="n">analyzerName</span><span class="o">></span> <span class="o">--</span><span class="n">preprocess</span> <span class="o">--</span><span class="n">unigram</span> <span class="o">--</span><span class="n">help</span><span class="p">]</span>
-<span class="n">Options</span>
- <span class="o">--</span><span class="n">input</span> <span class="p">(</span><span class="o">-</span><span class="nb">i</span><span class="p">)</span> <span class="n">input</span> <span class="n">The</span> <span class="n">Path</span> <span class="k">for</span> <span class="n">input</span> <span class="n">files</span><span class="p">.</span>
- <span class="o">--</span><span class="n">output</span> <span class="p">(</span><span class="o">-</span><span class="n">o</span><span class="p">)</span> <span class="n">output</span> <span class="n">The</span> <span class="n">Path</span> <span class="n">write</span> <span class="n">output</span> <span class="n">to</span>
- <span class="o">--</span><span class="n">maxNGramSize</span> <span class="p">(</span><span class="o">-</span><span class="n">ng</span><span class="p">)</span> <span class="n">ngramSize</span> <span class="p">(</span><span class="n">Optional</span><span class="p">)</span> <span class="n">The</span> <span class="n">maximum</span> <span class="nb">size</span> <span class="n">of</span> <span class="n">ngrams</span>
+
+<span class="n">Options</span>
+
+ <span class="o">--</span><span class="n">input</span> <span class="p">(</span><span class="o">-</span><span class="nb">i</span><span class="p">)</span> <span class="n">input</span> <span class="n">The</span> <span class="n">Path</span> <span class="k">for</span> <span class="n">input</span> <span class="n">files</span><span class="p">.</span>
+
+ <span class="o">--</span><span class="n">output</span> <span class="p">(</span><span class="o">-</span><span class="n">o</span><span class="p">)</span> <span class="n">output</span> <span class="n">The</span> <span class="n">Path</span> <span class="n">write</span> <span class="n">output</span> <span class="n">to</span>
+
+ <span class="o">--</span><span class="n">maxNGramSize</span> <span class="p">(</span><span class="o">-</span><span class="n">ng</span><span class="p">)</span> <span class="n">ngramSize</span> <span class="p">(</span><span class="n">Optional</span><span class="p">)</span> <span class="n">The</span> <span class="n">maximum</span> <span class="nb">size</span> <span class="n">of</span> <span class="n">ngramsto</span>
+ <span class="n">create</span> <span class="p">(</span>2 <span class="p">=</span> <span class="n">bigrams</span><span class="p">,</span> 3 <span class="p">=</span> <span class="n">trigrams</span><span class="p">,</span><span class="n">etc</span><span class="p">)</span>
+ <span class="n">Default</span> <span class="n">Value</span><span class="p">:</span>2
+
+ <span class="o">--</span><span class="n">overwrite</span> <span class="p">(</span><span class="o">-</span><span class="n">w</span><span class="p">)</span> <span class="n">If</span> <span class="n">set</span><span class="p">,</span> <span class="n">overwrite</span> <span class="n">the</span> <span class="n">outputdirectory</span>
+
+ <span class="o">--</span><span class="n">minSupport</span> <span class="p">(</span><span class="o">-</span><span class="n">s</span><span class="p">)</span> <span class="n">minSupport</span> <span class="p">(</span><span class="n">Optional</span><span class="p">)</span> <span class="n">Minimum</span> <span class="n">Support</span><span class="p">.</span> <span class="n">Default</span>
+ <span class="n">Value</span><span class="p">:</span> 2
+
+ <span class="o">--</span><span class="n">minLLR</span> <span class="p">(</span><span class="o">-</span><span class="n">ml</span><span class="p">)</span> <span class="n">minLLR</span> <span class="p">(</span><span class="n">Optional</span><span class="p">)</span><span class="n">The</span> <span class="n">minimum</span> <span class="n">Log</span> <span class="n">Likelihood</span>
+ <span class="n">Ratio</span><span class="p">(</span><span class="n">Float</span><span class="p">)</span> <span class="n">Default</span> <span class="n">is</span> 1<span class="p">.</span>0
+
+ <span class="o">--</span><span class="n">numReducers</span> <span class="p">(</span><span class="o">-</span><span class="n">nr</span><span class="p">)</span> <span class="n">numReducers</span> <span class="p">(</span><span class="n">Optional</span><span class="p">)</span> <span class="n">Number</span> <span class="n">of</span> <span class="n">reduce</span> <span class="n">tasks</span><span class="p">.</span>
+ <span class="n">Default</span> <span class="n">Value</span><span class="p">:</span> 1
+
+ <span class="o">--</span><span class="n">analyzerName</span> <span class="p">(</span><span class="o">-</span><span class="n">a</span><span class="p">)</span> <span class="n">analyzerName</span> <span class="n">The</span> <span class="n">class</span> <span class="n">name</span> <span class="n">of</span> <span class="n">the</span> <span class="n">analyzer</span>
+
+ <span class="o">--</span><span class="n">preprocess</span> <span class="p">(</span><span class="o">-</span><span class="n">p</span><span class="p">)</span> <span class="n">If</span> <span class="n">set</span><span class="p">,</span> <span class="n">input</span> <span class="n">is</span> <span class="n">SequenceFile</span><span class="o"><</span><span class="n">Text</span><span class="p">,</span><span class="n">Text</span><span class="o">></span>
+ <span class="n">where</span> <span class="n">the</span> <span class="n">value</span> <span class="n">is</span> <span class="n">the</span> <span class="n">document</span><span class="p">,</span> <span class="n">which</span>
+ <span class="n">will</span> <span class="n">be</span> <span class="n">tokenized</span> <span class="n">using</span> <span class="n">the</span> <span class="n">specified</span>
+ <span class="n">analyzer</span><span class="p">.</span>
+
+ <span class="o">--</span><span class="n">unigram</span> <span class="p">(</span><span class="o">-</span><span class="n">u</span><span class="p">)</span> <span class="n">If</span> <span class="n">set</span><span class="p">,</span> <span class="n">unigrams</span> <span class="n">will</span> <span class="n">be</span> <span class="n">emitted</span> <span class="n">inthe</span>
+ <span class="n">final</span> <span class="n">output</span> <span class="n">alongside</span> <span class="n">collocations</span>
+
+ <span class="o">--</span><span class="n">help</span> <span class="p">(</span><span class="o">-</span><span class="n">h</span><span class="p">)</span> <span class="n">Print</span> <span class="n">out</span> <span class="n">help</span>
</pre></div>
-<p>to<br />
- create (2 = bigrams, 3 = trigrams,
-etc) <br />
- Default Value:2 <br />
- --overwrite (-w) If set, overwrite the output
-directory <br />
- --minSupport (-s) minSupport (Optional) Minimum Support. Default <br />
- Value: 2 <br />
- --minLLR (-ml) minLLR (Optional)The minimum Log Likelihood<br />
- Ratio(Float) Default is 1.0 <br />
- --numReducers (-nr) numReducers (Optional) Number of reduce tasks. <br />
- Default Value: 1 <br />
- --analyzerName (-a) analyzerName The class name of the analyzer <br />
- --preprocess (-p) If set, input is
-SequenceFile<Text,Text><br />
- where the value is the document,
-which <br />
- will be tokenized using the specified
- analyzer. <br />
- --unigram (-u) If set, unigrams will be emitted in
-the <br />
- final output alongside collocations <br />
- --help (-h) Print out help </p>
<p><a name="Collocations-Algorithmdetails"></a></p>
<h2 id="algorithm-details">Algorithm details</h2>
<p>This section describes the implementation of the collocation identification
@@ -580,8 +579,11 @@ frequencies are collected across the ent
<p>Once this is done, ngrams are split into head and tail portions. A key of type GramKey is generated which is used later to join ngrams with their heads and tails in the reducer phase. The GramKey is a composite key made up of a string n-gram fragement as the primary key and a secondary key used for grouping and sorting in the reduce phase. The secondary key will either be EMPTY in the case where we are collecting either the head or tail of an ngram as the value or it will contain the byte<a href=".html"></a>
form of the ngram when collecting an ngram as the value.</p>
<div class="codehilite"><pre><span class="n">head_key</span><span class="p">(</span><span class="n">EMPTY</span><span class="p">)</span> <span class="o">-></span> <span class="p">(</span><span class="n">head</span> <span class="n">subgram</span><span class="p">,</span> <span class="n">head</span> <span class="n">frequency</span><span class="p">)</span>
-<span class="n">head_key</span><span class="p">(</span><span class="n">ngram</span><span class="p">)</span> <span class="o">-></span> <span class="p">(</span><span class="n">ngram</span><span class="p">,</span> <span class="n">ngram</span> <span class="n">frequency</span><span class="p">)</span>
+
+<span class="n">head_key</span><span class="p">(</span><span class="n">ngram</span><span class="p">)</span> <span class="o">-></span> <span class="p">(</span><span class="n">ngram</span><span class="p">,</span> <span class="n">ngram</span> <span class="n">frequency</span><span class="p">)</span>
+
<span class="n">tail_key</span><span class="p">(</span><span class="n">EMPTY</span><span class="p">)</span> <span class="o">-></span> <span class="p">(</span><span class="n">tail</span> <span class="n">subgram</span><span class="p">,</span> <span class="n">tail</span> <span class="n">frequency</span><span class="p">)</span>
+
<span class="n">tail_key</span><span class="p">(</span><span class="n">ngram</span><span class="p">)</span> <span class="o">-></span> <span class="p">(</span><span class="n">ngram</span><span class="p">,</span> <span class="n">ngram</span> <span class="n">frequency</span><span class="p">)</span>
</pre></div>
@@ -699,18 +701,6 @@ the Skipped.LESS_THAN_MIN_LLR counter is
CollocDriver, unigrams (single tokens) will be passed through the job and
each token's frequency will be calculated. As with ngrams, unigrams are
subject to filtering with minSupport and minLLR.</p>
-<p><a name="Collocations-References"></a></p>
-<h2 id="references">References</h2>
-<p>[1](1.html)
- http://en.wikipedia.org/wiki/Collocation
-[2](2.html)
- http://tdunning.blogspot.com/2008/03/surprise-and-coincidence.html</p>
-<p><a name="Collocations-Discussion"></a></p>
-<h2 id="discussion">Discussion</h2>
-<ul>
-<li>http://comments.gmane.org/gmane.comp.apache.mahout.user/5685 - Reuters
-example</li>
-</ul>
</div>
</div>
</div>