You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mahout.apache.org by ra...@apache.org on 2017/05/13 08:00:58 UTC

[06/51] [abbrv] [partial] mahout git commit: WEBSITE Emergency Patches for go-live closes apache/mahout#317

http://git-wip-us.apache.org/repos/asf/mahout/blob/7c0babd7/website/oldsite/_site/users/basics/algorithms.html
----------------------------------------------------------------------
diff --git a/website/oldsite/_site/users/basics/algorithms.html b/website/oldsite/_site/users/basics/algorithms.html
deleted file mode 100644
index a253f50..0000000
--- a/website/oldsite/_site/users/basics/algorithms.html
+++ /dev/null
@@ -1,615 +0,0 @@
-
-
-<!DOCTYPE html>
-<html lang="en">
-<head>
-  <meta charset="utf-8">
-  <meta http-equiv="X-UA-Compatible" content="IE=edge">
-
-  <title>Algorithms</title>
-  
-  <meta name="author" content="The Apache Software Foundation">
-
-  <!-- Enable responsive viewport -->
-  <meta name="viewport" content="width=device-width, initial-scale=1.0">
-
-  <!-- Bootstrap styles -->
-  <link href="/assets/themes/mahout3/css/bootstrap.min.css" rel="stylesheet">
-  <!-- Optional theme -->
-  <link href="/assets/themes/mahout3/css/bootstrap-theme.min.css" rel="stylesheet">
-  <!-- Sticky Footer -->
-  <link href="/assets/themes/mahout3/css/bs-sticky-footer.css" rel="stylesheet">
-
-  <!-- Custom styles -->
-  <link href="/assets/themes/mahout3/css/style.css" rel="stylesheet" type="text/css" media="all">
-
-  <!-- HTML5 Shim and Respond.js IE8 support of HTML5 elements and media queries -->
-  <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
-  <!--[if lt IE 9]>
-  <script src="https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"></script>
-  <script src="https://oss.maxcdn.com/libs/respond.js/1.3.0/respond.min.js"></script>
-  <![endif]-->
-
-  <!-- Fav and touch icons -->
-  <!-- Update these with your own images
-    <link rel="shortcut icon" href="images/favicon.ico">
-    <link rel="apple-touch-icon" href="images/apple-touch-icon.png">
-    <link rel="apple-touch-icon" sizes="72x72" href="images/apple-touch-icon-72x72.png">
-    <link rel="apple-touch-icon" sizes="114x114" href="images/apple-touch-icon-114x114.png">
-  -->
-
-  <!-- atom & rss feed -->
-  <link href="/atom.xml" type="application/atom+xml" rel="alternate" title="Sitewide ATOM Feed">
-  <link href="/rss.xml" type="application/rss+xml" rel="alternate" title="Sitewide RSS Feed">
-  <script type="text/x-mathjax-config">
-  MathJax.Hub.Config({
-    tex2jax: {
-      skipTags: ['script', 'noscript', 'style', 'textarea', 'pre']
-    }
-  });
-  MathJax.Hub.Queue(function() {
-    var all = MathJax.Hub.getAllJax(), i;
-    for(i = 0; i < all.length; i += 1) {
-      all[i].SourceElement().parentNode.className += ' has-jax';
-    }
-  });
-  </script>
-  <script type="text/javascript">
-    var mathjax = document.createElement('script');
-    mathjax.type = 'text/javascript';
-    mathjax.async = true;
-
-    mathjax.src = ('https:' == document.location.protocol) ?
-        'https://c328740.ssl.cf1.rackcdn.com/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML' :
-        'http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML';
-
-      var s = document.getElementsByTagName('script')[0];
-    s.parentNode.insertBefore(mathjax, s);
-  </script>
-</head>
-
-<nav class="navbar navbar-default navbar-fixed-top">
-  <div class="container-fluid">
-    <!-- Brand and toggle get grouped for better mobile display -->
-    <div class="navbar-header">
-      <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1" aria-expanded="false">
-        <span class="sr-only">Toggle navigation</span>
-        <span class="icon-bar"></span>
-        <span class="icon-bar"></span>
-        <span class="icon-bar"></span>
-      </button>
-      <a class="navbar-brand" href="/">
-        <img src="/assets/img/Mahout-logo-82x100.png" height="30" alt="I'm mahout">
-      </a>
-    </div>
-
-    <!--<div class="nav-collapse collapse">-->
-<div class="collapse navbar-collapse" id="main-navbar">
-    <ul class="nav navbar-nav">
-        <!-- <li><a href="/">Home</a></li> -->
-        <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">General<b class="caret"></b></a>
-            <ul class="dropdown-menu">
-                <li><a href="/general/downloads.html">Downloads</a>
-                <li><a href="/general/who-we-are.html">Who we are</a>
-                <li><a href="/general/mailing-lists,-irc-and-archives.html">Mailing Lists</a>
-                <li><a href="/general/release-notes.html">Release Notes</a>
-                <li><a href="/general/books-tutorials-and-talks.html">Books, Tutorials, Talks</a></li>
-                <li><a href="/general/powered-by-mahout.html">Powered By Mahout</a>
-                <li><a href="/general/professional-support.html">Professional Support</a>
-                <li class="divider"></li>
-                <li class="nav-header">Resources</li>
-                <li><a href="/general/reference-reading.html">Reference Reading</a>
-                <li><a href="/general/faq.html">FAQ</a>
-                <li class="divider"></li>
-                <li class="nav-header">Legal</li>
-                <li><a href="http://www.apache.org/licenses/">License</a></li>
-                <li><a href="http://www.apache.org/security/">Security</a></li>
-                <li><a href="/general/privacy-policy.html">Privacy Policy</a>
-            </ul>
-        </li>
-        <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developers<b class="caret"></b></a>
-            <ul class="dropdown-menu">
-                <li><a href="/developers/developer-resources.html">Developer resources</a></li>
-                <li><a href="/developers/version-control.html">Version control</a></li>
-                <li><a href="/developers/buildingmahout.html">Build from source</a></li>
-                <li><a href="/developers/issue-tracker.html">Issue tracker</a></li>
-                <li><a href="https://builds.apache.org/job/Mahout-Quality/" target="_blank">Code quality reports</a></li>
-                <li class="divider"></li>
-                <li class="nav-header">Contributions</li>
-                <li><a href="/developers/how-to-contribute.html">How to contribute</a></li>
-                <li><a href="/developers/how-to-become-a-committer.html">How to become a committer</a></li>
-                <li><a href="/developers/gsoc.html">GSoC</a></li>
-                <li class="divider"></li>
-                <li class="nav-header">For committers</li>
-                <li><a href="/developers/how-to-update-the-website.html">How to update the website</a></li>
-                <li><a href="/developers/patch-check-list.html">Patch check list</a></li>
-                <li><a href="/developers/github.html">Handling Github PRs</a></li>
-                <li><a href="/developers/how-to-release.html">How to release</a></li>
-                <li><a href="/developers/thirdparty-dependencies.html">Third party dependencies</a></li>
-            </ul>
-        </li>
-        <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Mahout-Samsara<b class="caret"></b></a>
-            <ul class="dropdown-menu">
-                <li><a href="/users/sparkbindings/home.html">Scala &amp; Spark Bindings Overview</a></li>
-                <li><a href="/users/sparkbindings/faq.html">FAQ</a></li>
-                <li><a href="/users/flinkbindings/playing-with-samsara-flink.html">Flink Bindings Overview</a></li>
-                <li class="nav-header">Engines</li>
-                <li><a href="/users/sparkbindings/home.html">Spark</a></li>
-                <li><a href="/users/environment/h2o-internals.html">H2O</a></li>
-                <li><a href="/users/flinkbindings/flink-internals.html">Flink</a></li>
-                <li class="nav-header">References</li>
-                <li><a href="/users/environment/in-core-reference.html">In-Core Algebraic DSL Reference</a></li>
-                <li><a href="/users/environment/out-of-core-reference.html">Distributed Algebraic DSL Reference</a></li>
-                <li class="nav-header">Tutorials</li>
-                <li><a href="/users/sparkbindings/play-with-shell.html">Playing with Mahout's Spark Shell</a></li>
-                <li><a href="/users/environment/how-to-build-an-app.html">How to build an app</a></li>
-                <li><a href="/users/environment/classify-a-doc-from-the-shell.html">Building a text classifier in Mahout's Spark Shell</a></li>
-            </ul>
-        </li>
-        <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Algorithms<b class="caret"></b></a>
-            <ul class="dropdown-menu">
-                <li><a href="/users/basics/algorithms.html">List of algorithms</a>
-                <li class="nav-header">Distributed Matrix Decomposition</li>
-                <li><a href="/users/algorithms/d-qr.html">Cholesky QR</a></li>
-                <li><a href="/users/algorithms/d-ssvd.html">SSVD</a></li>
-                <li><a href="/users/algorithms/d-als.html">Distributed ALS</a></li>
-                <li><a href="/users/algorithms/d-spca.html">SPCA</a></li>
-                <li class="nav-header">Recommendations</li>
-                <li><a href="/users/algorithms/recommender-overview.html">Recommender Overview</a></li>
-                <li><a href="/users/algorithms/intro-cooccurrence-spark.html">Intro to cooccurrence-based<br/> recommendations with Spark</a></li>
-                <li class="nav-header">Classification</li>
-                <li><a href="/users/algorithms/spark-naive-bayes.html">Spark Naive Bayes</a></li>
-            </ul>
-        </li>
-        <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">MapReduce Basics<b class="caret"></b></a>
-            <ul class="dropdown-menu">
-                <li><a href="/users/basics/algorithms.html">List of algorithms</a>
-                <li><a href="/users/basics/quickstart.html">Overview</a>
-                <li class="divider"></li>
-                <li class="nav-header">Working with text</li>
-                <li><a href="/users/basics/creating-vectors-from-text.html">Creating vectors from text</a>
-                <li><a href="/users/basics/collocations.html">Collocations</a>
-                <li class="divider"></li>
-                <li class="nav-header">Dimensionality reduction</li>
-                <li><a href="/users/dim-reduction/dimensional-reduction.html">Singular Value Decomposition</a></li>
-                <li><a href="/users/dim-reduction/ssvd.html">Stochastic SVD</a></li>
-                <li class="divider"></li>
-                <li class="nav-header">Topic Models</li>
-                <li><a href="/users/clustering/latent-dirichlet-allocation.html">Latent Dirichlet Allocation</a></li>
-            </ul>
-        </li>
-        <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Mahout MapReduce<b class="caret"></b></a>
-            <ul class="dropdown-menu">
-                <li class="nav-header">Classification</li>
-                <li><a href="/users/classification/bayesian.html">Naive Bayes</a></li>
-                <li><a href="/users/classification/hidden-markov-models.html">Hidden Markov Models</a></li>
-                <li><a href="/users/classification/logistic-regression.html">Logistic Regression (Single Machine)</a></li>
-                <li><a href="/users/classification/partial-implementation.html">Random Forest</a></li>
-                <li class="nav-header">Classification Examples</li>
-                <li><a href="/users/classification/breiman-example.html">Breiman example</a></li>
-                <li><a href="/users/classification/twenty-newsgroups.html">20 newsgroups example</a></li>
-                <li><a href="/users/classification/bankmarketing-example.html">SGD classifier bank marketing</a></li>
-                <li><a href="/users/classification/wikipedia-classifier-example.html">Wikipedia XML parser and classifier</a></li>
-                <li class="nav-header">Clustering</li>
-                <li><a href="/users/clustering/k-means-clustering.html">k-Means</a></li>
-                <li><a href="/users/clustering/canopy-clustering.html">Canopy</a></li>
-                <li><a href="/users/clustering/fuzzy-k-means.html">Fuzzy k-Means</a></li>
-                <li><a href="/users/clustering/streaming-k-means.html">Streaming KMeans</a></li>
-                <li><a href="/users/clustering/spectral-clustering.html">Spectral Clustering</a></li>
-                <li class="nav-header">Clustering Commandline usage</li>
-                <li><a href="/users/clustering/k-means-commandline.html">Options for k-Means</a></li>
-                <li><a href="/users/clustering/canopy-commandline.html">Options for Canopy</a></li>
-                <li><a href="/users/clustering/fuzzy-k-means-commandline.html">Options for Fuzzy k-Means</a></li>
-                <li class="nav-header">Clustering Examples</li>
-                <li><a href="/users/clustering/clustering-of-synthetic-control-data.html">Synthetic data</a></li>
-                <li class="nav-header">Cluster Post processing</li>
-                <li><a href="/users/clustering/cluster-dumper.html">Cluster Dumper tool</a></li>
-                <li><a href="/users/clustering/visualizing-sample-clusters.html">Cluster visualisation</a></li>
-                <li class="nav-header">Recommendations</li>
-                <li><a href="/users/recommender/recommender-first-timer-faq.html">First Timer FAQ</a></li>
-                <li><a href="/users/recommender/userbased-5-minutes.html">A user-based recommender <br/>in 5 minutes</a></li>
-                <li><a href="/users/recommender/matrix-factorization.html">Matrix factorization-based<br/> recommenders</a></li>
-                <li><a href="/users/recommender/recommender-documentation.html">Overview</a></li>
-                <li><a href="/users/recommender/intro-itembased-hadoop.html">Intro to item-based recommendations<br/> with Hadoop</a></li>
-                <li><a href="/users/recommender/intro-als-hadoop.html">Intro to ALS recommendations<br/> with Hadoop</a></li>
-            </ul>
-        </li>
-        <!--  <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Recommendations<b class="caret"></b></a>
-          <ul class="dropdown-menu">
-
-          </ul> -->
-        </li>
-    </ul>
-</div><!--/.nav-collapse -->
-  </div><!-- /.container-fluid -->
-</nav>
-
-<body>
-
-<div id="wrap">
-  <body class="">
-
-  <div class="container">
-    <hr />
-<p><em>Mahout 0.12.0 Features by Engine</em> <br />
-—</p>
-
-<table>
-  <thead>
-    <tr>
-      <th> </th>
-      <th style="text-align: center"><strong>Single Machine</strong></th>
-      <th style="text-align: center"><a href="http://hadoop.apache.org/"><strong>MapReduce</strong></a></th>
-      <th style="text-align: center"><a href="https://spark.apache.org/"><strong>Spark</strong></a></th>
-      <th style="text-align: center"><a href="http://0xdata.com/"><strong>H2O</strong></a></th>
-      <th style="text-align: center"><a href="https://flink.apache.org/"><strong>Flink</strong></a></th>
-    </tr>
-  </thead>
-  <tbody>
-    <tr>
-      <td><strong>Mahout Math-Scala Core Library and Scala DSL</strong></td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-    </tr>
-    <tr>
-      <td><a href="http://mahout.apache.org/users/sparkbindings/home.html">Mahout Distributed BLAS. Distributed Row Matrix API with R and Matlab like operators. Distributed ALS, SPCA, SSVD, thin-QR. Similarity Analysis</a>.</td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"><a href="https://mahout.apache.org/users/sparkbindings/ScalaSparkBindings.pdf">x</a></td>
-      <td style="text-align: center"><a href="https://github.com/apache/mahout/tree/master/h2o">x</a></td>
-      <td style="text-align: center"><a href="https://github.com/apache/mahout/tree/flink-binding/flink">x</a></td>
-    </tr>
-    <tr>
-      <td> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-    </tr>
-    <tr>
-      <td><strong>Mahout Interactive Shell</strong></td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-    </tr>
-    <tr>
-      <td><a href="http://mahout.apache.org/users/sparkbindings/play-with-shell.html">Interactive REPL shell for Spark optimized Mahout DSL</a></td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center">x</td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-    </tr>
-    <tr>
-      <td> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-    </tr>
-    <tr>
-      <td><strong>Collaborative Filtering</strong> <em>with CLI drivers</em></td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-    </tr>
-    <tr>
-      <td>User-Based Collaborative Filtering</td>
-      <td style="text-align: center"><em>deprecated</em></td>
-      <td style="text-align: center"><em>deprecated</em></td>
-      <td style="text-align: center"><a href="https://mahout.apache.org/users/algorithms/intro-cooccurrence-spark.html">x</a></td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-    </tr>
-    <tr>
-      <td>Item-Based Collaborative Filtering</td>
-      <td style="text-align: center">x</td>
-      <td style="text-align: center"><a href="https://mahout.apache.org/users/recommender/intro-itembased-hadoop.html">x</a></td>
-      <td style="text-align: center"><a href="https://mahout.apache.org/users/algorithms/intro-cooccurrence-spark.html">x</a></td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-    </tr>
-    <tr>
-      <td>Matrix Factorization with ALS</td>
-      <td style="text-align: center">x</td>
-      <td style="text-align: center"><a href="https://mahout.apache.org/users/recommender/intro-als-hadoop.html">x</a></td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-    </tr>
-    <tr>
-      <td>Matrix Factorization with ALS on Implicit Feedback</td>
-      <td style="text-align: center">x</td>
-      <td style="text-align: center"><a href="https://mahout.apache.org/users/recommender/intro-als-hadoop.html">x</a></td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-    </tr>
-    <tr>
-      <td>Weighted Matrix Factorization, SVD++</td>
-      <td style="text-align: center">x</td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-    </tr>
-    <tr>
-      <td> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-    </tr>
-    <tr>
-      <td><strong>Classification</strong> <em>with CLI drivers</em></td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-    </tr>
-    <tr>
-      <td>Logistic Regression - trained via SGD</td>
-      <td style="text-align: center"><a href="http://mahout.apache.org/users/classification/logistic-regression.html"><em>deprecated</em></a></td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-    </tr>
-    <tr>
-      <td>Naive Bayes / Complementary Naive Bayes</td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"><a href="https://mahout.apache.org/users/classification/bayesian.html"><em>deprecated</em></a></td>
-      <td style="text-align: center"><a href="https://mahout.apache.org/users/algorithms/spark-naive-bayes.html">x</a></td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-    </tr>
-    <tr>
-      <td>Hidden Markov Models</td>
-      <td style="text-align: center"><a href="https://mahout.apache.org/users/classification/hidden-markov-models.html"><em>deprecated</em></a></td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-    </tr>
-    <tr>
-      <td> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-    </tr>
-    <tr>
-      <td><strong>Clustering</strong> <em>with CLI drivers</em></td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-    </tr>
-    <tr>
-      <td>Canopy Clustering</td>
-      <td style="text-align: center"><a href="https://mahout.apache.org/users/clustering/canopy-clustering.html"><em>deprecated</em></a></td>
-      <td style="text-align: center"><a href="https://mahout.apache.org/users/clustering/canopy-clustering.html"><em>deprecated</em></a></td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-    </tr>
-    <tr>
-      <td>k-Means Clustering</td>
-      <td style="text-align: center"><a href="https://mahout.apache.org/users/clustering/k-means-clustering.html"><em>deprecated</em></a></td>
-      <td style="text-align: center"><a href="https://mahout.apache.org/users/clustering/k-means-clustering.html"><em>deprecated</em></a></td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-    </tr>
-    <tr>
-      <td>Fuzzy k-Means</td>
-      <td style="text-align: center"><a href="https://mahout.apache.org/users/clustering/fuzzy-k-means.html"><em>deprecated</em></a></td>
-      <td style="text-align: center"><a href="https://mahout.apache.org/users/clustering/fuzzy-k-means.html"><em>deprecated</em></a></td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-    </tr>
-    <tr>
-      <td>Streaming k-Means</td>
-      <td style="text-align: center"><a href="https://mahout.apache.org/users/clustering/streaming-k-means.html"><em>deprecated</em></a></td>
-      <td style="text-align: center"><a href="https://mahout.apache.org/users/clustering/streaming-k-means.html"><em>deprecated</em></a></td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-    </tr>
-    <tr>
-      <td>Spectral Clustering</td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"><a href="https://mahout.apache.org/users/clustering/spectral-clustering.html"><em>deprecated</em></a></td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-    </tr>
-    <tr>
-      <td> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-    </tr>
-    <tr>
-      <td><strong>Dimensionality Reduction</strong> <em>note: most scala-based dimensionality reduction algorithms are available through the <a href="https://mahout.apache.org/users/sparkbindings/home.html">Mahout Math-Scala Core Library for all engines</a></em></td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-    </tr>
-    <tr>
-      <td>Singular Value Decomposition</td>
-      <td style="text-align: center"><em>deprecated</em></td>
-      <td style="text-align: center"><em>deprecated</em></td>
-      <td style="text-align: center"><a href="http://mahout.apache.org/users/sparkbindings/home.html">x</a></td>
-      <td style="text-align: center"><a href="http://mahout.apache.org/users/environment/h2o-internals.html">x</a></td>
-      <td style="text-align: center"><a href="http://mahout.apache.org/users/flinkbindings/flink-internals.html">x</a></td>
-    </tr>
-    <tr>
-      <td>Lanczos Algorithm</td>
-      <td style="text-align: center"><em>deprecated</em></td>
-      <td style="text-align: center"><em>deprecated</em></td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-    </tr>
-    <tr>
-      <td>Stochastic SVD</td>
-      <td style="text-align: center"><a href="https://mahout.apache.org/users/dim-reduction/ssvd.html"><em>deprecated</em></a></td>
-      <td style="text-align: center"><a href="https://mahout.apache.org/users/dim-reduction/ssvd.html"><em>deprecated</em></a></td>
-      <td style="text-align: center"><a href="http://mahout.apache.org/users/algorithms/d-ssvd.html">x</a></td>
-      <td style="text-align: center"><a href="http://mahout.apache.org/users/algorithms/d-ssvd.html">x</a></td>
-      <td style="text-align: center"><a href="http://mahout.apache.org/users/algorithms/d-ssvd.html">x</a></td>
-    </tr>
-    <tr>
-      <td>PCA (via Stochastic SVD)</td>
-      <td style="text-align: center"><em>deprecated</em></td>
-      <td style="text-align: center"><em>deprecated</em></td>
-      <td style="text-align: center"><a href="http://mahout.apache.org/users/sparkbindings/home.html">x</a></td>
-      <td style="text-align: center"><a href="http://mahout.apache.org/users/environment/h2o-internals.html">x</a></td>
-      <td style="text-align: center"><a href="http://mahout.apache.org/users/flinkbindings/flink-internals.html">x</a></td>
-    </tr>
-    <tr>
-      <td>QR Decomposition</td>
-      <td style="text-align: center"><em>deprecated</em></td>
-      <td style="text-align: center"><em>deprecated</em></td>
-      <td style="text-align: center"><a href="http://mahout.apache.org/users/algorithms/d-qr.html">x</a></td>
-      <td style="text-align: center"><a href="http://mahout.apache.org/users/algorithms/d-qr.html">x</a></td>
-      <td style="text-align: center"><a href="http://mahout.apache.org/users/algorithms/d-qr.html">x</a></td>
-    </tr>
-    <tr>
-      <td> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-    </tr>
-    <tr>
-      <td><strong>Topic Models</strong></td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-    </tr>
-    <tr>
-      <td>Latent Dirichlet Allocation</td>
-      <td style="text-align: center"><em>deprecated</em></td>
-      <td style="text-align: center"><em>deprecated</em></td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-    </tr>
-    <tr>
-      <td> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-    </tr>
-    <tr>
-      <td><strong>Miscellaneous</strong></td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-    </tr>
-    <tr>
-      <td>RowSimilarityJob</td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"><em>deprecated</em></td>
-      <td style="text-align: center"><a href="https://github.com/apache/mahout/blob/master/spark/src/test/scala/org/apache/mahout/drivers/RowSimilarityDriverSuite.scala">x</a></td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-    </tr>
-    <tr>
-      <td>Collocations</td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"><a href="https://mahout.apache.org/users/basics/collocations.html"><em>deprecated</em></a></td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-    </tr>
-    <tr>
-      <td>Sparse TF-IDF Vectors from Text</td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"><a href="https://mahout.apache.org/users/basics/creating-vectors-from-text.html"><em>deprecated</em></a></td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-    </tr>
-    <tr>
-      <td>XML Parsing</td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"><a href="https://issues.apache.org/jira/browse/MAHOUT-1479?jql=text%20~%20%22wikipedia%20mahout%22"><em>deprecated</em></a></td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-    </tr>
-    <tr>
-      <td>Email Archive Parsing</td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"><a href="https://github.com/apache/mahout/tree/master/integration/src/main/java/org/apache/mahout/text"><em>deprecated</em></a></td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-    </tr>
-    <tr>
-      <td>Evolutionary Processes</td>
-      <td style="text-align: center"><a href="https://github.com/apache/mahout/tree/master/mr/src/main/java/org/apache/mahout/ep">x</a></td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-      <td style="text-align: center"> </td>
-    </tr>
-  </tbody>
-</table>
-
-
-  </div>
-
-
-</div>
-
-<div id="footer">
-  <div class="container">
-    <p>&copy; 2017 The Apache Software Foundation
-      with help from <a href="http://jekyllbootstrap.com" target="_blank" title="The Definitive Jekyll Blogging Framework">Jekyll Bootstrap</a>
-      and <a href="http://getbootstrap.com" target="_blank">Bootstrap</a>
-    </p>
-  </div>
-</div>
-
-
-
-
-
-
-
-<!-- Latest compiled and minified JavaScript, requires jQuery 1.x (2.x not supported in IE8) -->
-<!-- Placed at the end of the document so the pages load faster -->
-<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
-<script src="/assets/themes/mahout3/js/bootstrap.min.js"></script>
-</body>
-</html>
-

http://git-wip-us.apache.org/repos/asf/mahout/blob/7c0babd7/website/oldsite/_site/users/basics/collections.html
----------------------------------------------------------------------
diff --git a/website/oldsite/_site/users/basics/collections.html b/website/oldsite/_site/users/basics/collections.html
deleted file mode 100644
index 41a02f4..0000000
--- a/website/oldsite/_site/users/basics/collections.html
+++ /dev/null
@@ -1,365 +0,0 @@
-
-
-<!DOCTYPE html>
-<html lang="en">
-<head>
-  <meta charset="utf-8">
-  <meta http-equiv="X-UA-Compatible" content="IE=edge">
-
-  <title>Collections</title>
-  
-  <meta name="author" content="The Apache Software Foundation">
-
-  <!-- Enable responsive viewport -->
-  <meta name="viewport" content="width=device-width, initial-scale=1.0">
-
-  <!-- Bootstrap styles -->
-  <link href="/assets/themes/mahout3/css/bootstrap.min.css" rel="stylesheet">
-  <!-- Optional theme -->
-  <link href="/assets/themes/mahout3/css/bootstrap-theme.min.css" rel="stylesheet">
-  <!-- Sticky Footer -->
-  <link href="/assets/themes/mahout3/css/bs-sticky-footer.css" rel="stylesheet">
-
-  <!-- Custom styles -->
-  <link href="/assets/themes/mahout3/css/style.css" rel="stylesheet" type="text/css" media="all">
-
-  <!-- HTML5 Shim and Respond.js IE8 support of HTML5 elements and media queries -->
-  <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
-  <!--[if lt IE 9]>
-  <script src="https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"></script>
-  <script src="https://oss.maxcdn.com/libs/respond.js/1.3.0/respond.min.js"></script>
-  <![endif]-->
-
-  <!-- Fav and touch icons -->
-  <!-- Update these with your own images
-    <link rel="shortcut icon" href="images/favicon.ico">
-    <link rel="apple-touch-icon" href="images/apple-touch-icon.png">
-    <link rel="apple-touch-icon" sizes="72x72" href="images/apple-touch-icon-72x72.png">
-    <link rel="apple-touch-icon" sizes="114x114" href="images/apple-touch-icon-114x114.png">
-  -->
-
-  <!-- atom & rss feed -->
-  <link href="/atom.xml" type="application/atom+xml" rel="alternate" title="Sitewide ATOM Feed">
-  <link href="/rss.xml" type="application/rss+xml" rel="alternate" title="Sitewide RSS Feed">
-  <script type="text/x-mathjax-config">
-  MathJax.Hub.Config({
-    tex2jax: {
-      skipTags: ['script', 'noscript', 'style', 'textarea', 'pre']
-    }
-  });
-  MathJax.Hub.Queue(function() {
-    var all = MathJax.Hub.getAllJax(), i;
-    for(i = 0; i < all.length; i += 1) {
-      all[i].SourceElement().parentNode.className += ' has-jax';
-    }
-  });
-  </script>
-  <script type="text/javascript">
-    var mathjax = document.createElement('script');
-    mathjax.type = 'text/javascript';
-    mathjax.async = true;
-
-    mathjax.src = ('https:' == document.location.protocol) ?
-        'https://c328740.ssl.cf1.rackcdn.com/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML' :
-        'http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML';
-
-      var s = document.getElementsByTagName('script')[0];
-    s.parentNode.insertBefore(mathjax, s);
-  </script>
-</head>
-
-<nav class="navbar navbar-default navbar-fixed-top">
-  <div class="container-fluid">
-    <!-- Brand and toggle get grouped for better mobile display -->
-    <div class="navbar-header">
-      <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1" aria-expanded="false">
-        <span class="sr-only">Toggle navigation</span>
-        <span class="icon-bar"></span>
-        <span class="icon-bar"></span>
-        <span class="icon-bar"></span>
-      </button>
-      <a class="navbar-brand" href="/">
-        <img src="/assets/img/Mahout-logo-82x100.png" height="30" alt="I'm mahout">
-      </a>
-    </div>
-
-    <!--<div class="nav-collapse collapse">-->
-<div class="collapse navbar-collapse" id="main-navbar">
-    <ul class="nav navbar-nav">
-        <!-- <li><a href="/">Home</a></li> -->
-        <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">General<b class="caret"></b></a>
-            <ul class="dropdown-menu">
-                <li><a href="/general/downloads.html">Downloads</a>
-                <li><a href="/general/who-we-are.html">Who we are</a>
-                <li><a href="/general/mailing-lists,-irc-and-archives.html">Mailing Lists</a>
-                <li><a href="/general/release-notes.html">Release Notes</a>
-                <li><a href="/general/books-tutorials-and-talks.html">Books, Tutorials, Talks</a></li>
-                <li><a href="/general/powered-by-mahout.html">Powered By Mahout</a>
-                <li><a href="/general/professional-support.html">Professional Support</a>
-                <li class="divider"></li>
-                <li class="nav-header">Resources</li>
-                <li><a href="/general/reference-reading.html">Reference Reading</a>
-                <li><a href="/general/faq.html">FAQ</a>
-                <li class="divider"></li>
-                <li class="nav-header">Legal</li>
-                <li><a href="http://www.apache.org/licenses/">License</a></li>
-                <li><a href="http://www.apache.org/security/">Security</a></li>
-                <li><a href="/general/privacy-policy.html">Privacy Policy</a>
-            </ul>
-        </li>
-        <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developers<b class="caret"></b></a>
-            <ul class="dropdown-menu">
-                <li><a href="/developers/developer-resources.html">Developer resources</a></li>
-                <li><a href="/developers/version-control.html">Version control</a></li>
-                <li><a href="/developers/buildingmahout.html">Build from source</a></li>
-                <li><a href="/developers/issue-tracker.html">Issue tracker</a></li>
-                <li><a href="https://builds.apache.org/job/Mahout-Quality/" target="_blank">Code quality reports</a></li>
-                <li class="divider"></li>
-                <li class="nav-header">Contributions</li>
-                <li><a href="/developers/how-to-contribute.html">How to contribute</a></li>
-                <li><a href="/developers/how-to-become-a-committer.html">How to become a committer</a></li>
-                <li><a href="/developers/gsoc.html">GSoC</a></li>
-                <li class="divider"></li>
-                <li class="nav-header">For committers</li>
-                <li><a href="/developers/how-to-update-the-website.html">How to update the website</a></li>
-                <li><a href="/developers/patch-check-list.html">Patch check list</a></li>
-                <li><a href="/developers/github.html">Handling Github PRs</a></li>
-                <li><a href="/developers/how-to-release.html">How to release</a></li>
-                <li><a href="/developers/thirdparty-dependencies.html">Third party dependencies</a></li>
-            </ul>
-        </li>
-        <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Mahout-Samsara<b class="caret"></b></a>
-            <ul class="dropdown-menu">
-                <li><a href="/users/sparkbindings/home.html">Scala &amp; Spark Bindings Overview</a></li>
-                <li><a href="/users/sparkbindings/faq.html">FAQ</a></li>
-                <li><a href="/users/flinkbindings/playing-with-samsara-flink.html">Flink Bindings Overview</a></li>
-                <li class="nav-header">Engines</li>
-                <li><a href="/users/sparkbindings/home.html">Spark</a></li>
-                <li><a href="/users/environment/h2o-internals.html">H2O</a></li>
-                <li><a href="/users/flinkbindings/flink-internals.html">Flink</a></li>
-                <li class="nav-header">References</li>
-                <li><a href="/users/environment/in-core-reference.html">In-Core Algebraic DSL Reference</a></li>
-                <li><a href="/users/environment/out-of-core-reference.html">Distributed Algebraic DSL Reference</a></li>
-                <li class="nav-header">Tutorials</li>
-                <li><a href="/users/sparkbindings/play-with-shell.html">Playing with Mahout's Spark Shell</a></li>
-                <li><a href="/users/environment/how-to-build-an-app.html">How to build an app</a></li>
-                <li><a href="/users/environment/classify-a-doc-from-the-shell.html">Building a text classifier in Mahout's Spark Shell</a></li>
-            </ul>
-        </li>
-        <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Algorithms<b class="caret"></b></a>
-            <ul class="dropdown-menu">
-                <li><a href="/users/basics/algorithms.html">List of algorithms</a>
-                <li class="nav-header">Distributed Matrix Decomposition</li>
-                <li><a href="/users/algorithms/d-qr.html">Cholesky QR</a></li>
-                <li><a href="/users/algorithms/d-ssvd.html">SSVD</a></li>
-                <li><a href="/users/algorithms/d-als.html">Distributed ALS</a></li>
-                <li><a href="/users/algorithms/d-spca.html">SPCA</a></li>
-                <li class="nav-header">Recommendations</li>
-                <li><a href="/users/algorithms/recommender-overview.html">Recommender Overview</a></li>
-                <li><a href="/users/algorithms/intro-cooccurrence-spark.html">Intro to cooccurrence-based<br/> recommendations with Spark</a></li>
-                <li class="nav-header">Classification</li>
-                <li><a href="/users/algorithms/spark-naive-bayes.html">Spark Naive Bayes</a></li>
-            </ul>
-        </li>
-        <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">MapReduce Basics<b class="caret"></b></a>
-            <ul class="dropdown-menu">
-                <li><a href="/users/basics/algorithms.html">List of algorithms</a>
-                <li><a href="/users/basics/quickstart.html">Overview</a>
-                <li class="divider"></li>
-                <li class="nav-header">Working with text</li>
-                <li><a href="/users/basics/creating-vectors-from-text.html">Creating vectors from text</a>
-                <li><a href="/users/basics/collocations.html">Collocations</a>
-                <li class="divider"></li>
-                <li class="nav-header">Dimensionality reduction</li>
-                <li><a href="/users/dim-reduction/dimensional-reduction.html">Singular Value Decomposition</a></li>
-                <li><a href="/users/dim-reduction/ssvd.html">Stochastic SVD</a></li>
-                <li class="divider"></li>
-                <li class="nav-header">Topic Models</li>
-                <li><a href="/users/clustering/latent-dirichlet-allocation.html">Latent Dirichlet Allocation</a></li>
-            </ul>
-        </li>
-        <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Mahout MapReduce<b class="caret"></b></a>
-            <ul class="dropdown-menu">
-                <li class="nav-header">Classification</li>
-                <li><a href="/users/classification/bayesian.html">Naive Bayes</a></li>
-                <li><a href="/users/classification/hidden-markov-models.html">Hidden Markov Models</a></li>
-                <li><a href="/users/classification/logistic-regression.html">Logistic Regression (Single Machine)</a></li>
-                <li><a href="/users/classification/partial-implementation.html">Random Forest</a></li>
-                <li class="nav-header">Classification Examples</li>
-                <li><a href="/users/classification/breiman-example.html">Breiman example</a></li>
-                <li><a href="/users/classification/twenty-newsgroups.html">20 newsgroups example</a></li>
-                <li><a href="/users/classification/bankmarketing-example.html">SGD classifier bank marketing</a></li>
-                <li><a href="/users/classification/wikipedia-classifier-example.html">Wikipedia XML parser and classifier</a></li>
-                <li class="nav-header">Clustering</li>
-                <li><a href="/users/clustering/k-means-clustering.html">k-Means</a></li>
-                <li><a href="/users/clustering/canopy-clustering.html">Canopy</a></li>
-                <li><a href="/users/clustering/fuzzy-k-means.html">Fuzzy k-Means</a></li>
-                <li><a href="/users/clustering/streaming-k-means.html">Streaming KMeans</a></li>
-                <li><a href="/users/clustering/spectral-clustering.html">Spectral Clustering</a></li>
-                <li class="nav-header">Clustering Commandline usage</li>
-                <li><a href="/users/clustering/k-means-commandline.html">Options for k-Means</a></li>
-                <li><a href="/users/clustering/canopy-commandline.html">Options for Canopy</a></li>
-                <li><a href="/users/clustering/fuzzy-k-means-commandline.html">Options for Fuzzy k-Means</a></li>
-                <li class="nav-header">Clustering Examples</li>
-                <li><a href="/users/clustering/clustering-of-synthetic-control-data.html">Synthetic data</a></li>
-                <li class="nav-header">Cluster Post processing</li>
-                <li><a href="/users/clustering/cluster-dumper.html">Cluster Dumper tool</a></li>
-                <li><a href="/users/clustering/visualizing-sample-clusters.html">Cluster visualisation</a></li>
-                <li class="nav-header">Recommendations</li>
-                <li><a href="/users/recommender/recommender-first-timer-faq.html">First Timer FAQ</a></li>
-                <li><a href="/users/recommender/userbased-5-minutes.html">A user-based recommender <br/>in 5 minutes</a></li>
-                <li><a href="/users/recommender/matrix-factorization.html">Matrix factorization-based<br/> recommenders</a></li>
-                <li><a href="/users/recommender/recommender-documentation.html">Overview</a></li>
-                <li><a href="/users/recommender/intro-itembased-hadoop.html">Intro to item-based recommendations<br/> with Hadoop</a></li>
-                <li><a href="/users/recommender/intro-als-hadoop.html">Intro to ALS recommendations<br/> with Hadoop</a></li>
-            </ul>
-        </li>
-        <!--  <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Recommendations<b class="caret"></b></a>
-          <ul class="dropdown-menu">
-
-          </ul> -->
-        </li>
-    </ul>
-</div><!--/.nav-collapse -->
-  </div><!-- /.container-fluid -->
-</nav>
-
-<body>
-
-<div id="wrap">
-  <body class="">
-
-  <div class="container">
-    <p>TODO: Organize these somehow, add one-line blurbs</p>
-
-<p>Organize by usage? (classification, recommendation etc.)</p>
-
-<p><a name="Collections-CollectionsofCollections"></a></p>
-<h2 id="collections-of-collections">Collections of Collections</h2>
-
-<ul>
-  <li><a href="http://mldata.org/about/">ML Data</a>
- … repository supported by Pascal 2.</li>
-  <li><a href="http://wiki.dbpedia.org/Downloads30">DBPedia</a></li>
-  <li><a href="http://archive.ics.uci.edu/ml/">UCI Machine Learning Repo</a></li>
-  <li><a href="http://mloss.org/community/blog/2008/sep/19/data-sources/">http://mloss.org/community/blog/2008/sep/19/data-sources/</a></li>
-  <li><a href="http://ckan.net/group/lld">Linked Library Data</a>
- via CKAN</li>
-  <li><a href="http://infochimps.com/">InfoChimps</a>
- Free and purchasable datasets</li>
-  <li><a href="http://www.linkedin.com/groupItem?view=&amp;srchtype=discussedNews&amp;gid=3638279&amp;item=35736572&amp;type=member&amp;trk=EML_anet_ac_pst_ttle">http://www.linkedin.com/groupItem?view=&amp;srchtype=discussedNews&amp;gid=3638279&amp;item=35736572&amp;type=member&amp;trk=EML_anet_ac_pst_ttle</a>
- LinkedIn discussion of lots of data sets</li>
-</ul>
-
-<p><a name="Collections-CategorizationData"></a></p>
-<h2 id="categorization-data">Categorization Data</h2>
-
-<ul>
-  <li><a href="http://people.csail.mit.edu/jrennie/20Newsgroups/">20Newsgroups</a></li>
-  <li><a href="http://jmlr.csail.mit.edu/papers/volume5/lewis04a/lyrl2004_rcv1v2_README.htm">RCV1 data set</a></li>
-  <li><a href="http://direct.dei.unipd.it/">10 years of CLEF Data</a></li>
-  <li><a href="http://ece.ut.ac.ir/DBRG/Hamshahri/">http://ece.ut.ac.ir/DBRG/Hamshahri/</a>
- (Approximately 160k categorized docs)
-There is a newer beta verson here:<a href="http://ece.ut.ac.ir/DBRG/Hamshahri/ham2/">http://ece.ut.ac.ir/DBRG/Hamshahri/ham2/</a>
- (Approximately 320k categorized docs)</li>
-  <li>Lending Club load data <a href="https://www.lendingclub.com/info/download-data.action">https://www.lendingclub.com/info/download-data.action</a></li>
-</ul>
-
-<p><a name="Collections-RecommendationData"></a></p>
-<h2 id="recommendation-data">Recommendation Data</h2>
-
-<ul>
-  <li><a href="http://library.hud.ac.uk/data/usagedata/">Book usage and recommendation data from the University of Huddersfield</a></li>
-  <li><a href="http://denoiserthebetter.posterous.com/music-recommendation-datasets">Last.fm</a>
- - Non-commercial use only</li>
-  <li><a href="http://www.cs.uic.edu/~liub/FBS/sentiment-analysis.html">Amazon Product Review Data via Jindal and Liu</a>
- – Scroll down</li>
-  <li><a href="http://www.grouplens.org/node/73">GroupLens/MovieLens Movie Review Dataset</a></li>
-</ul>
-
-<p><a name="Collections-MultilingualData"></a></p>
-<h2 id="multilingual-data">Multilingual Data</h2>
-
-<ul>
-  <li><a href="http://urd.let.rug.nl/tiedeman/OPUS/OpenSubtitles.php">http://urd.let.rug.nl/tiedeman/OPUS/OpenSubtitles.php</a>
- - 308,000 subtitle files covering about 18,900 movies in 59 languages
-(July 2006 numbers). This is a curated collection of subtitles from an
-aggregation site, [http://www.openSubTitles.org]
-The original site, OpenSubtitles.org, is up to 1.6m subtitles files.</li>
-  <li><a href="http://www.statmt.org/">Statistical Machine Translation</a>
- - devoted to all things language translation. Includes multilingual
-corpuses of European and Canadian legal tomes.</li>
-</ul>
-
-<p><a name="Collections-Geospatial"></a></p>
-<h2 id="geospatial">Geospatial</h2>
-
-<ul>
-  <li><a href="http://www.naturalearthdata.com/">Natural Earth Data</a></li>
-  <li><a href="http://wiki.openstreetmap.org/wiki/Main_Page">Open Street Maps</a>
-And other crowd-sourced mapping data sites.</li>
-</ul>
-
-<p><a name="Collections-Airline"></a></p>
-<h2 id="airline">Airline</h2>
-
-<ul>
-  <li><a href="http://openflights.org/">Open Flights</a>
- - Crowd-sourced database of airlines, flights, airports, times, etc.</li>
-  <li><a href="http://stat-computing.org/dataexpo/2009/">Airline on-time information - 1987-2008</a>
- - 120m CSV records, 12G uncompressed</li>
-</ul>
-
-<p><a name="Collections-GeneralResources"></a></p>
-<h2 id="general-resources">General Resources</h2>
-
-<ul>
-  <li><a href="http://theinfo.org/">theinfo</a></li>
-  <li><a href="http://wordnet.princeton.edu/obtain">WordNet</a></li>
-  <li><a href="http://www.commoncrawl.org/">Common Crawl</a>
- - freely available web crawl on EC2</li>
-</ul>
-
-<p><a name="Collections-Stuff"></a></p>
-<h2 id="stuff">Stuff</h2>
-
-<ul>
-  <li><a href="http://www.cs.technion.ac.il/~gabr/resources/data/ne_datasets.html">http://www.cs.technion.ac.il/~gabr/resources/data/ne_datasets.html</a></li>
-  <li><a href="http://www-2.cs.cmu.edu/afs/cs.cmu.edu/project/theo-20/www/data/">4 Universities Data Set</a></li>
-  <li><a href="http://an.kaist.ac.kr/traces/WWW2010.html">Large crawl of Twitter</a></li>
-  <li><a href="http://beta.uniprot.org/">UniProt</a></li>
-  <li><a href="http://www.icwsm.org/2009/data/">http://www.icwsm.org/2009/data/</a></li>
-  <li><a href="http://data.gov">http://data.gov</a></li>
-  <li><a href="http://www.ckan.net/">http://www.ckan.net/</a></li>
-  <li><a href="http://www.guardian.co.uk/news/datablog/2010/jan/07/government-data-world">http://www.guardian.co.uk/news/datablog/2010/jan/07/government-data-world</a></li>
-  <li><a href="http://data.gov.uk/">http://data.gov.uk/</a></li>
-  <li><a href="http://www.ark.cs.cmu.edu/bills/">51,000 US Congressional Bills tagged</a></li>
-</ul>
-
-  </div>
-
-
-</div>
-
-<div id="footer">
-  <div class="container">
-    <p>&copy; 2017 The Apache Software Foundation
-      with help from <a href="http://jekyllbootstrap.com" target="_blank" title="The Definitive Jekyll Blogging Framework">Jekyll Bootstrap</a>
-      and <a href="http://getbootstrap.com" target="_blank">Bootstrap</a>
-    </p>
-  </div>
-</div>
-
-
-
-
-
-
-
-<!-- Latest compiled and minified JavaScript, requires jQuery 1.x (2.x not supported in IE8) -->
-<!-- Placed at the end of the document so the pages load faster -->
-<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
-<script src="/assets/themes/mahout3/js/bootstrap.min.js"></script>
-</body>
-</html>
-

http://git-wip-us.apache.org/repos/asf/mahout/blob/7c0babd7/website/oldsite/_site/users/basics/collocations.html
----------------------------------------------------------------------
diff --git a/website/oldsite/_site/users/basics/collocations.html b/website/oldsite/_site/users/basics/collocations.html
deleted file mode 100644
index ba57f09..0000000
--- a/website/oldsite/_site/users/basics/collocations.html
+++ /dev/null
@@ -1,636 +0,0 @@
-
-
-<!DOCTYPE html>
-<html lang="en">
-<head>
-  <meta charset="utf-8">
-  <meta http-equiv="X-UA-Compatible" content="IE=edge">
-
-  <title>Collocations</title>
-  
-  <meta name="author" content="The Apache Software Foundation">
-
-  <!-- Enable responsive viewport -->
-  <meta name="viewport" content="width=device-width, initial-scale=1.0">
-
-  <!-- Bootstrap styles -->
-  <link href="/assets/themes/mahout3/css/bootstrap.min.css" rel="stylesheet">
-  <!-- Optional theme -->
-  <link href="/assets/themes/mahout3/css/bootstrap-theme.min.css" rel="stylesheet">
-  <!-- Sticky Footer -->
-  <link href="/assets/themes/mahout3/css/bs-sticky-footer.css" rel="stylesheet">
-
-  <!-- Custom styles -->
-  <link href="/assets/themes/mahout3/css/style.css" rel="stylesheet" type="text/css" media="all">
-
-  <!-- HTML5 Shim and Respond.js IE8 support of HTML5 elements and media queries -->
-  <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
-  <!--[if lt IE 9]>
-  <script src="https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"></script>
-  <script src="https://oss.maxcdn.com/libs/respond.js/1.3.0/respond.min.js"></script>
-  <![endif]-->
-
-  <!-- Fav and touch icons -->
-  <!-- Update these with your own images
-    <link rel="shortcut icon" href="images/favicon.ico">
-    <link rel="apple-touch-icon" href="images/apple-touch-icon.png">
-    <link rel="apple-touch-icon" sizes="72x72" href="images/apple-touch-icon-72x72.png">
-    <link rel="apple-touch-icon" sizes="114x114" href="images/apple-touch-icon-114x114.png">
-  -->
-
-  <!-- atom & rss feed -->
-  <link href="/atom.xml" type="application/atom+xml" rel="alternate" title="Sitewide ATOM Feed">
-  <link href="/rss.xml" type="application/rss+xml" rel="alternate" title="Sitewide RSS Feed">
-  <script type="text/x-mathjax-config">
-  MathJax.Hub.Config({
-    tex2jax: {
-      skipTags: ['script', 'noscript', 'style', 'textarea', 'pre']
-    }
-  });
-  MathJax.Hub.Queue(function() {
-    var all = MathJax.Hub.getAllJax(), i;
-    for(i = 0; i < all.length; i += 1) {
-      all[i].SourceElement().parentNode.className += ' has-jax';
-    }
-  });
-  </script>
-  <script type="text/javascript">
-    var mathjax = document.createElement('script');
-    mathjax.type = 'text/javascript';
-    mathjax.async = true;
-
-    mathjax.src = ('https:' == document.location.protocol) ?
-        'https://c328740.ssl.cf1.rackcdn.com/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML' :
-        'http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML';
-
-      var s = document.getElementsByTagName('script')[0];
-    s.parentNode.insertBefore(mathjax, s);
-  </script>
-</head>
-
-<nav class="navbar navbar-default navbar-fixed-top">
-  <div class="container-fluid">
-    <!-- Brand and toggle get grouped for better mobile display -->
-    <div class="navbar-header">
-      <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1" aria-expanded="false">
-        <span class="sr-only">Toggle navigation</span>
-        <span class="icon-bar"></span>
-        <span class="icon-bar"></span>
-        <span class="icon-bar"></span>
-      </button>
-      <a class="navbar-brand" href="/">
-        <img src="/assets/img/Mahout-logo-82x100.png" height="30" alt="I'm mahout">
-      </a>
-    </div>
-
-    <!--<div class="nav-collapse collapse">-->
-<div class="collapse navbar-collapse" id="main-navbar">
-    <ul class="nav navbar-nav">
-        <!-- <li><a href="/">Home</a></li> -->
-        <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">General<b class="caret"></b></a>
-            <ul class="dropdown-menu">
-                <li><a href="/general/downloads.html">Downloads</a>
-                <li><a href="/general/who-we-are.html">Who we are</a>
-                <li><a href="/general/mailing-lists,-irc-and-archives.html">Mailing Lists</a>
-                <li><a href="/general/release-notes.html">Release Notes</a>
-                <li><a href="/general/books-tutorials-and-talks.html">Books, Tutorials, Talks</a></li>
-                <li><a href="/general/powered-by-mahout.html">Powered By Mahout</a>
-                <li><a href="/general/professional-support.html">Professional Support</a>
-                <li class="divider"></li>
-                <li class="nav-header">Resources</li>
-                <li><a href="/general/reference-reading.html">Reference Reading</a>
-                <li><a href="/general/faq.html">FAQ</a>
-                <li class="divider"></li>
-                <li class="nav-header">Legal</li>
-                <li><a href="http://www.apache.org/licenses/">License</a></li>
-                <li><a href="http://www.apache.org/security/">Security</a></li>
-                <li><a href="/general/privacy-policy.html">Privacy Policy</a>
-            </ul>
-        </li>
-        <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developers<b class="caret"></b></a>
-            <ul class="dropdown-menu">
-                <li><a href="/developers/developer-resources.html">Developer resources</a></li>
-                <li><a href="/developers/version-control.html">Version control</a></li>
-                <li><a href="/developers/buildingmahout.html">Build from source</a></li>
-                <li><a href="/developers/issue-tracker.html">Issue tracker</a></li>
-                <li><a href="https://builds.apache.org/job/Mahout-Quality/" target="_blank">Code quality reports</a></li>
-                <li class="divider"></li>
-                <li class="nav-header">Contributions</li>
-                <li><a href="/developers/how-to-contribute.html">How to contribute</a></li>
-                <li><a href="/developers/how-to-become-a-committer.html">How to become a committer</a></li>
-                <li><a href="/developers/gsoc.html">GSoC</a></li>
-                <li class="divider"></li>
-                <li class="nav-header">For committers</li>
-                <li><a href="/developers/how-to-update-the-website.html">How to update the website</a></li>
-                <li><a href="/developers/patch-check-list.html">Patch check list</a></li>
-                <li><a href="/developers/github.html">Handling Github PRs</a></li>
-                <li><a href="/developers/how-to-release.html">How to release</a></li>
-                <li><a href="/developers/thirdparty-dependencies.html">Third party dependencies</a></li>
-            </ul>
-        </li>
-        <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Mahout-Samsara<b class="caret"></b></a>
-            <ul class="dropdown-menu">
-                <li><a href="/users/sparkbindings/home.html">Scala &amp; Spark Bindings Overview</a></li>
-                <li><a href="/users/sparkbindings/faq.html">FAQ</a></li>
-                <li><a href="/users/flinkbindings/playing-with-samsara-flink.html">Flink Bindings Overview</a></li>
-                <li class="nav-header">Engines</li>
-                <li><a href="/users/sparkbindings/home.html">Spark</a></li>
-                <li><a href="/users/environment/h2o-internals.html">H2O</a></li>
-                <li><a href="/users/flinkbindings/flink-internals.html">Flink</a></li>
-                <li class="nav-header">References</li>
-                <li><a href="/users/environment/in-core-reference.html">In-Core Algebraic DSL Reference</a></li>
-                <li><a href="/users/environment/out-of-core-reference.html">Distributed Algebraic DSL Reference</a></li>
-                <li class="nav-header">Tutorials</li>
-                <li><a href="/users/sparkbindings/play-with-shell.html">Playing with Mahout's Spark Shell</a></li>
-                <li><a href="/users/environment/how-to-build-an-app.html">How to build an app</a></li>
-                <li><a href="/users/environment/classify-a-doc-from-the-shell.html">Building a text classifier in Mahout's Spark Shell</a></li>
-            </ul>
-        </li>
-        <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Algorithms<b class="caret"></b></a>
-            <ul class="dropdown-menu">
-                <li><a href="/users/basics/algorithms.html">List of algorithms</a>
-                <li class="nav-header">Distributed Matrix Decomposition</li>
-                <li><a href="/users/algorithms/d-qr.html">Cholesky QR</a></li>
-                <li><a href="/users/algorithms/d-ssvd.html">SSVD</a></li>
-                <li><a href="/users/algorithms/d-als.html">Distributed ALS</a></li>
-                <li><a href="/users/algorithms/d-spca.html">SPCA</a></li>
-                <li class="nav-header">Recommendations</li>
-                <li><a href="/users/algorithms/recommender-overview.html">Recommender Overview</a></li>
-                <li><a href="/users/algorithms/intro-cooccurrence-spark.html">Intro to cooccurrence-based<br/> recommendations with Spark</a></li>
-                <li class="nav-header">Classification</li>
-                <li><a href="/users/algorithms/spark-naive-bayes.html">Spark Naive Bayes</a></li>
-            </ul>
-        </li>
-        <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">MapReduce Basics<b class="caret"></b></a>
-            <ul class="dropdown-menu">
-                <li><a href="/users/basics/algorithms.html">List of algorithms</a>
-                <li><a href="/users/basics/quickstart.html">Overview</a>
-                <li class="divider"></li>
-                <li class="nav-header">Working with text</li>
-                <li><a href="/users/basics/creating-vectors-from-text.html">Creating vectors from text</a>
-                <li><a href="/users/basics/collocations.html">Collocations</a>
-                <li class="divider"></li>
-                <li class="nav-header">Dimensionality reduction</li>
-                <li><a href="/users/dim-reduction/dimensional-reduction.html">Singular Value Decomposition</a></li>
-                <li><a href="/users/dim-reduction/ssvd.html">Stochastic SVD</a></li>
-                <li class="divider"></li>
-                <li class="nav-header">Topic Models</li>
-                <li><a href="/users/clustering/latent-dirichlet-allocation.html">Latent Dirichlet Allocation</a></li>
-            </ul>
-        </li>
-        <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Mahout MapReduce<b class="caret"></b></a>
-            <ul class="dropdown-menu">
-                <li class="nav-header">Classification</li>
-                <li><a href="/users/classification/bayesian.html">Naive Bayes</a></li>
-                <li><a href="/users/classification/hidden-markov-models.html">Hidden Markov Models</a></li>
-                <li><a href="/users/classification/logistic-regression.html">Logistic Regression (Single Machine)</a></li>
-                <li><a href="/users/classification/partial-implementation.html">Random Forest</a></li>
-                <li class="nav-header">Classification Examples</li>
-                <li><a href="/users/classification/breiman-example.html">Breiman example</a></li>
-                <li><a href="/users/classification/twenty-newsgroups.html">20 newsgroups example</a></li>
-                <li><a href="/users/classification/bankmarketing-example.html">SGD classifier bank marketing</a></li>
-                <li><a href="/users/classification/wikipedia-classifier-example.html">Wikipedia XML parser and classifier</a></li>
-                <li class="nav-header">Clustering</li>
-                <li><a href="/users/clustering/k-means-clustering.html">k-Means</a></li>
-                <li><a href="/users/clustering/canopy-clustering.html">Canopy</a></li>
-                <li><a href="/users/clustering/fuzzy-k-means.html">Fuzzy k-Means</a></li>
-                <li><a href="/users/clustering/streaming-k-means.html">Streaming KMeans</a></li>
-                <li><a href="/users/clustering/spectral-clustering.html">Spectral Clustering</a></li>
-                <li class="nav-header">Clustering Commandline usage</li>
-                <li><a href="/users/clustering/k-means-commandline.html">Options for k-Means</a></li>
-                <li><a href="/users/clustering/canopy-commandline.html">Options for Canopy</a></li>
-                <li><a href="/users/clustering/fuzzy-k-means-commandline.html">Options for Fuzzy k-Means</a></li>
-                <li class="nav-header">Clustering Examples</li>
-                <li><a href="/users/clustering/clustering-of-synthetic-control-data.html">Synthetic data</a></li>
-                <li class="nav-header">Cluster Post processing</li>
-                <li><a href="/users/clustering/cluster-dumper.html">Cluster Dumper tool</a></li>
-                <li><a href="/users/clustering/visualizing-sample-clusters.html">Cluster visualisation</a></li>
-                <li class="nav-header">Recommendations</li>
-                <li><a href="/users/recommender/recommender-first-timer-faq.html">First Timer FAQ</a></li>
-                <li><a href="/users/recommender/userbased-5-minutes.html">A user-based recommender <br/>in 5 minutes</a></li>
-                <li><a href="/users/recommender/matrix-factorization.html">Matrix factorization-based<br/> recommenders</a></li>
-                <li><a href="/users/recommender/recommender-documentation.html">Overview</a></li>
-                <li><a href="/users/recommender/intro-itembased-hadoop.html">Intro to item-based recommendations<br/> with Hadoop</a></li>
-                <li><a href="/users/recommender/intro-als-hadoop.html">Intro to ALS recommendations<br/> with Hadoop</a></li>
-            </ul>
-        </li>
-        <!--  <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Recommendations<b class="caret"></b></a>
-          <ul class="dropdown-menu">
-
-          </ul> -->
-        </li>
-    </ul>
-</div><!--/.nav-collapse -->
-  </div><!-- /.container-fluid -->
-</nav>
-
-<body>
-
-<div id="wrap">
-  <body class="">
-
-  <div class="container">
-    <p><a name="Collocations-CollocationsinMahout"></a></p>
-<h1 id="collocations-in-mahout">Collocations in Mahout</h1>
-
-<p>A collocation is defined as a sequence of words or terms which co-occur
-more often than would be expected by chance. Statistically relevant
-combinations of terms identify additional lexical units which can be
-treated as features in a vector-based representation of a text. A detailed
-discussion of collocations can be found on <a href="http://en.wikipedia.org/wiki/Collocation">Wikipedia</a>.</p>
-
-<p>See there for a more detailed discussion of collocations in the <a href="http://comments.gmane.org/gmane.comp.apache.mahout.user/5685">Reuters example</a>.</p>
-
-<p><a name="Collocations-Log-LikelihoodbasedCollocationIdentification"></a></p>
-<h2 id="theory-behind-implementation-log-likelihood-based-collocation-identification">Theory behind implementation: Log-Likelihood based Collocation Identification</h2>
-
-<p>Mahout provides an implementation of a collocation identification algorithm
-which scores collocations using log-likelihood ratio. The log-likelihood
-score indicates the relative usefulness of a collocation with regards other
-term combinations in the text. Collocations with the highest scores in a
-particular corpus will generally be more useful as features.</p>
-
-<p>Calculating the LLR is very straightforward and is described concisely in
-<a href="http://tdunning.blogspot.com/2008/03/surprise-and-coincidence.html">Ted Dunning’s blog post</a>
-. Ted describes the series of counts reqired to calculate the LLR for two
-events A and B in order to determine if they co-occur more often than pure
-chance. These counts include the number of times the events co-occur (k11),
-the number of times the events occur without each other (k12 and k21), and
-the number of times anything occurs. These counts are summarized in the
-following table:</p>
-
-<table>
-<tr><td> </td><td> Event A </td><td> Everything but Event A </td></tr>
-<tr><td> Event B </td><td> A and B together (k11) </td><td>  B but not A (k12) </td></tr>
-<tr><td> Everything but Event B </td><td> A but not B (k21) </td><td> Neither B nor A (k22) </td></tr>
-</table>
-
-<p>For the purposes of collocation identification, it is useful to begin by
-thinking in word pairs, bigrams. In this case the leading or head term from
-the pair corresponds to A from the table above, B corresponds to the
-trailing or tail term, while neither B nor A is the total number of word
-pairs in the corpus less those containing B, A or both B and A.</p>
-
-<p>Given the word pair of ‘oscillation overthruster’, the Log-Likelihood ratio
-is computed by looking at the number of occurences of that word pair in the
-corpus, the number of word pairs that begin with ‘oscillation’ but end with
-something other than ‘overthruster’, the number of word pairs that end with
-‘overthruster’ begin with something other than ‘oscillation’ and the number
-of word pairs in the corpus that contain neither ‘oscillation’ and
-overthruster.</p>
-
-<p>This can be extended from bigrams to trigrams, 4-grams and beyond. In these
-cases, the current algorithm uses the first token of the ngram as the head
-of the ngram and the remaining n-1 tokens from the ngram, the n-1gram as it
-were, as the tail. Given the trigram ‘hong kong cavaliers’, ‘hong’ is
-treated as the head while ‘kong cavaliers’ is treated as the tail. Future
-versions of this algorithm will allow for variations in which tokens of the
-ngram are treated as the head and tail.</p>
-
-<p>Beyond ngrams, it is often useful to inspect cases where individual words
-occur around other interesting features of the text such as sentence
-boundaries.</p>
-
-<p><a name="Collocations-GeneratingNGrams"></a></p>
-<h2 id="generating-ngrams">Generating NGrams</h2>
-
-<p>The tools that the collocation identification algorithm are embeeded within
-either consume tokenized text as input or provide the ability to specify an
-implementation of the Lucene Analyzer class perform tokenization in order
-to form ngrams. The tokens are passed through a Lucene ShingleFilter to
-produce NGrams of the desired length.</p>
-
-<p>Given the text “Alice was beginning to get very tired” as an example,
-Lucene’s StandardAnalyzer produces the tokens ‘alice’, ‘beginning’, ‘get’,
-‘very’ and ‘tired’, while the ShingleFilter with a max NGram size set to 3
-produces the shingles ‘alice beginning’, ‘alice beginning get’, ‘beginning
-get’, ‘beginning get very’, ‘get very’, ‘get very tired’ and ‘very tired’.
-Note that both bigrams and trigrams are produced here. A future enhancement
-to the existing algorithm would involve limiting the output to a particular
-gram size as opposed to solely specifiying a max ngram size.</p>
-
-<p><a name="Collocations-RunningtheCollocationIdentificationAlgorithm."></a></p>
-<h2 id="running-the-collocation-identification-algorithm">Running the Collocation Identification Algorithm.</h2>
-
-<p>There are a couple ways to run the llr-based collocation algorithm in
-mahout</p>
-
-<p><a name="Collocations-Whencreatingvectorsfromasequencefile"></a></p>
-<h3 id="when-creating-vectors-from-a-sequence-file">When creating vectors from a sequence file</h3>
-
-<p>The llr collocation identifier is integrated into the process that is used
-to create vectors from sequence files of text keys and values. Collocations
-are generated when the –maxNGramSize (-ng) option is not specified and
-defaults to 2 or is set to a number of 2 or greater. The –minLLR option
-can be used to control the cutoff that prevents collocations below the
-specified LLR score from being emitted, and the –minSupport argument can
-be used to filter out collocations that appear below a certain number of
-times.</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>bin/mahout seq2sparse
-
-Usage:									    
-     [--minSupport &lt;minSupport&gt; --analyzerName &lt;analyzerName&gt; --chunkSize &lt;chunkSize&gt;
-      --output &lt;output&gt; --input &lt;input&gt; --minDF &lt;minDF&gt;
-      --maxDFPercent&lt;maxDFPercent&gt; --weight &lt;weight&gt; --norm &lt;norm&gt; --minLLR &lt;minLLR&gt;
-      --numReducers  &lt;numReducers&gt; --maxNGramSize &lt;ngramSize&gt; --overwrite --help		    
-      --sequentialAccessVector]
-Options 								    
-
-  --minSupport (-s) minSupport	  (Optional) Minimum Support. Default Value: 2				    
-
-  --analyzerName (-a) analyzerName    The class name of the analyzer
-
-  --chunkSize (-chunk) chunkSize      The chunkSize in MegaBytes. 100-10000MB
-
-  --output (-o) output		 The output directory
-
-  --input (-i) input		   Input dir containing the documents in sequence file format
-
-  --minDF (-md) minDF		  The minimum document frequency. Default is 1
-
-  --maxDFPercent (-x) maxDFPercent    The max percentage of docs for the DF. Can be used to remove 
-                                      really high frequency terms. Expressed as an
-                                      integer between 0 and 100. Default is 99.     
-
-  --weight (-wt) weight 	      The kind of weight to use. Currently TF   
-				      or TFIDF				    
-
-  --norm (-n) norm		      The norm to use, expressed as either a    
-				      float or "INF" if you want to use the 
-				      Infinite norm.  Must be greater orequal  
-				      to 0.  The default is not to normalize    
-
-  --minLLR (-ml) minLLR 	      (Optional)The minimum Log Likelihood  
-				      Ratio(Float)  Default is 1.0
-    
-  --numReducers (-nr) numReducers     (Optional) Number of reduce tasks.    
-				      Default Value: 1			    
-
-  --maxNGramSize (-ng) ngramSize      (Optional) The maximum size of ngrams to  
-				      create (2 = bigrams, 3 = trigrams, etc)   
-				      Default Value:2			 
-   
-  --overwrite (-w)		      If set, overwrite the output directory    
-  --help (-h)			      Print out help			    
-  --sequentialAccessVector (-seq)     (Optional) Whether output vectors should	
-				      be SequentialAccessVectors If set true	
-				      else false 
-</code></pre>
-</div>
-
-<p><a name="Collocations-CollocDriver"></a></p>
-<h3 id="collocdriver">CollocDriver</h3>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>bin/mahout org.apache.mahout.vectorizer.collocations.llr.CollocDriver
-
-Usage:									    
- [--input &lt;input&gt; --output &lt;output&gt; --maxNGramSize &lt;ngramSize&gt; --overwrite    
---minSupport &lt;minSupport&gt; --minLLR &lt;minLLR&gt; --numReducers &lt;numReducers&gt;     
---analyzerName &lt;analyzerName&gt; --preprocess --unigram --help]
-
-Options 								    
-
-  --input (-i) input		      The Path for input files. 	    
-
-  --output (-o) output		      The Path write output to		    
-
-  --maxNGramSize (-ng) ngramSize      (Optional) The maximum size of ngramsto  
-				      create (2 = bigrams, 3 = trigrams,etc)   
-				      Default Value:2			
-
-  --overwrite (-w)		      If set, overwrite the outputdirectory    
-
-  --minSupport (-s) minSupport	      (Optional) Minimum Support. Default   
-				      Value: 2				    
-
-  --minLLR (-ml) minLLR 	      (Optional)The minimum Log Likelihood
-				      Ratio(Float)  Default is 1.0	  
-  
-  --numReducers (-nr) numReducers     (Optional) Number of reduce tasks.    
-				      Default Value: 1			    
-
-  --analyzerName (-a) analyzerName    The class name of the analyzer	    
-
-  --preprocess (-p)		      If set, input is SequenceFile&lt;Text,Text&gt;  
-				      where the value is the document, which	
-				      will be tokenized using the specified 
-				      analyzer. 			
-
-  --unigram (-u)		      If set, unigrams will be emitted inthe   
-				      final output alongside collocations
-   
-  --help (-h)			      Print out help	      
-</code></pre>
-</div>
-
-<p><a name="Collocations-Algorithmdetails"></a></p>
-<h2 id="algorithm-details">Algorithm details</h2>
-
-<p>This section describes the implementation of the collocation identification
-algorithm in terms of the map-reduce phases that are used to generate
-ngrams and count the frequencies required to perform the log-likelihood
-calculation. Unless otherwise noted, classes that are indicated in
-CamelCase can be found in the mahout-utils module under the package
-org.apache.mahout.utils.nlp.collocations.llr</p>
-
-<p>The algorithm is implemented in two map-reduce passes:</p>
-
-<p><a name="Collocations-Pass1:CollocDriver.generateCollocations(...)"></a></p>
-<h3 id="pass-1-collocdrivergeneratecollocations">Pass 1: CollocDriver.generateCollocations(…)</h3>
-
-<p>Generates NGrams and counts frequencies for ngrams, head and tail subgrams.</p>
-
-<p><a name="Collocations-Map:CollocMapper"></a></p>
-<h4 id="map-collocmapper">Map: CollocMapper</h4>
-
-<p>Input k: Text (documentId), v: StringTuple (tokens)</p>
-
-<p>Each call to the mapper passes in the full set of tokens for the
-corresponding document using a StringTuple. The ShingleFilter is run across
-these tokens to produce ngrams of the desired length. ngrams and
-frequencies are collected across the entire document.</p>
-
-<p>Once this is done, ngrams are split into head and tail portions. A key of type GramKey is generated which is used later to join ngrams with their heads and tails in the reducer phase. The GramKey is a composite key made up of a string n-gram fragement as the primary key and a secondary key used for grouping and sorting in the reduce phase. The secondary key will either be EMPTY in the case where we are collecting either the head or tail of an ngram as the value or it will contain the byte<a href=".html"></a>
- form of the ngram when collecting an ngram as the value.</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>head_key(EMPTY) -&gt; (head subgram, head frequency)
-
-head_key(ngram) -&gt; (ngram, ngram frequency) 
-
-tail_key(EMPTY) -&gt; (tail subgram, tail frequency)
-
-tail_key(ngram) -&gt; (ngram, ngram frequency)
-</code></pre>
-</div>
-
-<p>subgram and ngram values are packaged in Gram objects.</p>
-
-<p>For each ngram found, the Count.NGRAM_TOTAL counter is incremented. When
-the pass is complete, this counter will hold the total number of ngrams
-encountered in the input which is used as a part of the LLR calculation.</p>
-
-<p>Output k: GramKey (head or tail subgram), v: Gram (head, tail or ngram with
-frequency)</p>
-
-<p><a name="Collocations-Combiner:CollocCombiner"></a></p>
-<h4 id="combiner-colloccombiner">Combiner: CollocCombiner</h4>
-
-<p>Input k: GramKey, v:Gram (as above)</p>
-
-<p>This phase merges the counts for unique ngrams or ngram fragments across
-multiple documents. The combiner treats the entire GramKey as the key and
-as such, identical tuples from separate documents are passed into a single
-call to the combiner’s reduce method, their frequencies are summed and a
-single tuple is passed out via the collector.</p>
-
-<p>Output k: GramKey, v:Gram</p>
-
-<p><a name="Collocations-Reduce:CollocReducer"></a></p>
-<h4 id="reduce-collocreducer">Reduce: CollocReducer</h4>
-
-<p>Input k: GramKey, v: Gram (as above)</p>
-
-<p>The CollocReducer employs the Hadoop secondary sort strategy to avoid
-caching ngram tuples in memory in order to calculate total ngram and
-subgram frequencies. The GramKeyPartitioner ensures that tuples with the
-same primary key are sent to the same reducer while the
-GramKeyGroupComparator ensures that iterator provided by the reduce method
-first returns the subgram and then returns ngram values grouped by ngram.
-This eliminates the need to cache the values returned by the iterator in
-order to calculate total frequencies for both subgrams and ngrams. There
-input will consist of multiple frequencies for each (subgram_key, subgram)
-or (subgram_key, ngram) tuple; one from each map task executed in which the
-particular subgram was found.
-The input will be traversed in the following order:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>(head subgram, frequency 1)
-(head subgram, frequency 2)
-... 
-(head subgram, frequency N)
-(ngram 1, frequency 1)
-(ngram 1, frequency 2)
-...
-(ngram 1, frequency N)
-(ngram 2, frequency 1)
-(ngram 2, frequency 2)
-...
-(ngram 2, frequency N)
-...
-(ngram N, frequency 1)
-(ngram N, frequency 2)
-...
-(ngram N, frequency N)
-</code></pre>
-</div>
-
-<p>Where all of the ngrams above share the same head. Data is presented in the
-same manner for the tail subgrams.</p>
-
-<p>As the values for a subgram or ngram are traversed, frequencies are
-accumulated. Once all values for a subgram or ngram are processed the
-resulting key/value pairs are passed to the collector as long as the ngram
-frequency is equal to or greater than the specified minSupport. When an
-ngram is skipped in this way the Skipped.LESS_THAN_MIN_SUPPORT counter to
-be incremented.</p>
-
-<p>Pairs are passed to the collector in the following format:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>ngram, ngram frequency -&gt; subgram subgram frequency
-</code></pre>
-</div>
-
-<p>In this manner, the output becomes an unsorted version of the following:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>ngram 1, frequency -&gt; ngram 1 head, head frequency
-ngram 1, frequency -&gt; ngram 1 tail, tail frequency
-ngram 2, frequency -&gt; ngram 2 head, head frequency
-ngram 2, frequency -&gt; ngram 2 tail, tail frequency
-ngram N, frequency -&gt; ngram N head, head frequency
-ngram N, frequency -&gt; ngram N tail, tail frequency
-</code></pre>
-</div>
-
-<p>Output is in the format k:Gram (ngram, frequency), v:Gram (subgram,
-frequency)</p>
-
-<p><a name="Collocations-Pass2:CollocDriver.computeNGramsPruneByLLR(...)"></a></p>
-<h3 id="pass-2-collocdrivercomputengramsprunebyllr">Pass 2: CollocDriver.computeNGramsPruneByLLR(…)</h3>
-
-<p>Pass 1 has calculated full frequencies for ngrams and subgrams, Pass 2
-performs the LLR calculation.</p>
-
-<p><a name="Collocations-MapPhase:IdentityMapper(org.apache.hadoop.mapred.lib.IdentityMapper)"></a></p>
-<h4 id="map-phase-identitymapper-orgapachehadoopmapredlibidentitymapper">Map Phase: IdentityMapper (org.apache.hadoop.mapred.lib.IdentityMapper)</h4>
-
-<p>This phase is a no-op. The data is passed through unchanged. The rest of
-the work for llr calculation is done in the reduce phase.</p>
-
-<p><a name="Collocations-ReducePhase:LLRReducer"></a></p>
-<h4 id="reduce-phase-llrreducer">Reduce Phase: LLRReducer</h4>
-
-<p>Input is k:Gram, v:Gram (as above)</p>
-
-<p>This phase receives the head and tail subgrams and their frequencies for
-each ngram (with frequency) produced for the input:</p>
-
-<div class="highlighter-rouge"><pre class="highlight"><code>ngram 1, frequency -&gt; ngram 1 head, frequency; ngram 1 tail, frequency
-ngram 2, frequency -&gt; ngram 2 head, frequency; ngram 2 tail, frequency
-...
-ngram 1, frequency -&gt; ngram N head, frequency; ngram N tail, frequency
-</code></pre>
-</div>
-
-<p>It also reads the full ngram count obtained from the first pass, passed in
-as a configuration option. The parameters to the llr calculation are
-calculated as follows:</p>
-
-<p>k11 = f_n
-k12 = f_h - f_n
-k21 = f_t - f_n
-k22 = N - ((f_h + f_t) - f_n)</p>
-
-<p>Where f_n is the ngram frequency, f_h and f_t the frequency of head and
-tail and N is the total number of ngrams.</p>
-
-<p>Tokens with a llr below that of the specified minimum llr are dropped and
-the Skipped.LESS_THAN_MIN_LLR counter is incremented.</p>
-
-<p>Output is k: Text (ngram), v: DoubleWritable (llr score)</p>
-
-<p><a name="Collocations-Unigrampass-through."></a></p>
-<h3 id="unigram-pass-through">Unigram pass-through.</h3>
-
-<p>By default in seq2sparse, or if the -u option is provided to the
-CollocDriver, unigrams (single tokens) will be passed through the job and
-each token’s frequency will be calculated. As with ngrams, unigrams are
-subject to filtering with minSupport and minLLR.</p>
-
-
-  </div>
-
-
-</div>
-
-<div id="footer">
-  <div class="container">
-    <p>&copy; 2017 The Apache Software Foundation
-      with help from <a href="http://jekyllbootstrap.com" target="_blank" title="The Definitive Jekyll Blogging Framework">Jekyll Bootstrap</a>
-      and <a href="http://getbootstrap.com" target="_blank">Bootstrap</a>
-    </p>
-  </div>
-</div>
-
-
-
-
-
-
-
-<!-- Latest compiled and minified JavaScript, requires jQuery 1.x (2.x not supported in IE8) -->
-<!-- Placed at the end of the document so the pages load faster -->
-<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
-<script src="/assets/themes/mahout3/js/bootstrap.min.js"></script>
-</body>
-</html>
-