You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by ct...@apache.org on 2021/01/28 16:33:32 UTC
svn commit: r1070647 [28/35] - in
/websites/production/lucene/content/solr/guide/8_8: ./
images/math-expressions/ meta-docs/
Modified: websites/production/lucene/content/solr/guide/8_8/statistics.html
==============================================================================
--- websites/production/lucene/content/solr/guide/8_8/statistics.html (original)
+++ websites/production/lucene/content/solr/guide/8_8/statistics.html Thu Jan 28 16:33:25 2021
@@ -8,7 +8,7 @@
<meta name="description" content="">
<meta name="keywords" content=" ">
-<title>Statistics | Apache Solr Reference Guide 8.8-DRAFT</title>
+<title>Statistics | Apache Solr Reference Guide 8.8</title>
<link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.1.3/css/bootstrap.min.css" integrity="sha384-MCw98/SFnGE8fJT3GXwEOngsV7Zt27NXFoaoApmYm81iuXoPkFOJwJ8ERdknLPMO" crossorigin="anonymous">
<link rel="stylesheet" type="text/css" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.5.0/css/font-awesome.min.css">
@@ -68,7 +68,7 @@
</script>
</head>
-<body class="DRAFT" id="statistics">
+<body class="" id="statistics">
<div class="container-fluid">
<div class="row">
<nav id="sidebar" class="col-2 d-none d-md-block">
@@ -76,11 +76,6 @@
<div class="sidebar-header">
<div class="sidebarTitle text-center">Apache Solr Reference Guide</div>
- <p class="draft-notice">
- This is an unofficial DRAFT of the Guide for 8.8.
- <a href="https://lucene.apache.org/solr/guide/">Official releases are available from the Solr website</a>.
- </p>
-
<!--comment out this block if you want to hide search-->
<!--start search-->
@@ -872,11 +867,36 @@
</li>
<li class="sb-level2">
- <a href="math-expressions.html">Math Expressions</a>
+ <a href="math-expressions.html">Streaming Expressions and Math Expressions</a>
<ul>
<li class="sb-level3">
+ <a href="visualization.html">Visualization</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="math-start.html">Getting Started</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="loading.html">Loading Data</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="search-sample.html">Searching, Sampling and Aggregation</a>
+
+ </li>
+
+ <li class="sb-level3">
+ <a href="transform.html">Transforming Data</a>
+
+ </li>
+
+ <li class="sb-level3">
<a href="scalar-math.html">Scalar Math</a>
</li>
@@ -897,12 +917,12 @@
</li>
<li class="sb-level3">
- <a href="vectorization.html">Streams and Vectorization</a>
+ <a href="term-vectors.html">Text Analysis and Term Vectors</a>
</li>
<li class="sb-level3">
- <a href="term-vectors.html">Text Analysis and Term Vectors</a>
+ <a href="probability-distributions.html">Probability Distributions</a>
</li>
@@ -912,12 +932,12 @@
</li>
<li class="sb-level3">
- <a href="probability-distributions.html">Probability Distributions</a>
+ <a href="regression.html">Linear Regression</a>
</li>
<li class="sb-level3">
- <a href="simulations.html">Monte Carlo Simulations</a>
+ <a href="curve-fitting.html">Curve Fitting</a>
</li>
@@ -927,32 +947,32 @@
</li>
<li class="sb-level3">
- <a href="regression.html">Linear Regression</a>
+ <a href="numerical-analysis.html">Interpolation, Derivatives and Integrals</a>
</li>
<li class="sb-level3">
- <a href="numerical-analysis.html">Interpolation, Derivatives and Integrals</a>
+ <a href="dsp.html">Digital Signal Processing</a>
</li>
<li class="sb-level3">
- <a href="curve-fitting.html">Curve Fitting</a>
+ <a href="simulations.html">Monte Carlo Simulations</a>
</li>
<li class="sb-level3">
- <a href="dsp.html">Digital Signal Processing</a>
+ <a href="machine-learning.html">Machine Learning</a>
</li>
<li class="sb-level3">
- <a href="machine-learning.html">Machine Learning</a>
+ <a href="computational-geometry.html">Computational Geometry</a>
</li>
<li class="sb-level3">
- <a href="computational-geometry.html">Computational Geometry</a>
+ <a href="logs.html">Log Analytics</a>
</li>
@@ -1658,14 +1678,19 @@
<li><a href="#histograms-and-frequency-tables">Histograms and Frequency Tables</a>
<ul class="sectlevel2">
<li><a href="#histograms">histograms</a></li>
+<li><a href="#custom-histograms">Custom Histograms</a></li>
<li><a href="#frequency-tables">Frequency Tables</a></li>
</ul>
</li>
-<li><a href="#percentiles">Percentiles</a></li>
-<li><a href="#covariance-and-correlation">Covariance and Correlation</a>
+<li><a href="#percentiles">Percentiles</a>
+<ul class="sectlevel2">
+<li><a href="#quantile-plots">Quantile Plots</a></li>
+</ul>
+</li>
+<li><a href="#correlation-and-covariance">Correlation and Covariance</a>
<ul class="sectlevel2">
-<li><a href="#covariance-and-covariance-matrices">Covariance and Covariance Matrices</a></li>
<li><a href="#correlation-and-correlation-matrices">Correlation and Correlation Matrices</a></li>
+<li><a href="#covariance-and-covariance-matrices">Covariance and Covariance Matrices</a></li>
</ul>
</li>
<li><a href="#statistical-inference-tests">Statistical Inference Tests</a></li>
@@ -1679,234 +1704,213 @@
<section class="content">
<section id="preamble" aria-label="Preamble"><p>This section of the user guide covers the core statistical functions
available in math expressions.</p></section>
-<section class="sect1"><h2 id="descriptive-statistics">Descriptive Statistics</h2><p>The <code>describe</code> function can be used to return descriptive statistics about a
+<section class="sect1"><h2 id="descriptive-statistics">Descriptive Statistics</h2><p>The <code>describe</code> function returns descriptive statistics for a
numeric array. The <code>describe</code> function returns a single <strong>tuple</strong> with name/value
-pairs containing descriptive statistics.</p>
-<p>Below is a simple example that selects a random sample of documents,
-vectorizes the <strong>price_f</strong> field in the result set and uses the <code>describe</code> function to
-return descriptive statistics about the vector:</p>
-<div class="listingblock"><pre class="rouge highlight"><code class="language-text" data-lang="text"><pre class="highlight"><code>let(a=random(collection1, q="*:*", rows="1500", fl="price_f"),
- b=col(a, price_f),
+pairs containing the descriptive statistics.</p>
+<p>Below is a simple example that selects a random sample of documents from the <strong>logs</strong> collection,
+vectorizes the <code>response_d</code> field in the result set and uses the <code>describe</code> function to
+return descriptive statistics about the vector.</p>
+<div class="listingblock"><pre class="rouge highlight"><code class="language-text" data-lang="text"><pre class="highlight"><code>let(a=random(logs, q="*:*", fl="response_d", rows="50000"),
+ b=col(a, response_d),
c=describe(b))</code></pre></code></pre></div>
<p>When this expression is sent to the <code>/stream</code> handler it responds with:</p>
<div class="listingblock"><pre class="rouge highlight"><code class="language-json" data-lang="json"><pre class="highlight"><code><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
</span><span style="color: #000080">"result-set"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
</span><span style="color: #000080">"docs"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">[</span><span style="color: #bbbbbb">
</span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"c"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"sumsq"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">4999.041975263254</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"max"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.99995726</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"var"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.08344429493940454</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"geometricMean"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.36696588922559575</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"sum"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">7497.460565552007</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"kurtosis"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">-1.2000739963006035</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"N"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">15000</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"min"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.00012338161</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"mean"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.49983070437013266</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"popVar"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.08343873198640858</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"skewness"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">-0.001735537500095477</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"stdev"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.28886726179926403</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">}</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"sumsq"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">36674200601.78738</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"max"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">1068.854686837548</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"var"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">1957.9752647562789</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"geometricMean"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">854.1445499569674</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"sum"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">42764648.83319176</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"kurtosis"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.013189848821424377</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"N"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">50000</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"min"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">656.023249311864</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"mean"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">855.2929766638425</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"popVar"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">1957.936105250984</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"skewness"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.0014560741802307174</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"stdev"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">44.24901428005237</span><span style="color: #bbbbbb">
</span><span style="background-color: #f8f8f8">},</span><span style="color: #bbbbbb">
</span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
</span><span style="color: #000080">"EOF"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #000000;font-weight: bold">true</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"RESPONSE_TIME"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">305</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"RESPONSE_TIME"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">430</span><span style="color: #bbbbbb">
</span><span style="background-color: #f8f8f8">}</span><span style="color: #bbbbbb">
</span><span style="background-color: #f8f8f8">]</span><span style="color: #bbbbbb">
</span><span style="background-color: #f8f8f8">}</span><span style="color: #bbbbbb">
-</span><span style="background-color: #f8f8f8">}</span></code></pre></code></pre></div></section>
-<section class="sect1"><h2 id="histograms-and-frequency-tables">Histograms and Frequency Tables</h2><p>Histograms and frequency tables are are tools for understanding the distribution
+</span><span style="background-color: #f8f8f8">}</span></code></pre></code></pre></div>
+<p>Notice that the random sample contains 50,000 records and the response
+time is only 430 milliseconds. Samples of this size can be used to
+reliably estimate the statistics for very large underlying
+data sets with sub-second performance.</p>
+<p>The <code>describe</code> function can also be visualized in a table with Zeppelin-Solr:</p>
+<div class="imageblock"><img src="images/math-expressions/describe.png" alt="describe"></div></section>
+<section class="sect1"><h2 id="histograms-and-frequency-tables">Histograms and Frequency Tables</h2><p>Histograms and frequency tables are tools for visualizing the distribution
of a random variable.</p>
<p>The <code>hist</code> function creates a histogram designed for usage with continuous data. The
<code>freqTable</code> function creates a frequency table for use with discrete data.</p>
-<section class="sect2"><h3 id="histograms">histograms</h3><p>Below is an example that selects a random sample, creates a vector from the
-result set and uses the <code>hist</code> function to return a histogram with 5 bins.
-The <code>hist</code> function returns a list of tuples with summary statistics for each bin.</p>
-<div class="listingblock"><pre class="rouge highlight"><code class="language-text" data-lang="text"><pre class="highlight"><code>let(a=random(collection1, q="*:*", rows="15000", fl="price_f"),
- b=col(a, price_f),
- c=hist(b, 5))</code></pre></code></pre></div>
-<p>When this expression is sent to the <code>/stream</code> handler it responds with:</p>
-<div class="listingblock"><pre class="rouge highlight"><code class="language-json" data-lang="json"><pre class="highlight"><code><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"result-set"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"docs"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">[</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"c"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">[</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"prob"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.2057939717603699</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"min"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.000010371208</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"max"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.19996578</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"mean"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.10010319358402578</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"var"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.003366805016271609</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"cumProb"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.10293732468049072</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"sum"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">309.0185585938884</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"stdev"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.058024176136086666</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"N"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">3087</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">},</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"prob"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.19381868629885585</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"min"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.20007741</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"max"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.3999073</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"mean"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.2993590803885827</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"var"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.003401644034068929</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"cumProb"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.3025295802728267</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"sum"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">870.5362057700005</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"stdev"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.0583236147205309</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"N"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">2908</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">},</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"prob"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.20565789836690007</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"min"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.39995712</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"max"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.5999038</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"mean"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.4993620963792545</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"var"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.0033158364923609046</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"cumProb"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.5023006239697967</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"sum"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">1540.5320673300018</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"stdev"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.05758330046429177</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"N"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">3085</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">},</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"prob"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.19437108496008693</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"min"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.6000449</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"max"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.79973197</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"mean"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.7001752711861512</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"var"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.0033895105082360185</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"cumProb"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.7026537198687285</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"sum"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">2042.4112660500066</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"stdev"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.058219502816805456</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"N"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">2917</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">},</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"prob"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.20019582213899467</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"min"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.7999126</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"max"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.99987316</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"mean"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.8985428275824184</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"var"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.003312360017780078</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"cumProb"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.899450457219298</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"sum"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">2698.3241112299997</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"stdev"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.05755310606544253</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"N"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">3003</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">}</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">]</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">},</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"EOF"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #000000;font-weight: bold">true</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"RESPONSE_TIME"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">322</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">}</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">]</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">}</span><span style="color: #bbbbbb">
-</span><span style="background-color: #f8f8f8">}</span></code></pre></code></pre></div>
-<p>The <code>col</code> function can be used to <strong>vectorize</strong> a column of data from the list of tuples
-returned by the <code>hist</code> function.</p>
-<p>In the example below, the <strong>N</strong> field,
-which is the number of observations in the each bin, is returned as a vector.</p>
-<div class="listingblock"><pre class="rouge highlight"><code class="language-text" data-lang="text"><pre class="highlight"><code>let(a=random(collection1, q="*:*", rows="15000", fl="price_f"),
- b=col(a, price_f),
- c=hist(b, 11),
- d=col(c, N))</code></pre></code></pre></div>
+<section class="sect2"><h3 id="histograms">histograms</h3><p>In the example below a histogram is used to visualize a random sample of
+response times from the logs collection. The example retrieves the
+random sample with the <code>random</code> function and creates a vector from the <code>response_d</code> field
+in the result set. Then the <code>hist</code> function is applied to the vector
+to return a histogram with 22 bins. The <code>hist</code> function returns a
+list of tuples with summary statistics for each bin.</p>
+<div class="listingblock"><pre class="rouge highlight"><code class="language-text" data-lang="text"><pre class="highlight"><code>let(a=random(logs, q="*:*", fl="response_d", rows="50000"),
+ b=col(a, response_d),
+ c=hist(b, 22))</code></pre></code></pre></div>
<p>When this expression is sent to the <code>/stream</code> handler it responds with:</p>
-<div class="listingblock"><pre class="rouge highlight"><code class="language-json" data-lang="json"><pre class="highlight"><code><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"result-set"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"docs"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">[</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"d"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">[</span><span style="color: #bbbbbb">
- </span><span style="color: #009999">1387</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #009999">1396</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #009999">1391</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #009999">1357</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #009999">1384</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #009999">1360</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #009999">1367</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #009999">1375</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #009999">1307</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #009999">1310</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #009999">1366</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">]</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">},</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"EOF"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #000000;font-weight: bold">true</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"RESPONSE_TIME"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">307</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">}</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">]</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">}</span><span style="color: #bbbbbb">
-</span><span style="background-color: #f8f8f8">}</span></code></pre></code></pre></div></section>
+<div class="listingblock"><pre class="rouge highlight"><code class="language-text" data-lang="text"><pre class="highlight"><code>{
+ "result-set": {
+ "docs": [
+ {
+ "prob": 0.00004896007228311655,
+ "min": 675.573084576817,
+ "max": 688.3309631697003,
+ "mean": 683.805542728906,
+ "var": 50.9974629924082,
+ "cumProb": 0.000030022417162809913,
+ "sum": 2051.416628186718,
+ "stdev": 7.141250800273591,
+ "N": 3
+ },
+ {
+ "prob": 0.00029607514624062624,
+ "min": 696.2875238591652,
+ "max": 707.9706315779541,
+ "mean": 702.1110569558929,
+ "var": 14.136444379466969,
+ "cumProb": 0.00022705264963879807,
+ "sum": 11233.776911294284,
+ "stdev": 3.759846323916307,
+ "N": 16
+ },
+ {
+ "prob": 0.0011491235433157194,
+ "min": 709.1574910598678,
+ "max": 724.9027194369135,
+ "mean": 717.8554290699951,
+ "var": 20.6935845290122,
+ "cumProb": 0.0009858515418689757,
+ "sum": 41635.61488605971,
+ "stdev": 4.549020172412098,
+ "N": 58
+ },
+ ...
+ ]}}</code></pre></code></pre></div>
+<p>With Zeppelin-Solr the histogram can be first visualized as a table:</p>
+<div class="imageblock"><img src="images/math-expressions/histtable.png" alt="histtable"></div>
+<p>Then the histogram can be visualized with an area chart by plotting the <strong>mean</strong> of
+the bins on the <strong>x-axis</strong> and the <strong>prob</strong> (probability) on the <strong>y-axis</strong>:</p>
+<div class="imageblock"><img src="images/math-expressions/hist.png" alt="hist"></div>
+<p>The cumulative probability can be plotted by switching the <strong>y-axis</strong> to the <strong>cumProb</strong> column:</p>
+<div class="imageblock"><img src="images/math-expressions/cumProb.png" alt="cumProb"></div></section>
+<section class="sect2"><h3 id="custom-histograms">Custom Histograms</h3><p>Custom histograms can be defined and visualized by combining the output from multiple
+<code>stats</code> functions into a single histogram. Instead of automatically binning a numeric
+field the custom histogram allows for comparison of bins based on queries.</p>
+<div class="admonitionblock note">
+<table>
+<tr>
+<td class="icon">
+<i class="fa icon-note" title="Note"></i>
+</td>
+<td class="content">
+The <code>stats</code> function is first discussed in the <strong>Searching, Sampling and Aggregation</strong> section of the
+user guide.
+</td>
+</tr>
+</table>
+</div>
+<p>A simple example will illustrate how to define and visualize a custom histogram.</p>
+<p>In this example, three <code>stats</code> functions are wrapped in a <code>plist</code> function. The
+<code>plist</code> (parallel list) function executes each of its internal functions in parallel
+and concatenates the results into a single stream. <code>plist</code> also maintains the order
+of the outputs from each of the sub-functions. In this example each <code>stats</code> function
+computes the count of documents that match a specific query. In this case they count the
+number of documents that contain the terms copper, gold and silver. The list of tuples
+with the counts is then stored in variable <strong>a</strong>.</p>
+<p>Then an <code>array</code> of labels is created and set to variable <strong>l</strong>.</p>
+<p>Finally the <code>zplot</code> function is used to plot the labels vector and the <code>count(*)</code> column.
+Notice the <code>col</code> function is used inside of the <code>zplot</code> function to extract the
+counts from the <code>stats</code> results.</p>
+<div class="imageblock"><img src="images/math-expressions/custom-hist.png" alt="custom hist"></div></section>
<section class="sect2"><h3 id="frequency-tables">Frequency Tables</h3><p>The <code>freqTable</code> function returns a frequency distribution for a discrete data set.
The <code>freqTable</code> function doesn’t create bins like the histogram. Instead it counts
the occurrence of each discrete data value and returns a list of tuples with the
-frequency statistics for each value. Fields from a frequency table can be vectorized using
-using the <code>col</code> function in the same manner as a histogram.</p>
-<p>Below is a simple example of a frequency table built from a random sample of
-a discrete variable.</p>
-<div class="listingblock"><pre class="rouge highlight"><code class="language-text" data-lang="text"><pre class="highlight"><code>let(a=random(collection1, q="*:*", rows="15000", fl="day_i"),
- b=col(a, day_i),
- c=freqTable(b))</code></pre></code></pre></div>
+frequency statistics for each value.</p>
+<p>Below is an example of a frequency table built from a result set
+of rounded <strong>differences</strong> in daily opening stock prices for the stock ticker <strong>amzn</strong>.</p>
+<p>This example is interesting because it shows a multi-step process to arrive
+at the result. The first step is to <strong>search</strong> for records in the the <strong>stocks</strong>
+collection with a ticker of <strong>amzn</strong>. Notice that the result set is sorted by
+date ascending and it returns the <code>open_d</code> field which is the opening price for
+the day.</p>
+<p>The <code>open_d</code> field is then vectorized and set to variable <strong>b</strong>, which now contains
+a vector of opening prices ordered by date ascending.</p>
+<p>The <code>diff</code> function is then used to calculate the <strong>first difference</strong> for the
+vector of opening prices. The first difference simply subtracts the previous value
+from each value in the array. This will provide an array of price differences
+for each day which will show daily change in opening price.</p>
+<p>Then the <code>round</code> function is used to round the price differences to the nearest
+integer to create a vector of discrete values. The <code>round</code> function in this
+example is effectively <strong>binning</strong> continuous data at integer boundaries.</p>
+<p>Finally the <code>freqTable</code> function is run on the discrete values to calculate
+the frequency table.</p>
+<div class="listingblock"><pre class="rouge highlight"><code class="language-text" data-lang="text"><pre class="highlight"><code>let(a=search(stocks,
+ q="ticker_s:amzn",
+ fl="open_d, date_dt",
+ sort="date_dt asc",
+ rows=25000),
+ b=col(a, open_d),
+ c=diff(b),
+ d=round(c),
+ e=freqTable(d))</code></pre></code></pre></div>
<p>When this expression is sent to the <code>/stream</code> handler it responds with:</p>
-<div class="listingblock"><pre class="rouge highlight"><code class="language-json" data-lang="json"><pre class="highlight"><code><span style="color: #bbbbbb"> </span><span style="color: #d14">"result-set"</span><span style="color: #a61717;background-color: #e3d2d2">:</span><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"docs"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">[</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"c"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">[</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"pct"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.0318</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"count"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">477</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"cumFreq"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">477</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"cumPct"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.0318</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"value"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">},</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"pct"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.033133333333333334</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"count"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">497</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"cumFreq"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">974</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"cumPct"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.06493333333333333</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"value"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">1</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">},</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"pct"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.03426666666666667</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"count"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">514</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"cumFreq"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">1488</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"cumPct"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.0992</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"value"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">2</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">},</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"pct"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.0346</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"count"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">519</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"cumFreq"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">2007</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"cumPct"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.1338</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"value"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">3</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">},</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"pct"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.03133333333333333</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"count"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">470</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"cumFreq"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">2477</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"cumPct"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.16513333333333333</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"value"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">4</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">},</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"pct"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.03333333333333333</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"count"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">500</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"cumFreq"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">2977</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"cumPct"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.19846666666666668</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"value"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">5</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">}</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">]</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">},</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"EOF"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #000000;font-weight: bold">true</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"RESPONSE_TIME"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">281</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">}</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">]</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">}</span><span style="color: #bbbbbb">
-</span><span style="color: #a61717;background-color: #e3d2d2">}</span></code></pre></code></pre></div></section></section>
+<div class="listingblock"><pre class="rouge highlight"><code class="language-text" data-lang="text"><pre class="highlight"><code> {
+ "result-set": {
+ "docs": [
+ {
+ "pct": 0.00019409937888198756,
+ "count": 1,
+ "cumFreq": 1,
+ "cumPct": 0.00019409937888198756,
+ "value": -57
+ },
+ {
+ "pct": 0.00019409937888198756,
+ "count": 1,
+ "cumFreq": 2,
+ "cumPct": 0.00038819875776397513,
+ "value": -51
+ },
+ {
+ "pct": 0.00019409937888198756,
+ "count": 1,
+ "cumFreq": 3,
+ "cumPct": 0.0005822981366459627,
+ "value": -49
+ },
+ ...
+ ]}}</code></pre></code></pre></div>
+<p>With Zeppelin-Solr the frequency table can be first visualized as a table:</p>
+<div class="imageblock"><img src="images/math-expressions/freqTable.png" alt="freqTable"></div>
+<p>The frequency table can then be plotted by switching to a scatter chart and selecting
+the <strong>value</strong> column for the <strong>x-axis</strong> and the <strong>count</strong> column for the <strong>y-axis</strong></p>
+<div class="imageblock"><img src="images/math-expressions/freqTable1.png" alt="freqTable1"></div>
+<p>Notice that the visualization nicely displays the frequency of daily change in stock prices
+rounded to integers. The most frequently occurring value is 0 with 1494 occurrences followed by
+ -1 and 1 with around 700 occurrences.</p></section></section>
<section class="sect1"><h2 id="percentiles">Percentiles</h2><p>The <code>percentile</code> function returns the estimated value for a specific percentile in
-a sample set. The example below returns the estimation for the 95th percentile
-of the <strong>price_f</strong> field.</p>
-<div class="listingblock"><pre class="rouge highlight"><code class="language-text" data-lang="text"><pre class="highlight"><code>let(a=random(collection1, q="*:*", rows="15000", fl="price_f"),
- b=col(a, price_f),
- c=percentile(b, 95))</code></pre></code></pre></div>
+a sample set. The example below returns a random sample containing the <code>response_d</code> field
+from the logs collection. The <code>response_d</code> field is vectorized and the 20th percentile
+is calculated for the vector:</p>
+<div class="listingblock"><pre class="rouge highlight"><code class="language-text" data-lang="text"><pre class="highlight"><code>let(a=random(logs, q="*:*", rows="15000", fl="response_d"),
+ b=col(a, response_d),
+ c=percentile(b, 20))</code></pre></code></pre></div>
<p>When this expression is sent to the <code>/stream</code> handler it responds with:</p>
<div class="listingblock"><pre class="rouge highlight"><code class="language-json" data-lang="json"><pre class="highlight"><code><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
</span><span style="color: #000080">"result-set"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
</span><span style="color: #000080">"docs"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">[</span><span style="color: #bbbbbb">
</span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"c"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">312.94</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"c"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">818.073554</span><span style="color: #bbbbbb">
</span><span style="background-color: #f8f8f8">},</span><span style="color: #bbbbbb">
</span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
</span><span style="color: #000080">"EOF"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #000000;font-weight: bold">true</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
@@ -1915,10 +1919,10 @@ of the <strong>price_f</strong> field.</
</span><span style="background-color: #f8f8f8">]</span><span style="color: #bbbbbb">
</span><span style="background-color: #f8f8f8">}</span><span style="color: #bbbbbb">
</span><span style="background-color: #f8f8f8">}</span></code></pre></code></pre></div>
-<p>The <code>percentile</code> function also operates on an array of percentile values.
+<p>The <code>percentile</code> function can also compute an array of percentile values.
The example below is computing the 20th, 40th, 60th and 80th percentiles for a random sample
-of the <strong>response_d</strong> field:</p>
-<div class="listingblock"><pre class="rouge highlight"><code class="language-text" data-lang="text"><pre class="highlight"><code>let(a=random(collection2, q="*:*", rows="15000", fl="response_d"),
+of the <code>response_d</code> field:</p>
+<div class="listingblock"><pre class="rouge highlight"><code class="language-text" data-lang="text"><pre class="highlight"><code>let(a=random(logs, q="*:*", rows="15000", fl="response_d"),
b=col(a, response_d),
c=percentile(b, array(20,40,60,80)))</code></pre></code></pre></div>
<p>When this expression is sent to the <code>/stream</code> handler it responds with:</p>
@@ -1939,100 +1943,114 @@ of the <strong>response_d</strong> field
</span><span style="background-color: #f8f8f8">}</span><span style="color: #bbbbbb">
</span><span style="background-color: #f8f8f8">]</span><span style="color: #bbbbbb">
</span><span style="background-color: #f8f8f8">}</span><span style="color: #bbbbbb">
-</span><span style="background-color: #f8f8f8">}</span></code></pre></code></pre></div></section>
-<section class="sect1"><h2 id="covariance-and-correlation">Covariance and Correlation</h2><p>Covariance and Correlation measure how random variables move
+</span><span style="background-color: #f8f8f8">}</span></code></pre></code></pre></div>
+<section class="sect2"><h3 id="quantile-plots">Quantile Plots</h3><p>Quantile plots or QQ Plots are powerful tools for visually comparing two or more distributions.</p>
+<p>A quantile plot, plots the percentiles from two or more distributions in the same visualization. This allows
+for visual comparison of the distributions at each percentile. A simple example will help illustrate the power
+of quantile plots.</p>
+<p>In this example the distribution of daily stock price changes for two stock tickers, <strong>goog</strong> and
+<strong>amzn</strong>, are visualized with a quantile plot.</p>
+<p>The example first creates an array of values representing the percentiles that will be calculated and sets this array
+to variable <strong>p</strong>. Then random samples of the <code>change_d</code> field are drawn for the tickers <strong>amzn</strong> and <strong>goog</strong>. The <code>change_d</code> field
+represents the change in stock price for one day. Then the <code>change_d</code> field is vectorized for both samples and placed
+in the variables <strong>amzn</strong> and <strong>goog</strong>. The <code>percentile</code> function is then used to calculate the percentiles for both vectors. Notice that
+the variable <strong>p</strong> is used to specify the list of percentiles that are calculated.</p>
+<p>Finally <code>zplot</code> is used to plot the percentiles sequence on the <strong>x-axis</strong> and the calculated
+percentile values for both distributions on the <strong>y-axis</strong>. And a line plot is used
+to visualize the QQ plot.</p>
+<div class="imageblock"><img src="images/math-expressions/quantile-plot.png" alt="quantile plot"></div>
+<p>This quantile plot provides a clear picture of the distributions of daily price changes for <strong>amzn</strong>
+and <strong>googl</strong>. In the plot the <strong>x-axis</strong> is the percentiles and the <strong>y-axis</strong> is the percentile value calculated.</p>
+<p>Notice that the <strong>goog</strong> percentile value starts lower and ends higher than the <strong>amzn</strong> plot and that there is a
+steeper slope. This shows the greater variability in the <strong>goog</strong> price change distribution. The plot gives a clear picture
+of the difference
+in the distributions across the full range of percentiles.</p></section></section>
+<section class="sect1"><h2 id="correlation-and-covariance">Correlation and Covariance</h2><p>Correlation and Covariance measure how random variables fluctuate
together.</p>
-<section class="sect2"><h3 id="covariance-and-covariance-matrices">Covariance and Covariance Matrices</h3><p>The <code>cov</code> function calculates the covariance of two sample sets of data.</p>
-<p>In the example below covariance is calculated for two numeric
-arrays.</p>
-<p>The example below uses arrays created by the <code>array</code> function. Its important to note that
-vectorized data from SolrCloud collections can be used with any function that
-operates on arrays.</p>
-<div class="listingblock"><pre class="rouge highlight"><code class="language-text" data-lang="text"><pre class="highlight"><code>let(a=array(1, 2, 3, 4, 5),
- b=array(100, 200, 300, 400, 500),
- c=cov(a, b))</code></pre></code></pre></div>
-<p>When this expression is sent to the <code>/stream</code> handler it responds with:</p>
-<div class="listingblock"><pre class="rouge highlight"><code class="language-json" data-lang="json"><pre class="highlight"><code><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"result-set"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"docs"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">[</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"c"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.9484775349999998</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">},</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"EOF"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #000000;font-weight: bold">true</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"RESPONSE_TIME"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">286</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">}</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">]</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">}</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">}</span></code></pre></code></pre></div>
+<section class="sect2"><h3 id="correlation-and-correlation-matrices">Correlation and Correlation Matrices</h3><p>Correlation is a measure of the linear correlation between two vectors. Correlation is scaled between
+-1 and 1.</p>
+<p>Three correlation types are supported:</p>
+<div class="ulist"><ul><li><p><strong>pearsons</strong> (default)</p></li><li><p><strong>kendalls</strong></p></li><li><p><strong>spearmans</strong></p></li></ul></div>
+<p>The type of correlation is specified by adding the <strong>type</strong> named parameter in the
+function call.</p>
+<p>In the example below a random sample containing two fields, <code>filesize_d</code> and <code>response_d</code>, is drawn from
+the logs collection using the <code>random</code> function. The fields are vectorized into the
+variables <strong>x</strong> and <strong>y</strong> and then <strong>Spearman’s</strong> correlation for
+the two vectors is calculated using the <code>corr</code> function.</p>
+<div class="imageblock"><img src="images/math-expressions/correlation.png" alt="correlation"></div>
+<section class="sect3"><h4 id="correlation-matrices">Correlation Matrices</h4><p>Correlation matrices are powerful tools for visualizing the correlation between two or more
+vectors.</p>
+<p>The <code>corr</code> function builds a correlation matrix
+if a matrix is passed as the parameter. The correlation matrix is computed by correlating the <strong>columns</strong>
+of the matrix.</p>
+<p>The example below demonstrates the power of correlation matrices combined with 2 dimensional faceting.</p>
+<p>In this example the <code>facet2D</code> function is used to generate a two dimensional facet aggregation
+over the fields <code>complaint_type_s</code> and <code>zip_s</code> from the <strong>nyc311</strong> complaints database.
+The <strong>top 20</strong> complaint types and the <strong>top 25</strong> zip codes for each complaint type are aggregated.
+The result is a stream of tuples each containing the fields <code>complaint_type_s</code>, <code>zip_s</code> and
+the count for the pair.</p>
+<p>The <code>pivot</code> function is then used to pivot the fields into a <strong>matrix</strong> with the <code>zip_s</code>
+field as the <strong>rows</strong> and the <code>complaint_type_s</code> field as the <strong>columns</strong>. The <code>count(*)</code> field populates
+the values in the cells of the matrix.</p>
+<p>The <code>corr</code> function is then used correlate the <strong>columns</strong> of the matrix. This produces a correlation matrix
+that shows how complaint types are correlated based on the zip codes they appear in. Another way to look at this
+is it shows how the different complaint types tend to co-occur across zip codes.</p>
+<p>Finally the <code>zplot</code> function is used to plot the correlation matrix as a heat map.</p>
+<div class="imageblock"><img src="images/math-expressions/corrmatrix.png" alt="corrmatrix"></div>
+<p>Notice in the example the correlation matrix is square with complaint types shown on both
+the <strong>x</strong> and y-axises. The color of the cells in the heat map shows the
+intensity of the correlation between the complaint types.</p>
+<p>The heat map is interactive, so mousing over one of the cells pops up the values
+for the cell.</p>
+<div class="imageblock"><img src="images/math-expressions/corrmatrix2.png" alt="corrmatrix2"></div>
+<p>Notice that HEAT/HOT WATER and UNSANITARY CONDITION complaints have a correlation of 8 (rounded to the nearest
+tenth).</p></section></section>
+<section class="sect2"><h3 id="covariance-and-covariance-matrices">Covariance and Covariance Matrices</h3><p>Covariance is an unscaled measure of correlation.</p>
+<p>The <code>cov</code> function calculates the covariance of two vectors of data.</p>
+<p>In the example below a random sample containing two fields, <code>filesize_d</code> and <code>response_d</code>, is drawn from
+the logs collection using the <code>random</code> function. The fields are vectorized into the
+variables <strong>x</strong> and <strong>y</strong> and then the covariance for
+the two vectors is calculated using the <code>cov</code> function.</p>
+<div class="imageblock"><img src="images/math-expressions/covariance.png" alt="covariance"></div>
<p>If a matrix is passed to the <code>cov</code> function it will automatically compute a covariance
-matrix for the columns of the matrix.</p>
-<p>Notice in the example three numeric arrays are added as rows
-in a matrix. The matrix is then transposed to turn the rows into
-columns, and the covariance matrix is computed for the columns of the
-matrix.</p>
-<div class="listingblock"><pre class="rouge highlight"><code class="language-text" data-lang="text"><pre class="highlight"><code>let(a=array(1, 2, 3, 4, 5),
- b=array(100, 200, 300, 400, 500),
- c=array(30, 40, 80, 90, 110),
- d=transpose(matrix(a, b, c)),
- e=cov(d))</code></pre></code></pre></div>
+matrix for the <strong>columns</strong> of the matrix.</p>
+<p>Notice in the example below that the <strong>x</strong> and <strong>y</strong> vectors are added to a matrix.
+The matrix is then transposed to turn the rows into columns,
+and the covariance matrix is computed for the columns of the matrix.</p>
+<div class="listingblock"><pre class="rouge highlight"><code class="language-text" data-lang="text"><pre class="highlight"><code>let(a=random(logs, q="*:*", fl="filesize_d, response_d", rows=50000),
+ x=col(a, filesize_d),
+ y=col(a, response_d),
+ m=transpose(matrix(x, y)),
+ covariance=cov(m))</code></pre></code></pre></div>
<p>When this expression is sent to the <code>/stream</code> handler it responds with:</p>
<div class="listingblock"><pre class="rouge highlight"><code class="language-json" data-lang="json"><pre class="highlight"><code><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
</span><span style="color: #000080">"result-set"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
</span><span style="color: #000080">"docs"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">[</span><span style="color: #bbbbbb">
</span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"e"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">[</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">[</span><span style="color: #bbbbbb">
- </span><span style="color: #009999">2.5</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #009999">250</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #009999">52.5</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">],</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"covariance"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">[</span><span style="color: #bbbbbb">
</span><span style="background-color: #f8f8f8">[</span><span style="color: #bbbbbb">
- </span><span style="color: #009999">250</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #009999">25000</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #009999">5250</span><span style="color: #bbbbbb">
+ </span><span style="color: #009999">4018404.072532102</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #009999">80243.3948172242</span><span style="color: #bbbbbb">
</span><span style="background-color: #f8f8f8">],</span><span style="color: #bbbbbb">
</span><span style="background-color: #f8f8f8">[</span><span style="color: #bbbbbb">
- </span><span style="color: #009999">52.5</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #009999">5250</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #009999">1150</span><span style="color: #bbbbbb">
+ </span><span style="color: #009999">80243.3948172242</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
+ </span><span style="color: #009999">1948.3216661122592</span><span style="color: #bbbbbb">
</span><span style="background-color: #f8f8f8">]</span><span style="color: #bbbbbb">
</span><span style="background-color: #f8f8f8">]</span><span style="color: #bbbbbb">
</span><span style="background-color: #f8f8f8">},</span><span style="color: #bbbbbb">
</span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
</span><span style="color: #000080">"EOF"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #000000;font-weight: bold">true</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"RESPONSE_TIME"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">2</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">}</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">]</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">}</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">}</span></code></pre></code></pre></div></section>
-<section class="sect2"><h3 id="correlation-and-correlation-matrices">Correlation and Correlation Matrices</h3><p>Correlation is measure of covariance that has been scaled between
--1 and 1.</p>
-<p>Three correlation types are supported:</p>
-<div class="ulist"><ul><li><p><strong>pearsons</strong> (default)</p></li><li><p><strong>kendalls</strong></p></li><li><p><strong>spearmans</strong></p></li></ul></div>
-<p>The type of correlation is specified by adding the <strong>type</strong> named parameter in the
-function call. The example below demonstrates the use of the <strong>type</strong>
-named parameter.</p>
-<div class="listingblock"><pre class="rouge highlight"><code class="language-text" data-lang="text"><pre class="highlight"><code>let(a=array(1, 2, 3, 4, 5),
- b=array(100, 200, 300, 400, 5000),
- c=corr(a, b, type=spearmans))</code></pre></code></pre></div>
-<p>When this expression is sent to the <code>/stream</code> handler it responds with:</p>
-<div class="listingblock"><pre class="rouge highlight"><code class="language-json" data-lang="json"><pre class="highlight"><code><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"result-set"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"docs"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="background-color: #f8f8f8">[</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"c"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0.7432941462471664</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">},</span><span style="color: #bbbbbb">
- </span><span style="background-color: #f8f8f8">{</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"EOF"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #000000;font-weight: bold">true</span><span style="background-color: #f8f8f8">,</span><span style="color: #bbbbbb">
- </span><span style="color: #000080">"RESPONSE_TIME"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">0</span><span style="color: #bbbbbb">
+ </span><span style="color: #000080">"RESPONSE_TIME"</span><span style="background-color: #f8f8f8">:</span><span style="color: #bbbbbb"> </span><span style="color: #009999">534</span><span style="color: #bbbbbb">
</span><span style="background-color: #f8f8f8">}</span><span style="color: #bbbbbb">
</span><span style="background-color: #f8f8f8">]</span><span style="color: #bbbbbb">
</span><span style="background-color: #f8f8f8">}</span><span style="color: #bbbbbb">
</span><span style="background-color: #f8f8f8">}</span></code></pre></code></pre></div>
-<p>Like the <code>cov</code> function, the <code>corr</code> function automatically builds a correlation matrix
-if a matrix is passed as a parameter. The correlation matrix is built by correlating the columns
-of the matrix passed in.</p></section></section>
+<p>The covariance matrix contains both the variance for the two vectors and the covariance between the vectors
+in the following format:</p>
+<div class="listingblock"><pre class="rouge highlight"><code class="language-text" data-lang="text"><pre class="highlight"><code> x y
+ x [4018404.072532102, 80243.3948172242],
+ y [80243.3948172242, 1948.3216661122592]</code></pre></code></pre></div>
+<p>The covariance matrix is always square. So a covariance matrix created from 3 vectors will produce a 3 x 3 matrix.</p></section></section>
<section class="sect1"><h2 id="statistical-inference-tests">Statistical Inference Tests</h2><p>Statistical inference tests test a hypothesis on <strong>random samples</strong> and return p-values which
can be used to infer the reliability of the test for the entire population.</p>
<p>The following statistical inference tests are available:</p>
@@ -2042,7 +2060,7 @@ random samples.</p></li><li><p><code>pai
in the means of two random samples with paired data.</p></li><li><p><code>gTestDataSet</code>: The G-test tests if two samples of binned discrete data were drawn
from the same population.</p></li><li><p><code>chiSquareDataset</code>: The Chi-Squared test tests if two samples of binned discrete data were
drawn from the same population.</p></li><li><p><code>mannWhitney</code>: The Mann-Whitney test is a non-parametric test that tests if two
-samples of continuous were pulled
+samples of continuous data were pulled
from the same population. The Mann-Whitney test is often used instead of the T-test when the
underlying assumptions of the T-test are not
met.</p></li><li><p><code>ks</code>: The Kolmogorov-Smirnov test tests if two samples of continuous data were drawn from
@@ -2227,10 +2245,10 @@ is the number of standard deviations a n
<nav class="scrollnav row">
<div class="col-lg-12">
- <a class="btn btn-primary prev float-left" href="term-vectors.html">Text Analysis and Term Vectors</a>
+ <a class="btn btn-primary prev float-left" href="probability-distributions.html">Probability Distributions</a>
- <a class="btn btn-primary next float-right" href="probability-distributions.html">Probability Distributions</a>
+ <a class="btn btn-primary next float-right" href="regression.html">Linear Regression</a>
</div>
</nav>
@@ -2242,7 +2260,7 @@ is the number of standard deviations a n
<div class="row">
<div class="col-lg-12 footer">
©2021 Apache Software Foundation. All rights reserved. <br />
- Site Version: 8.8-DRAFT <br />Site last generated: 2021-01-19 <br />
+ Site Version: 8.8 <br />Site last generated: 2021-01-28 <br />
<p><img src="images/solr-sunOnly-small.png" alt="Apache Solr"/></p>
</div>
</div>