You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@madlib.apache.org by xt...@apache.org on 2016/04/07 23:47:29 UTC
[26/51] [abbrv] [partial] incubator-madlib-site git commit: Update
doc for 1.9 release
http://git-wip-us.apache.org/repos/asf/incubator-madlib-site/blob/c506dd05/docs/latest/group__grp__path.html
----------------------------------------------------------------------
diff --git a/docs/latest/group__grp__path.html b/docs/latest/group__grp__path.html
index dee1dbb..bb4227f 100644
--- a/docs/latest/group__grp__path.html
+++ b/docs/latest/group__grp__path.html
@@ -6,7 +6,7 @@
<meta http-equiv="X-UA-Compatible" content="IE=9"/>
<meta name="generator" content="Doxygen 1.8.10"/>
<meta name="keywords" content="madlib,postgres,greenplum,machine learning,data mining,deep learning,ensemble methods,data science,market basket analysis,affinity analysis,pca,lda,regression,elastic net,huber white,proportional hazards,k-means,latent dirichlet allocation,bayes,support vector machines,svm"/>
-<title>MADlib: Grp_path</title>
+<title>MADlib: Path Functions</title>
<link href="tabs.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="jquery.js"></script>
<script type="text/javascript" src="dynsections.js"></script>
@@ -24,22 +24,17 @@
<script type="text/javascript">
$(document).ready(function() { init_search(); });
</script>
-<script type="text/x-mathjax-config">
- MathJax.Hub.Config({
- extensions: ["tex2jax.js", "TeX/AMSmath.js", "TeX/AMSsymbols.js"],
- jax: ["input/TeX","output/HTML-CSS"],
-});
-</script><script src="../mathjax/MathJax.js"></script>
<!-- hack in the navigation tree -->
-<script type="text/javascript" src="navtree_hack.js"></script>
+<script type="text/javascript" src="eigen_navtree_hacks.js"></script>
<link href="doxygen.css" rel="stylesheet" type="text/css" />
+<link href="madlib_extra.css" rel="stylesheet" type="text/css"/>
<!-- google analytics -->
<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
- ga('create', 'UA-45382226-1', 'auto');
+ ga('create', 'UA-45382226-1', 'madlib.net');
ga('send', 'pageview');
</script>
</head>
@@ -49,10 +44,10 @@
<table cellspacing="0" cellpadding="0">
<tbody>
<tr style="height: 56px;">
- <td id="projectlogo"><a href="http://madlib.incubator.apache.org"><img alt="Logo" src="madlib.png" height="50" style="padding-left:0.5em;" border="0"/ ></a></td>
+ <td id="projectlogo"><a href="http://madlib.net"><img alt="Logo" src="madlib.png" height="50" style="padding-left:0.5em;" border="0"/ ></a></td>
<td style="padding-left: 0.5em;">
<div id="projectname">
- <span id="projectnumber">1.8</span>
+ <span id="projectnumber">1.9</span>
</div>
<div id="projectbrief">User Documentation for MADlib</div>
</td>
@@ -111,7 +106,7 @@ $(document).ready(function(){initNavTree('group__grp__path.html','');});
<div class="header">
<div class="headertitle">
-<div class="title">Grp_path</div> </div>
+<div class="title">Path Functions<div class="ingroups"><a class="el" href="group__grp__utility__functions.html">Utility Functions</a></div></div> </div>
</div><!--header-->
<div class="contents">
<div class="toc"><b>Contents</b> </p><ul>
@@ -119,24 +114,31 @@ $(document).ready(function(){initNavTree('group__grp__path.html','');});
<a href="#syntax">Function Syntax</a> </li>
<li>
<a href="#examples">Examples</a> </li>
+<li>
+<a href="#nomenclature">Nomenclature</a> </li>
+<li>
+<a href="#literature">Literature</a> </li>
</ul>
-</div><p>The goal of the MADlib path function is to perform regular pattern matching over a sequence of rows, and to extract useful information about the matches. The useful information could be a simple count of matches or something more involved like aggregation.</p>
-<p>There are many use cases where path functions are typically used:</p><ul>
-<li>Web analytics (clickstream)</li>
-<li>Marketing revenue attribution</li>
-<li>Telephone calling patterns</li>
-<li>Stock market trading sequences</li>
-<li>Predictive maintenance</li>
-<li>Genomics sequencing</li>
-</ul>
+</div><p>The goal of the MADlib path function is to perform regular pattern matching over a sequence of rows, and to extract useful information about the pattern matches. The useful information could be a simple count of matches or something more involved like aggregations or window functions.</p>
+<p>Symbols are used to identify particular rows of interest. Then, standard PostgreSQL pattern matching using symbols can be applied to identify patterns across the rows of interest. (This is similar in concept to regular expressions which match patterns within strings of text.)</p>
+<p>For example, a symbol can be defined for purchase events by on-line shoppers. Then, preceding events that led to the purchase can be identified and operated on, perhaps to find the common actions that resulted in a purchase. Or conversely, to find actions that resulted in an exit without a purchase having been made.</p>
+<p>Steps on how to use path functions:</p>
+<ol type="1">
+<li>Partition input rows.</li>
+<li>Order the partitions.</li>
+<li>Define symbols to match rows of interest.</li>
+<li>Define regular expression of symbols and operators to define patterns to match in your ordered partitions.</li>
+<li>Define an aggregate function to compute for each pattern match.</li>
+<li>If desired, output the pattern matches for inspection or to operate on them with subsequent queries.</li>
+</ol>
<p><a class="anchor" id="syntax"></a></p><dl class="section user"><dt>Function Syntax</dt><dd><pre class="syntax">
path(
source_table,
output_table,
partition_expr,
order_expr,
- pattern,
symbol,
+ pattern,
aggregate_func,
persist_rows
)
@@ -147,19 +149,29 @@ path(
<p class="enddd"></p>
</dd>
<dt>output_table </dt>
-<dd><p class="startdd">VARCHAR. Name of the result table. This table contains columns predicated by the <em>result</em> argument (shown below).</p>
+<dd><p class="startdd">VARCHAR. Name of the result table.</p>
<p class="enddd"></p>
</dd>
<dt>partition_expr </dt>
-<dd><p class="startdd">VARCHAR. The 'partition_expr' can be a list of columns or expressions (separated by comma) to divide all rows into groups, or partitions, that share the same values of the partition expression(s). For each row, the matching is applied across the rows that fall into the same partition. This can be NULL or '' to indicate the matching is to be applied on the whole table.</p>
+<dd><p class="startdd">VARCHAR. The 'partition_expr' can be a single column or a list of comma-separated columns/expressions to divide all rows into groups, or partitions. Matching is applied across the rows that fall into t he same partition. This can be NULL or '' to indicate the matching is to be applied to the whole table.</p>
<p class="enddd"></p>
</dd>
<dt>order_expr </dt>
-<dd><p class="startdd">VARCHAR. This expression controls the order in which rows are processed or matched in a partition. </p>
+<dd><p class="startdd">VARCHAR. This expression controls the order in which rows are processed or matched in a partition. For example, time is a common way to order partitions. </p>
+<p class="enddd"></p>
+</dd>
+<dt>symbol </dt>
+<dd><p class="startdd">VARCHAR. Symbols enable you to express patterns of interest in a simple way (see definition of ‘pattern’ argument below). A symbol identifies a row of a particular type that you’re searching for as part of a pattern match. Symbol definition uses the standard PostgreSQL assignment statement 'identifier := expression;' [1]. A given row can only match one symbol. If a row matches multiple symbols, the symbol that comes first in the symbol definition list will take precedence. </p>
<p class="enddd"></p>
</dd>
<dt>pattern </dt>
-<dd><p class="startdd">VARCHAR. The PATTERN clause defines the pattern that path searches for. You express the PATTERN using symbols and operators. Symbols defined using more than 1 characters need to be wrapped in parentheses '()'. The following pattern-matching metacharacters are supported: </p><ul>
+<dd><p class="startdd">VARCHAR. The 'pattern' clause defines the pattern that the path algorithm searches for. You express the pattern using symbols and operators following regular PostgreSQL pattern matching syntax and rules [2].</p>
+<p><a class="anchor" id="note"></a></p><dl class="section note"><dt>Note</dt><dd>Symbols defined using more than one (1) character need to be enclosed in parentheses '()' when referenced in the 'pattern' argument. For example:<ul>
+<li>a symbol defined as 'a' in the 'symbol' argument can be used directly in the 'pattern' argument</li>
+<li>a symbol defined as 'abc' in the 'symbol' argument must be written as '(abc)' in the 'pattern' argument</li>
+</ul>
+</dd></dl>
+<p>The following pattern matching metacharacters are supported: </p><ul>
<li>
| denotes alternation (either of two alternatives). </li>
<li>
@@ -179,91 +191,231 @@ Parentheses () can be used to group items into a single logical item. </li>
</ul>
<p class="enddd"></p>
</dd>
-<dt>symbol </dt>
-<dd><p class="startdd">VARCHAR. A symbol represents a row of a particular type that you’re searching for as part of a row sequence. In the SYMBOLS clause, you write a predicate to define the type of row that matches the symbol. </p>
-<p class="enddd"></p>
-</dd>
<dt>aggregate_func </dt>
-<dd><p class="startdd">VARCHAR. A comma-separated list of window functions and aggregates to be applied on the matched window. </p>
+<dd><p class="startdd">VARCHAR. A comma-separated list of aggregates to be applied to the pattern matches [3]. Please note that window functions cannot currently be used in the parameter 'aggregate_func'. If you want to use a window function [4], output the pattern matches and write a SQL query with a window function over the output tuples (see 'persist_rows' parameter below).</p>
+<p>If you just want to output the pattern matched rows and not compute any aggregates, you can put NULL or '' in the 'aggregate_func' parameter. </p>
<p class="enddd"></p>
</dd>
<dt>persist_rows </dt>
-<dd><p class="startdd">BOOLEAN. If TRUE the matched rows are persisted in another table. This table is named as <output_table>_tuples (the string "_tuples" is added as suffix to the value of <em>output_table</em>). </p>
+<dd><p class="startdd">BOOLEAN. If TRUE the matched rows are persisted in a separate output table. This table is named as <output_table>_tuples (the string "_tuples" is added as suffix to the value of <em>output_table</em>). </p>
<p class="enddd"></p>
</dd>
</dl>
<p><a class="anchor" id="examples"></a></p><dl class="section user"><dt>Examples</dt><dd></dd></dl>
-<ul>
-<li>Build sample dataset <pre class="example">
-CREATE TABLE data (id integer, sessionid integer, starttime timestamp, topic varchar, portfolio integer);
-COPY data FROM STDIN DELIMITER AS '|';
-1|1|1/01/00 5:00 AM|Real-time Equity pricing|769
-1|2|1/01/00 5:30 AM|Real-time Index pricing|9898
-1|3|1/01/00 6:00 AM|Real-time Index pricing|9898
-1|4|1/01/00 5:00 AM|Calendar|98977
-1|5|1/01/00 10:00 AM|Real-time Equity pricing|769
-1|6|1/01/00 10:30 AM|Real-time Equity pricing|9898
-1|7|1/01/00 12:00 PM|Calendar|1325
-1|8|1/01/00 12:00 PM|Calendar|6777
-1|9|1/01/00 4:05 PM|Real-time Equity pricing|769
-1|10|1/01/00 4:10 PM|Report Dates and Analyst Recommendations|34346
-1|11|1/01/00 4:15 PM|Real-time Index pricing|9898
-1|12|1/01/00 4:20 PM|Snapshot Futures pricing|5568
-1|13|1/01/00 4:21 PM|Real-time Equity pricing|769
-1|14|1/01/00 4:25 PM|Bond security master|789
-1|15|1/01/00 10:00 PM|Bond pricing|55
-1|16|1/01/00 10:00 PM|Bond pricing|55
-1|17|1/01/00 10:00 PM|Bond pricing|55
-1|18|1/01/00 10:00 PM|Bond Yield Analysis|655
-1|19|1/01/00 10:00 PM|Bond pricing|55
-1|20|1/01/00 10:00 PM|Bond pricing|55
-2|1|1/04/00 12:00 AM|Real-time Equity pricing|769
-2|2|1/04/00 12:00 AM|Real-time Equity pricing|234234
-2|3|1/04/00 12:00 AM|Calendar|1325
-2|4|1/04/00 12:00 AM|Calendar|6777
-2|5|1/04/00 12:00 AM|Real-time Equity pricing|769
-2|6|1/04/00 12:00 AM|Report Dates and Analyst Recommendations|34346
-2|7|1/04/00 12:00 AM|Real-time Index pricing|9898
-2|8|1/04/00 12:00 AM|Snapshot Futures pricing|5568
-2|9|1/04/00 12:00 AM|Real-time Equity pricing|769
-2|10|1/04/00 12:00 AM|Bond security master|789
-2|11|1/04/00 12:00 AM|Bond pricing|55
-2|12|1/04/00 12:00 AM|Bond pricing|55
-2|13|1/04/00 12:00 AM|Bond pricing|55
-2|14|1/04/00 12:00 AM|Bond pricing|55
-\.
- 
-CREATE TABLE trades AS
-SELECT *, starttimestamp::date startdate,
- CASE WHEN THEN 'before'
- WHEN THEN 'market'
- WHEN THEN 'close'
- WHEN THEN 'after'
- END tradingperiod
-FROM sessiontable
+<p>The data set describes shopper behavior on a notional web site that sells beer and wine. A beacon fires an event to a log file when the shopper visits different pages on the site: landing page, beer selection page, wine selection page, and checkout. Other pages on the site like help pages show up in the logs as well. Let’s assume that the log has been sessionized.</p>
+<p>Create the date table:</p>
+<pre class="example">
+DROP TABLE IF EXISTS eventlog, path_output, path_output_tuples;
+CREATE TABLE eventlog (event_timestamp TIMESTAMP,
+ user_id INT,
+ session_id INT,
+ page TEXT,
+ revenue FLOAT);
+INSERT INTO eventlog VALUES
+('04/15/2015 01:03:00', 100821, 100, 'LANDING', 0),
+('04/15/2015 01:04:00', 100821, 100, 'WINE', 0),
+('04/15/2015 01:05:00', 100821, 100, 'CHECKOUT', 39),
+('04/15/2015 02:06:00', 100821, 101, 'WINE', 0),
+('04/15/2015 02:09:00', 100821, 101, 'WINE', 0),
+('04/15/2015 01:15:00', 101121, 102, 'LANDING', 0),
+('04/15/2015 01:16:00', 101121, 102, 'WINE', 0),
+('04/15/2015 01:17:00', 101121, 102, 'CHECKOUT', 15),
+('04/15/2015 01:18:00', 101121, 102, 'LANDING', 0),
+('04/15/2015 01:19:00', 101121, 102, 'HELP', 0),
+('04/15/2015 01:21:00', 101121, 102, 'WINE', 0),
+('04/15/2015 01:22:00', 101121, 102, 'CHECKOUT', 23),
+('04/15/2015 02:15:00', 101331, 103, 'LANDING', 0),
+('04/15/2015 02:16:00', 101331, 103, 'WINE', 0),
+('04/15/2015 02:17:00', 101331, 103, 'HELP', 0),
+('04/15/2015 02:18:00', 101331, 103, 'WINE', 0),
+('04/15/2015 02:19:00', 101331, 103, 'CHECKOUT', 16),
+('04/15/2015 02:22:00', 101443, 104, 'BEER', 0),
+('04/15/2015 02:25:00', 101443, 104, 'CHECKOUT', 12),
+('04/15/2015 02:29:00', 101881, 105, 'LANDING', 0),
+('04/15/2015 02:30:00', 101881, 105, 'BEER', 0),
+('04/15/2015 01:05:00', 102201, 106, 'LANDING', 0),
+('04/15/2015 01:06:00', 102201, 106, 'HELP', 0),
+('04/15/2015 01:09:00', 102201, 106, 'LANDING', 0),
+('04/15/2015 02:15:00', 102201, 107, 'WINE', 0),
+('04/15/2015 02:16:00', 102201, 107, 'BEER', 0),
+('04/15/2015 02:17:00', 102201, 107, 'WINE', 0),
+('04/15/2015 02:18:00', 102871, 108, 'BEER', 0),
+('04/15/2015 02:19:00', 102871, 108, 'WINE', 0),
+('04/15/2015 02:22:00', 102871, 108, 'CHECKOUT', 21),
+('04/15/2015 02:25:00', 102871, 108, 'LANDING', 0),
+('04/15/2015 02:17:00', 103711, 109, 'BEER', 0),
+('04/15/2015 02:18:00', 103711, 109, 'LANDING', 0),
+('04/15/2015 02:19:00', 103711, 109, 'WINE', 0);
+</pre><ol type="1">
+<li>Calculate the revenue by checkout: <pre class="example">
+ SELECT madlib.path(
+ 'eventlog', -- Name of input table
+ 'path_output', -- Table name to store path results
+ 'session_id', -- Partition input table by session
+ 'event_timestamp ASC', -- Order partitions in input table by time
+ 'buy:=page=''CHECKOUT''', -- Define a symbol for checkout events
+ '(buy)', -- Pattern search: purchase
+ 'sum(revenue) as checkout_rev', -- Aggregate: sum revenue by checkout
+ TRUE -- Persist matches
+ );
+SELECT * FROM path_output ORDER BY session_id, match_id;
+</pre> Result: <pre class="result">
+ session_id | match_id | checkout_rev
+------------+----------+--------------
+ 100 | 1 | 39
+ 102 | 1 | 15
+ 102 | 2 | 23
+ 103 | 1 | 16
+ 104 | 1 | 12
+ 108 | 1 | 21
+(6 rows)
+</pre> Note that there are 2 checkouts within session 102, which is apparent from the 'match_id' column. This serves to illustrate that the 'aggregate_func' operates on a <em>per pattern match</em> basis, not on a <em>per partition</em> basis. If in fact we wanted revenue by partition ('session_id' in this example), then we could do: <pre class="example">
+SELECT session_id, sum(checkout_rev) FROM path_output GROUP BY session_id ORDER BY session_id;
+</pre> Result: <pre class="result">
+ session_id | sum
+------------+-----
+ 100 | 39
+ 102 | 38
+ 103 | 16
+ 104 | 12
+ 108 | 21
+(5 rows)
+</pre> Since we set TRUE for 'persist_rows', we can view the associated pattern matches: <pre class="example">
+SELECT * FROM path_output_tuples ORDER BY session_id ASC, event_timestamp ASC;
+</pre> Result: <pre class="result">
+ event_timestamp | user_id | session_id | page | revenue | symbol | match_id
+---------------------+---------+------------+----------+---------+--------+----------
+ 2015-04-15 01:05:00 | 100821 | 100 | CHECKOUT | 39 | buy | 1
+ 2015-04-15 01:17:00 | 101121 | 102 | CHECKOUT | 15 | buy | 1
+ 2015-04-15 01:22:00 | 101121 | 102 | CHECKOUT | 23 | buy | 2
+ 2015-04-15 02:19:00 | 101331 | 103 | CHECKOUT | 16 | buy | 1
+ 2015-04-15 02:25:00 | 101443 | 104 | CHECKOUT | 12 | buy | 1
+ 2015-04-15 02:22:00 | 102871 | 108 | CHECKOUT | 21 | buy | 1
+(6 rows)
+</pre> Notice that the 'symbol' and 'match_id' columns are added to the right of the matched rows.</li>
+<li>We are interested in sessions with an order placed within 4 pages of entering the shopping site via the landing page. We represent this by the regular expression: '(land)[^(land)(buy)]{0,2}(buy)'. In other words, visit to the landing page followed by from 0 to 2 non-entry, non-sale pages, followed by a purchase. The SQL is as follows: <pre class="example">
+SELECT madlib.path(
+ 'eventlog', -- Name of input table
+ 'path_output', -- Table name to store path results
+ 'session_id', -- Partition input table by session
+ 'event_timestamp ASC', -- Order partitions in input table by time
+ 'land:=page=''LANDING'',
+ wine:=page=''WINE'',
+ beer:=page=''BEER'',
+ buy:=page=''CHECKOUT'',
+ other:=page<>''LANDING'' AND page<>''WINE'' AND page<>''BEER'' AND page<>''CHECKOUT''', -- Symbols for page types
+ '(land)[^(land)(buy)]{0,2}(buy)', -- Purchase within 4 pages entering site
+ 'sum(revenue) as checkout_rev', -- Aggregate: sum revenue by checkout
+ TRUE -- Persist matches
+ );
+SELECT * FROM path_output ORDER BY session_id, match_id;
+</pre> Result: <pre class="result">
+ session_id | match_id | session_rev
+------------+----------+-------------
+ 100 | 1 | 39
+ 102 | 1 | 15
+ 102 | 2 | 23
+(3 rows)
+</pre> Now view the associated pattern matches: <pre class="example">
+SELECT * FROM path_output_tuples ORDER BY session_id ASC, event_timestamp ASC;
+</pre> Result: <pre class="result">
+ event_timestamp | user_id | session_id | page | revenue | symbol | match_id
+---------------------+---------+------------+----------+---------+--------+----------
+ 2015-04-15 01:03:00 | 100821 | 100 | LANDING | 0 | land | 1
+ 2015-04-15 01:04:00 | 100821 | 100 | WINE | 0 | wine | 1
+ 2015-04-15 01:05:00 | 100821 | 100 | CHECKOUT | 39 | buy | 1
+ 2015-04-15 01:15:00 | 101121 | 102 | LANDING | 0 | land | 1
+ 2015-04-15 01:16:00 | 101121 | 102 | WINE | 0 | wine | 1
+ 2015-04-15 01:17:00 | 101121 | 102 | CHECKOUT | 15 | buy | 1
+ 2015-04-15 01:18:00 | 101121 | 102 | LANDING | 0 | land | 2
+ 2015-04-15 01:19:00 | 101121 | 102 | HELP | 0 | other | 2
+ 2015-04-15 01:21:00 | 101121 | 102 | WINE | 0 | wine | 2
+ 2015-04-15 01:22:00 | 101121 | 102 | CHECKOUT | 23 | buy | 2
+(10 rows)
</pre></li>
-<li>Compare within each day for the same Topic/Portfolio across every user <pre class="example">
- SELECT madlib.path(
- 'trades',
- 'trades_out',
- 'startdate, topic, portfolio' -- each day of activity is looked at independently
- 'starttime' -- order by time
- 'BEFORE*.MARKET+.CLOSE+.AFTER*' -- at least one event during each of MARKET and CLOSE, but gather up the rest
- 'BEFORE:=starttimestamp::time >= ''0:00:00'' and starttimestamp::time < ''9:30:00''::time,
- MARKET:=starttimestamp::time >= ''9:30:00'' and starttimestamp::time < ''16:00:00''::time,
- CLOSE:= starttimestamp::time <= ''16:00:00'' and starttimestamp::time < ''16:30:00''::time,
- AFTER:= starttimestamp::time <= ''16:30:00'' and starttimestamp::time < ''24:00:00''::time
- ',
- 'first(startdate) as starttime, array_agg(id) as all_users, count(*) as num_matches'
- )
-</pre> </li>
+<li>We may want to use a window function instead of an aggregate. Currently, only aggregates are supported in the core path function in the parameter 'aggregate_func'. However, you can write window functions on the output tuples to achieve the desired result.   Continuing the previous example, let’s say we want to compute average revenue for checkouts within 4 pages of entering the shopping site via the landing page: <pre class="example">
+SELECT DATE(event_timestamp), user_id, session_id, revenue,
+ avg(revenue) OVER (PARTITION BY DATE(event_timestamp)) as avg_checkout_rev
+ FROM path_output_tuples
+ WHERE page='CHECKOUT'
+ ORDER BY user_id, session_id;
+</pre> Result: <pre class="result">
+ date | user_id | session_id | revenue | avg_checkout_rev
+------------+---------+------------+---------+------------------
+ 2015-04-15 | 100821 | 100 | 39 | 25.6666666666667
+ 2015-04-15 | 101121 | 102 | 15 | 25.6666666666667
+ 2015-04-15 | 101121 | 102 | 23 | 25.6666666666667
+(3 rows)
+</pre> Here we are partitioning the window function by day because we want daily averages, although our sample data set only has a single day.</li>
+<li>Now we want to do a golden path analysis to find the most successful shopper paths through the site. Since our data set is small, we decide this means the most frequently viewed page just before a checkout is made: <pre class="example">
+SELECT madlib.path(
+ 'eventlog', -- Name of input table
+ 'path_output', -- Table name to store path results
+ 'session_id', -- Partition input table by session
+ 'event_timestamp ASC', -- Order partitions in input table by time
+ 'land:=page=''LANDING'',
+ wine:=page=''WINE'',
+ beer:=page=''BEER'',
+ buy:=page=''CHECKOUT'',
+ other:=page<>''LANDING'' AND page<>''WINE'' AND page<>''BEER'' AND page<>''CHECKOUT''', -- Symbols for page types
+ '[^(buy)](buy)', -- Pattern to match
+ 'array_agg(page ORDER BY session_id ASC, event_timestamp ASC) as page_path', -- Build array with shopper paths
+ FALSE -- Don't persist matches
+ );
+</pre></li>
+</ol>
+<p>Now count the common paths and print the most frequent:</p>
+<pre class="example">
+SELECT count(*), page_path from
+ (SELECT * FROM path_output) q
+GROUP BY page_path
+ORDER BY count(*) DESC
+LIMIT 10;
+</pre><p>Result: </p><pre class="result">
+ count | page_path
+-------+-----------------
+ 5 | {WINE,CHECKOUT}
+ 1 | {BEER,CHECKOUT}
+(2 rows)
+</pre><p>There are only 2 different paths. The wine page is viewed more frequently than the beer page just before checkout.</p>
+<p><a class="anchor" id="note"></a></p><dl class="section note"><dt>Note</dt><dd>Please note some current limitations of the path algorithm. These limitations will be addressed in subsequent releases.<ul>
+<li>Window functions cannot currently be used in the parameter 'aggregate_func'. Instead, output the pattern matches and write a SQL query with a window function over the output tuples.</li>
+<li>Overlapping pattern matches are not supported. That is, a given row can only belong to one pattern match (non-overlapping).</li>
+<li>A given row can only match one symbol. If a row matches multiple symbols, the symbol that comes <em>first</em> in the symbol definition list will take precedence.</li>
+<li>Maximum number of symbols that can be defined is 35.</li>
+<li>The columns 'match_id' and 'symbol' are generated by the path algorithm. If coincidently you have columns in your input data named 'match_id' or 'symbol', the system generated column names will be changed to "__madlib_path_match_id__" and "__madlib_path_symbol__"</li>
+</ul>
+</dd></dl>
+<p><a class="anchor" id="nomenclature"></a></p><dl class="section user"><dt>Nomenclature</dt><dd></dd></dl>
+<p>Partition</p><ul>
+<li>scope of rows to be searched for pattern match</li>
+<li>typical examples: user id, session id, portfolio id</li>
+</ul>
+<p>Order</p><ul>
+<li>sort order of input rows in partition</li>
+<li>typical example: time</li>
+</ul>
+<p>Symbol</p><ul>
+<li>a row of a particular type that you’re searching for, that you want to include in a pattern</li>
+</ul>
+<p>Pattern</p><ul>
+<li>regular PostgreSQL pattern match expression of symbols and operators that you want to match across rows</li>
+</ul>
+<p>Pattern match</p><ul>
+<li>rows that result from a pattern match expression of symbols</li>
+<li>can be multiple matches per partition</li>
</ul>
+<p><a class="anchor" id="literature"></a></p><dl class="section user"><dt>Literature</dt><dd></dd></dl>
+<p>[1] PostgreSQL basic statements/assignment operator, <a href="http://www.postgresql.org/docs/8.2/static/plpgsql-statements.html">http://www.postgresql.org/docs/8.2/static/plpgsql-statements.html</a></p>
+<p>[2] PostgreSQL pattern matching, <a href="http://www.postgresql.org/docs/current/static/functions-matching.html">http://www.postgresql.org/docs/current/static/functions-matching.html</a></p>
+<p>[3] PostgreSQL aggregate functions, <a href="http://www.postgresql.org/docs/8.2/static/tutorial-agg.html">http://www.postgresql.org/docs/8.2/static/tutorial-agg.html</a></p>
+<p>[4] PostgreSQL window functions, <a href="http://www.postgresql.org/docs/8.4/static/tutorial-window.html">http://www.postgresql.org/docs/8.4/static/tutorial-window.html</a> </p>
</div><!-- contents -->
</div><!-- doc-content -->
<!-- start footer part -->
<div id="nav-path" class="navpath"><!-- id is needed for treeview function! -->
<ul>
- <li class="footer">Generated on Tue Mar 22 2016 17:55:59 for MADlib by
+ <li class="footer">Generated on Thu Apr 7 2016 14:24:11 for MADlib by
<a href="http://www.doxygen.org/index.html">
<img class="footer" src="doxygen.png" alt="doxygen"/></a> 1.8.10 </li>
</ul>
http://git-wip-us.apache.org/repos/asf/incubator-madlib-site/blob/c506dd05/docs/latest/group__grp__pca.html
----------------------------------------------------------------------
diff --git a/docs/latest/group__grp__pca.html b/docs/latest/group__grp__pca.html
index f4a176c..80e7174 100644
--- a/docs/latest/group__grp__pca.html
+++ b/docs/latest/group__grp__pca.html
@@ -24,14 +24,8 @@
<script type="text/javascript">
$(document).ready(function() { init_search(); });
</script>
-<script type="text/x-mathjax-config">
- MathJax.Hub.Config({
- extensions: ["tex2jax.js", "TeX/AMSmath.js", "TeX/AMSsymbols.js"],
- jax: ["input/TeX","output/HTML-CSS"],
-});
-</script><script src="../mathjax/MathJax.js"></script>
<!-- hack in the navigation tree -->
-<script type="text/javascript" src="navtree_hack.js"></script>
+<script type="text/javascript" src="eigen_navtree_hacks.js"></script>
<link href="doxygen.css" rel="stylesheet" type="text/css" />
<link href="madlib_extra.css" rel="stylesheet" type="text/css"/>
<!-- google analytics -->
@@ -40,7 +34,7 @@
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
- ga('create', 'UA-45382226-1', 'auto');
+ ga('create', 'UA-45382226-1', 'madlib.net');
ga('send', 'pageview');
</script>
</head>
@@ -50,10 +44,10 @@
<table cellspacing="0" cellpadding="0">
<tbody>
<tr style="height: 56px;">
- <td id="projectlogo"><a href="http://madlib.incubator.apache.org"><img alt="Logo" src="madlib.png" height="50" style="padding-left:0.5em;" border="0"/ ></a></td>
+ <td id="projectlogo"><a href="http://madlib.net"><img alt="Logo" src="madlib.png" height="50" style="padding-left:0.5em;" border="0"/ ></a></td>
<td style="padding-left: 0.5em;">
<div id="projectname">
- <span id="projectnumber">1.8</span>
+ <span id="projectnumber">1.9</span>
</div>
<div id="projectbrief">User Documentation for MADlib</div>
</td>
@@ -114,7 +108,7 @@ $(document).ready(function(){initNavTree('group__grp__pca.html','');});
<div class="summary">
<a href="#groups">Modules</a> </div>
<div class="headertitle">
-<div class="title">Dimensionality Reduction<div class="ingroups"><a class="el" href="group__grp__utility__functions.html">Utility Functions</a></div></div> </div>
+<div class="title">Dimensionality Reduction<div class="ingroups"><a class="el" href="group__grp__datatrans.html">Data Types and Transformations</a></div></div> </div>
</div><!--header-->
<div class="contents">
<a name="details" id="details"></a><h2 class="groupheader">Detailed Description</h2>
@@ -134,7 +128,7 @@ Modules</h2></td></tr>
<!-- start footer part -->
<div id="nav-path" class="navpath"><!-- id is needed for treeview function! -->
<ul>
- <li class="footer">Generated on Mon Jul 27 2015 20:37:45 for MADlib by
+ <li class="footer">Generated on Thu Apr 7 2016 14:24:10 for MADlib by
<a href="http://www.doxygen.org/index.html">
<img class="footer" src="doxygen.png" alt="doxygen"/></a> 1.8.10 </li>
</ul>
http://git-wip-us.apache.org/repos/asf/incubator-madlib-site/blob/c506dd05/docs/latest/group__grp__pca__project.html
----------------------------------------------------------------------
diff --git a/docs/latest/group__grp__pca__project.html b/docs/latest/group__grp__pca__project.html
index a9518e5..8d2fbf0 100644
--- a/docs/latest/group__grp__pca__project.html
+++ b/docs/latest/group__grp__pca__project.html
@@ -24,14 +24,8 @@
<script type="text/javascript">
$(document).ready(function() { init_search(); });
</script>
-<script type="text/x-mathjax-config">
- MathJax.Hub.Config({
- extensions: ["tex2jax.js", "TeX/AMSmath.js", "TeX/AMSsymbols.js"],
- jax: ["input/TeX","output/HTML-CSS"],
-});
-</script><script src="../mathjax/MathJax.js"></script>
<!-- hack in the navigation tree -->
-<script type="text/javascript" src="navtree_hack.js"></script>
+<script type="text/javascript" src="eigen_navtree_hacks.js"></script>
<link href="doxygen.css" rel="stylesheet" type="text/css" />
<link href="madlib_extra.css" rel="stylesheet" type="text/css"/>
<!-- google analytics -->
@@ -40,7 +34,7 @@
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
- ga('create', 'UA-45382226-1', 'auto');
+ ga('create', 'UA-45382226-1', 'madlib.net');
ga('send', 'pageview');
</script>
</head>
@@ -50,10 +44,10 @@
<table cellspacing="0" cellpadding="0">
<tbody>
<tr style="height: 56px;">
- <td id="projectlogo"><a href="http://madlib.incubator.apache.org"><img alt="Logo" src="madlib.png" height="50" style="padding-left:0.5em;" border="0"/ ></a></td>
+ <td id="projectlogo"><a href="http://madlib.net"><img alt="Logo" src="madlib.png" height="50" style="padding-left:0.5em;" border="0"/ ></a></td>
<td style="padding-left: 0.5em;">
<div id="projectname">
- <span id="projectnumber">1.8</span>
+ <span id="projectnumber">1.9</span>
</div>
<div id="projectbrief">User Documentation for MADlib</div>
</td>
@@ -112,7 +106,7 @@ $(document).ready(function(){initNavTree('group__grp__pca__project.html','');});
<div class="header">
<div class="headertitle">
-<div class="title">Principal Component Projection<div class="ingroups"><a class="el" href="group__grp__utility__functions.html">Utility Functions</a> » <a class="el" href="group__grp__pca.html">Dimensionality Reduction</a></div></div> </div>
+<div class="title">Principal Component Projection<div class="ingroups"><a class="el" href="group__grp__datatrans.html">Data Types and Transformations</a> » <a class="el" href="group__grp__pca.html">Dimensionality Reduction</a></div></div> </div>
</div><!--header-->
<div class="contents">
<div class="toc"><b>Contents</b> </p><ul>
@@ -127,7 +121,7 @@ $(document).ready(function(){initNavTree('group__grp__pca__project.html','');});
<li class="level1">
<a href="#related">Related Topics</a> </li>
</ul>
-</div><p>Principal component projection is a mathematical procedure that projects high dimensional data onto a lower dimensional space. This lower dimensional space is defined by the \( k \) principal components with the highest variance in the training data. More details on the mathematics of PCA can be found in <a class="el" href="pca_8sql__in.html#a31abf88e67a446a4f789764aa2c61e85">pca_train</a> and some details about the principal component projection calculations can be found in the <a class="el" href="group__grp__mlogreg.html#background">Technical Background</a>.</p>
+</div><p>Principal component projection is a mathematical procedure that projects high dimensional data onto a lower dimensional space. This lower dimensional space is defined by the <img class="formulaInl" alt="$ k $" src="form_97.png"/> principal components with the highest variance in the training data. More details on the mathematics of PCA can be found in <a class="el" href="pca_8sql__in.html#a31abf88e67a446a4f789764aa2c61e85">pca_train</a> and some details about the principal component projection calculations can be found in the <a class="el" href="group__grp__mlogreg.html#background">Technical Background</a>.</p>
<p><a class="anchor" id="project"></a></p><dl class="section user"><dt>Projection Function</dt><dd>The projection functions have the following formats: <pre class="syntax">
madlib.pca_project( source_table,
pc_table,
@@ -151,7 +145,7 @@ madlib.pca_sparse_project( source_table,
</pre></dd></dl>
<dl class="section user"><dt>Arguments</dt><dd><dl class="arglist">
<dt>source_table </dt>
-<dd><p class="startdd">TEXT. Source table name. Identical to <a class="el" href="pca_8sql__in.html#a31abf88e67a446a4f789764aa2c61e85">pca_train</a>, the input data matrix should have \( N \) rows and \( M \) columns, where \( N \) is the number of data points, and \( M \) is the number of features for each data point.</p>
+<dd><p class="startdd">TEXT. Source table name. Identical to <a class="el" href="pca_8sql__in.html#a31abf88e67a446a4f789764aa2c61e85">pca_train</a>, the input data matrix should have <img class="formulaInl" alt="$ N $" src="form_218.png"/> rows and <img class="formulaInl" alt="$ M $" src="form_174.png"/> columns, where <img class="formulaInl" alt="$ N $" src="form_218.png"/> is the number of data points, and <img class="formulaInl" alt="$ M $" src="form_174.png"/> is the number of features for each data point.</p>
<p>The input table for <em> pca_project </em> is expected to be in the one of the two standard MADlib dense matrix formats, and the sparse input table for <em> pca_sparse_project </em> should be in the standard MADlib sparse matrix format. These formats are described in the documentation for <a class="el" href="pca_8sql__in.html#a31abf88e67a446a4f789764aa2c61e85">pca_train</a>.</p>
<p class="enddd"></p>
</dd>
@@ -168,15 +162,14 @@ madlib.pca_sparse_project( source_table,
<tr>
<th>row_vec </th><td>A vector containing elements in the row of the matrix. </td></tr>
</table>
-<p></p>
<p class="enddd"></p>
</dd>
<dt>row_id </dt>
-<dd><p class="startdd">TEXT. Column name containing the row IDs in the input source table.</p>
+<dd><p class="startdd">TEXT. Column name containing the row IDs in the input source table. The column needs to be of type that can be cast to INT and should only contain values between 1 and <em>N</em>. For dense matrix format, it should contain all continguous integers from 1 to <em>N</em>.</p>
<p class="enddd"></p>
</dd>
<dt>col_id </dt>
-<dd><p class="startdd">TEXT. Name of 'col_id' column in sparse matrix representation (sparse matrices only). </p>
+<dd><p class="startdd">TEXT. Column name of containing the col IDS in sparse matrix representation (sparse matrices only). The column should be of type that can be cast to INT and contain values between 1 and <em>M</em>.</p>
<p class="enddd"></p>
</dd>
<dt>val_id </dt>
@@ -225,14 +218,13 @@ CREATE TABLE mat (
row_id integer,
row_vec double precision[]
);
-COPY mat (row_id, row_vec) FROM stdin;
-1 {1,2,5}
-0 {4,7,5}
-3 {9,2,4}
-2 {7,4,4}
-5 {0,5,5}
-4 {8,5,7}
-\.
+INSERT INTO mat VALUES
+(1, ARRAY[4,7,5]),
+(2, ARRAY[1,2,5]),
+(3, ARRAY[7,4,4]),
+(4, ARRAY[9,2,4]),
+(5, ARRAY[8,5,7]),
+(6, ARRAY[0,5,5]);
</pre></li>
<li>Run the PCA function and keep only the top two PCs: <pre class="example">
DROP TABLE IF EXISTS result_table;
@@ -268,19 +260,19 @@ SELECT * FROM result_summary_table;
</ul>
</dd></dl>
<p><a class="anchor" id="background"></a></p><dl class="section user"><dt>Technical Background</dt><dd></dd></dl>
-<p>Given a table containing some principal components \( \boldsymbol P \) and some input data \( \boldsymbol X \), the low-dimensional representation \( {\boldsymbol X}' \) is computed as </p><p class="formulaDsp">
-\begin{align*} {\boldsymbol {\hat{X}}} & = {\boldsymbol X} - \vec{e} \hat{x}^T \\ {\boldsymbol X}' & = {\boldsymbol {\hat {X}}} {\boldsymbol P}. \end{align*}
+<p>Given a table containing some principal components <img class="formulaInl" alt="$ \boldsymbol P $" src="form_229.png"/> and some input data <img class="formulaInl" alt="$ \boldsymbol X $" src="form_219.png"/>, the low-dimensional representation <img class="formulaInl" alt="$ {\boldsymbol X}' $" src="form_230.png"/> is computed as </p><p class="formulaDsp">
+<img class="formulaDsp" alt="\begin{align*} {\boldsymbol {\hat{X}}} & = {\boldsymbol X} - \vec{e} \hat{x}^T \\ {\boldsymbol X}' & = {\boldsymbol {\hat {X}}} {\boldsymbol P}. \end{align*}" src="form_231.png"/>
</p>
-<p> where \(\hat{x} \) is the column means of \( \boldsymbol X \) and \( \vec{e} \) is the vector of all ones. This step is equivalent to centering the data around the origin.</p>
-<p>The residual table \( \boldsymbol R \) is a measure of how well the low-dimensional representation approximates the true input data, and is computed as </p><p class="formulaDsp">
-\[ {\boldsymbol R} = {\boldsymbol {\hat{X}}} - {\boldsymbol X}' {\boldsymbol P}^T. \]
+<p> where <img class="formulaInl" alt="$\hat{x} $" src="form_232.png"/> is the column means of <img class="formulaInl" alt="$ \boldsymbol X $" src="form_219.png"/> and <img class="formulaInl" alt="$ \vec{e} $" src="form_224.png"/> is the vector of all ones. This step is equivalent to centering the data around the origin.</p>
+<p>The residual table <img class="formulaInl" alt="$ \boldsymbol R $" src="form_233.png"/> is a measure of how well the low-dimensional representation approximates the true input data, and is computed as </p><p class="formulaDsp">
+<img class="formulaDsp" alt="\[ {\boldsymbol R} = {\boldsymbol {\hat{X}}} - {\boldsymbol X}' {\boldsymbol P}^T. \]" src="form_234.png"/>
</p>
<p> A residual matrix with entries mostly close to zero indicates a good representation.</p>
-<p>The residual norm \( r \) is simply </p><p class="formulaDsp">
-\[ r = \|{\boldsymbol R}\|_F \]
+<p>The residual norm <img class="formulaInl" alt="$ r $" src="form_235.png"/> is simply </p><p class="formulaDsp">
+<img class="formulaDsp" alt="\[ r = \|{\boldsymbol R}\|_F \]" src="form_236.png"/>
</p>
-<p> where \( \|\cdot\|_F \) is the Frobenius norm. The relative residual norm \( r' \) is </p><p class="formulaDsp">
-\[ r' = \frac{ \|{\boldsymbol R}\|_F }{\|{\boldsymbol X}\|_F } \]
+<p> where <img class="formulaInl" alt="$ \|\cdot\|_F $" src="form_237.png"/> is the Frobenius norm. The relative residual norm <img class="formulaInl" alt="$ r' $" src="form_238.png"/> is </p><p class="formulaDsp">
+<img class="formulaDsp" alt="\[ r' = \frac{ \|{\boldsymbol R}\|_F }{\|{\boldsymbol X}\|_F } \]" src="form_239.png"/>
</p>
<p><a class="anchor" id="related"></a></p><dl class="section user"><dt>Related Topics</dt><dd>File <a class="el" href="pca__project_8sql__in.html" title="Principal Component Analysis Projection. ">pca_project.sql_in</a> documenting the SQL functions</dd></dl>
<p><a class="el" href="group__grp__pca__train.html">Principal Component Analysis</a> </p>
@@ -289,7 +281,7 @@ SELECT * FROM result_summary_table;
<!-- start footer part -->
<div id="nav-path" class="navpath"><!-- id is needed for treeview function! -->
<ul>
- <li class="footer">Generated on Mon Jul 27 2015 20:37:45 for MADlib by
+ <li class="footer">Generated on Thu Apr 7 2016 14:24:10 for MADlib by
<a href="http://www.doxygen.org/index.html">
<img class="footer" src="doxygen.png" alt="doxygen"/></a> 1.8.10 </li>
</ul>
http://git-wip-us.apache.org/repos/asf/incubator-madlib-site/blob/c506dd05/docs/latest/group__grp__pca__train.html
----------------------------------------------------------------------
diff --git a/docs/latest/group__grp__pca__train.html b/docs/latest/group__grp__pca__train.html
index 1cff7d8..94d6ce6 100644
--- a/docs/latest/group__grp__pca__train.html
+++ b/docs/latest/group__grp__pca__train.html
@@ -24,14 +24,8 @@
<script type="text/javascript">
$(document).ready(function() { init_search(); });
</script>
-<script type="text/x-mathjax-config">
- MathJax.Hub.Config({
- extensions: ["tex2jax.js", "TeX/AMSmath.js", "TeX/AMSsymbols.js"],
- jax: ["input/TeX","output/HTML-CSS"],
-});
-</script><script src="../mathjax/MathJax.js"></script>
<!-- hack in the navigation tree -->
-<script type="text/javascript" src="navtree_hack.js"></script>
+<script type="text/javascript" src="eigen_navtree_hacks.js"></script>
<link href="doxygen.css" rel="stylesheet" type="text/css" />
<link href="madlib_extra.css" rel="stylesheet" type="text/css"/>
<!-- google analytics -->
@@ -40,7 +34,7 @@
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
- ga('create', 'UA-45382226-1', 'auto');
+ ga('create', 'UA-45382226-1', 'madlib.net');
ga('send', 'pageview');
</script>
</head>
@@ -50,10 +44,10 @@
<table cellspacing="0" cellpadding="0">
<tbody>
<tr style="height: 56px;">
- <td id="projectlogo"><a href="http://madlib.incubator.apache.org"><img alt="Logo" src="madlib.png" height="50" style="padding-left:0.5em;" border="0"/ ></a></td>
+ <td id="projectlogo"><a href="http://madlib.net"><img alt="Logo" src="madlib.png" height="50" style="padding-left:0.5em;" border="0"/ ></a></td>
<td style="padding-left: 0.5em;">
<div id="projectname">
- <span id="projectnumber">1.8</span>
+ <span id="projectnumber">1.9</span>
</div>
<div id="projectbrief">User Documentation for MADlib</div>
</td>
@@ -112,7 +106,7 @@ $(document).ready(function(){initNavTree('group__grp__pca__train.html','');});
<div class="header">
<div class="headertitle">
-<div class="title">Principal Component Analysis<div class="ingroups"><a class="el" href="group__grp__utility__functions.html">Utility Functions</a> » <a class="el" href="group__grp__pca.html">Dimensionality Reduction</a></div></div> </div>
+<div class="title">Principal Component Analysis<div class="ingroups"><a class="el" href="group__grp__datatrans.html">Data Types and Transformations</a> » <a class="el" href="group__grp__pca.html">Dimensionality Reduction</a></div></div> </div>
</div><!--header-->
<div class="contents">
<div class="toc"><b>Contents</b> </p><ul>
@@ -135,7 +129,7 @@ $(document).ready(function(){initNavTree('group__grp__pca__train.html','');});
pca_train( source_table,
out_table,
row_id,
- k,
+ components_param,
grouping_cols,
lanczos_iter,
use_correlation,
@@ -149,7 +143,7 @@ pca_sparse_train( source_table,
val_id,
row_dim,
col_dim,
- k,
+ components_param,
grouping_cols,
lanczos_iter,
use_correlation,
@@ -158,7 +152,7 @@ pca_sparse_train( source_table,
</pre></dd></dl>
<p><b>Arguments</b> </p><dl class="arglist">
<dt>source_table </dt>
-<dd><p class="startdd">TEXT. Name of the input table containing the data for PCA training. The input data matrix should have \( N \) rows and \( M \) columns, where \( N \) is the number of data points, and \( M \) is the number of features for each data point.</p>
+<dd><p class="startdd">TEXT. Name of the input table containing the data for PCA training. The input data matrix should have <img class="formulaInl" alt="$ N $" src="form_218.png"/> rows and <img class="formulaInl" alt="$ M $" src="form_174.png"/> columns, where <img class="formulaInl" alt="$ N $" src="form_218.png"/> is the number of data points, and <img class="formulaInl" alt="$ M $" src="form_174.png"/> is the number of features for each data point.</p>
<p>A dense input table is expected to be in the one of the two standard MADlib dense matrix formats, and a sparse input table should be in the standard MADlib sparse matrix format.</p>
<p>The two standard MADlib dense matrix formats are </p><pre>{TABLE|VIEW} <em>source_table</em> (
<em>row_id</em> INTEGER,
@@ -168,7 +162,7 @@ pca_sparse_train( source_table,
<em>col1</em> FLOAT8,
<em>col2</em> FLOAT8,
...
-)</pre><p>Note that the column name <em>row_id</em> is taken as an input parameter, and should be an INTEGER and contain a list of continguous row indices from 0 to N-1, where N = number of rows.</p>
+)</pre><p>Note that the column name <em>row_id</em> is taken as an input parameter, and should contain a continguous list of row indices (starting at 1) for the input matrix.</p>
<p>The input table for sparse PCA is expected to be in the form:</p>
<pre>{TABLE|VIEW} <em>source_table</em> (
...
@@ -181,27 +175,29 @@ pca_sparse_train( source_table,
</dd>
<dt>out_table </dt>
<dd><p class="startdd">TEXT. The name of the table that will contain the output. The output is divided into three tables.</p>
-<p>The primary output table (<em>out_table</em>) encodes the principal components with the <em>k</em> highest eigenvalues. The table has the following columns: </p><table class="output">
+<p>The primary output table (<em>out_table</em>) encodes the principal components with the <em>k</em> highest eigenvalues where <em>k</em> is either directly provided by the user or computed according to the proportion of variance. The table has the following columns: </p><table class="output">
<tr>
<th>row_id </th><td>Eigenvalue rank in descending order of the eigenvalue size. </td></tr>
<tr>
<th>principal_components </th><td>Vectors containing elements of the principal components. </td></tr>
<tr>
-<th>eigen_values </th><td>The eigenvalues associated with each principal component. </td></tr>
+<th>std_dev </th><td>The standart deviation of each principal component. </td></tr>
+<tr>
+<th>proportion </th><td>The proportion of variance covered by the principal component. </td></tr>
</table>
-<p>The table <em>out_table</em>_means contains the column means. This table has just one column: </p><table class="output">
+<p>The table <em>out_table</em>_mean contains the column means. This table has just one column: </p><table class="output">
<tr>
-<th>column_means </th><td>A vector containing the column means for the input matrix. </td></tr>
+<th>column_mean </th><td>A vector containing the column means for the input matrix. </td></tr>
</table>
<p>The optional table <em>result_summary_table</em> contains information about the performance of the PCA. The contents of this table are described under the <em>result_summary_table</em> argument. </p>
<p class="enddd"></p>
</dd>
<dt>row_id </dt>
-<dd><p class="startdd">TEXT. Column name containing the row IDs in the input source table.</p>
+<dd><p class="startdd">TEXT. Column name containing the row IDs in the input source table. The column needs to be of type that can be cast to INT and should only contain values between 1 and <em>N</em>. For dense matrix format, it should contain all continguous integers from 1 to <em>N</em>.</p>
<p class="enddd"></p>
</dd>
<dt>col_id </dt>
-<dd><p class="startdd">TEXT. Name of 'col_id' column in sparse matrix representation (sparse matrices only). </p>
+<dd><p class="startdd">TEXT. Column name of containing the col IDS in sparse matrix representation (sparse matrices only). The column should be of type that can be cast to INT and contain values between 1 and <em>M</em>.</p>
<p class="enddd"></p>
</dd>
<dt>val_id </dt>
@@ -216,17 +212,17 @@ pca_sparse_train( source_table,
<dd><p class="startdd">INTEGER. The number of columns in the sparse matrix (sparse matrices only). </p>
<p class="enddd"></p>
</dd>
-<dt>k </dt>
-<dd><p class="startdd">INTEGER. The number of principal components to calculate from the input data. </p>
-<p class="enddd"></p>
+<dt>components_param </dt>
+<dd><p class="startdd">INTEGER or FLOAT. The parameter to control the number of principal components to calculate from the input data. If 'components_param' is INTEGER, it is used to denote the number of principal components (<em>k</em>) to compute. If 'components_param' is FLOAT, the algorithm will return enough principal vectors so that the ratio of the sum of the eigenvalues collected thus far to the sum of all eigenvalues is greater than this parameter (proportion of variance). The value of 'components_param' must be either a positive INTEGER or a FLOAT in the range (0.0,1.0]</p>
+<dl class="section note"><dt>Note</dt><dd>The difference in interpretation between INTEGER and FLOAT was introduced to maintain backward campatibility after the proportion of variance feature was introduced. A special case to be aware of: 'components_param' = 1 (INTEGER) will return 1 principal component, but 'components_param' = 1.0 (FLOAT) will return all principal components, i.e., proportion of variance of 100%.</dd></dl>
</dd>
<dt>grouping_cols (optional) </dt>
-<dd><p class="startdd">TEXT, default: NULL. Currently <em>grouping_cols</em> is present as a placeholder for forward compatibility. The parameter is planned to be implemented as a comma-separated list of column names, with the source data grouped using the combination of all the columns. An independent PCA model will be computed for each combination of the grouping columns.</p>
-<p class="enddd"></p>
+<dd><p class="startdd">TEXT, default: NULL.</p>
+<dl class="section note"><dt>Note</dt><dd><em>Not currently implemented. Any non-NULL value is ignored. Grouping support will be added in a future release. </em> The parameter is planned to be implemented as a comma-separated list of column names, with the source data grouped using the combination of all the columns. An independent PCA model will be computed for each combination of the grouping columns.</dd></dl>
</dd>
<dt>lanczos_iter (optional) </dt>
<dd><p class="startdd">INTEGER, default: minimum of {k+40, smallest matrix dimension}. The number of Lanczos iterations for the SVD calculation. The Lanczos iteration number roughly corresponds to the accuracy of the SVD calculation, and a higher iteration number corresponds to greater accuracy but longer computation time. The number of iterations must be at least as large as the value of <em>k</em>, but no larger than the smallest dimension of the matrix. If the iteration number is given as zero, then the default number of iterations is used.</p>
-<p class="enddd"></p>
+<dl class="section note"><dt>Note</dt><dd>If both 'lanczos_iter' and proportion of variance (via the 'components_param' parameter) are defined, 'lanczos_iter' will take precedence in determining the number of principal components (i.e. the number of principal components will not be greater than 'lanczos_iter' even if the target proportion had not been reached).</dd></dl>
</dd>
<dt>use_correlation (optional) </dt>
<dd><p class="startdd">BOOLEAN, default FALSE. Whether to use the correlation matrix for calculating the principal components instead of the covariance matrix. Currently <em>use_correlation</em> is a placeholder for forward compatibility, and this value must be set to false.</p>
@@ -261,14 +257,15 @@ CREATE TABLE mat (
row_id integer,
row_vec double precision[]
);
-COPY mat (row_id, row_vec) FROM stdin;
-0 {1,2,3}
-1 {2,1,2}
-2 {3,2,1}
+COPY mat (row_id, row_vec) FROM stdin DELIMITER '|';
+1|{1,2,3}
+2|{2,1,2}
+3|{3,2,1}
\.
</pre></li>
-<li>Run the PCA function: <pre class="example">
-DROP TABLE result_table;
+<li>Run the PCA function for a fixed number of components: <pre class="example">
+DROP TABLE IF EXISTS result_table;
+DROP TABLE IF EXISTS result_table_mean;
SELECT pca_train( 'mat',
'result_table',
'row_id',
@@ -278,27 +275,46 @@ SELECT pca_train( 'mat',
<li>View the PCA results: <pre class="example">
SELECT * FROM result_table;
</pre> Result <pre class="result">
- row_id | principal_components | eigen_values
---------+--------------------------------------------------------------+----------------------
- 0 | {0.707106781186547,0.408248290459781,-0.577350269192513} | 2
- 2 | {-0.707106781186547,0.408248290459781,-0.577350269192512} | 1.26294130828989e-08
- 1 | {2.08166817117217e-17,-0.816496580931809,-0.577350269183852} | 0.816496580927726
+ row_id | principal_components | std_dev | proportion
+--------+--------------------------------------------------------------+----------------------+----------------------
+ 1 | {-0.707106781186547,-1.6306400674182e-16,0.707106781186547} | 1.41421356237309 | 0.857142857142245
+ 2 | {-1.66533453693773e-16,1,5.55111512312578e-17} | 0.577350269189626 | 0.142857142857041
+ 3 | {-0.707106781186548,1.11022302462516e-16,-0.707106781186547} | 1.59506745224211e-16 | 1.09038864737157e-32
+</pre></li>
+<li>Run the PCA function for a proportion of variance: <pre class="example">
+DROP TABLE IF EXISTS result_table;
+DROP TABLE IF EXISTS result_table_mean;
+SELECT pca_train( 'mat',
+ 'result_table',
+ 'row_id',
+ 0.9
+ );
+</pre></li>
+<li>View the PCA results: <pre class="example">
+SELECT * FROM result_table;
+</pre> Result <pre class="result">
+ row_id | principal_components | std_dev | proportion
+--------+--------------------------------------------------------------+-------------------+-------------------
+ 1 | {-0.707106781186548,-3.46944695195361e-17,0.707106781186548} | 1.4142135623731 | 0.857142857142245
+ 2 | {2.22044604925031e-16,-1,1.11022302462516e-16} | 0.577350269189626 | 0.142857142857041
</pre></li>
</ol>
<p><a class="anchor" id="notes"></a></p><dl class="section user"><dt>Notes</dt><dd></dd></dl>
<ul>
<li>Table names can be optionally schema qualified (current_schemas() would be searched if a schema name is not provided) and all table and column names should follow case-sensitivity and quoting rules per the database. (For instance, 'mytable' and 'MyTable' both resolve to the same entity, i.e. 'mytable'. If mixed-case or multi-byte characters are desired for entity names then the string should be double-quoted; in this case the input would be '"MyTable"').</li>
<li>Because of the centering step in PCA (see <a class="el" href="group__grp__pca__train.html#background_pca">Technical Background</a>), sparse matrices almost always become dense during the training process. Thus, this implementation automatically densifies sparse matrix input, and there should be no expected performance improvement in using sparse matrix input over dense matrix input.</li>
+<li>For the parameter 'components_param', INTEGER and FLOAT are interpreted differently. A special case to be aware of: 'components_param' = 1 (INTEGER) will return 1 principal component, but 'components_param' = 1.0 (FLOAT) will return all principal components, i.e., proportion of variance of 100%.</li>
+<li>If both 'lanczos_iter' and proportion of variance (via the 'components_param' parameter) are defined, 'lanczos_iter' will take precedence in determining the number of principal components (i.e. the number of principal components will not be greater than 'lanczos_iter' even if the target proportion had not been reached).</li>
</ul>
<p><a class="anchor" id="background_pca"></a></p><dl class="section user"><dt>Technical Background</dt><dd></dd></dl>
-<p>The PCA implemented here uses an SVD decomposition implementation to recover the principal components (as opposed to the directly computing the eigenvectors of the covariance matrix). Let \( \boldsymbol X \) be the data matrix, and let \( \hat{x} \) be a vector of the column averages of \( \boldsymbol{X}\). PCA computes the matrix \( \hat{\boldsymbol X} \) as </p><p class="formulaDsp">
-\[ \hat{\boldsymbol X} = {\boldsymbol X} - \vec{e} \hat{x}^T \]
+<p>The PCA implemented here uses an SVD decomposition implementation to recover the principal components (as opposed to the directly computing the eigenvectors of the covariance matrix). Let <img class="formulaInl" alt="$ \boldsymbol X $" src="form_219.png"/> be the data matrix, and let <img class="formulaInl" alt="$ \hat{x} $" src="form_220.png"/> be a vector of the column averages of <img class="formulaInl" alt="$ \boldsymbol{X}$" src="form_221.png"/>. PCA computes the matrix <img class="formulaInl" alt="$ \hat{\boldsymbol X} $" src="form_222.png"/> as </p><p class="formulaDsp">
+<img class="formulaDsp" alt="\[ \hat{\boldsymbol X} = {\boldsymbol X} - \vec{e} \hat{x}^T \]" src="form_223.png"/>
</p>
-<p> where \( \vec{e} \) is the vector of all ones.</p>
+<p> where <img class="formulaInl" alt="$ \vec{e} $" src="form_224.png"/> is the vector of all ones.</p>
<p>PCA then computes the SVD matrix factorization </p><p class="formulaDsp">
-\[ \hat{\boldsymbol X} = {\boldsymbol U}{\boldsymbol \Sigma}{\boldsymbol V}^T \]
+<img class="formulaDsp" alt="\[ \hat{\boldsymbol X} = {\boldsymbol U}{\boldsymbol \Sigma}{\boldsymbol V}^T \]" src="form_225.png"/>
</p>
-<p> where \( {\boldsymbol \Sigma} \) is a diagonal matrix. The eigenvalues are recovered as the entries of \( {\boldsymbol \Sigma}/(\sqrt{N-1}) \), and the principal components are the rows of \( {\boldsymbol V} \).</p>
+<p> where <img class="formulaInl" alt="$ {\boldsymbol \Sigma} $" src="form_226.png"/> is a diagonal matrix. The eigenvalues are recovered as the entries of <img class="formulaInl" alt="$ {\boldsymbol \Sigma}/(\sqrt{(N-1)} $" src="form_546.png"/>, and the principal components are the rows of <img class="formulaInl" alt="$ {\boldsymbol V} $" src="form_228.png"/>. The reasoning behind using N − 1 instead of N to calculate the covariance is <a href="https://en.wikipedia.org/wiki/Bessel%27s_correction">Bessel's correction</a>.</p>
<p>It is important to note that the PCA implementation assumes that the user will use only the principal components that have non-zero eigenvalues. The SVD calculation is done with the Lanczos method, with does not guarantee correctness for singular vectors with zero-valued eigenvalues. Consequently, principal components with zero-valued eigenvalues are not guaranteed to be correct. Generally, this will not be problem unless the user wants to use the principal components for the entire eigenspectrum.</p>
<p><a class="anchor" id="literature"></a></p><dl class="section user"><dt>Literature</dt><dd></dd></dl>
<p>[1] Principal Component Analysis. <a href="http://en.wikipedia.org/wiki/Principal_component_analysis">http://en.wikipedia.org/wiki/Principal_component_analysis</a></p>
@@ -311,7 +327,7 @@ SELECT * FROM result_table;
<!-- start footer part -->
<div id="nav-path" class="navpath"><!-- id is needed for treeview function! -->
<ul>
- <li class="footer">Generated on Mon Jul 27 2015 20:37:45 for MADlib by
+ <li class="footer">Generated on Thu Apr 7 2016 14:24:10 for MADlib by
<a href="http://www.doxygen.org/index.html">
<img class="footer" src="doxygen.png" alt="doxygen"/></a> 1.8.10 </li>
</ul>
http://git-wip-us.apache.org/repos/asf/incubator-madlib-site/blob/c506dd05/docs/latest/group__grp__pmml.html
----------------------------------------------------------------------
diff --git a/docs/latest/group__grp__pmml.html b/docs/latest/group__grp__pmml.html
index bdef3d5..3d063f5 100644
--- a/docs/latest/group__grp__pmml.html
+++ b/docs/latest/group__grp__pmml.html
@@ -24,14 +24,8 @@
<script type="text/javascript">
$(document).ready(function() { init_search(); });
</script>
-<script type="text/x-mathjax-config">
- MathJax.Hub.Config({
- extensions: ["tex2jax.js", "TeX/AMSmath.js", "TeX/AMSsymbols.js"],
- jax: ["input/TeX","output/HTML-CSS"],
-});
-</script><script src="../mathjax/MathJax.js"></script>
<!-- hack in the navigation tree -->
-<script type="text/javascript" src="navtree_hack.js"></script>
+<script type="text/javascript" src="eigen_navtree_hacks.js"></script>
<link href="doxygen.css" rel="stylesheet" type="text/css" />
<link href="madlib_extra.css" rel="stylesheet" type="text/css"/>
<!-- google analytics -->
@@ -40,7 +34,7 @@
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
- ga('create', 'UA-45382226-1', 'auto');
+ ga('create', 'UA-45382226-1', 'madlib.net');
ga('send', 'pageview');
</script>
</head>
@@ -50,10 +44,10 @@
<table cellspacing="0" cellpadding="0">
<tbody>
<tr style="height: 56px;">
- <td id="projectlogo"><a href="http://madlib.incubator.apache.org"><img alt="Logo" src="madlib.png" height="50" style="padding-left:0.5em;" border="0"/ ></a></td>
+ <td id="projectlogo"><a href="http://madlib.net"><img alt="Logo" src="madlib.png" height="50" style="padding-left:0.5em;" border="0"/ ></a></td>
<td style="padding-left: 0.5em;">
<div id="projectname">
- <span id="projectnumber">1.8</span>
+ <span id="projectnumber">1.9</span>
</div>
<div id="projectbrief">User Documentation for MADlib</div>
</td>
@@ -324,7 +318,7 @@ SELECT madlib.pmml('patients_logregr_grouping',
<!-- start footer part -->
<div id="nav-path" class="navpath"><!-- id is needed for treeview function! -->
<ul>
- <li class="footer">Generated on Mon Jul 27 2015 20:37:45 for MADlib by
+ <li class="footer">Generated on Thu Apr 7 2016 14:24:11 for MADlib by
<a href="http://www.doxygen.org/index.html">
<img class="footer" src="doxygen.png" alt="doxygen"/></a> 1.8.10 </li>
</ul>
http://git-wip-us.apache.org/repos/asf/incubator-madlib-site/blob/c506dd05/docs/latest/group__grp__prob.html
----------------------------------------------------------------------
diff --git a/docs/latest/group__grp__prob.html b/docs/latest/group__grp__prob.html
index 1bca282..0a396b5 100644
--- a/docs/latest/group__grp__prob.html
+++ b/docs/latest/group__grp__prob.html
@@ -24,14 +24,8 @@
<script type="text/javascript">
$(document).ready(function() { init_search(); });
</script>
-<script type="text/x-mathjax-config">
- MathJax.Hub.Config({
- extensions: ["tex2jax.js", "TeX/AMSmath.js", "TeX/AMSsymbols.js"],
- jax: ["input/TeX","output/HTML-CSS"],
-});
-</script><script src="../mathjax/MathJax.js"></script>
<!-- hack in the navigation tree -->
-<script type="text/javascript" src="navtree_hack.js"></script>
+<script type="text/javascript" src="eigen_navtree_hacks.js"></script>
<link href="doxygen.css" rel="stylesheet" type="text/css" />
<link href="madlib_extra.css" rel="stylesheet" type="text/css"/>
<!-- google analytics -->
@@ -40,7 +34,7 @@
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
- ga('create', 'UA-45382226-1', 'auto');
+ ga('create', 'UA-45382226-1', 'madlib.net');
ga('send', 'pageview');
</script>
</head>
@@ -50,10 +44,10 @@
<table cellspacing="0" cellpadding="0">
<tbody>
<tr style="height: 56px;">
- <td id="projectlogo"><a href="http://madlib.incubator.apache.org"><img alt="Logo" src="madlib.png" height="50" style="padding-left:0.5em;" border="0"/ ></a></td>
+ <td id="projectlogo"><a href="http://madlib.net"><img alt="Logo" src="madlib.png" height="50" style="padding-left:0.5em;" border="0"/ ></a></td>
<td style="padding-left: 0.5em;">
<div id="projectname">
- <span id="projectnumber">1.8</span>
+ <span id="projectnumber">1.9</span>
</div>
<div id="projectbrief">User Documentation for MADlib</div>
</td>
@@ -128,9 +122,9 @@ $(document).ready(function(){initNavTree('group__grp__prob.html','');});
</div><p>The Probability Functions module provides cumulative distribution, density/mass, and quantile functions for a wide range of probability distributions.</p>
<p>Unless otherwise documented, all of these functions are wrappers around functionality provided by the boost C++ library [1, “<a href="http://www.boost.org/doc/libs/1_49_0/libs/math/doc/sf_and_dist/html/math_toolkit/dist.html">Statistical Distributions and Functions</a>”].</p>
<p>For convenience, all cumulative distribution and density/mass functions (CDFs and PDF/PMFs in short) are defined over the range of all floating-point numbers including infinity. Inputs that are <code>NULL</code> or <code>NaN</code> (not a number) will always produce a <code>NULL</code> or <code>NaN</code> result, respectively. Inputs that are plus or minus infinity will return the respective limits.</p>
-<p>A quantile function for a probability distrution with CDF \( F \) takes a probability argument \( p \in [0,1] \) and returns the value \( x \) so that \( F(x) = p \), provided such an \( x \) exists and it is unique. If it does not, the result will be \( \sup \{ x \in D \mid F(x) \leq p \} \) (interpreted as 0 if the supremum is over an empty set) if \( p < 0.5 \), and \( \inf \{ x \in D \mid F(x) \geq p \} \) if \( p \geq 0.5 \). Here \( D \) denotes the domain of the distribution, which is the set of reals \( \mathbb R \) for continuous and the set of nonnegative integers \( \mathbb N_0 \) for discrete distributions.</p>
-<p>Intuitively, the formulas in the previous paragraph deal with the following special cases. The 0-quantile will always be the “left end” of the support, and the 1-quantile will be the “right end” of the support of the distribution. For discrete distributions, most values of \( p \in [0,1] \) do not admit an \( x \) with \( F(x) = p \). Instead, there is an \( x \in \mathbb N_0 \) so that \( F(x) < p < F(x + 1) \). The above formulas mean that the value returned as \( p \)-quantile is \( x \) if \( p < 0.5 \), and it is \( x + 1 \) if \( p \geq 0.5 \). (As a special case, in order to ensure that quantiles are always within the support, the \( p \)-quantile will be 0 if \( p < F(0) \)).</p>
-<p>The rationale for choosing this behavior is that \(p\)-quantiles for \( p < 0.5 \) are typically requested when interested in the value \( x \) such that with confidence level <b>at least</b> \( 1 - p \) a random variable will be \( > x \) (or equivalently, with probability <b>at most</b> \( p \), it will be \( \leq x \)). Likewise, \(p\)-quantiles for \( p \geq 0.5 \) are typically requested when interested in the value \( x \) such that with confidence level <b>at least</b> \( p \) a random variable will be \( \leq x \). See also [1, “<a href="http://www.boost.org/doc/libs/1_46_1/libs/math/doc/sf_and_dist/html/math_toolkit/policy/pol_tutorial/understand_dis_quant.html">Understanding Quantiles of Discrete Distributions</a>”].</p>
+<p>A quantile function for a probability distrution with CDF <img class="formulaInl" alt="$ F $" src="form_112.png"/> takes a probability argument <img class="formulaInl" alt="$ p \in [0,1] $" src="form_240.png"/> and returns the value <img class="formulaInl" alt="$ x $" src="form_178.png"/> so that <img class="formulaInl" alt="$ F(x) = p $" src="form_241.png"/>, provided such an <img class="formulaInl" alt="$ x $" src="form_178.png"/> exists and it is unique. If it does not, the result will be <img class="formulaInl" alt="$ \sup \{ x \in D \mid F(x) \leq p \} $" src="form_242.png"/> (interpreted as 0 if the supremum is over an empty set) if <img class="formulaInl" alt="$ p < 0.5 $" src="form_243.png"/>, and <img class="formulaInl" alt="$ \inf \{ x \in D \mid F(x) \geq p \} $" src="form_244.png"/> if <img class="formulaInl" alt="$ p \geq 0.5 $" src="form_245.png"/>. Here <img class="formulaInl" alt="$ D $" src="form_246.png"/> denotes the domain of the distribution, which is the set
of reals <img class="formulaInl" alt="$ \mathbb R $" src="form_247.png"/> for continuous and the set of nonnegative integers <img class="formulaInl" alt="$ \mathbb N_0 $" src="form_248.png"/> for discrete distributions.</p>
+<p>Intuitively, the formulas in the previous paragraph deal with the following special cases. The 0-quantile will always be the “left end” of the support, and the 1-quantile will be the “right end” of the support of the distribution. For discrete distributions, most values of <img class="formulaInl" alt="$ p \in [0,1] $" src="form_240.png"/> do not admit an <img class="formulaInl" alt="$ x $" src="form_178.png"/> with <img class="formulaInl" alt="$ F(x) = p $" src="form_241.png"/>. Instead, there is an <img class="formulaInl" alt="$ x \in \mathbb N_0 $" src="form_249.png"/> so that <img class="formulaInl" alt="$ F(x) < p < F(x + 1) $" src="form_250.png"/>. The above formulas mean that the value returned as <img class="formulaInl" alt="$ p $" src="form_110.png"/>-quantile is <img class="formulaInl" alt="$ x $" src="form_178.png"/> if <img class="formulaInl" alt="$ p < 0.5 $" src="form_243.png"/>, and it is <img class="formulaInl" alt="$ x + 1 $" src="form_251.png"/> if <img c
lass="formulaInl" alt="$ p \geq 0.5 $" src="form_245.png"/>. (As a special case, in order to ensure that quantiles are always within the support, the <img class="formulaInl" alt="$ p $" src="form_110.png"/>-quantile will be 0 if <img class="formulaInl" alt="$ p < F(0) $" src="form_252.png"/>).</p>
+<p>The rationale for choosing this behavior is that <img class="formulaInl" alt="$p$" src="form_253.png"/>-quantiles for <img class="formulaInl" alt="$ p < 0.5 $" src="form_243.png"/> are typically requested when interested in the value <img class="formulaInl" alt="$ x $" src="form_178.png"/> such that with confidence level <b>at least</b> <img class="formulaInl" alt="$ 1 - p $" src="form_254.png"/> a random variable will be <img class="formulaInl" alt="$ > x $" src="form_255.png"/> (or equivalently, with probability <b>at most</b> <img class="formulaInl" alt="$ p $" src="form_110.png"/>, it will be <img class="formulaInl" alt="$ \leq x $" src="form_256.png"/>). Likewise, <img class="formulaInl" alt="$p$" src="form_253.png"/>-quantiles for <img class="formulaInl" alt="$ p \geq 0.5 $" src="form_245.png"/> are typically requested when interested in the value <img class="formulaInl" alt="$ x $" src="form_178.png"/> such that with confidence level <b>at least</b> <img class="formulaInl"
alt="$ p $" src="form_110.png"/> a random variable will be <img class="formulaInl" alt="$ \leq x $" src="form_256.png"/>. See also [1, “<a href="http://www.boost.org/doc/libs/1_46_1/libs/math/doc/sf_and_dist/html/math_toolkit/policy/pol_tutorial/understand_dis_quant.html">Understanding Quantiles of Discrete Distributions</a>”].</p>
<p><a class="anchor" id="syntax"></a></p><dl class="section user"><dt>Function Syntax</dt><dd></dd></dl>
<p>Cumulative distribution functions:</p>
<pre class="syntax"><em>distribution</em>_cdf(<em>random variate</em>[, <em>parameter1</em> [, <em>parameter2</em> [, <em>parameter3</em>] ] ])</pre><p>Probability density/mass functions: </p><pre class="syntax"><em>distribution</em>_{pdf|pmf}(<em>random variate</em>[, <em>parameter1</em> [, <em>parameter2</em> [, <em>parameter3</em>] ] ])</pre><p>Quantile functions: </p><pre class="syntax"><em>distribution</em>_quantile(<em>probability</em>[, <em>parameter1</em> [, <em>parameter2</em> [, <em>parameter3</em>] ] ])</pre><p>For concrete function signatures, see <a class="el" href="prob_8sql__in.html">prob.sql_in</a>.</p>
@@ -156,7 +150,7 @@ SELECT madlib.normal_quantile(0.5, 0, 1);
<!-- start footer part -->
<div id="nav-path" class="navpath"><!-- id is needed for treeview function! -->
<ul>
- <li class="footer">Generated on Mon Jul 27 2015 20:37:45 for MADlib by
+ <li class="footer">Generated on Thu Apr 7 2016 14:24:10 for MADlib by
<a href="http://www.doxygen.org/index.html">
<img class="footer" src="doxygen.png" alt="doxygen"/></a> 1.8.10 </li>
</ul>
http://git-wip-us.apache.org/repos/asf/incubator-madlib-site/blob/c506dd05/docs/latest/group__grp__profile.html
----------------------------------------------------------------------
diff --git a/docs/latest/group__grp__profile.html b/docs/latest/group__grp__profile.html
index 8218103..ce8d6b4 100644
--- a/docs/latest/group__grp__profile.html
+++ b/docs/latest/group__grp__profile.html
@@ -24,14 +24,8 @@
<script type="text/javascript">
$(document).ready(function() { init_search(); });
</script>
-<script type="text/x-mathjax-config">
- MathJax.Hub.Config({
- extensions: ["tex2jax.js", "TeX/AMSmath.js", "TeX/AMSsymbols.js"],
- jax: ["input/TeX","output/HTML-CSS"],
-});
-</script><script src="../mathjax/MathJax.js"></script>
<!-- hack in the navigation tree -->
-<script type="text/javascript" src="navtree_hack.js"></script>
+<script type="text/javascript" src="eigen_navtree_hacks.js"></script>
<link href="doxygen.css" rel="stylesheet" type="text/css" />
<link href="madlib_extra.css" rel="stylesheet" type="text/css"/>
<!-- google analytics -->
@@ -40,7 +34,7 @@
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
- ga('create', 'UA-45382226-1', 'auto');
+ ga('create', 'UA-45382226-1', 'madlib.net');
ga('send', 'pageview');
</script>
</head>
@@ -50,10 +44,10 @@
<table cellspacing="0" cellpadding="0">
<tbody>
<tr style="height: 56px;">
- <td id="projectlogo"><a href="http://madlib.incubator.apache.org"><img alt="Logo" src="madlib.png" height="50" style="padding-left:0.5em;" border="0"/ ></a></td>
+ <td id="projectlogo"><a href="http://madlib.net"><img alt="Logo" src="madlib.png" height="50" style="padding-left:0.5em;" border="0"/ ></a></td>
<td style="padding-left: 0.5em;">
<div id="projectname">
- <span id="projectnumber">1.8</span>
+ <span id="projectnumber">1.9</span>
</div>
<div id="projectbrief">User Documentation for MADlib</div>
</td>
@@ -204,7 +198,7 @@ SELECT * FROM profile_full( 'pg_catalog.pg_tables',
<!-- start footer part -->
<div id="nav-path" class="navpath"><!-- id is needed for treeview function! -->
<ul>
- <li class="footer">Generated on Mon Jul 27 2015 20:37:46 for MADlib by
+ <li class="footer">Generated on Thu Apr 7 2016 14:24:11 for MADlib by
<a href="http://www.doxygen.org/index.html">
<img class="footer" src="doxygen.png" alt="doxygen"/></a> 1.8.10 </li>
</ul>
http://git-wip-us.apache.org/repos/asf/incubator-madlib-site/blob/c506dd05/docs/latest/group__grp__quantile.html
----------------------------------------------------------------------
diff --git a/docs/latest/group__grp__quantile.html b/docs/latest/group__grp__quantile.html
index ee4b14b..b03557f 100644
--- a/docs/latest/group__grp__quantile.html
+++ b/docs/latest/group__grp__quantile.html
@@ -24,14 +24,8 @@
<script type="text/javascript">
$(document).ready(function() { init_search(); });
</script>
-<script type="text/x-mathjax-config">
- MathJax.Hub.Config({
- extensions: ["tex2jax.js", "TeX/AMSmath.js", "TeX/AMSsymbols.js"],
- jax: ["input/TeX","output/HTML-CSS"],
-});
-</script><script src="../mathjax/MathJax.js"></script>
<!-- hack in the navigation tree -->
-<script type="text/javascript" src="navtree_hack.js"></script>
+<script type="text/javascript" src="eigen_navtree_hacks.js"></script>
<link href="doxygen.css" rel="stylesheet" type="text/css" />
<link href="madlib_extra.css" rel="stylesheet" type="text/css"/>
<!-- google analytics -->
@@ -40,7 +34,7 @@
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
- ga('create', 'UA-45382226-1', 'auto');
+ ga('create', 'UA-45382226-1', 'madlib.net');
ga('send', 'pageview');
</script>
</head>
@@ -50,10 +44,10 @@
<table cellspacing="0" cellpadding="0">
<tbody>
<tr style="height: 56px;">
- <td id="projectlogo"><a href="http://madlib.incubator.apache.org"><img alt="Logo" src="madlib.png" height="50" style="padding-left:0.5em;" border="0"/ ></a></td>
+ <td id="projectlogo"><a href="http://madlib.net"><img alt="Logo" src="madlib.png" height="50" style="padding-left:0.5em;" border="0"/ ></a></td>
<td style="padding-left: 0.5em;">
<div id="projectname">
- <span id="projectnumber">1.8</span>
+ <span id="projectnumber">1.9</span>
</div>
<div id="projectbrief">User Documentation for MADlib</div>
</td>
@@ -165,7 +159,7 @@ SELECT quantile( 'tab1',
<!-- start footer part -->
<div id="nav-path" class="navpath"><!-- id is needed for treeview function! -->
<ul>
- <li class="footer">Generated on Mon Jul 27 2015 20:37:46 for MADlib by
+ <li class="footer">Generated on Thu Apr 7 2016 14:24:11 for MADlib by
<a href="http://www.doxygen.org/index.html">
<img class="footer" src="doxygen.png" alt="doxygen"/></a> 1.8.10 </li>
</ul>
http://git-wip-us.apache.org/repos/asf/incubator-madlib-site/blob/c506dd05/docs/latest/group__grp__random__forest.html
----------------------------------------------------------------------
diff --git a/docs/latest/group__grp__random__forest.html b/docs/latest/group__grp__random__forest.html
index fa6f520..6269ba0 100644
--- a/docs/latest/group__grp__random__forest.html
+++ b/docs/latest/group__grp__random__forest.html
@@ -24,14 +24,8 @@
<script type="text/javascript">
$(document).ready(function() { init_search(); });
</script>
-<script type="text/x-mathjax-config">
- MathJax.Hub.Config({
- extensions: ["tex2jax.js", "TeX/AMSmath.js", "TeX/AMSsymbols.js"],
- jax: ["input/TeX","output/HTML-CSS"],
-});
-</script><script src="../mathjax/MathJax.js"></script>
<!-- hack in the navigation tree -->
-<script type="text/javascript" src="navtree_hack.js"></script>
+<script type="text/javascript" src="eigen_navtree_hacks.js"></script>
<link href="doxygen.css" rel="stylesheet" type="text/css" />
<link href="madlib_extra.css" rel="stylesheet" type="text/css"/>
<!-- google analytics -->
@@ -40,7 +34,7 @@
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
- ga('create', 'UA-45382226-1', 'auto');
+ ga('create', 'UA-45382226-1', 'madlib.net');
ga('send', 'pageview');
</script>
</head>
@@ -50,10 +44,10 @@
<table cellspacing="0" cellpadding="0">
<tbody>
<tr style="height: 56px;">
- <td id="projectlogo"><a href="http://madlib.incubator.apache.org"><img alt="Logo" src="madlib.png" height="50" style="padding-left:0.5em;" border="0"/ ></a></td>
+ <td id="projectlogo"><a href="http://madlib.net"><img alt="Logo" src="madlib.png" height="50" style="padding-left:0.5em;" border="0"/ ></a></td>
<td style="padding-left: 0.5em;">
<div id="projectname">
- <span id="projectnumber">1.8</span>
+ <span id="projectnumber">1.9</span>
</div>
<div id="projectbrief">User Documentation for MADlib</div>
</td>
@@ -454,7 +448,7 @@ SELECT madlib.forest_train('dt_golf', -- source table
'train_output', -- output model table
'id', -- id column
'class', -- response
- '"OUTLOOK", temperature, humidity, windy', -- features
+ '"OUTLOOK", temperature, humidity, windy', -- features
NULL, -- exclude columns
NULL, -- grouping columns
20::integer, -- number of trees
@@ -464,8 +458,7 @@ SELECT madlib.forest_train('dt_golf', -- source table
8::integer, -- max depth
3::integer, -- min split
1::integer, -- min bucket
- 10::integer, -- number of splits per continuous variable
- FALSE -- verbose
+ 10::integer -- number of splits per continuous variable
);
\x on
SELECT * FROM train_output_summary;
@@ -708,7 +701,7 @@ ORDER BY am, id;
<!-- start footer part -->
<div id="nav-path" class="navpath"><!-- id is needed for treeview function! -->
<ul>
- <li class="footer">Generated on Mon Jul 27 2015 20:37:45 for MADlib by
+ <li class="footer">Generated on Thu Apr 7 2016 14:24:10 for MADlib by
<a href="http://www.doxygen.org/index.html">
<img class="footer" src="doxygen.png" alt="doxygen"/></a> 1.8.10 </li>
</ul>