You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@madlib.apache.org by ri...@apache.org on 2016/03/26 01:42:24 UTC
incubator-madlib git commit: Docs: Multiple edits in mainpage, modules
Repository: incubator-madlib
Updated Branches:
refs/heads/master 3dcb96ad3 -> da90d620a
Docs: Multiple edits in mainpage, modules
JIRA: MADLIB-982
Closes #33
Project: http://git-wip-us.apache.org/repos/asf/incubator-madlib/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-madlib/commit/da90d620
Tree: http://git-wip-us.apache.org/repos/asf/incubator-madlib/tree/da90d620
Diff: http://git-wip-us.apache.org/repos/asf/incubator-madlib/diff/da90d620
Branch: refs/heads/master
Commit: da90d620abbf73629ebb776053cc33995d2eb48b
Parents: 3dcb96a
Author: Frank McQuillan <fm...@pivotal.io>
Authored: Fri Mar 25 17:40:49 2016 -0700
Committer: Rahul Iyer <ri...@pivotal.io>
Committed: Fri Mar 25 17:42:07 2016 -0700
----------------------------------------------------------------------
doc/imgs/madlib.png | Bin 20085 -> 2209 bytes
doc/mainpage.dox.in | 37 +--
.../modules/elastic_net/elastic_net.sql_in | 9 +-
src/ports/postgres/modules/linalg/svd.sql_in | 268 +++++++++++--------
src/ports/postgres/modules/pca/pca.sql_in | 59 ++--
.../postgres/modules/summary/summary.sql_in | 49 ++--
6 files changed, 261 insertions(+), 161 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-madlib/blob/da90d620/doc/imgs/madlib.png
----------------------------------------------------------------------
diff --git a/doc/imgs/madlib.png b/doc/imgs/madlib.png
index e4aa714..be014c6 100644
Binary files a/doc/imgs/madlib.png and b/doc/imgs/madlib.png differ
http://git-wip-us.apache.org/repos/asf/incubator-madlib/blob/da90d620/doc/mainpage.dox.in
----------------------------------------------------------------------
diff --git a/doc/mainpage.dox.in b/doc/mainpage.dox.in
index 040611d..46f1488 100644
--- a/doc/mainpage.dox.in
+++ b/doc/mainpage.dox.in
@@ -1,32 +1,37 @@
/**
@mainpage
-MADlib is an open-source library for scalable in-database analytics. It provides
-data-parallel implementations of mathematical, statistical and machine learning
-methods for structured and unstructured data.
+Apache MADlib (incubating) is an open-source library for scalable
+in-database analytics. It provides data-parallel implementations of
+mathematical, statistical and machine learning methods for structured
+and unstructured data.
The MADlib mission: to foster widespread development of scalable analytic
-skills, by harnessing efforts from commercial practice, academic research, and
-open-source development.
+skills, by harnessing efforts from commercial practice, academic research,
+and open-source development.
Useful links:
<ul>
-<li>MADlib project site http://madlib.net/</li>
-<li>MADlib bug reporting site: http://jira.madlib.net/ and quick guide: https://github.com/madlib/madlib/wiki/Bug-reporting</li>
+<li><a href="http://madlib.incubator.apache.org">MADlib web site</a></li>
+<li><a href="https://cwiki.apache.org/confluence/display/MADLIB">MADlib wiki</a></li>
+<li><a href="https://issues.apache.org/jira/browse/MADLIB/">JIRAs for reporting bugs and reviewing backlog</a></li>
+<li><a href="https://mail-archives.apache.org/mod_mbox/incubator-madlib-user/">User mailing list</a></li>
+<li><a href="https://mail-archives.apache.org/mod_mbox/incubator-madlib-dev/">Dev mailing list</a></li>
<li>User documentation for earlier releases:
- <a href="http://doc.madlib.net/v1.7.1/index.html">v1.7.1</a>,
- <a href="http://doc.madlib.net/v1.7/index.html">v1.7</a>,
- <a href="http://doc.madlib.net/v1.6/index.html">v1.6</a>,
- <a href="http://doc.madlib.net/v1.5/index.html">v1.5</a>,
- <a href="http://doc.madlib.net/v1.4/index.html">v1.4</a>,
- <a href="http://doc.madlib.net/v1.3/index.html">v1.3</a>,
- <a href="http://doc.madlib.net/v1.2/index.html">v1.2</a>
+ <a href="../v1.8/index.html">v1.8</a>,
+ <a href="../v1.7.1/index.html">v1.7.1</a>,
+ <a href="../v1.7/index.html">v1.7</a>,
+ <a href="../v1.6/index.html">v1.6</a>,
+ <a href="../v1.5/index.html">v1.5</a>,
+ <a href="../v1.4/index.html">v1.4</a>,
+ <a href="../v1.3/index.html">v1.3</a>,
+ <a href="../v1.2/index.html">v1.2</a>
</li>
</ul>
-Please refer to the <a href="@DOXYGEN_README_FILE@">Read-Me</a> file for information
+Please refer to the <a href="https://github.com/apache/incubator-madlib/blob/master/ReadMe.txt">Read-Me</a> file for information
about incorporated third-party material. License information regarding MADlib
and included third-party libraries can be found inside the
-<a href="@DOXYGEN_LICENSE_DIR@">license</a> directory.
+<a href="https://github.com/apache/incubator-madlib/blob/master/LICENSE">License</a> directory.
@defgroup grp_datatrans Data Types and Transformations
@{Data types and transformation operations @}
http://git-wip-us.apache.org/repos/asf/incubator-madlib/blob/da90d620/src/ports/postgres/modules/elastic_net/elastic_net.sql_in
----------------------------------------------------------------------
diff --git a/src/ports/postgres/modules/elastic_net/elastic_net.sql_in b/src/ports/postgres/modules/elastic_net/elastic_net.sql_in
index a99547e..5709b5d 100644
--- a/src/ports/postgres/modules/elastic_net/elastic_net.sql_in
+++ b/src/ports/postgres/modules/elastic_net/elastic_net.sql_in
@@ -130,7 +130,14 @@ the dependent variable expression to the <em>excluded</em> string.</DD>
<DD>BOOLEAN, default: TRUE. Whether to normalize the data. Setting this to TRUE usually yields better results and faster convergence.</DD>
<DT>grouping_col (optional)</DT>
-<DD>TEXT, default: NULL. <em>Not currently implemented. Any non-NULL value is ignored.</em> An expression list used to group the input dataset into discrete groups, running one regression per group. Similar to the SQL <tt>GROUP BY</tt> clause. When this value is NULL, no grouping is used and a single result model is generated.</DD>
+<DD>TEXT, default: NULL.
+
+@note <em>Not currently implemented. Any non-NULL value is ignored.
+Grouping support will be added in a future release. </em>
+When implemented, an expression list will be used to group the input dataset
+into discrete groups,
+running one regression per group. Similar to the SQL <tt>GROUP BY</tt> clause.
+When this value is NULL, no grouping is used and a single result model is generated.</DD>
<DT>optimizer (optional)</DT>
<DD>TEXT, default: 'fista'. Name of optimizer, either 'fista' or 'igd'.</DD>
http://git-wip-us.apache.org/repos/asf/incubator-madlib/blob/da90d620/src/ports/postgres/modules/linalg/svd.sql_in
----------------------------------------------------------------------
diff --git a/src/ports/postgres/modules/linalg/svd.sql_in b/src/ports/postgres/modules/linalg/svd.sql_in
index 7393fd6..799e9cb 100644
--- a/src/ports/postgres/modules/linalg/svd.sql_in
+++ b/src/ports/postgres/modules/linalg/svd.sql_in
@@ -23,7 +23,7 @@ m4_include(`SQLCommon.m4')
</ul>
</div>
-@brief Performs factorization of dense, sparse, and block matrices.
+@brief Performs factorization of dense and sparse matrices.
In linear algebra, the singular value decomposition (SVD) is a factorization of
a real or complex matrix, with many useful applications in signal processing and
@@ -37,13 +37,13 @@ decomposed as follows:
where \f$U\f$ is a \f$m \times n\f$ orthonormal matrix,
\f$\Sigma\f$ is a \f$n \times n\f$ diagonal matrix, and \f$V\f$ is an
\f$n \times n\f$ orthonormal matrix. The diagonal elements of \f$\Sigma\f$ are
-called the \e {singular values}.
+called the <em>singular values</em>.
@anchor syntax
@par SVD Functions
-SVD factorizations are provided for dense matrices, sparse matrices, and block
-matrices. In addition, a native implementation is provided for sparse
+SVD factorizations are provided for dense and sparse matrices.
+In addition, a native implementation is provided for very sparse
matrices for improved performance.
<b>SVD Function for Dense Matrices</b>
@@ -62,9 +62,11 @@ svd( source_table,
<dt>source_table</dt>
<dd>TEXT. Source table name (dense matrix).
- The table contains a \c row_id column that identifies each row.
- Further, the other columns are assumed to be the data for the matrix
- represented in two possible forms, illustrated by the following 2x2 matrix example:
+ The table contains a \c row_id column that identifies each row,
+ with numbering starting from 1.
+ The other columns contain the data for the matrix.
+ There are two possible dense formats as illustrated by the 2x2 matrix example
+ below. You can use either of these dense formats:
-# <pre class="example">
row_id col1 col2
row1 1 1 0
@@ -77,23 +79,28 @@ row2 2 {0, 1}
</pre>
</dd>
<dt>output_table_prefix</dt>
- <dd>TEXT. Prefix for output tables. See <a href="#output">Output Tables</a>.</dd>
+ <dd>TEXT. Prefix for output tables. See
+ <a href="#output">Output Tables</a> below for a description
+ of the convention used.</dd>
<dt>row_id</dt>
<dd>TEXT. ID for each row.</dd>
<dt>k</dt>
- <dd>INTEGER. Number of singular vectors to compute.</dd>
- <dt>n_iterations (optional)</dt>
- <dd>INTEGER. Number of iterations to run.</dd>
+ <dd>INTEGER. Number of singular values to compute.</dd>
+ <dt>n_iterations (optional). </dd>
+ <dd>INTEGER. Number of iterations to run.
+ @note The number of iterations must be
+ in the range [k, column dimension], where
+ k is number of singular values.</dd>
<dt>result_summary_table (optional)</dt>
- <dd>TEXT. The name of the table to store result summary.</dd>
+ <dd>TEXT. The name of the table to store the result summary.</dd>
</dl>
<hr>
-<b>SVD Function for Sparse Matrix input</b>
+<b>SVD Function for Sparse Matrices</b>
Use this function for matrices that are represented in the sparse-matrix
-format (example below). <b>The input matrix is converted to a dense matrix
-before the SVD operation.</b>
+format (example below). <b>Note that the input matrix is converted to a
+dense matrix before the SVD operation, for efficient computation reasons. </b>
<pre class="syntax">
svd_sparse( source_table,
@@ -113,22 +120,31 @@ svd_sparse( source_table,
<dt>source_table</dt>
<dd>TEXT. Source table name (sparse matrix).
-An example sparse matrix representation is given below:
+A sparse matrix is represented using the row and column indices for each
+non-zero entry of the matrix. This representation is useful for matrices
+containing multiple zero elements. Below is an example of a sparse 4x7 matrix
+with just 6 out of 28 entries being non-zero. The dimensionality of the matrix is
+inferred using the max value in <em>row</em> and <em>col</em> columns. Note the
+last entry is included (even though it is 0) to provide the dimensionality of the
+matrix, indicating that the 4th row and 7th column contain all zeros.
<pre class="example">
- row_id col_id value
-row1 1 1 2
-row2 2 1 1
-row3 3 2 1
+ row_id | col_id | value
+--------+--------+-------
+ 1 | 1 | 9
+ 1 | 5 | 6
+ 1 | 6 | 6
+ 2 | 1 | 8
+ 3 | 1 | 3
+ 3 | 2 | 9
+ 4 | 7 | 0
+(6 rows)
</pre>
-The \c row_id represents the row number, \c col_id represents the column
-number and the \c value represents the matrix value at [\c row_id, \c col_id].
-The \c row_id and \c col_id values are indexed starting from 0. Thus the
-\c row_id ranges from 1 to \c row_dim, while the \c col_id ranges from 1 to
-\c col_dim
</dd>
<dt>output_table_prefix</dt>
-<dd>TEXT. Prefix for output tables. See <a href="#output">Output Tables</a>.</dd>
+<dd>TEXT. Prefix for output tables. See
+ <a href="#output">Output Tables</a> below for a description
+ of the convention used. </dd>
<dt>row_id</dt>
<dd>TEXT. Name of the column containing the row index for each entry in sparse matrix.</dd>
<dt>col_id</dt>
@@ -140,20 +156,26 @@ The \c row_id and \c col_id values are indexed starting from 0. Thus the
<dt>col_dim</dt>
<dd>INTEGER. Number of columns in matrix.</dd>
<dt>k</dt>
-<dd>INTEGER. Number of singular vectors to compute.</dd>
+<dd>INTEGER. Number of singular values to compute.</dd>
<dt>n_iterations (optional)</dt>
-<dd>INTEGER. Number of iterations to run.</dd>
+<dd>INTEGER. Number of iterations to run.
+ @note The number of iterations must be
+ in the range [k, column dimension], where
+ k is number of singular values.</dd>
<dt>result_summary_table (optional)</dt>
-<dd>TEXT, default: NULL. The name of the table to store a summary of the results.</dd>
+<dd>TEXT. The name of the table to store the result summary.</dd>
</dl>
<hr>
-<b>Native implementation for sparse matrix</b>
+<b>Native Implementation for Sparse Matrices</b>
Use this function for matrices that are represented in the sparse-matrix
-format (example below). This function use the native sparse representation while
-computing the SVD. <b>This function should be favored if the matrix is highly sparse.</b>
+format (see sparse matrix example above). This function uses the
+native sparse representation while computing the SVD.
+@note Note that this function should be favored if the matrix is
+highly sparse, since it computes very sparse matrices
+efficiently. </b>
<pre class="syntax">
svd_sparse_native( source_table,
@@ -173,7 +195,9 @@ svd_sparse_native( source_table,
<dt>source_table</dt>
<dd>TEXT. Source table name (sparse matrix - see example above).</dd>
<dt>output_table_prefix</dt>
- <dd>TEXT. Prefix for output tables. See <a href="#output">Output Tables</a>.</dd>
+ <dd>TEXT. Prefix for output tables. See
+ <a href="#output">Output Tables</a> below for a description
+ of the convention used.</dd>
<dt>row_id</dt>
<dd>TEXT. ID for each row.</dd>
<dt>col_id</dt>
@@ -185,51 +209,31 @@ svd_sparse_native( source_table,
<dt>col_dim</dt>
<dd>INTEGER. Col dimension of sparse matrix.</dd>
<dt>k</dt>
- <dd>INTEGER. Number of singular vectors to compute.</dd>
+ <dd>INTEGER. Number of singular values to compute.</dd>
<dt>n_iterations (optional)</dt>
- <dd>INTEGER. Number of iterations to run.</dd>
+ <dd>INTEGER. Number of iterations to run.
+ @note The number of iterations must be
+ in the range [k, column dimension], where
+ k is number of singular values.</dd>
<dt>result_summary_table (optional)</dt>
<dd>TEXT. Table name to store result summary.</dd>
</dl>
<hr>
-<b>Block matrices</b>
-
-<pre class="syntax">
-svd_block( source_table,
- output_table_prefix,
- k,
- n_iterations,
- result_summary_table
- );
-</pre>
-\b Arguments
-<dl class="arglist">
- <dt>source_table</dt>
- <dd>TEXT. Source table name (block matrix).</dd>
- <dt>output_table_prefix</dt>
- <dd>TEXT. Prefix for output tables. See <a href="#output">Output Tables</a>.</dd>
- <dt>k</dt>
- <dd>INTEGER. Number of singular vectors to compute.</dd>
- <dt>n_iterations (optional)</dt>
- <dd>INTEGER. Number of iterations to run.</dd>
- <dt>result_summary_table (optional)</dt>
- <dd>TEXT. Table name to store result summary.</dd>
-</dl>
@anchor output
@par Output Tables
-Output for eigen vectors/values is in the following three tables:
- - Left singular matrix: Table named \<output_table_prefix\>_left (e.g. ‘netflix_u’)
- - Right singular matrix: Table named \<output_table_prefix\>_right (e.g. ‘netflix_v’)
- - Singular values: Table named \<output_table_prefix\>_s (e.g. ‘netflix_s’)
+Output for eigenvectors/values is in the following three tables:
+ - Left singular matrix: Table is named \<output_table_prefix\>_u (e.g. ‘netflix_u’)
+ - Right singular matrix: Table is named \<output_table_prefix\>_v (e.g. ‘netflix_v’)
+ - Singular values: Table is named \<output_table_prefix\>_s (e.g. ‘netflix_s’)
-The singular vector tables are of the format:
+The left and right singular vector tables are of the format:
<table class="output">
<tr>
<th>row_id</th>
- <td>INTEGER. The ID corresponding to each eigen value (in decreasing order).</td>
+ <td>INTEGER. The ID corresponding to each eigenvalue (in decreasing order).</td>
</tr>
<tr>
<th>row_vec</th>
@@ -237,23 +241,23 @@ The singular vector tables are of the format:
</tr>
</table>
-The singular values table is in a sparse table format, since only the diagonal
+The singular values table is in sparse table format, since only the diagonal
elements of the matrix are non-zero:
<table class="output">
<tr>
<th>row_id</th>
- <td>INTEGER. \e i for \e ith eigen value.</td>
+ <td>INTEGER. \e i for \e ith eigenvalue.</td>
</tr>
<tr>
<th>col_id</th>
- <td>INTEGER. \e i for \e ith eigen value (same as row_id).</td>
+ <td>INTEGER. \e i for \e ith eigenvalue (same as row_id).</td>
</tr>
<tr>
<th>value</th>
- <td>FLOAT8. Eigen Value.</td>
+ <td>FLOAT8. Eigenvalue.</td>
</tr>
</table>
-All \c row_id (and \c col_id) in the above tables start from 0.
+All \c row_id and \c col_id in the above tables start from 1.
The result summary table has the following columns:
@@ -276,7 +280,7 @@ The result summary table has the following columns:
</tr>
<tr>
<th>relative_recon_error</th>
- <td>FLOAT8. relative quality score.</td>
+ <td>FLOAT8. Relative quality score.</td>
</tr>
</table>
@@ -296,65 +300,105 @@ SELECT madlib.svd();
-# Create an input dataset (dense matrix).
<pre class="example">
+DROP TABLE IF EXISTS mat, mat_sparse, svd_summary_table, svd_u, svd_v, svd_s;
CREATE TABLE mat (
row_id integer,
row_vec double precision[]
);
-COPY mat (row_id, row_vec) FROM stdin delimiter '|';
-1|{396,840,353,446,318,886,15,584,159,383}
-2|{691,58,899,163,159,533,604,582,269,390}
-3|{293,742,298,75,404,857,941,662,846,2}
-4|{462,532,787,265,982,306,600,608,212,885}
-5|{304,151,337,387,643,753,603,531,459,652}
-6|{327,946,368,943,7,516,272,24,591,204}
-7|{877,59,260,302,891,498,710,286,864,675}
-8|{458,959,774,376,228,354,300,669,718,565}
-9|{824,390,818,844,180,943,424,520,65,913}
-10|{882,761,398,688,761,405,125,484,222,873}
-11|{528,1,860,18,814,242,314,965,935,809}
-12|{492,220,576,289,321,261,173,1,44,241}
-13|{415,701,221,503,67,393,479,218,219,916}
-14|{350,192,211,633,53,783,30,444,176,932}
-15|{909,472,871,695,930,455,398,893,693,838}
-16|{739,651,678,577,273,935,661,47,373,618}
-\\.
+INSERT INTO mat VALUES
+(1,'{396,840,353,446,318,886,15,584,159,383}'),
+(2,'{691,58,899,163,159,533,604,582,269,390}'),
+(3,'{293,742,298,75,404,857,941,662,846,2}'),
+(4,'{462,532,787,265,982,306,600,608,212,885}'),
+(5,'{304,151,337,387,643,753,603,531,459,652}'),
+(6,'{327,946,368,943,7,516,272,24,591,204}'),
+(7,'{877,59,260,302,891,498,710,286,864,675}'),
+(8,'{458,959,774,376,228,354,300,669,718,565}'),
+(9,'{824,390,818,844,180,943,424,520,65,913}'),
+(10,'{882,761,398,688,761,405,125,484,222,873}'),
+(11,'{528,1,860,18,814,242,314,965,935,809}'),
+(12,'{492,220,576,289,321,261,173,1,44,241}'),
+(13,'{415,701,221,503,67,393,479,218,219,916}'),
+(14,'{350,192,211,633,53,783,30,444,176,932}'),
+(15,'{909,472,871,695,930,455,398,893,693,838}'),
+(16,'{739,651,678,577,273,935,661,47,373,618}');
</pre>
-# Run SVD function for a dense matrix.
<pre class="example">
-DROP TABLE IF EXISTS svd_u;
-DROP TABLE IF EXISTS svd_v;
-DROP TABLE IF EXISTS svd_s;
-SELECT madlib.svd( 'mat',
- 'svd',
- 'row_id',
- 10
+SELECT madlib.svd( 'mat', -- Input table
+ 'svd', -- Output table prefix
+ 'row_id', -- Column name with row index
+ 10, -- Number of singular values to compute
+ NULL, -- Use default number of iterations
+ 'svd_summary_table' -- Result summary table
);
</pre>
-
+-# Print out the singular values and the summary table. For the singular values:
+<pre class="example">
+SELECT * FROM svd_s ORDER BY row_id;
+</pre>
+Result:
+<pre class="result">
+ row_id | col_id | value
+ --------+--------+------------------
+ 1 | 1 | 6475.67225281804
+ 2 | 2 | 1875.18065580415
+ 3 | 3 | 1483.25228429636
+ 4 | 4 | 1159.72262897427
+ 5 | 5 | 1033.86092570574
+ 6 | 6 | 948.437358703966
+ 7 | 7 | 795.379572772455
+ 8 | 8 | 709.086240684469
+ 9 | 9 | 462.473775959371
+ 10 | 10 | 365.875217945698
+ 10 | 10 |
+(11 rows)
+</pre>
+For the summary table:
+<pre class="example">
+SELECT * FROM svd_summary_table;
+</pre>
+Result:
+<pre class="result">
+ rows_used | exec_time (ms) | iter | recon_error | relative_recon_error
+ -----------+----------------+------+-------------------+----------------------
+ 16 | 1332.47 | 10 | 4.36920148766e-13 | 7.63134130332e-16
+(1 row)
+</pre>
-# Create a sparse matrix by running the \ref matrix_sparsify() utility function on the dense matrix.
<pre class="example">
-DROP TABLE IF EXISTS mat_sparse;
-SELECT madlib.matrix_sparsify( 'mat',
- 'mat_sparse',
- FALSE
- );
-DROP TABLE IF EXISTS svd_u;
-DROP TABLE IF EXISTS svd_v;
-DROP TABLE IF EXISTS svd_s;
+SELECT madlib.matrix_sparsify('mat',
+ 'row=row_id, val=row_vec',
+ 'mat_sparse',
+ 'row=row_id, col=col_id, val=value');
</pre>
-# Run the SVD function for a sparse matrix.
<pre class="example">
-SELECT madlib.svd_sparse( 'mat_sparse',
- 'svd',
- 'row_id',
- 'col_id',
- 'value',
- 10
- );
+SELECT madlib.svd_sparse( 'mat_sparse', -- Input table
+ 'svd', -- Output table prefix
+ 'row_id', -- Column name with row index
+ 'col_id', -- Column name with column index
+ 'value', -- Matrix cell value
+ 16, -- Number of rows in matrix
+ 10, -- Number of columns in matrix
+ 10 -- Number of singular values to compute
+ );
</pre>
+-# Run the SVD function for a very sparse matrix.
+<pre class="example">
+SELECT madlib.svd_sparse_native ( 'mat_sparse', -- Input table
+ 'svd', -- Output table prefix
+ 'row_id', -- Column name with row index
+ 'col_id', -- Column name with column index
+ 'value', -- Matrix cell value
+ 16, -- Number of rows in matrix
+ 10, -- Number of columns in matrix
+ 10 -- Number of singular values to compute
+ );
+</pre>
@anchor background
@par Technical Background
In linear algebra, the singular value decomposition (SVD) is a factorization of
@@ -368,7 +412,7 @@ decomposed as follows:
where \f$U\f$ is a \f$m \times n\f$ orthonormal matrix,
\f$\Sigma\f$ is a \f$n \times n\f$ diagonal matrix, and \f$V\f$ is an
\f$n \times n\f$ orthonormal matrix. The diagonal elements of \f$\Sigma\f$ are
-called the \e {singular values}.
+called the <em>singular values</em>.
It is possible to formulate the problem of computing the singular triplets
(\f$\sigma_i, u_i, v_i\f$) of \f$A\f$ as an eigenvalue problem involving a Hermitian
matrix related to \f$A\f$. There are two possible ways of achieving this:
http://git-wip-us.apache.org/repos/asf/incubator-madlib/blob/da90d620/src/ports/postgres/modules/pca/pca.sql_in
----------------------------------------------------------------------
diff --git a/src/ports/postgres/modules/pca/pca.sql_in b/src/ports/postgres/modules/pca/pca.sql_in
index cbf921a..7873bdf 100644
--- a/src/ports/postgres/modules/pca/pca.sql_in
+++ b/src/ports/postgres/modules/pca/pca.sql_in
@@ -177,19 +177,30 @@ contain values between 1 and <em>M</em>.</DD>
<DT>components_param</DT>
<DD>INTEGER or FLOAT. The parameter to control the number of principal
-components to calculate from the input data. If components_param is INTEGER,
-it is used for denoting the number of principal components (<em>k</em>) to
-compute. If components_param is FLOAT, the algorithm would return enough
+components to calculate from the input data. If 'components_param' is INTEGER,
+it is used to denote the number of principal components (<em>k</em>) to
+compute. If 'components_param' is FLOAT, the algorithm will return enough
principal vectors so that the ratio of the sum of the eigenvalues collected
-thus far to the sum of all eigenvalues is greater than this parameter.
-This value has to be either a positive INTEGER or a FLOAT in the range
-(0.0,1.0]</DD>
+thus far to the sum of all eigenvalues is greater than this parameter
+(proportion of variance). The value of 'components_param' must be either
+a positive INTEGER or a FLOAT in the range (0.0,1.0]</DD>
+
+@note The difference in interpretation between INTEGER and FLOAT was
+introduced to maintain backward campatibility after the proportion of
+variance feature was introduced. A special case to be aware of:
+'components_param' = 1 (INTEGER) will return 1 principal
+component, but 'components_param' = 1.0 (FLOAT) will return all
+principal components, i.e., proportion of variance of 100%.
<DT>grouping_cols (optional)</DT>
-<DD>TEXT, default: NULL. Currently <em>grouping_cols</em> is present as a placeholder for forward
- compatibility. The parameter is planned to be implemented as a comma-separated
- list of column names, with the source data grouped using the combination of all the columns.
- An independent PCA model will be computed for each combination of the grouping columns.</DD>
+<DD>TEXT, default: NULL.
+
+@note <em>Not currently implemented. Any non-NULL value is ignored.
+ Grouping support will be added in a future release. </em> The parameter
+ is planned to be implemented as a
+ comma-separated list of column names, with the source data grouped using
+ the combination of all the columns. An independent PCA model will be
+ computed for each combination of the grouping columns.</DD>
<DT>lanczos_iter (optional)</DT>
<DD>INTEGER, default: minimum of {k+40, smallest matrix dimension}.
@@ -201,6 +212,13 @@ large as the value of <em>k</em>, but no larger than the smallest dimension
of the matrix. If the iteration number is given as zero, then the default
number of iterations is used.</DD>
+@note If both 'lanczos_iter' and proportion of variance (via
+the 'components_param' parameter) are defined, 'lanczos_iter'
+will take precedence in determining the number of
+principal components (i.e. the number of principal components
+will not be greater than 'lanczos_iter' even if the target
+proportion had not been reached).
+
<DT>use_correlation (optional)</DT>
<DD>BOOLEAN, default FALSE. Whether to use the correlation matrix for calculating the principal components instead of the covariance matrix. Currently
<em>use_correlation</em> is a placeholder for forward compatibility, and
@@ -324,13 +342,20 @@ string should be double-quoted; in this case the input would be '"MyTable"').
\ref background_pca "Technical Background"), sparse matrices almost always
become dense during the training process. Thus, this implementation
automatically densifies sparse matrix input, and there should be no expected
- performance improvement in using sparse matrix input over dense matrix input.
-
-- If both <em>lanczos_iter</em> and proportion of variance (via the
-<em>grouping_cols</em>) is defined, <em>lanczos_iter</em> will
-take precedence in determining the number of principal components (i.e. the
-number of principal components will not be greater than <em>lanczos_iter</em>
-even if the target proportion is not reached).
+performance improvement in using sparse matrix input over dense matrix input.
+
+- For the parameter 'components_param', INTEGER and FLOAT are
+interpreted differently. A special case to be aware of:
+'components_param' = 1 (INTEGER) will return 1 principal
+component, but 'components_param' = 1.0 (FLOAT) will return all
+principal components, i.e., proportion of variance of 100%.
+
+- If both 'lanczos_iter' and proportion of variance (via
+the 'components_param' parameter) are defined, 'lanczos_iter'
+will take precedence in determining the number of
+principal components (i.e. the number of principal components
+will not be greater than 'lanczos_iter' even if the target
+proportion had not been reached).
@anchor background_pca
@par Technical Background
http://git-wip-us.apache.org/repos/asf/incubator-madlib/blob/da90d620/src/ports/postgres/modules/summary/summary.sql_in
----------------------------------------------------------------------
diff --git a/src/ports/postgres/modules/summary/summary.sql_in b/src/ports/postgres/modules/summary/summary.sql_in
index 3f25502..5457463 100644
--- a/src/ports/postgres/modules/summary/summary.sql_in
+++ b/src/ports/postgres/modules/summary/summary.sql_in
@@ -25,9 +25,9 @@ m4_include(`SQLCommon.m4')
@brief Calculates general descriptive statistics for any data table.
-The MADlib \b summary() function is a generic function that produces summary
-statistics from any data table. The function invokes various methods from
-the MADlib library to provide an overview of the data.
+The MADlib \b summary() function produces summary
+statistics for any data table. The function invokes various methods from
+the MADlib library to provide the data overview.
@anchor usage
@par Summary Function Syntax
@@ -45,13 +45,6 @@ summary ( source_table,
get_estimates
)
</pre>
-\b Arguments
-<DL class="arglist">
-<dt>source_table</dt>
-<dd>TEXT. The name of the table containing the input data.</dd>
-<dt>output_table</dt>
-<dd>TEXT. The name of the table to contain the output summary data.
-
The \b summary() function returns a composite type containing three fields:
<table class="output">
<tr>
@@ -67,10 +60,15 @@ The \b summary() function returns a composite type containing three fields:
<td>FLOAT8. The time taken (in seconds) to compute the summary.</td>
</tr>
</table>
-
+\b Arguments
+<DL class="arglist">
+<dt>source_table</dt>
+<dd>TEXT. The name of the table containing the input data.</dd>
+<dt>output_table</dt>
+<dd>TEXT. The name of the table to contain the output summary data.
Summary statistics are saved in a table with the name specifed in the
-<em>output_table</em> argument. The table can contain the
+<em>output_table</em> argument. The table contains the
following columns:
<table class="output">
<tr>
@@ -87,7 +85,7 @@ following columns:
</tr>
<tr>
<th>column_number</th>
- <td>Physical column number for the target column, as described in \e pg_attribute.</td>
+ <td>Physical column number for the target column, as described in \e pg_attribute</td> catalog.
</tr>
<tr>
<th>data_type</th>
@@ -167,13 +165,24 @@ following columns:
<dt>target_columns (optional)</dt>
<dd>TEXT, default NULL. A comma-separated list of columns to summarize. If NULL, summaries are produced for all columns.</dd>
<dt>grouping_cols (optional)</dt>
-<dd>TEXT, default: null. A comma-separated list of columns on which to group results. If NULL, summaries are produced on the complete table.</dd>
+<dd>TEXT, default: null. A comma-separated list of columns on which to
+group results. If NULL, summaries are produced on the complete table.</dd>
+@note Please note that summary statistics are calculated for each grouping
+column independently. That is, grouping columns are not combined together
+as in the regular PostgreSQL style GROUP BY directive. (This was done
+to reduce long run time and huge output table size which would otherwise
+result in the case of large input tables with a lot of grouping_cols and
+target_cols specified.)
<dt>get_distinct (optional)</dt>
<dd>BOOLEAN, default TRUE. If true, distinct values are counted.</dd>
<dt>get_quartiles (optional)</dt>
-<dd>BOOLEAN, default TRUE. If TRUE, qartiles are computed.</dd>
+<dd>BOOLEAN, default TRUE. If TRUE, quartiles are computed.</dd>
<dt>ntile_array (optional)</dt>
<dd>FLOAT8[], default NULL. An array of quantile values to compute. If NULL, quantile values are not computed.</dd>
+@note Quartile and quantile functions are not available for PostgreSQL 9.3 or
+lower. If you are using PostgreSQL 9.3 or lower, the output table will not
+contain these values, even if you set 'get_quartiles' = TRUE or
+provide an array of quantile values for the parameter 'ntile_array'.
<dt>how_many_mfv (optional)</dt>
<dd>INTEGER, default: 10. The number of most-frequent-values to compute.</dd>
<dt>get_estimates (optional)</dt>
@@ -285,6 +294,16 @@ string should be double-quoted; in this case the input would be '"MyTable"').
in a slow but exact method. The most frequent values are computed using a
faithful implementation that preserves the approximation guarantees of
the Cormode/Muthukrishnan method (more information in \ref grp_mfvsketch).
+- Summary statistics are calculated for each grouping
+column independently. That is, grouping columns are not combined together
+as in the regular PostgreSQL style GROUP BY directive. (This was done
+to reduce long run time and huge output table size which would otherwise
+result in the case of large input tables with a lot of grouping_cols and
+target_cols specified.)
+- Quartile and quantile functions are not available for PostgreSQL 9.3 or
+lower. If you are using PostgreSQL 9.3 or lower, the output table will not
+contain these values, even if you set 'get_quartiles' = TRUE or
+provide an array of quantile values for the parameter 'ntile_array'.
@anchor related