You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@madlib.apache.org by ri...@apache.org on 2018/07/26 18:34:17 UTC
madlib git commit: Multiple: Clean and update documentation
Repository: madlib
Updated Branches:
refs/heads/master 2aac41897 -> 836759e69
Multiple: Clean and update documentation
Closes #298
Project: http://git-wip-us.apache.org/repos/asf/madlib/repo
Commit: http://git-wip-us.apache.org/repos/asf/madlib/commit/836759e6
Tree: http://git-wip-us.apache.org/repos/asf/madlib/tree/836759e6
Diff: http://git-wip-us.apache.org/repos/asf/madlib/diff/836759e6
Branch: refs/heads/master
Commit: 836759e69ae617ffc0cd7640cb7ca76b25e69c1d
Parents: 2aac418
Author: Frank McQuillan <fm...@pivotal.io>
Authored: Tue Jul 24 17:20:18 2018 -0700
Committer: Rahul Iyer <ri...@apache.org>
Committed: Thu Jul 26 11:33:34 2018 -0700
----------------------------------------------------------------------
doc/mainpage.dox.in | 174 +++++++++----------
src/ports/postgres/modules/convex/mlp.sql_in | 3 +-
.../modules/utilities/encode_categorical.sql_in | 3 +-
.../postgres/modules/utilities/path.sql_in | 3 +-
.../postgres/modules/utilities/pivot.sql_in | 3 +-
.../modules/utilities/sessionize.sql_in | 3 +-
.../postgres/modules/utilities/utilities.sql_in | 5 +-
7 files changed, 94 insertions(+), 100 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/madlib/blob/836759e6/doc/mainpage.dox.in
----------------------------------------------------------------------
diff --git a/doc/mainpage.dox.in b/doc/mainpage.dox.in
index c2c9a7a..8f97491 100644
--- a/doc/mainpage.dox.in
+++ b/doc/mainpage.dox.in
@@ -2,13 +2,9 @@
@mainpage
Apache MADlib is an open-source library for scalable
in-database analytics. It provides data-parallel implementations of
-mathematical, statistical and machine learning methods for structured
+mathematical, statistical, graph and machine learning methods for structured
and unstructured data.
-The MADlib mission: to foster widespread development of scalable analytic
-skills, by harnessing efforts from commercial practice, academic research,
-and open-source development.
-
Useful links:
<ul>
<li><a href="http://madlib.apache.org">MADlib web site</a></li>
@@ -21,32 +17,22 @@ Useful links:
<a href="../v1.13/index.html">v1.13</a>,
<a href="../v1.12/index.html">v1.12</a>,
<a href="../v1.11/index.html">v1.11</a>,
- <a href="../v1.10/index.html">v1.10.0</a>,
- <a href="../v1.9.1/index.html">v1.9.1</a>,
- <a href="../v1.9/index.html">v1.9</a>,
- <a href="../v1.8/index.html">v1.8</a>,
- <a href="../v1.7.1/index.html">v1.7.1</a>,
- <a href="../v1.7/index.html">v1.7</a>,
- <a href="../v1.6/index.html">v1.6</a>,
- <a href="../v1.5/index.html">v1.5</a>,
- <a href="../v1.4/index.html">v1.4</a>,
- <a href="../v1.3/index.html">v1.3</a>,
- <a href="../v1.2/index.html">v1.2</a>
+ <a href="../v1.10/index.html">v1.10</a>
</li>
</ul>
Please refer to the
<a href="https://github.com/apache/madlib/blob/master/README.md">ReadMe</a>
file for information about incorporated third-party material. License information
-regarding MADlib and included third-party libraries can be found inside the
+regarding MADlib and included third-party libraries can be found in the
<a href="https://github.com/apache/madlib/blob/master/LICENSE">
License</a> directory.
@defgroup grp_datatrans Data Types and Transformations
-@{Data types and transformation operations @}
+@details Data types and operations that transform and shape data.
@defgroup grp_arraysmatrix Arrays and Matrices
@ingroup grp_datatrans
- @brief Mathematical operations for arrays and matrices
+ @brief Mathematical operations for arrays and matrices.
@details
These modules provide basic mathematical operations to be run on array and matrices.
@@ -100,13 +86,14 @@ complete matrix stored as a distributed table.
@defgroup grp_matrix Matrix Operations
@defgroup grp_matrix_factorization Matrix Factorization
- @brief Matrix Factorization methods including Singular Value Decomposition and Low-rank Matrix Factorization
+ @brief Linear algebra methods that factorize a matrix into a product of matrices.
+ @details Linear algebra methods that factorize a matrix into a product of matrices.
@{
@defgroup grp_lmf Low-Rank Matrix Factorization
@defgroup grp_svd Singular Value Decomposition
@}
- @defgroup grp_linalg Norms and Distance functions
+ @defgroup grp_linalg Norms and Distance Functions
@defgroup grp_svec Sparse Vectors
@}
@@ -126,49 +113,58 @@ complete matrix stored as a distributed table.
@ingroup grp_datatrans
@defgroup grp_graph Graph
-Contains graph algorithms.
+@brief Graph algorithms and measures associated with graphs.
+@details Graph algorithms and measures associated with graphs.
@{
@defgroup grp_apsp All Pairs Shortest Path
@defgroup grp_bfs Breadth-First Search
@defgroup grp_hits HITS
+
@defgroup grp_graph_measures Measures
- Graph Measures
+ @brief A collection of metrics computed on a graph.
+ @details A collection of metrics computed on a graph.
@{
@defgroup grp_graph_avg_path_length Average Path Length
@defgroup grp_graph_closeness Closeness
@defgroup grp_graph_diameter Graph Diameter
@defgroup grp_graph_vertex_degrees In-Out Degree
@}
+
@defgroup grp_pagerank PageRank
@defgroup grp_sssp Single Source Shortest Path
@defgroup grp_wcc Weakly Connected Components
@}
@defgroup grp_mdl Model Selection
-@{Contains functions for model selection and model evaluation. @}
+@brief Functions for model selection and model evaluation.
+@details Functions for model selection and model evaluation.
+@{
@defgroup grp_validation Cross Validation
@ingroup grp_mdl
@defgroup grp_pred Prediction Metrics
@ingroup grp_mdl
@defgroup grp_train_test_split Train-Test Split
@ingroup grp_mdl
+@}
@defgroup grp_sampling Sampling
- @ingroup grp_sampling
- @{A collection of methods for sampling from a population. @}
- @defgroup grp_balance_sampling Balanced Sampling
- @ingroup grp_sampling
- @defgroup grp_strs Stratified Sampling
- @ingroup grp_sampling
+@brief A collection of methods for sampling from a population.
+@details A collection of methods for sampling from a population.
+@{
+ @defgroup grp_balance_sampling Balanced Sampling
+ @defgroup grp_strs Stratified Sampling
+@}
@defgroup grp_stats Statistics
-@{Contains statistics modules @}
+@brief A collection of probability and statistics modules.
+@details A collection of probability and statistics modules.
+@{
@defgroup grp_desc_stats Descriptive Statistics
- @ingroup grp_stats
- @{A collection of methods to compute descriptive statistics of the dataset @}
-
+ @brief Methods to compute descriptive statistics of a dataset.
+ @details Methods to compute descriptive statistics of a dataset.
+ @{
@defgroup grp_sketches Cardinality Estimators
- @ingroup grp_desc_stats
+ @brief Methods to estimate the number of unique values contained in data.
@{
@defgroup grp_countmin CountMin (Cormode-Muthukrishnan)
@defgroup grp_fmsketch FM (Flajolet-Martin)
@@ -176,31 +172,28 @@ Contains graph algorithms.
@}
@defgroup grp_correlation Covariance and Correlation
- @ingroup grp_desc_stats
-
@defgroup grp_summary Summary
- @ingroup grp_desc_stats
+ @}
@defgroup grp_inf_stats Inferential Statistics
- @ingroup grp_stats
- @{A collection of methods to compute inferential statistics on a dataset.@}
+ @brief Methods to compute inferential statistics of a dataset.
+ @details Methods to compute inferential statistics of a dataset.
+ @{
@defgroup grp_stats_tests Hypothesis Tests
- @ingroup grp_inf_stats
+ @}
@defgroup grp_prob Probability Functions
- @ingroup grp_stats
+@}
@defgroup grp_super Supervised Learning
-@{Contains methods which perform supervised learning tasks @}
+@brief Methods to perform a variety of supervised learning tasks.
+@details Methods to perform a variety of supervised learning tasks.
+@{
@defgroup grp_crf Conditional Random Field
- @ingroup grp_super
-
@defgroup grp_nn Neural Network
- @ingroup grp_super
-
@defgroup grp_regml Regression Models
- @ingroup grp_super
- A collection of methods for modeling conditional expectation of a response variable.
+ @brief A collection of methods for modeling conditional expectation of a response variable.
+ @details A collection of methods for modeling conditional expectation of a response variable.
@{
@defgroup grp_clustered_errors Clustered Variance
@defgroup grp_cox_prop_hazards Cox-Proportional Hazards Regression
@@ -215,82 +208,78 @@ Contains graph algorithms.
@}
@defgroup grp_svm Support Vector Machines
- @ingroup grp_super
-
@defgroup grp_tree Tree Methods
- @ingroup grp_super
- @{A collection of recursive partitioning (tree) methods. @}
+ @brief A collection of recursive partitioning (tree) methods.
+ @details A collection of recursive partitioning (tree) methods.
+ @{
@defgroup grp_decision_tree Decision Tree
- @ingroup grp_tree
-
@defgroup grp_random_forest Random Forest
- @ingroup grp_tree
+ @}
+@}
@defgroup grp_tsa Time Series Analysis
-@{A collection of methods to analyze time series data. @}
+@brief A collection of methods to analyze time series data.
+@details A collection of methods to analyze time series data.
+@{
@defgroup grp_arima ARIMA
- @ingroup grp_tsa
+@}
@defgroup grp_unsupervised Unsupervised Learning
-@{A collection of methods for unsupervised learning tasks @}
+@brief A collection of methods for unsupervised learning tasks.
+@details A collection of methods for unsupervised learning tasks.
+@{
@defgroup grp_association_rules Association Rules
- @ingroup grp_unsupervised
- @{A collection of methods used to uncover interesting patterns in transactional datasets. @}
+ @brief Methods used to discover patterns in transactional datasets.
+ @details Methods used to discover patterns in transactional datasets.
+ @{
@defgroup grp_assoc_rules Apriori Algorithm
- @ingroup grp_association_rules
+ @}
@defgroup grp_clustering Clustering
- @ingroup grp_unsupervised
- @{A collection of methods for clustering data@}
+ @brief Methods for clustering data.
+ @details Methods for clustering data.
+ @{
@defgroup grp_kmeans k-Means Clustering
- @ingroup grp_clustering
+ @}
@defgroup grp_pca Dimensionality Reduction
- @ingroup grp_unsupervised
- @brief A collection of methods for dimensionality reduction.
- @details A collection of methods for dimensionality reduction.
+ @brief Methods for reducing the number of variables in a dataset to obtain a set of principle variables.
+ @details Methods for reducing the number of variables in a dataset to obtain a set of principle variables.
@{
@defgroup grp_pca_train Principal Component Analysis
@defgroup grp_pca_project Principal Component Projection
@}
@defgroup grp_topic_modelling Topic Modelling
- @ingroup grp_unsupervised
- @{A collection of methods to uncover abstract topics in a document corpus @}
+ @brief A collection of methods to uncover abstract topics in a document corpus.
+ @details A collection of methods to uncover abstract topics in a document corpus.
+ @{
@defgroup grp_lda Latent Dirichlet Allocation
- @ingroup grp_topic_modelling
+ @}
+@}
-@defgroup grp_other_functions Other Functions
+@defgroup grp_other_functions Utilities
+@details Useful utilities for data science workflows.
+@{
@defgroup grp_cols2vec Columns to Vector
- @ingroup grp_other_functions
+ @defgroup @grp_utilities Database Functions
@defgroup grp_linear_solver Linear Solvers
- @ingroup grp_other_functions
- @{A collection of methods that implement solutions for systems of consistent linear equations. @}
-
+ @brief Methods that implement solutions for systems of consistent linear equations.
+ @details Methods that implement solutions for systems of consistent linear equations.
+ @{
@defgroup grp_dense_linear_solver Dense Linear Systems
- @ingroup grp_linear_solver
-
@defgroup grp_sparse_linear_solver Sparse Linear Systems
- @ingroup grp_linear_solver
+ @}
@defgroup grp_minibatch_preprocessing Mini-Batch Preprocessor
- @ingroup grp_other_functions
-
@defgroup grp_pmml PMML Export
- @ingroup grp_other_functions
-
@defgroup grp_text_utilities Term Frequency
- @ingroup grp_other_functions
-
- @defgroup @grp_utilities Utilities
- @ingroup grp_other_functions
-
+@}
@defgroup grp_early_stage Early Stage Development
-@brief A collection of implementations which are in early stage of development.
-There may be some issues that will be addressed in a future version.
+@details Implementations which are in an early stage of development.
Interface and implementation are subject to change.
@{
@defgroup grp_cg Conjugate Gradient
@@ -300,8 +289,9 @@ Interface and implementation are subject to change.
@}
@defgroup grp_deprecated Deprecated Modules
-@brief A collection of deprecated modules. These functions will be removed in the
-next major version (2.0).
+Deprecated modules that will be removed in the
+next major version (2.0). There are newer MADlib modules
+that have replaced these functions.
@{
@defgroup grp_indicator Create Indicator Variables
@defgroup grp_mlogreg Multinomial Logistic Regression
http://git-wip-us.apache.org/repos/asf/madlib/blob/836759e6/src/ports/postgres/modules/convex/mlp.sql_in
----------------------------------------------------------------------
diff --git a/src/ports/postgres/modules/convex/mlp.sql_in b/src/ports/postgres/modules/convex/mlp.sql_in
index 9fba404..3e2468f 100644
--- a/src/ports/postgres/modules/convex/mlp.sql_in
+++ b/src/ports/postgres/modules/convex/mlp.sql_in
@@ -444,7 +444,8 @@ range 0 to 1, where 0 means no momentum.
</DD>
<DT>nesterov</dt>
-<DD>Default: TRUE. Nesterov momentum can provide better results than using
+<DD>Default: TRUE. Only used when the 'momentum' parameter is > 0.
+Nesterov momentum can provide better results than using
classical momentum alone, due to its look-ahead characteristics. In classical
momentum we correct the velocity and then update the model with that velocity,
whereas in Nesterov Accelerated Gradient method, we first move the model in the
http://git-wip-us.apache.org/repos/asf/madlib/blob/836759e6/src/ports/postgres/modules/utilities/encode_categorical.sql_in
----------------------------------------------------------------------
diff --git a/src/ports/postgres/modules/utilities/encode_categorical.sql_in b/src/ports/postgres/modules/utilities/encode_categorical.sql_in
index 69b7e34..06e418d 100644
--- a/src/ports/postgres/modules/utilities/encode_categorical.sql_in
+++ b/src/ports/postgres/modules/utilities/encode_categorical.sql_in
@@ -43,7 +43,8 @@ m4_include(`SQLCommon.m4')
</ul>
</div>
-@brief Provides functions to encode categorical variables
+@brief Functions to encode categorical variables to prepare data for
+input into predictive algorithms.
@anchor categorical
@par Coding Systems for Categorical Variables
http://git-wip-us.apache.org/repos/asf/madlib/blob/836759e6/src/ports/postgres/modules/utilities/path.sql_in
----------------------------------------------------------------------
diff --git a/src/ports/postgres/modules/utilities/path.sql_in b/src/ports/postgres/modules/utilities/path.sql_in
index 8fe03ec..c90855b 100644
--- a/src/ports/postgres/modules/utilities/path.sql_in
+++ b/src/ports/postgres/modules/utilities/path.sql_in
@@ -39,7 +39,8 @@ m4_include(`SQLCommon.m4')
</ul>
</div>
-@brief Path Functions
+@brief A function to perform complex pattern matching across rows
+and extract useful information about the matches.
@details The goal of the MADlib path function is to perform regular pattern
matching over a sequence of rows, and to extract useful information about
the pattern matches. The useful information could be a simple count of
http://git-wip-us.apache.org/repos/asf/madlib/blob/836759e6/src/ports/postgres/modules/utilities/pivot.sql_in
----------------------------------------------------------------------
diff --git a/src/ports/postgres/modules/utilities/pivot.sql_in b/src/ports/postgres/modules/utilities/pivot.sql_in
index 0c8989a..d1aba19 100644
--- a/src/ports/postgres/modules/utilities/pivot.sql_in
+++ b/src/ports/postgres/modules/utilities/pivot.sql_in
@@ -41,7 +41,8 @@ m4_include(`SQLCommon.m4')
</ul>
</div>
-@brief Provides pivoting functions helpful for data preparation before modeling
+@brief Pivoting and data summarization tools for preparing data
+for modeling operations.
@anchor pivoting
The goal of the MADlib pivot function is to provide a data summarization tool
http://git-wip-us.apache.org/repos/asf/madlib/blob/836759e6/src/ports/postgres/modules/utilities/sessionize.sql_in
----------------------------------------------------------------------
diff --git a/src/ports/postgres/modules/utilities/sessionize.sql_in b/src/ports/postgres/modules/utilities/sessionize.sql_in
index c66b417..f8ac6d7 100644
--- a/src/ports/postgres/modules/utilities/sessionize.sql_in
+++ b/src/ports/postgres/modules/utilities/sessionize.sql_in
@@ -38,7 +38,8 @@ m4_include(`SQLCommon.m4')
</ul>
</div>
-@brief Sessionize
+@brief Session reconstruction of data
+consisting of a time stampled sequence of events.
@details The MADlib sessionize function performs time-oriented session reconstruction
on a data set comprising a sequence of events. A defined period of inactivity
indicates the end of one session and beginning of the next session. Sessions can be
http://git-wip-us.apache.org/repos/asf/madlib/blob/836759e6/src/ports/postgres/modules/utilities/utilities.sql_in
----------------------------------------------------------------------
diff --git a/src/ports/postgres/modules/utilities/utilities.sql_in b/src/ports/postgres/modules/utilities/utilities.sql_in
index 4993afe..09f971d 100644
--- a/src/ports/postgres/modules/utilities/utilities.sql_in
+++ b/src/ports/postgres/modules/utilities/utilities.sql_in
@@ -25,9 +25,8 @@ m4_include(`SQLCommon.m4')
@brief Provides a collection of user-defined functions for performing common tasks in the database.
-The utility module consists of useful utility functions to assist data
-scientists in using the product. Several of these functions can be used
-while implementing new algorithms.
+Database functions are a collection of lower level utilities
+to assist data scientists and others in using MADlib.
@anchor utilities
@par Utility Functions