You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@madlib.apache.org by nj...@apache.org on 2017/09/20 17:58:46 UTC
madlib git commit: Docs: MLP docs to be corrected post grouping
support
Repository: madlib
Updated Branches:
refs/heads/master ba115d02d -> 714a82ec0
Docs: MLP docs to be corrected post grouping support
The MLP docs created during grouping support had some inadvertant
changes. This commit fixes those issues.
Closes #187
Project: http://git-wip-us.apache.org/repos/asf/madlib/repo
Commit: http://git-wip-us.apache.org/repos/asf/madlib/commit/714a82ec
Tree: http://git-wip-us.apache.org/repos/asf/madlib/tree/714a82ec
Diff: http://git-wip-us.apache.org/repos/asf/madlib/diff/714a82ec
Branch: refs/heads/master
Commit: 714a82ec0c9274c0e9c8ae0dd3c20c2f9a0d8267
Parents: ba115d0
Author: Nandish Jayaram <nj...@apache.org>
Authored: Tue Sep 19 14:14:03 2017 -0700
Committer: Nandish Jayaram <nj...@apache.org>
Committed: Wed Sep 20 10:57:47 2017 -0700
----------------------------------------------------------------------
src/ports/postgres/modules/convex/mlp.sql_in | 268 ++++++++++++----------
1 file changed, 153 insertions(+), 115 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/madlib/blob/714a82ec/src/ports/postgres/modules/convex/mlp.sql_in
----------------------------------------------------------------------
diff --git a/src/ports/postgres/modules/convex/mlp.sql_in b/src/ports/postgres/modules/convex/mlp.sql_in
index 55f401c..bafb4dd 100644
--- a/src/ports/postgres/modules/convex/mlp.sql_in
+++ b/src/ports/postgres/modules/convex/mlp.sql_in
@@ -49,16 +49,17 @@ Also called "vanilla neural networks", MLPs consist of several
fully connected hidden layers with non-linear activation
functions. In the case of classification, the final layer of the
neural net has as many nodes as classes, and the output of the
-neural net can be interpreted as the probability of a given input
-feature belonging to a specific class.
+neural net can be interpreted as the probability that a given input
+feature belongs to a specific class.
@brief Solves classification and regression problems with several
-fully connected layers and nonlinear activation functions.
+fully connected layers and non-linear activation functions.
@anchor mlp_classification
@par Classification Training Function
-The mlp classification training function has the following format:
+The MLP classification training function has the following format:
+
<pre class="syntax">
mlp_classification(
source_table,
@@ -74,39 +75,43 @@ mlp_classification(
grouping_col
)
</pre>
-\b Arguments
-<DL class="arglist">
- <DT>source_table</DT>
- <DD>TEXT. Name of the table containing the training data.</DD>
-
-
- <DT>output_table</DT>
- <DD>TEXT. Name of the output table containing the model. Details of the output
- tables are provided below.
- </DD>
-
- <DT>independent_varname</DT>
- <DD>TEXT. Expression list to evaluate for the
- independent variables. An intercept variable should not be included as part
- of this expression. <b>Please note that expression should be encoded properly.</b>
- All values are cast to DOUBLE PRECISION, so categorical variables should be
- one-hot or dummy encoded. See <a href="group__grp__encode__categorical.html">here</a>
- for more details.
- </DD>
-
- <DT>dependent_varname</DT>
- <DD> TEXT. Name of the dependent variable column. For classification, supported types are:
+\b Arguments
+<dl class="arglist">
+ <dt>source_table</dt>
+ <dd>TEXT. Name of the table containing the training data.</dd>
+
+ <dt>output_table</dt>
+ <dd>TEXT. Name of the output table containing the model. Details of the output
+ table are shown below.
+ </dd>
+
+ <dt>independent_varname</dt>
+ <dd>TEXT. Expression list to evaluate for the independent variables.
+
+ @note
+ Please note that an intercept variable should not be included as part
+ of this expression - this is different from other MADlib modules. Also
+ please note that <b>independent variables should be encoded properly.</b>
+ All values are cast to DOUBLE PRECISION, so categorical variables should be
+ one-hot or dummy encoded as appropriate.
+ See <a href="group__grp__encode__categorical.html">Encoding Categorical Variables</a>
+ for more details on how to do this.
+ </dd>
+
+ <dt>dependent_varname</dt>
+ <dd> TEXT. Name of the dependent variable column. For classification, supported types are:
text, varchar, character varying, char, character
- integer, smallint, bigint, and boolean. </DD>
+ integer, smallint, bigint, and boolean. </dd>
- <DT>hidden_layer_sizes </DT>
- <DD>INTEGER[]
+ <DT>hidden_layer_sizes (optional)</DT>
+ <DD>INTEGER[], default: ARRAY[100].
The number of neurons in each hidden layer. The length of this array will
- determine the number of hidden layers. NULL for no hidden layers.
+ determine the number of hidden layers. For example, ARRAY[5,10] means 2 hidden
+ layers, one with 5 neurons and the other with 10 neurons.
+ Use ARRAY[]::INTEGER[] for no hidden layers.
</DD>
-
<DT>optimizer_params (optional)</DT>
<DD>TEXT, default: NULL.
Parameters for optimization in a comma-separated string
@@ -117,27 +122,38 @@ mlp_classification(
<DD>TEXT, default: 'sigmoid'.
Activation function. Currently three functions are supported: 'sigmoid' (default),
'relu', and 'tanh'. The text can be any prefix of the three
- strings; for e.g., activation='s' will use the sigmoid activation.
+ strings; for e.g., specifying 's' will use sigmoid activation.
</DD>
-
<DT>weights (optional)</DT>
- <DD>TEXT, default: NULL.
+ <DD>TEXT, default: 1.
Weights for input rows. Column name which specifies the weight for each input row.
- This weight will be incorporated into the update during SGD, and will not be used
- for loss calculations. If not specified, weight for each row will default to 1.
- Column should be a numeric type.
+ This weight will be incorporated into the update during stochastic gradient descent (SGD),
+ but will not be used
+ for loss calculations. If not specified, weight for each row will default to 1 (equal
+ weights). Column should be a numeric type.
</DD>
<DT>warm_start (optional)</DT>
<DD>BOOLEAN, default: FALSE.
- Initalize weights with the coefficients from the last call. If true, weights will
- be initialized from output_table. Note that all parameters other than optimizer_params,
- and verbose must remain constant between calls to warm_start.
+ Initalize weights with the coefficients from the last call of the training function.
+ If set to true, weights will be initialized from the output_table generated by the
+ previous run. Note that all parameters
+ other than optimizer_params and verbose must remain constant
+ between calls when warm_start is used.
+
+ @note
+ The warm start feature works based on the name of the output_table.
+ When using warm start, do not drop the output table or the output table summary
+ before calling the training function, since these are needed to obtain the weights
+ from the previous run.
+ If you are not using warm start, the output table and the output table
+ summary must be dropped in the usual way before calling the training function.
</DD>
<DT>verbose (optional)</DT>
- <DD>BOOLEAN, default: FALSE. Provides verbose output of the results of training.</DD>
+ <DD>BOOLEAN, default: FALSE. Provides verbose output of the results of training, including
+ the value of loss at each iteration.</DD>
<DT>grouping_col (optional)</DT>
<DD>TEXT, default: NULL.
@@ -146,19 +162,19 @@ mlp_classification(
model per group. When this value is NULL, no grouping is used and
a single model is generated for all data.
</DD>
-</DL>
+</dl>
<b>Output tables</b>
<br>
- The model table produced by mlp contains the following columns:
+ The model table produced by MLP contains the following columns:
<table class="output">
<tr>
<th>coeffs</th>
- <td>FLOAT8[]. Flat array containing the weights of the neural net</td>
+ <td>FLOAT8[]. Flat array containing the weights of the neural net.</td>
</tr>
<tr>
<th>n_iterations</th>
- <td>INTEGER. Number of iterations completed by stochastic gradient descent
+ <td>INTEGER. Number of iterations completed by the stochastic gradient descent
algorithm. The algorithm either converged in this number of iterations
or hit the maximum number specified in the optimization parameters. </td>
</tr>
@@ -169,7 +185,6 @@ mlp_classification(
</tr>
</table>
-
A summary table named \<output_table\>_summary is also created, which has the following columns:
<table class="output">
<tr>
@@ -206,7 +221,7 @@ A summary table named \<output_table\>_summary is also created, which has the fo
</tr>
<tr>
<th>layer_sizes</th>
- <td>The number of units in each layer including the input and output layer.</td>
+ <td>The number of units in each layer including the input and output layers.</td>
</tr>
<tr>
<th>activation</th>
@@ -251,12 +266,11 @@ following columns:
</tr>
</table>
-
@anchor mlp_regression
@par Regression Training Function
-The mlp regression training function has the following format:
+The MLP regression training function has the following format:
<pre class="syntax">
-mlp_regression(source_table,
+mlp_regression(
source_table,
output_table,
independent_varname,
@@ -273,35 +287,38 @@ mlp_regression(source_table,
\b Arguments
-Specifications for regression are largely the same as for classification. In the
-model table, the loss will refer to mean square error instead of cross entropy. In the
+Parameters for regression are largely the same as for classification. In the
+model table, the loss refers to mean square error instead of cross entropy. In the
summary table, there is no classes column. The following
arguments have specifications which differ from mlp_classification:
<DL class="arglist">
<DT>dependent_varname</DT>
<DD>TEXT. Name of the dependent variable column.
- For regression supported types are any numeric type, or array
- or numeric types (for multiple regression).
+ For regression, supported types are any numeric type, or array
+ of numeric types (for multiple regression).
</DD>
</DL>
-
@anchor optimizer_params
@par Optimizer Parameters
Parameters in this section are supplied in the \e optimizer_params argument as a string
containing a comma-delimited list of name-value pairs. All of these named
-parameters are optional, and their order does not matter. You must use the
+parameters are optional and their order does not matter. You must use the
format "<param_name> = <value>" to specify the value of a parameter, otherwise
the parameter is ignored.
-
<pre class="syntax">
'learning_rate_init = <value>,
+ learning_rate_policy = <value>,
+ gamma = <value>,
+ power = <value>,
+ iterations_per_step = <value>,
n_iterations = <value>,
n_tries = <value>,
+ lambda = <value>,
tolerance = <value>'
</pre>
-\b Optimizer Parameters
+\b Optimizer \b Parameters
<DL class="arglist">
<DT>learning_rate_init</dt>
@@ -314,12 +331,12 @@ practice one often tunes this parameter.
<DT>learning_rate_policy</dt>
<DD>Default: constant.
-One of 'constant', 'exp', 'inv' or 'step' or any prefix of these.
-'constant': learning_rate = learning_rate_init
-'exp': learning_rate = learning_rate_init * gamma^(iter)
-'inv': learning_rate = learning_rate_init * (iter+1)^(-power)
-'step': learning_rate = learning_rate_init * gamma^(floor(iter/iterations_per_step))
-Where iter is the current iteration of SGD.
+One of 'constant', 'exp', 'inv' or 'step' or any prefix of these (e.g., 's' means 'step').
+These are defined below, where 'iter' is the current iteration of SGD:
+ - 'constant': learning_rate = learning_rate_init
+ - 'exp': learning_rate = learning_rate_init * gamma^(iter)
+ - 'inv': learning_rate = learning_rate_init * (iter+1)^(-power)
+ - 'step': learning_rate = learning_rate_init * gamma^(floor(iter/iterations_per_step))
</DD>
<DT>gamma</dt>
@@ -339,11 +356,11 @@ a factor of gamma. Valid for learning rate policy = 'step'.
</DD>
<DT>n_iterations</dt>
-<DD>Default: [100]. The maximum number of iterations allowed.
+<DD>Default: 100. The maximum number of iterations allowed.
</DD>
<DT>n_tries</dt>
-<DD>Default: [1]. Number of times to retrain the network with randomly initialized
+<DD>Default: 1. Number of times to retrain the network with randomly initialized
weights.
</DD>
@@ -354,21 +371,25 @@ weights.
<DT>tolerance</dt>
<DD>Default: 0.001. The criterion to end iterations. The training stops whenever
the difference between the training models of two consecutive iterations is
-smaller than \e tolerance or the iteration number is larger than \e max_iter.
+smaller than \e tolerance or the iteration number is larger than \e n_iterations.
+If you want to run the full number of iterations specified in \e n_interations,
+set tolerance=0.0
</DD>
</DL>
@anchor predict
@par Prediction Function
-Used to generate predictions given a previously trained model on novel data.
-The same syntax is used for classification, and regression.
+Used to generate predictions on novel data given a previously trained model.
+The same syntax is used for classification and regression.
<pre class="syntax">
-mlp_predict(model_table,
- data_table,
- id_col_name,
- output_table,
- pred_type)
+mlp_predict(
+ model_table,
+ data_table,
+ id_col_name,
+ output_table,
+ pred_type
+ )
</pre>
\b Arguments
@@ -379,16 +400,14 @@ mlp_predict(model_table,
<DT>data_table</DT>
<DD>TEXT. Name of the table containing the data for prediction. This table is expected
to contain the same input features that were used during training. The table should
- also contain id_col_name used for identifying each row. if model_table was created
- with grouping_col parameter, the data_table must also have those corresponding columns
- in it.</DD>
+ also contain id_col_name used for identifying each row.</DD>
<DT>id_col_name</DT>
- <DD>TEXT. The name of the id column in the input table.</DD>
+ <DD>TEXT. The name of the id column in data_table.</DD>
<DT>output_table</DT>
<DD>TEXT. Name of the table where output predictions are written. If this
-table name is already in use, then an error is returned. Table contains:</DD>
+table name is already in use, an error is returned. Table contains:</DD>
<table class="output">
<tr>
<th>id</th>
@@ -415,13 +434,11 @@ table name is already in use, then an error is returned. Table contains:</DD>
<DT>pred_type</DT>
- <DD>TEXT.
-
-The type of output requested:
+ <DD>TEXT. The type of output requested:
'response' gives the actual prediction,
'prob' gives the probability of each class.
For regression, only type='response' is defined.
-The name of the id column in the input table.</DD>
+</DD>
</DL>
</table>
@@ -565,10 +582,6 @@ coeff | {-0.166258065719,0.134984689989,-0.445383330023,0.997461636443,
loss | 0.0501651250932
num_iterations | 50
</pre>
-Notice that the loss is lower compared to the previous example, despite
-having the same values for every other parameter. This is because the algorithm
-learnt three different models starting with a different set of initial weights
-for the coefficients, and chose the best model among them.
-# Next, use the warm_start parameter to start learning a new model, using
the coefficients already present in mlp_model. Note that we must not drop
@@ -602,6 +615,11 @@ coeff | {-0.0269464483507,0.1800462291,-0.550842844362,1.14444526434,0.
loss | 0.00379153493882
num_iterations | 450
</pre>
+Notice that the loss is lower compared to the previous example, despite
+having the same values for every other parameter. This is because the algorithm
+learnt three different models starting with a different set of initial weights
+for the coefficients, and chose the best model among them as the initial
+weights for the coefficients when run with warm start.
-# Next, group the training data by state, and learn a different model for each state.
<pre class="example">
@@ -813,8 +831,8 @@ SELECT * FROM mlp_regress;
Result:
<pre class="result">
-[ RECORD 1 ]--+------------------------------------------------------------------------------------
-coeff | {-0.195979856539,0.359039037803,-0.303414234056,-0.310630096593,0.378147172817 ...
-loss | 0.221305210969
+coeff | {-0.0876810606854,0.179699128308,-0.145139175157,-0.147247509843,0.163163651648, ...
+loss | 0.199137729586
num_iterations | 450
</pre>
@@ -1015,6 +1033,16 @@ Result for the classification model:
52 | Iris_versicolor | {5.8,2.6,4.0,1.2} | Iris_versicolor | 2 | Tennessee
(52 rows)
</pre>
+Count the misclassifications:
+<pre class="example">
+SELECT COUNT(*) FROM mlp_prediction JOIN iris_data USING (id)
+WHERE mlp_prediction.estimated_class_text != iris_data.class_text;
+</pre>
+<pre class="result">
+ count
+-------+
+ 0
+</pre>
-# Prediction based on grouping using the state column:
<pre class="example">
@@ -1104,30 +1132,40 @@ SELECT * FROM lin_housing JOIN mlp_regress_prediction USING (id) ORDER BY id;
</pre>
Result for the regression model:
<pre class="result">
- id | x | zipcode | y | estimated_y
-----+-------------------------------------------------------------------------+---------+------+------------------
- 1 | {1,0.00632,18,2.31,0,0.538,6.575,65.2,4.09,1,296,15.3,396.9,4.98} | 94016 | 24 | 24.0416136183081
- 2 | {1,0.02731,0,7.07,0,0.469,6.421,78.9,4.9671,2,242,17.8,396.9,9.14} | 94016 | 21.6 | 21.7168649538176
- 3 | {1,0.02729,0,7.07,0,0.469,7.185,61.1,4.9671,2,242,17.8,392.83,4.03} | 94016 | 34.7 | 34.71119208276
- 4 | {1,0.03237,0,2.18,0,0.458,6.998,45.8,6.0622,3,222,18.7,394.63,2.94} | 94016 | 33.4 | 33.6026539746505
- 5 | {1,0.06905,0,2.18,0,0.458,7.147,54.2,6.0622,3,222,18.7,396.9,5.33} | 94016 | 36.2 | 36.4103569974181
- 6 | {1,0.02985,0,2.18,0,0.458,6.43,58.7,6.0622,3,222,18.7,394.12,5.21} | 94016 | 28.7 | 28.7448645460105
- 7 | {1,0.08829,12.5,7.87,0,0.524,6.012,66.6,5.5605,5,311,15.2,395.6,12.43} | 94016 | 22.9 | 22.8753734693458
- 8 | {1,0.14455,12.5,7.87,0,0.524,6.172,96.1,5.9505,5,311,15.2,396.9,19.15} | 94016 | 27.1 | 26.1765578928806
- 9 | {1,0.21124,12.5,7.87,0,0.524,5.631,100,6.0821,5,311,15.2,386.63,29.93} | 94016 | 16.5 | 16.2770923030797
- 10 | {1,0.17004,12.5,7.87,0,0.524,6.004,85.9,6.5921,5,311,15.2,386.71,17.1} | 94016 | 18.9 | 17.4836946218525
- 11 | {1,0.22489,12.5,7.87,0,0.524,6.377,94.3,6.3467,5,311,15.2,392.52,20.45} | 94016 | 15 | 14.9358249090426
- 12 | {1,0.11747,12.5,7.87,0,0.524,6.009,82.9,6.2267,5,311,15.2,396.9,13.27} | 20001 | 18.9 | 19.6315991336163
- 13 | {1,0.09378,12.5,7.87,0,0.524,5.889,39,5.4509,5,311,15.2,390.5,15.71} | 20001 | 21.7 | 21.6625603724929
- 14 | {1,0.62976,0,8.14,0,0.538,5.949,61.8,4.7075,4,307,21,396.9,8.26} | 20001 | 20.4 | 20.6080076082523
- 15 | {1,0.63796,0,8.14,0,0.538,6.096,84.5,4.4619,4,307,21,380.02,10.26} | 20001 | 18.2 | 18.3150013322586
- 16 | {1,0.62739,0,8.14,0,0.538,5.834,56.5,4.4986,4,307,21,395.62,8.47} | 20001 | 19.9 | 19.7240708857765
- 17 | {1,1.05393,0,8.14,0,0.538,5.935,29.3,4.4986,4,307,21,386.85,6.58} | 20001 | 23.1 | 23.0663190059655
- 18 | {1,0.7842,0,8.14,0,0.538,5.99,81.7,4.2579,4,307,21,386.75,14.67} | 20001 | 17.5 | 17.4301498103015
- 19 | {1,0.80271,0,8.14,0,0.538,5.456,36.6,3.7965,4,307,21,288.99,11.69} | 20001 | 20.2 | 20.182409396145
- 20 | {1,0.7258,0,8.14,0,0.538,5.727,69.5,3.7965,4,307,21,390.95,11.28} | 20001 | 18.2 | 18.2126532108106
+ id | x | zipcode | y | estimated_y
+----+----------------------------------------------------------------------------------+---------+------+------------------
+ 1 | {1,0.00632,18.00,2.310,0,0.5380,6.5750,65.20,4.0900,1,296.0,15.30,396.90,4.98} | 94016 | 24 | 23.9989087488259
+ 2 | {1,0.02731,0.00,7.070,0,0.4690,6.4210,78.90,4.9671,2,242.0,17.80,396.90,9.14} | 94016 | 21.6 | 21.5983177932005
+ 3 | {1,0.02729,0.00,7.070,0,0.4690,7.1850,61.10,4.9671,2,242.0,17.80,392.83,4.03} | 94016 | 34.7 | 34.7102398021623
+ 4 | {1,0.03237,0.00,2.180,0,0.4580,6.9980,45.80,6.0622,3,222.0,18.70,394.63,2.94} | 94016 | 33.4 | 33.4221257351015
+ 5 | {1,0.06905,0.00,2.180,0,0.4580,7.1470,54.20,6.0622,3,222.0,18.70,396.90,5.33} | 94016 | 36.2 | 36.1523886001663
+ 6 | {1,0.02985,0.00,2.180,0,0.4580,6.4300,58.70,6.0622,3,222.0,18.70,394.12,5.21} | 94016 | 28.7 | 28.723894783928
+ 7 | {1,0.08829,12.50,7.870,0,0.5240,6.0120,66.60,5.5605,5,311.0,15.20,395.60,12.43} | 94016 | 22.9 | 22.6515242795835
+ 8 | {1,0.14455,12.50,7.870,0,0.5240,6.1720,96.10,5.9505,5,311.0,15.20,396.90,19.15} | 94016 | 27.1 | 25.7615314879354
+ 9 | {1,0.21124,12.50,7.870,0,0.5240,5.6310,100.00,6.0821,5,311.0,15.20,386.63,29.93} | 94016 | 16.5 | 15.7368298351732
+ 10 | {1,0.17004,12.50,7.870,0,0.5240,6.0040,85.90,6.5921,5,311.0,15.20,386.71,17.10} | 94016 | 18.9 | 16.8850496141437
+ 11 | {1,0.22489,12.50,7.870,0,0.5240,6.3770,94.30,6.3467,5,311.0,15.20,392.52,20.45} | 94016 | 15 | 14.9150416339458
+ 12 | {1,0.11747,12.50,7.870,0,0.5240,6.0090,82.90,6.2267,5,311.0,15.20,396.90,13.27} | 20001 | 18.9 | 19.4541629864106
+ 13 | {1,0.09378,12.50,7.870,0,0.5240,5.8890,39.00,5.4509,5,311.0,15.20,390.50,15.71} | 20001 | 21.7 | 21.715554997762
+ 14 | {1,0.62976,0.00,8.140,0,0.5380,5.9490,61.80,4.7075,4,307.0,21.00,396.90,8.26} | 20001 | 20.4 | 20.3181247234996
+ 15 | {1,0.63796,0.00,8.140,0,0.5380,6.0960,84.50,4.4619,4,307.0,21.00,380.02,10.26} | 20001 | 18.2 | 18.5026399122209
+ 16 | {1,0.62739,0.00,8.140,0,0.5380,5.8340,56.50,4.4986,4,307.0,21.00,395.62,8.47} | 20001 | 19.9 | 19.9131696333521
+ 17 | {1,1.05393,0.00,8.140,0,0.5380,5.9350,29.30,4.4986,4,307.0,21.00,386.85,6.58} | 20001 | 23.1 | 23.1757650468106
+ 18 | {1,0.78420,0.00,8.140,0,0.5380,5.9900,81.70,4.2579,4,307.0,21.00,386.75,14.67} | 20001 | 17.5 | 17.2671872543377
+ 19 | {1,0.80271,0.00,8.140,0,0.5380,5.4560,36.60,3.7965,4,307.0,21.00,288.99,11.69} | 20001 | 20.2 | 20.1073474558796
+ 20 | {1,0.72580,0.00,8.140,0,0.5380,5.7270,69.50,3.7965,4,307.0,21.00,390.95,11.28} | 20001 | 18.2 | 18.2143446340975
(20 rows)
</pre>
+RMS error:
+<pre class="example">
+SELECT SQRT(SUM(ABS(y-estimated_y))/COUNT(y)) as rms_error FROM lin_housing
+JOIN mlp_regress_prediction USING (id);
+</pre>
+<pre class="result">
+ rms_error
+------------------+
+ 0.544960829104004
+</pre>
-# Prediction using the regression model for each group based on the zipcode:
<pre class="example">
@@ -1175,16 +1213,16 @@ Note that the results you get for all examples may vary with the platform you ar
@anchor background
@par Technical Background
-To train a neural net, the respective loss function is minimized using stochastic gradient descent.
+To train a neural net, the loss function is minimized using stochastic gradient descent.
In the case of classification, the loss function is cross entropy. For regression, mean square error
is used. Weights in the neural net are updated via the backpropogation process, which uses dynamic
programming to compute the partial derivative of each weight with respect to the overall loss. This
-partial derivative incorporates the respective activation function used, so this requires that the
+partial derivative incorporates the activation function used, which requires that the
activation function be differentiable.
-For an overview of multilayer perceptrons, see website [1].
+For an overview of multilayer perceptrons, see [1].
-For details on backpropogation, see the notes at [2].
+For details on backpropogation, see [2].
@anchor literature
@literature