You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@madlib.apache.org by cooper-sloan <gi...@git.apache.org> on 2017/07/11 16:31:30 UTC
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
GitHub user cooper-sloan opened a pull request:
https://github.com/apache/incubator-madlib/pull/149
MLP: Multilayer Perceptron
JIRA: MADLIB-413
Add train and predict for multilayer perceptron.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cooper-sloan/incubator-madlib mlp_phase1
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-madlib/pull/149.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #149
----
commit 3693c70178ea74fb3cb742715c4091ddcc265bdc
Author: Cooper Sloan <co...@gmail.com>
Date: 2017-06-17T00:41:07Z
MLP: Multilayer Perceptron
JIRA: MADLIB-413
Add train and predict for multilayer perceptron.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by iyerr3 <gi...@git.apache.org>.
Github user iyerr3 commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r128583687
--- Diff: doc/mainpage.dox.in ---
@@ -195,6 +195,9 @@ complete matrix stored as a distributed table.
@defgroup grp_robust Robust Variance
@}
+ @defgroup grp_mlp Multilayer Perceptron
--- End diff --
This needs to go above `Regression ...` to keep it ordered alphabetically.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by edespino <gi...@git.apache.org>.
Github user edespino commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r127083385
--- Diff: src/ports/postgres/modules/convex/mlp_igd.py_in ---
@@ -0,0 +1,754 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+@file mlp_igd.py_in
+
+@brief Multilayer perceptron using IGD: Driver functions
+
+@namespace mlp_igd
+"""
+import plpy
+
+from utilities.control import MinWarning
+from utilities.utilities import add_postfix
+from utilities.utilities import py_list_to_sql_string
+from utilities.utilities import extract_keyvalue_params
+from utilities.utilities import _assert
+from utilities.utilities import unique_string
+from utilities.utilities import strip_end_quotes
+
+from utilities.validate_args import cols_in_tbl_valid
+from utilities.validate_args import input_tbl_valid
+from utilities.validate_args import is_var_valid
+from utilities.validate_args import output_tbl_valid
+from utilities.validate_args import get_expr_type
+from utilities.validate_args import array_col_has_same_dimension
+from utilities.validate_args import array_col_dimension
+
+
+def mlp(schema_madlib, source_table, output_table, independent_varname,
+ dependent_varname, hidden_layer_sizes,
+ optimizer_param_str, activation, is_classification, **kwargs):
+ """
+ Args:
+ @param schema_madlib
+ @param source_table
+ @param output_table
+ @param independent_varname
+ @param dependent_varname
+ @param hidden_layer_sizes
+ @param optimizer_param_str
+
+ Returns:
+ None
+ """
+ with MinWarning('info'):
+ optimizer_params = _get_optimizer_params(optimizer_param_str or "")
+ _validate_args(source_table, output_table, independent_varname,
+ dependent_varname, hidden_layer_sizes,
+ optimizer_params, is_classification)
+
+ current_iteration = 1
+ prev_state = None
+ tolerance = optimizer_params["tolerance"]
+ n_iterations = optimizer_params["n_iterations"]
+ step_size = optimizer_params["step_size"]
+ n_tries = optimizer_params["n_tries"]
+ activation_name = _get_activation_function_name(activation)
+ activation_index = _get_activation_index(activation_name)
+ num_input_nodes = array_col_dimension(source_table, independent_varname)
+ num_output_nodes = 0
+ classes = []
+ dependent_type = get_expr_type(dependent_varname, source_table)
+ original_dependent_varname = dependent_varname
+
+ if(is_classification):
+ dependent_variable_sql = """
+ SELECT DISTINCT {dependent_varname}
+ FROM {source_table}
+ """.format(dependent_varname=dependent_varname,
+ source_table=source_table)
+ labels = plpy.execute(dependent_variable_sql)
+ one_hot_dependent_varname = 'ARRAY['
+ num_output_nodes = len(labels)
+ for x in labels:
+ label = _format_label(x[dependent_varname])
+ classes.append(label)
+ one_hot_dependent_varname += dependent_varname + "=" + str(label) + ","
+ # Remove the last comma
+ one_hot_dependent_varname = one_hot_dependent_varname[:-1]
+ one_hot_dependent_varname += ']::integer[]'
+ dependent_varname = one_hot_dependent_varname
+ if(not is_classification):
+ if "[]" not in dependent_type:
+ dependent_varname = "ARRAY[" + dependent_varname + "]"
+ num_output_nodes = array_col_dimension(source_table, dependent_varname)
+ layer_sizes = [num_input_nodes] + hidden_layer_sizes + [num_output_nodes]
+
+ while True:
+ if prev_state:
+ prev_state_str = py_list_to_sql_string(prev_state, array_type="double precision")
+ else:
+ prev_state_str = "(NULL)::DOUBLE PRECISION[]"
+ train_sql = """
+ SELECT
+ {schema_madlib}.mlp_igd_step(
+ ({independent_varname})::DOUBLE PRECISION[],
+ ({dependent_varname})::DOUBLE PRECISION[],
+ {prev_state},
+ {layer_sizes},
+ ({step_size})::FLOAT8,
+ {activation},
+ {is_classification}) as curr_state
+ FROM {source_table} AS _src
+ """.format(schema_madlib=schema_madlib,
+ independent_varname=independent_varname,
+ dependent_varname=dependent_varname,
+ prev_state=prev_state_str,
+ # C++ uses double internally
+ layer_sizes=py_list_to_sql_string(layer_sizes,
+ array_type="double precision"),
+ step_size=step_size,
+ source_table=source_table,
+ activation=activation_index,
+ is_classification=int(is_classification))
+ curr_state = plpy.execute(train_sql)[0]["curr_state"]
+ dist_sql = """
+ SELECT {schema_madlib}.internal_mlp_igd_distance(
+ {prev_state},
+ {curr_state}) as state_dist
+ """.format(schema_madlib=schema_madlib,
+ prev_state=prev_state_str,
+ curr_state=py_list_to_sql_string(curr_state, "double precision"),
+ tolerance=tolerance)
+ state_dist = plpy.execute(dist_sql)[0]["state_dist"]
+ if ((state_dist and state_dist < tolerance) or
+ current_iteration > n_iterations):
+ break
+ prev_state = curr_state
+ # plpy.info("Iteration: "+str(current_iteration)+"/"+str(n_iterations))
+ current_iteration += 1
+ _build_model_table(schema_madlib, output_table, curr_state, n_iterations)
+ layer_sizes_str=py_list_to_sql_string(layer_sizes, array_type="integer")
+ classes_str=py_list_to_sql_string([strip_end_quotes(cl, "'") for cl in classes],
+ array_type=dependent_type)
+ # TODO validation for summary
+ summary_table_creation_query = """
+ CREATE TABLE {output_table}_summary(
+ source_table TEXT,
+ independent_varname TEXT,
+ dependent_varname TEXT,
+ tolerance FLOAT,
+ step_size FLOAT,
+ n_iterations INTEGER,
+ n_tries INTEGER,
+ layer_sizes INTEGER[],
+ activation_function TEXT,
+ is_classification BOOLEAN,
+ classes {dependent_type}[]
+ )""".format(output_table=output_table,
+ dependent_type=dependent_type)
+
+ summary_table_update_query = """
+ INSERT INTO {output_table}_summary VALUES(
+ '{source_table}',
+ '{independent_varname}',
+ '{original_dependent_varname}',
+ {tolerance},
+ {step_size},
+ {n_iterations},
+ {n_tries},
+ {layer_sizes_str},
+ '{activation_name}',
+ {is_classification},
+ {classes_str}
+ )
+ """.format(**locals())
+ plpy.execute(summary_table_creation_query)
+ plpy.execute(summary_table_update_query)
+# ----------------------------------------------------------------------
+
+
+def _build_model_table(schema_madlib, output_table, final_state, n_iterations):
+ final_state_str = py_list_to_sql_string(final_state, array_type="double precision")
+
+ model_table_query = """
+ CREATE TABLE {output_table} AS
+ SELECT
+ (result).coeff AS coeff,
+ (result).loss AS loss,
+ {n_iterations} AS num_iterations
+ -- (result).num_rows_processed AS num_rows_processed,
+ -- n_tuples_including_nulls - (result).num_rows_processed
+ FROM (
+ SELECT
+ {schema_madlib}.internal_mlp_igd_result(
+ {final_state_str}
+ ) AS result
+ ) rel_state_subq
+ """.format(**locals())
+ plpy.execute(model_table_query)
+# ----------------------------------------------------------------------
+
+
+def _get_optimizer_params(param_str):
+ params_defaults = {
+ "step_size": (0.001, float),
+ "n_iterations": (100, int),
+ "n_tries": (1, int),
+ "tolerance": (0.001, float),
+ }
+ param_defaults = dict([(k, v[0]) for k, v in params_defaults.items()])
+ param_types = dict([(k, v[1]) for k, v in params_defaults.items()])
+
+ if not param_str:
+ return param_defaults
+
+ name_value = extract_keyvalue_params(param_str, param_types, param_defaults,
+ ignore_invalid=True)
--- End diff --
Is there a specific reason why invalid optimizer parameters are ignored (ignore_invalid=True)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by edespino <gi...@git.apache.org>.
Github user edespino commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r127324219
--- Diff: src/ports/postgres/modules/convex/test/mlp.sql_in ---
@@ -0,0 +1,832 @@
+/* ----------------------------------------------------------------------- *//**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ *//* ----------------------------------------------------------------------- */
+
+
+
+
+-- The data were collected by Anderson, Edgar (1935).
+-- The irises of the Gaspe Peninsula, Bulletin of the American Iris Society, 59, 2–5.
+
+-- Classification
+
+
+SELECT setseed(0.5);
+DROP TABLE IF EXISTS iris_data, iris_test, mlp_class, mlp_class_summary CASCADE;
+CREATE TABLE iris_data(
+ id integer,
+ attributes numeric[],
+ class_text varchar,
+ class integer
+);
+
+INSERT INTO iris_data VALUES
+(1,ARRAY[5.1,3.5,1.4,0.2],'Iris-setosa',1),
+(2,ARRAY[4.9,3.0,1.4,0.2],'Iris-setosa',1),
+(3,ARRAY[4.7,3.2,1.3,0.2],'Iris-setosa',1),
+(4,ARRAY[4.6,3.1,1.5,0.2],'Iris-setosa',1),
+(5,ARRAY[5.0,3.6,1.4,0.2],'Iris-setosa',1),
+(6,ARRAY[5.4,3.9,1.7,0.4],'Iris-setosa',1),
+(7,ARRAY[4.6,3.4,1.4,0.3],'Iris-setosa',1),
+(8,ARRAY[5.0,3.4,1.5,0.2],'Iris-setosa',1),
+(9,ARRAY[4.4,2.9,1.4,0.2],'Iris-setosa',1),
+(10,ARRAY[4.9,3.1,1.5,0.1],'Iris-setosa',1),
+(11,ARRAY[5.4,3.7,1.5,0.2],'Iris-setosa',1),
+(12,ARRAY[4.8,3.4,1.6,0.2],'Iris-setosa',1),
+(13,ARRAY[4.8,3.0,1.4,0.1],'Iris-setosa',1),
+(14,ARRAY[4.3,3.0,1.1,0.1],'Iris-setosa',1),
+(15,ARRAY[5.8,4.0,1.2,0.2],'Iris-setosa',1),
+(16,ARRAY[5.7,4.4,1.5,0.4],'Iris-setosa',1),
+(17,ARRAY[5.4,3.9,1.3,0.4],'Iris-setosa',1),
+(18,ARRAY[5.1,3.5,1.4,0.3],'Iris-setosa',1),
+(19,ARRAY[5.7,3.8,1.7,0.3],'Iris-setosa',1),
+(20,ARRAY[5.1,3.8,1.5,0.3],'Iris-setosa',1),
+(21,ARRAY[5.4,3.4,1.7,0.2],'Iris-setosa',1),
+(22,ARRAY[5.1,3.7,1.5,0.4],'Iris-setosa',1),
+(23,ARRAY[4.6,3.6,1.0,0.2],'Iris-setosa',1),
+(24,ARRAY[5.1,3.3,1.7,0.5],'Iris-setosa',1),
+(25,ARRAY[4.8,3.4,1.9,0.2],'Iris-setosa',1),
+(26,ARRAY[5.0,3.0,1.6,0.2],'Iris-setosa',1),
+(27,ARRAY[5.0,3.4,1.6,0.4],'Iris-setosa',1),
+(28,ARRAY[5.2,3.5,1.5,0.2],'Iris-setosa',1),
+(29,ARRAY[5.2,3.4,1.4,0.2],'Iris-setosa',1),
+(30,ARRAY[4.7,3.2,1.6,0.2],'Iris-setosa',1),
+(31,ARRAY[4.8,3.1,1.6,0.2],'Iris-setosa',1),
+(32,ARRAY[5.4,3.4,1.5,0.4],'Iris-setosa',1),
+(33,ARRAY[5.2,4.1,1.5,0.1],'Iris-setosa',1),
+(34,ARRAY[5.5,4.2,1.4,0.2],'Iris-setosa',1),
+(35,ARRAY[4.9,3.1,1.5,0.1],'Iris-setosa',1),
+(36,ARRAY[5.0,3.2,1.2,0.2],'Iris-setosa',1),
+(37,ARRAY[5.5,3.5,1.3,0.2],'Iris-setosa',1),
+(38,ARRAY[4.9,3.1,1.5,0.1],'Iris-setosa',1),
+(39,ARRAY[4.4,3.0,1.3,0.2],'Iris-setosa',1),
+(40,ARRAY[5.1,3.4,1.5,0.2],'Iris-setosa',1),
+(41,ARRAY[5.0,3.5,1.3,0.3],'Iris-setosa',1),
+(42,ARRAY[4.5,2.3,1.3,0.3],'Iris-setosa',1),
+(43,ARRAY[4.4,3.2,1.3,0.2],'Iris-setosa',1),
+(44,ARRAY[5.0,3.5,1.6,0.6],'Iris-setosa',1),
+(45,ARRAY[5.1,3.8,1.9,0.4],'Iris-setosa',1),
+(46,ARRAY[4.8,3.0,1.4,0.3],'Iris-setosa',1),
+(47,ARRAY[5.1,3.8,1.6,0.2],'Iris-setosa',1),
+(48,ARRAY[4.6,3.2,1.4,0.2],'Iris-setosa',1),
+(49,ARRAY[5.3,3.7,1.5,0.2],'Iris-setosa',1),
+(50,ARRAY[5.0,3.3,1.4,0.2],'Iris-setosa',1),
+(51,ARRAY[7.0,3.2,4.7,1.4],'Iris-versicolor',2),
+(52,ARRAY[6.4,3.2,4.5,1.5],'Iris-versicolor',2),
+(53,ARRAY[6.9,3.1,4.9,1.5],'Iris-versicolor',2),
+(54,ARRAY[5.5,2.3,4.0,1.3],'Iris-versicolor',2),
+(55,ARRAY[6.5,2.8,4.6,1.5],'Iris-versicolor',2),
+(56,ARRAY[5.7,2.8,4.5,1.3],'Iris-versicolor',2),
+(57,ARRAY[6.3,3.3,4.7,1.6],'Iris-versicolor',2),
+(58,ARRAY[4.9,2.4,3.3,1.0],'Iris-versicolor',2),
+(59,ARRAY[6.6,2.9,4.6,1.3],'Iris-versicolor',2),
+(60,ARRAY[5.2,2.7,3.9,1.4],'Iris-versicolor',2),
+(61,ARRAY[5.0,2.0,3.5,1.0],'Iris-versicolor',2),
+(62,ARRAY[5.9,3.0,4.2,1.5],'Iris-versicolor',2),
+(63,ARRAY[6.0,2.2,4.0,1.0],'Iris-versicolor',2),
+(64,ARRAY[6.1,2.9,4.7,1.4],'Iris-versicolor',2),
+(65,ARRAY[5.6,2.9,3.6,1.3],'Iris-versicolor',2),
+(66,ARRAY[6.7,3.1,4.4,1.4],'Iris-versicolor',2),
+(67,ARRAY[5.6,3.0,4.5,1.5],'Iris-versicolor',2),
+(68,ARRAY[5.8,2.7,4.1,1.0],'Iris-versicolor',2),
+(69,ARRAY[6.2,2.2,4.5,1.5],'Iris-versicolor',2),
+(70,ARRAY[5.6,2.5,3.9,1.1],'Iris-versicolor',2),
+(71,ARRAY[5.9,3.2,4.8,1.8],'Iris-versicolor',2),
+(72,ARRAY[6.1,2.8,4.0,1.3],'Iris-versicolor',2),
+(73,ARRAY[6.3,2.5,4.9,1.5],'Iris-versicolor',2),
+(74,ARRAY[6.1,2.8,4.7,1.2],'Iris-versicolor',2),
+(75,ARRAY[6.4,2.9,4.3,1.3],'Iris-versicolor',2),
+(76,ARRAY[6.6,3.0,4.4,1.4],'Iris-versicolor',2),
+(77,ARRAY[6.8,2.8,4.8,1.4],'Iris-versicolor',2),
+(78,ARRAY[6.7,3.0,5.0,1.7],'Iris-versicolor',2),
+(79,ARRAY[6.0,2.9,4.5,1.5],'Iris-versicolor',2),
+(80,ARRAY[5.7,2.6,3.5,1.0],'Iris-versicolor',2),
+(81,ARRAY[5.5,2.4,3.8,1.1],'Iris-versicolor',2),
+(82,ARRAY[5.5,2.4,3.7,1.0],'Iris-versicolor',2),
+(83,ARRAY[5.8,2.7,3.9,1.2],'Iris-versicolor',2),
+(84,ARRAY[6.0,2.7,5.1,1.6],'Iris-versicolor',2),
+(85,ARRAY[5.4,3.0,4.5,1.5],'Iris-versicolor',2),
+(86,ARRAY[6.0,3.4,4.5,1.6],'Iris-versicolor',2),
+(87,ARRAY[6.7,3.1,4.7,1.5],'Iris-versicolor',2),
+(88,ARRAY[6.3,2.3,4.4,1.3],'Iris-versicolor',2),
+(89,ARRAY[5.6,3.0,4.1,1.3],'Iris-versicolor',2),
+(90,ARRAY[5.5,2.5,4.0,1.3],'Iris-versicolor',2),
+(91,ARRAY[5.5,2.6,4.4,1.2],'Iris-versicolor',2),
+(92,ARRAY[6.1,3.0,4.6,1.4],'Iris-versicolor',2),
+(93,ARRAY[5.8,2.6,4.0,1.2],'Iris-versicolor',2),
+(94,ARRAY[5.0,2.3,3.3,1.0],'Iris-versicolor',2),
+(95,ARRAY[5.6,2.7,4.2,1.3],'Iris-versicolor',2),
+(96,ARRAY[5.7,3.0,4.2,1.2],'Iris-versicolor',2),
+(97,ARRAY[5.7,2.9,4.2,1.3],'Iris-versicolor',2),
+(98,ARRAY[6.2,2.9,4.3,1.3],'Iris-versicolor',2),
+(99,ARRAY[5.1,2.5,3.0,1.1],'Iris-versicolor',2),
+(100,ARRAY[5.7,2.8,4.1,1.3],'Iris-versicolor',2),
+(101,ARRAY[6.3,3.3,6.0,2.5],'Iris-virginica',3),
+(102,ARRAY[5.8,2.7,5.1,1.9],'Iris-virginica',3),
+(103,ARRAY[7.1,3.0,5.9,2.1],'Iris-virginica',3),
+(104,ARRAY[6.3,2.9,5.6,1.8],'Iris-virginica',3),
+(105,ARRAY[6.5,3.0,5.8,2.2],'Iris-virginica',3),
+(106,ARRAY[7.6,3.0,6.6,2.1],'Iris-virginica',3),
+(107,ARRAY[4.9,2.5,4.5,1.7],'Iris-virginica',3),
+(108,ARRAY[7.3,2.9,6.3,1.8],'Iris-virginica',3),
+(109,ARRAY[6.7,2.5,5.8,1.8],'Iris-virginica',3),
+(110,ARRAY[7.2,3.6,6.1,2.5],'Iris-virginica',3),
+(111,ARRAY[6.5,3.2,5.1,2.0],'Iris-virginica',3),
+(112,ARRAY[6.4,2.7,5.3,1.9],'Iris-virginica',3),
+(113,ARRAY[6.8,3.0,5.5,2.1],'Iris-virginica',3),
+(114,ARRAY[5.7,2.5,5.0,2.0],'Iris-virginica',3),
+(115,ARRAY[5.8,2.8,5.1,2.4],'Iris-virginica',3),
+(116,ARRAY[6.4,3.2,5.3,2.3],'Iris-virginica',3),
+(117,ARRAY[6.5,3.0,5.5,1.8],'Iris-virginica',3),
+(118,ARRAY[7.7,3.8,6.7,2.2],'Iris-virginica',3),
+(119,ARRAY[7.7,2.6,6.9,2.3],'Iris-virginica',3),
+(120,ARRAY[6.0,2.2,5.0,1.5],'Iris-virginica',3),
+(121,ARRAY[6.9,3.2,5.7,2.3],'Iris-virginica',3),
+(122,ARRAY[5.6,2.8,4.9,2.0],'Iris-virginica',3),
+(123,ARRAY[7.7,2.8,6.7,2.0],'Iris-virginica',3),
+(124,ARRAY[6.3,2.7,4.9,1.8],'Iris-virginica',3),
+(125,ARRAY[6.7,3.3,5.7,2.1],'Iris-virginica',3),
+(126,ARRAY[7.2,3.2,6.0,1.8],'Iris-virginica',3),
+(127,ARRAY[6.2,2.8,4.8,1.8],'Iris-virginica',3),
+(128,ARRAY[6.1,3.0,4.9,1.8],'Iris-virginica',3),
+(129,ARRAY[6.4,2.8,5.6,2.1],'Iris-virginica',3),
+(130,ARRAY[7.2,3.0,5.8,1.6],'Iris-virginica',3),
+(131,ARRAY[7.4,2.8,6.1,1.9],'Iris-virginica',3),
+(132,ARRAY[7.9,3.8,6.4,2.0],'Iris-virginica',3),
+(133,ARRAY[6.4,2.8,5.6,2.2],'Iris-virginica',3),
+(134,ARRAY[6.3,2.8,5.1,1.5],'Iris-virginica',3),
+(135,ARRAY[6.1,2.6,5.6,1.4],'Iris-virginica',3),
+(136,ARRAY[7.7,3.0,6.1,2.3],'Iris-virginica',3),
+(137,ARRAY[6.3,3.4,5.6,2.4],'Iris-virginica',3),
+(138,ARRAY[6.4,3.1,5.5,1.8],'Iris-virginica',3),
+(139,ARRAY[6.0,3.0,4.8,1.8],'Iris-virginica',3),
+(140,ARRAY[6.9,3.1,5.4,2.1],'Iris-virginica',3),
+(141,ARRAY[6.7,3.1,5.6,2.4],'Iris-virginica',3),
+(142,ARRAY[6.9,3.1,5.1,2.3],'Iris-virginica',3),
+(143,ARRAY[5.8,2.7,5.1,1.9],'Iris-virginica',3),
+(144,ARRAY[6.8,3.2,5.9,2.3],'Iris-virginica',3),
+(145,ARRAY[6.7,3.3,5.7,2.5],'Iris-virginica',3),
+(146,ARRAY[6.7,3.0,5.2,2.3],'Iris-virginica',3),
+(147,ARRAY[6.3,2.5,5.0,1.9],'Iris-virginica',3),
+(148,ARRAY[6.5,3.0,5.2,2.0],'Iris-virginica',3),
+(149,ARRAY[6.2,3.4,5.4,2.3],'Iris-virginica',3),
+(150,ARRAY[5.9,3.0,5.1,1.8],'Iris-virginica',3);
+
+
+SELECT mlp_classification(
+ 'iris_data', -- Source table
+ 'mlp_class', -- Desination table
+ 'attributes', -- Input features
+ 'class', -- Label
+ ARRAY[5], -- Number of units per layer
+ 'step_size=0.001,
+ n_iterations=1000,
+ tolerance=0',
+ 'tanh');
+
+
+SELECT assert(
+ -- Loss will improve much more if more iterations are run
+ loss < 30,
+ 'MLP: Loss is too high (> 30). Wrong result.'
+) FROM mlp_class;
+
+DROP TABLE IF EXISTS mlp_prediction;
+DROP TABLE IF EXISTS mlp_prediction_output;
+-- See prediction accuracy for training data
+SELECT mlp_predict(
+ 'mlp_class',
+ 'iris_data',
+ 'id',
+ 'mlp_prediction_output',
+ 'output');
+
+-- Sum over the probabilities where the prediction is correct
+-- If the predictions were perfect, the score would be 150
+SELECT assert(
+ score > 140,
+ 'MLP: Score is too low (< 140). Wrong result.'
+) FROM(
+SELECT SUM(
+mlp_prediction_output.estimated_prob_1*(iris_data.class=1)::INT+
+mlp_prediction_output.estimated_prob_2*(iris_data.class=2)::INT+
+mlp_prediction_output.estimated_prob_3*(iris_data.class=3)::INT
+) AS score
+FROM mlp_prediction_output INNER JOIN iris_data ON iris_data.id=mlp_prediction_output.id)q;
+
+
+SELECT mlp_predict(
+ 'mlp_class',
+ 'iris_data',
+ 'id',
+ 'mlp_prediction',
+ 'response');
+
+select * from mlp_prediction;
+SELECT assert(
+ -- Accuracy greater than 90%
+ COUNT(*)/150.0 > 0.95,
+ 'MLP: Accuracy is too low (< 95%). Wrong result.'
+) FROM
+ (SELECT iris_data.class AS actual, mlp_prediction.estimated_class as estimated
+ FROM mlp_prediction INNER JOIN iris_data ON
+ iris_data.id=mlp_prediction.id) q
+WHERE q.actual=q.estimated;
+
+DROP TABLE IF EXISTS mlp_prediction;
+DROP TABLE IF EXISTS mlp_prediction_output;
+-- Regression
+
+
+DROP TABLE IF EXISTS lin_housing_wi, mlp_regress CASCADE;
+CREATE TABLE lin_housing_wi (id serial, x float8[], grp_by_col int, y float8);
+COPY lin_housing_wi (x, grp_by_col, y) FROM STDIN NULL '?' DELIMITER '|';
+{1,0.00632,18.00,2.310,0,0.5380,6.5750,65.20,4.0900,1,296.0,15.30,396.90,4.98} | 1 | 24.00
+{1,0.02731,0.00,7.070,0,0.4690,6.4210,78.90,4.9671,2,242.0,17.80,396.90,9.14} | 1 | 21.60
+{1,0.02729,0.00,7.070,0,0.4690,7.1850,61.10,4.9671,2,242.0,17.80,392.83,4.03} | 1 | 34.70
+{1,0.03237,0.00,2.180,0,0.4580,6.9980,45.80,6.0622,3,222.0,18.70,394.63,2.94} | 1 | 33.40
+{1,0.06905,0.00,2.180,0,0.4580,7.1470,54.20,6.0622,3,222.0,18.70,396.90,5.33} | 1 | 36.20
+{1,0.02985,0.00,2.180,0,0.4580,6.4300,58.70,6.0622,3,222.0,18.70,394.12,5.21} | 1 | 28.70
+{1,0.08829,12.50,7.870,0,0.5240,6.0120,66.60,5.5605,5,311.0,15.20,395.60,12.43} | 1 | 22.90
+{1,0.14455,12.50,7.870,0,0.5240,6.1720,96.10,5.9505,5,311.0,15.20,396.90,19.15} | 1 | 27.10
+{1,0.21124,12.50,7.870,0,0.5240,5.6310,100.00,6.0821,5,311.0,15.20,386.63,29.93} | 1 | 16.50
+{1,0.17004,12.50,7.870,0,0.5240,6.0040,85.90,6.5921,5,311.0,15.20,386.71,17.10} | 1 | 18.90
+{1,0.22489,12.50,7.870,0,0.5240,6.3770,94.30,6.3467,5,311.0,15.20,392.52,20.45} | 1 | 15.00
+{1,0.11747,12.50,7.870,0,0.5240,6.0090,82.90,6.2267,5,311.0,15.20,396.90,13.27} | 1 | 18.90
+{1,0.09378,12.50,7.870,0,0.5240,5.8890,39.00,5.4509,5,311.0,15.20,390.50,15.71} | 1 | 21.70
+{1,0.62976,0.00,8.140,0,0.5380,5.9490,61.80,4.7075,4,307.0,21.00,396.90,8.26} | 1 | 20.40
+{1,0.63796,0.00,8.140,0,0.5380,6.0960,84.50,4.4619,4,307.0,21.00,380.02,10.26} | 1 | 18.20
+{1,0.62739,0.00,8.140,0,0.5380,5.8340,56.50,4.4986,4,307.0,21.00,395.62,8.47} | 1 | 19.90
+{1,1.05393,0.00,8.140,0,0.5380,5.9350,29.30,4.4986,4,307.0,21.00,386.85,6.58} | 1 | 23.10
+{1,0.78420,0.00,8.140,0,0.5380,5.9900,81.70,4.2579,4,307.0,21.00,386.75,14.67} | 1 | 17.50
+{1,0.80271,0.00,8.140,0,0.5380,5.4560,36.60,3.7965,4,307.0,21.00,288.99,11.69} | 1 | 20.20
+{1,0.72580,0.00,8.140,0,0.5380,5.7270,69.50,3.7965,4,307.0,21.00,390.95,11.28} | 1 | 18.20
+{1,1.25179,0.00,8.140,0,0.5380,5.5700,98.10,3.7979,4,307.0,21.00,376.57,21.02} | 1 | 13.60
+{1,0.85204,0.00,8.140,0,0.5380,5.9650,89.20,4.0123,4,307.0,21.00,392.53,13.83} | 1 | 19.60
+{1,1.23247,0.00,8.140,0,0.5380,6.1420,91.70,3.9769,4,307.0,21.00,396.90,18.72} | 1 | 15.20
+{1,0.98843,0.00,8.140,0,0.5380,5.8130,100.00,4.0952,4,307.0,21.00,394.54,19.88} | 1 | 14.50
+{1,0.75026,0.00,8.140,0,0.5380,5.9240,94.10,4.3996,4,307.0,21.00,394.33,16.30} | 1 | 15.60
+{1,0.84054,0.00,8.140,0,0.5380,5.5990,85.70,4.4546,4,307.0,21.00,303.42,16.51} | 1 | 13.90
+{1,0.67191,0.00,8.140,0,0.5380,5.8130,90.30,4.6820,4,307.0,21.00,376.88,14.81} | 1 | 16.60
+{1,0.95577,0.00,8.140,0,0.5380,6.0470,88.80,4.4534,4,307.0,21.00,306.38,17.28} | 1 | 14.80
+{1,0.77299,0.00,8.140,0,0.5380,6.4950,94.40,4.4547,4,307.0,21.00,387.94,12.80} | 1 | 18.40
+{1,1.00245,0.00,8.140,0,0.5380,6.6740,87.30,4.2390,4,307.0,21.00,380.23,11.98} | 1 | 21.00
+{1,1.13081,0.00,8.140,0,0.5380,5.7130,94.10,4.2330,4,307.0,21.00,360.17,22.60} | 1 | 12.70
+{1,1.35472,0.00,8.140,0,0.5380,6.0720,100.00,4.1750,4,307.0,21.00,376.73,13.04} | 1 | 14.50
+{1,1.38799,0.00,8.140,0,0.5380,5.9500,82.00,3.9900,4,307.0,21.00,232.60,27.71} | 1 | 13.20
+{1,1.15172,0.00,8.140,0,0.5380,5.7010,95.00,3.7872,4,307.0,21.00,358.77,18.35} | 1 | 13.10
+{1,1.61282,0.00,8.140,0,0.5380,6.0960,96.90,3.7598,4,307.0,21.00,248.31,20.34} | 1 | 13.50
+{1,0.06417,0.00,5.960,0,0.4990,5.9330,68.20,3.3603,5,279.0,19.20,396.90,9.68} | 1 | 18.90
+{1,0.09744,0.00,5.960,0,0.4990,5.8410,61.40,3.3779,5,279.0,19.20,377.56,11.41} | 1 | 20.00
+{1,0.08014,0.00,5.960,0,0.4990,5.8500,41.50,3.9342,5,279.0,19.20,396.90,8.77} | 1 | 21.00
+{1,0.17505,0.00,5.960,0,0.4990,5.9660,30.20,3.8473,5,279.0,19.20,393.43,10.13} | 1 | 24.70
+{1,0.02763,75.00,2.950,0,0.4280,6.5950,21.80,5.4011,3,252.0,18.30,395.63,4.32} | 1 | 30.80
+{1,0.03359,75.00,2.950,0,0.4280,7.0240,15.80,5.4011,3,252.0,18.30,395.62,1.98} | 1 | 34.90
+{1,0.12744,0.00,6.910,0,0.4480,6.7700,2.90,5.7209,3,233.0,17.90,385.41,4.84} | 1 | 26.60
+{1,0.14150,0.00,6.910,0,0.4480,6.1690,6.60,5.7209,3,233.0,17.90,383.37,5.81} | 1 | 25.30
+{1,0.15936,0.00,6.910,0,0.4480,6.2110,6.50,5.7209,3,233.0,17.90,394.46,7.44} | 1 | 24.70
+{1,0.12269,0.00,6.910,0,0.4480,6.0690,40.00,5.7209,3,233.0,17.90,389.39,9.55} | 1 | 21.20
+{1,0.17142,0.00,6.910,0,0.4480,5.6820,33.80,5.1004,3,233.0,17.90,396.90,10.21} | 1 | 19.30
+{1,0.18836,0.00,6.910,0,0.4480,5.7860,33.30,5.1004,3,233.0,17.90,396.90,14.15} | 1 | 20.00
+{1,0.22927,0.00,6.910,0,0.4480,6.0300,85.50,5.6894,3,233.0,17.90,392.74,18.80} | 1 | 16.60
+{1,0.25387,0.00,6.910,0,0.4480,5.3990,95.30,5.8700,3,233.0,17.90,396.90,30.81} | 1 | 14.40
+{1,0.21977,0.00,6.910,0,0.4480,5.6020,62.00,6.0877,3,233.0,17.90,396.90,16.20} | 1 | 19.40
+{1,0.08873,21.00,5.640,0,0.4390,5.9630,45.70,6.8147,4,243.0,16.80,395.56,13.45} | 1 | 19.70
+{1,0.04337,21.00,5.640,0,0.4390,6.1150,63.00,6.8147,4,243.0,16.80,393.97,9.43} | 1 | 20.50
+{1,0.05360,21.00,5.640,0,0.4390,6.5110,21.10,6.8147,4,243.0,16.80,396.90,5.28} | 1 | 25.00
+{1,0.04981,21.00,5.640,0,0.4390,5.9980,21.40,6.8147,4,243.0,16.80,396.90,8.43} | 1 | 23.40
+{1,0.01360,75.00,4.000,0,0.4100,5.8880,47.60,7.3197,3,469.0,21.10,396.90,14.80} | 1 | 18.90
+{1,0.01311,90.00,1.220,0,0.4030,7.2490,21.90,8.6966,5,226.0,17.90,395.93,4.81} | 1 | 35.40
+{1,0.02055,85.00,0.740,0,0.4100,6.3830,35.70,9.1876,2,313.0,17.30,396.90,5.77} | 1 | 24.70
+{1,0.01432,100.00,1.320,0,0.4110,6.8160,40.50,8.3248,5,256.0,15.10,392.90,3.95} | 1 | 31.60
+{1,0.15445,25.00,5.130,0,0.4530,6.1450,29.20,7.8148,8,284.0,19.70,390.68,6.86} | 1 | 23.30
+{1,0.10328,25.00,5.130,0,0.4530,5.9270,47.20,6.9320,8,284.0,19.70,396.90,9.22} | 1 | 19.60
+{1,0.14932,25.00,5.130,0,0.4530,5.7410,66.20,7.2254,8,284.0,19.70,395.11,13.15} | 1 | 18.70
+{1,0.17171,25.00,5.130,0,0.4530,5.9660,93.40,6.8185,8,284.0,19.70,378.08,14.44} | 1 | 16.00
+{1,0.11027,25.00,5.130,0,0.4530,6.4560,67.80,7.2255,8,284.0,19.70,396.90,6.73} | 1 | 22.20
+{1,0.12650,25.00,5.130,0,0.4530,6.7620,43.40,7.9809,8,284.0,19.70,395.58,9.50} | 1 | 25.00
+{1,0.01951,17.50,1.380,0,0.4161,7.1040,59.50,9.2229,3,216.0,18.60,393.24,8.05} | 1 | 33.00
+{1,0.03584,80.00,3.370,0,0.3980,6.2900,17.80,6.6115,4,337.0,16.10,396.90,4.67} | 1 | 23.50
+{1,0.04379,80.00,3.370,0,0.3980,5.7870,31.10,6.6115,4,337.0,16.10,396.90,10.24} | 1 | 19.40
+{1,0.05789,12.50,6.070,0,0.4090,5.8780,21.40,6.4980,4,345.0,18.90,396.21,8.10} | 1 | 22.00
+{1,0.13554,12.50,6.070,0,0.4090,5.5940,36.80,6.4980,4,345.0,18.90,396.90,13.09} | 1 | 17.40
+{1,0.12816,12.50,6.070,0,0.4090,5.8850,33.00,6.4980,4,345.0,18.90,396.90,8.79} | 1 | 20.90
+{1,0.08826,0.00,10.810,0,0.4130,6.4170,6.60,5.2873,4,305.0,19.20,383.73,6.72} | 1 | 24.20
+{1,0.15876,0.00,10.810,0,0.4130,5.9610,17.50,5.2873,4,305.0,19.20,376.94,9.88} | 1 | 21.70
+{1,0.09164,0.00,10.810,0,0.4130,6.0650,7.80,5.2873,4,305.0,19.20,390.91,5.52} | 1 | 22.80
+{1,0.19539,0.00,10.810,0,0.4130,6.2450,6.20,5.2873,4,305.0,19.20,377.17,7.54} | 1 | 23.40
+{1,0.07896,0.00,12.830,0,0.4370,6.2730,6.00,4.2515,5,398.0,18.70,394.92,6.78} | 1 | 24.10
+{1,0.09512,0.00,12.830,0,0.4370,6.2860,45.00,4.5026,5,398.0,18.70,383.23,8.94} | 1 | 21.40
+{1,0.10153,0.00,12.830,0,0.4370,6.2790,74.50,4.0522,5,398.0,18.70,373.66,11.97} | 1 | 20.00
+{1,0.08707,0.00,12.830,0,0.4370,6.1400,45.80,4.0905,5,398.0,18.70,386.96,10.27} | 1 | 20.80
+{1,0.05646,0.00,12.830,0,0.4370,6.2320,53.70,5.0141,5,398.0,18.70,386.40,12.34} | 1 | 21.20
+{1,0.08387,0.00,12.830,0,0.4370,5.8740,36.60,4.5026,5,398.0,18.70,396.06,9.10} | 1 | 20.30
+{1,0.04113,25.00,4.860,0,0.4260,6.7270,33.50,5.4007,4,281.0,19.00,396.90,5.29} | 1 | 28.00
+{1,0.04462,25.00,4.860,0,0.4260,6.6190,70.40,5.4007,4,281.0,19.00,395.63,7.22} | 1 | 23.90
+{1,0.03659,25.00,4.860,0,0.4260,6.3020,32.20,5.4007,4,281.0,19.00,396.90,6.72} | 1 | 24.80
+{1,0.03551,25.00,4.860,0,0.4260,6.1670,46.70,5.4007,4,281.0,19.00,390.64,7.51} | 1 | 22.90
+{1,0.05059,0.00,4.490,0,0.4490,6.3890,48.00,4.7794,3,247.0,18.50,396.90,9.62} | 1 | 23.90
+{1,0.05735,0.00,4.490,0,0.4490,6.6300,56.10,4.4377,3,247.0,18.50,392.30,6.53} | 1 | 26.60
+{1,0.05188,0.00,4.490,0,0.4490,6.0150,45.10,4.4272,3,247.0,18.50,395.99,12.86} | 1 | 22.50
+{1,0.07151,0.00,4.490,0,0.4490,6.1210,56.80,3.7476,3,247.0,18.50,395.15,8.44} | 1 | 22.20
+{1,0.05660,0.00,3.410,0,0.4890,7.0070,86.30,3.4217,2,270.0,17.80,396.90,5.50} | 1 | 23.60
+{1,0.05302,0.00,3.410,0,0.4890,7.0790,63.10,3.4145,2,270.0,17.80,396.06,5.70} | 1 | 28.70
+{1,0.04684,0.00,3.410,0,0.4890,6.4170,66.10,3.0923,2,270.0,17.80,392.18,8.81} | 1 | 22.60
+{1,0.03932,0.00,3.410,0,0.4890,6.4050,73.90,3.0921,2,270.0,17.80,393.55,8.20} | 1 | 22.00
+{1,0.04203,28.00,15.040,0,0.4640,6.4420,53.60,3.6659,4,270.0,18.20,395.01,8.16} | 1 | 22.90
+{1,0.02875,28.00,15.040,0,0.4640,6.2110,28.90,3.6659,4,270.0,18.20,396.33,6.21} | 1 | 25.00
+{1,0.04294,28.00,15.040,0,0.4640,6.2490,77.30,3.6150,4,270.0,18.20,396.90,10.59} | 1 | 20.60
+{1,0.12204,0.00,2.890,0,0.4450,6.6250,57.80,3.4952,2,276.0,18.00,357.98,6.65} | 1 | 28.40
+{1,0.11504,0.00,2.890,0,0.4450,6.1630,69.60,3.4952,2,276.0,18.00,391.83,11.34} | 1 | 21.40
+{1,0.12083,0.00,2.890,0,0.4450,8.0690,76.00,3.4952,2,276.0,18.00,396.90,4.21} | 1 | 38.70
+{1,0.08187,0.00,2.890,0,0.4450,7.8200,36.90,3.4952,2,276.0,18.00,393.53,3.57} | 1 | 43.80
+{1,0.06860,0.00,2.890,0,0.4450,7.4160,62.50,3.4952,2,276.0,18.00,396.90,6.19} | 1 | 33.20
+{1,0.14866,0.00,8.560,0,0.5200,6.7270,79.90,2.7778,5,384.0,20.90,394.76,9.42} | 1 | 27.50
+{1,0.11432,0.00,8.560,0,0.5200,6.7810,71.30,2.8561,5,384.0,20.90,395.58,7.67} | 1 | 26.50
+{1,0.22876,0.00,8.560,0,0.5200,6.4050,85.40,2.7147,5,384.0,20.90,70.80,10.63} | 1 | 18.60
+{1,0.21161,0.00,8.560,0,0.5200,6.1370,87.40,2.7147,5,384.0,20.90,394.47,13.44} | 1 | 19.30
+{1,0.13960,0.00,8.560,0,0.5200,6.1670,90.00,2.4210,5,384.0,20.90,392.69,12.33} | 1 | 20.10
+{1,0.13262,0.00,8.560,0,0.5200,5.8510,96.70,2.1069,5,384.0,20.90,394.05,16.47} | 1 | 19.50
+{1,0.17120,0.00,8.560,0,0.5200,5.8360,91.90,2.2110,5,384.0,20.90,395.67,18.66} | 1 | 19.50
+{1,0.13117,0.00,8.560,0,0.5200,6.1270,85.20,2.1224,5,384.0,20.90,387.69,14.09} | 1 | 20.40
+{1,0.12802,0.00,8.560,0,0.5200,6.4740,97.10,2.4329,5,384.0,20.90,395.24,12.27} | 1 | 19.80
+{1,0.26363,0.00,8.560,0,0.5200,6.2290,91.20,2.5451,5,384.0,20.90,391.23,15.55} | 1 | 19.40
+{1,0.10793,0.00,8.560,0,0.5200,6.1950,54.40,2.7778,5,384.0,20.90,393.49,13.00} | 1 | 21.70
+{1,0.10084,0.00,10.010,0,0.5470,6.7150,81.60,2.6775,6,432.0,17.80,395.59,10.16} | 1 | 22.80
+{1,0.12329,0.00,10.010,0,0.5470,5.9130,92.90,2.3534,6,432.0,17.80,394.95,16.21} | 1 | 18.80
+{1,0.22212,0.00,10.010,0,0.5470,6.0920,95.40,2.5480,6,432.0,17.80,396.90,17.09} | 1 | 18.70
+{1,0.14231,0.00,10.010,0,0.5470,6.2540,84.20,2.2565,6,432.0,17.80,388.74,10.45} | 1 | 18.50
+{1,0.17134,0.00,10.010,0,0.5470,5.9280,88.20,2.4631,6,432.0,17.80,344.91,15.76} | 1 | 18.30
+{1,0.13158,0.00,10.010,0,0.5470,6.1760,72.50,2.7301,6,432.0,17.80,393.30,12.04} | 1 | 21.20
+{1,0.15098,0.00,10.010,0,0.5470,6.0210,82.60,2.7474,6,432.0,17.80,394.51,10.30} | 1 | 19.20
+{1,0.13058,0.00,10.010,0,0.5470,5.8720,73.10,2.4775,6,432.0,17.80,338.63,15.37} | 1 | 20.40
+{1,0.14476,0.00,10.010,0,0.5470,5.7310,65.20,2.7592,6,432.0,17.80,391.50,13.61} | 1 | 19.30
+{1,0.06899,0.00,25.650,0,0.5810,5.8700,69.70,2.2577,2,188.0,19.10,389.15,14.37} | 1 | 22.00
+{1,0.07165,0.00,25.650,0,0.5810,6.0040,84.10,2.1974,2,188.0,19.10,377.67,14.27} | 1 | 20.30
+{1,0.09299,0.00,25.650,0,0.5810,5.9610,92.90,2.0869,2,188.0,19.10,378.09,17.93} | 1 | 20.50
+{1,0.15038,0.00,25.650,0,0.5810,5.8560,97.00,1.9444,2,188.0,19.10,370.31,25.41} | 1 | 17.30
+{1,0.09849,0.00,25.650,0,0.5810,5.8790,95.80,2.0063,2,188.0,19.10,379.38,17.58} | 1 | 18.80
+{1,0.16902,0.00,25.650,0,0.5810,5.9860,88.40,1.9929,2,188.0,19.10,385.02,14.81} | 1 | 21.40
+{1,0.38735,0.00,25.650,0,0.5810,5.6130,95.60,1.7572,2,188.0,19.10,359.29,27.26} | 1 | 15.70
+{1,0.25915,0.00,21.890,0,0.6240,5.6930,96.00,1.7883,4,437.0,21.20,392.11,17.19} | 1 | 16.20
+{1,0.32543,0.00,21.890,0,0.6240,6.4310,98.80,1.8125,4,437.0,21.20,396.90,15.39} | 1 | 18.00
+{1,0.88125,0.00,21.890,0,0.6240,5.6370,94.70,1.9799,4,437.0,21.20,396.90,18.34} | 1 | 14.30
+{1,0.34006,0.00,21.890,0,0.6240,6.4580,98.90,2.1185,4,437.0,21.20,395.04,12.60} | 1 | 19.20
+{1,1.19294,0.00,21.890,0,0.6240,6.3260,97.70,2.2710,4,437.0,21.20,396.90,12.26} | 1 | 19.60
+{1,0.59005,0.00,21.890,0,0.6240,6.3720,97.90,2.3274,4,437.0,21.20,385.76,11.12} | 1 | 23.00
+{1,0.32982,0.00,21.890,0,0.6240,5.8220,95.40,2.4699,4,437.0,21.20,388.69,15.03} | 1 | 18.40
+{1,0.97617,0.00,21.890,0,0.6240,5.7570,98.40,2.3460,4,437.0,21.20,262.76,17.31} | 1 | 15.60
+{1,0.55778,0.00,21.890,0,0.6240,6.3350,98.20,2.1107,4,437.0,21.20,394.67,16.96} | 1 | 18.10
+{1,0.32264,0.00,21.890,0,0.6240,5.9420,93.50,1.9669,4,437.0,21.20,378.25,16.90} | 1 | 17.40
+{1,0.35233,0.00,21.890,0,0.6240,6.4540,98.40,1.8498,4,437.0,21.20,394.08,14.59} | 1 | 17.10
+{1,0.24980,0.00,21.890,0,0.6240,5.8570,98.20,1.6686,4,437.0,21.20,392.04,21.32} | 1 | 13.30
+{1,0.54452,0.00,21.890,0,0.6240,6.1510,97.90,1.6687,4,437.0,21.20,396.90,18.46} | 1 | 17.80
+{1,0.29090,0.00,21.890,0,0.6240,6.1740,93.60,1.6119,4,437.0,21.20,388.08,24.16} | 1 | 14.00
+{1,1.62864,0.00,21.890,0,0.6240,5.0190,100.00,1.4394,4,437.0,21.20,396.90,34.41} | 1 | 14.40
+{1,3.32105,0.00,19.580,1,0.8710,5.4030,100.00,1.3216,5,403.0,14.70,396.90,26.82} | 1 | 13.40
+{1,4.09740,0.00,19.580,0,0.8710,5.4680,100.00,1.4118,5,403.0,14.70,396.90,26.42} | 1 | 15.60
+{1,2.77974,0.00,19.580,0,0.8710,4.9030,97.80,1.3459,5,403.0,14.70,396.90,29.29} | 1 | 11.80
+{1,2.37934,0.00,19.580,0,0.8710,6.1300,100.00,1.4191,5,403.0,14.70,172.91,27.80} | 1 | 13.80
+{1,2.15505,0.00,19.580,0,0.8710,5.6280,100.00,1.5166,5,403.0,14.70,169.27,16.65} | 1 | 15.60
+{1,2.36862,0.00,19.580,0,0.8710,4.9260,95.70,1.4608,5,403.0,14.70,391.71,29.53} | 1 | 14.60
+{1,2.33099,0.00,19.580,0,0.8710,5.1860,93.80,1.5296,5,403.0,14.70,356.99,28.32} | 1 | 17.80
+{1,2.73397,0.00,19.580,0,0.8710,5.5970,94.90,1.5257,5,403.0,14.70,351.85,21.45} | 1 | 15.40
+{1,1.65660,0.00,19.580,0,0.8710,6.1220,97.30,1.6180,5,403.0,14.70,372.80,14.10} | 1 | 21.50
+{1,1.49632,0.00,19.580,0,0.8710,5.4040,100.00,1.5916,5,403.0,14.70,341.60,13.28} | 1 | 19.60
+{1,1.12658,0.00,19.580,1,0.8710,5.0120,88.00,1.6102,5,403.0,14.70,343.28,12.12} | 1 | 15.30
+{1,2.14918,0.00,19.580,0,0.8710,5.7090,98.50,1.6232,5,403.0,14.70,261.95,15.79} | 1 | 19.40
+{1,1.41385,0.00,19.580,1,0.8710,6.1290,96.00,1.7494,5,403.0,14.70,321.02,15.12} | 1 | 17.00
+{1,3.53501,0.00,19.580,1,0.8710,6.1520,82.60,1.7455,5,403.0,14.70,88.01,15.02} | 1 | 15.60
+{1,2.44668,0.00,19.580,0,0.8710,5.2720,94.00,1.7364,5,403.0,14.70,88.63,16.14} | 1 | 13.10
+{1,1.22358,0.00,19.580,0,0.6050,6.9430,97.40,1.8773,5,403.0,14.70,363.43,4.59} | 1 | 41.30
+{1,1.34284,0.00,19.580,0,0.6050,6.0660,100.00,1.7573,5,403.0,14.70,353.89,6.43} | 1 | 24.30
+{1,1.42502,0.00,19.580,0,0.8710,6.5100,100.00,1.7659,5,403.0,14.70,364.31,7.39} | 1 | 23.30
+{1,1.27346,0.00,19.580,1,0.6050,6.2500,92.60,1.7984,5,403.0,14.70,338.92,5.50} | 1 | 27.00
+{1,1.46336,0.00,19.580,0,0.6050,7.4890,90.80,1.9709,5,403.0,14.70,374.43,1.73} | 1 | 50.00
+{1,1.83377,0.00,19.580,1,0.6050,7.8020,98.20,2.0407,5,403.0,14.70,389.61,1.92} | 1 | 50.00
+{1,1.51902,0.00,19.580,1,0.6050,8.3750,93.90,2.1620,5,403.0,14.70,388.45,3.32} | 1 | 50.00
+{1,2.24236,0.00,19.580,0,0.6050,5.8540,91.80,2.4220,5,403.0,14.70,395.11,11.64} | 1 | 22.70
+{1,2.92400,0.00,19.580,0,0.6050,6.1010,93.00,2.2834,5,403.0,14.70,240.16,9.81} | 1 | 25.00
+{1,2.01019,0.00,19.580,0,0.6050,7.9290,96.20,2.0459,5,403.0,14.70,369.30,3.70} | 1 | 50.00
+{1,1.80028,0.00,19.580,0,0.6050,5.8770,79.20,2.4259,5,403.0,14.70,227.61,12.14} | 1 | 23.80
+{1,2.30040,0.00,19.580,0,0.6050,6.3190,96.10,2.1000,5,403.0,14.70,297.09,11.10} | 1 | 23.80
+{1,2.44953,0.00,19.580,0,0.6050,6.4020,95.20,2.2625,5,403.0,14.70,330.04,11.32} | 1 | 22.30
+{1,1.20742,0.00,19.580,0,0.6050,5.8750,94.60,2.4259,5,403.0,14.70,292.29,14.43} | 1 | 17.40
+{1,2.31390,0.00,19.580,0,0.6050,5.8800,97.30,2.3887,5,403.0,14.70,348.13,12.03} | 1 | 19.10
+{1,0.13914,0.00,4.050,0,0.5100,5.5720,88.50,2.5961,5,296.0,16.60,396.90,14.69} | 1 | 23.10
+{1,0.09178,0.00,4.050,0,0.5100,6.4160,84.10,2.6463,5,296.0,16.60,395.50,9.04} | 1 | 23.60
+{1,0.08447,0.00,4.050,0,0.5100,5.8590,68.70,2.7019,5,296.0,16.60,393.23,9.64} | 1 | 22.60
+{1,0.06664,0.00,4.050,0,0.5100,6.5460,33.10,3.1323,5,296.0,16.60,390.96,5.33} | 1 | 29.40
+{1,0.07022,0.00,4.050,0,0.5100,6.0200,47.20,3.5549,5,296.0,16.60,393.23,10.11} | 1 | 23.20
+{1,0.05425,0.00,4.050,0,0.5100,6.3150,73.40,3.3175,5,296.0,16.60,395.60,6.29} | 1 | 24.60
+{1,0.06642,0.00,4.050,0,0.5100,6.8600,74.40,2.9153,5,296.0,16.60,391.27,6.92} | 1 | 29.90
+{1,0.05780,0.00,2.460,0,0.4880,6.9800,58.40,2.8290,3,193.0,17.80,396.90,5.04} | 1 | 37.20
+{1,0.06588,0.00,2.460,0,0.4880,7.7650,83.30,2.7410,3,193.0,17.80,395.56,7.56} | 1 | 39.80
+{1,0.06888,0.00,2.460,0,0.4880,6.1440,62.20,2.5979,3,193.0,17.80,396.90,9.45} | 1 | 36.20
+{1,0.09103,0.00,2.460,0,0.4880,7.1550,92.20,2.7006,3,193.0,17.80,394.12,4.82} | 1 | 37.90
+{1,0.10008,0.00,2.460,0,0.4880,6.5630,95.60,2.8470,3,193.0,17.80,396.90,5.68} | 1 | 32.50
+{1,0.08308,0.00,2.460,0,0.4880,5.6040,89.80,2.9879,3,193.0,17.80,391.00,13.98} | 1 | 26.40
+{1,0.06047,0.00,2.460,0,0.4880,6.1530,68.80,3.2797,3,193.0,17.80,387.11,13.15} | 1 | 29.60
+{1,0.05602,0.00,2.460,0,0.4880,7.8310,53.60,3.1992,3,193.0,17.80,392.63,4.45} | 1 | 50.00
+{1,0.07875,45.00,3.440,0,0.4370,6.7820,41.10,3.7886,5,398.0,15.20,393.87,6.68} | 1 | 32.00
+{1,0.12579,45.00,3.440,0,0.4370,6.5560,29.10,4.5667,5,398.0,15.20,382.84,4.56} | 1 | 29.80
+{1,0.08370,45.00,3.440,0,0.4370,7.1850,38.90,4.5667,5,398.0,15.20,396.90,5.39} | 1 | 34.90
+{1,0.09068,45.00,3.440,0,0.4370,6.9510,21.50,6.4798,5,398.0,15.20,377.68,5.10} | 1 | 37.00
+{1,0.06911,45.00,3.440,0,0.4370,6.7390,30.80,6.4798,5,398.0,15.20,389.71,4.69} | 1 | 30.50
+{1,0.08664,45.00,3.440,0,0.4370,7.1780,26.30,6.4798,5,398.0,15.20,390.49,2.87} | 1 | 36.40
+{1,0.02187,60.00,2.930,0,0.4010,6.8000,9.90,6.2196,1,265.0,15.60,393.37,5.03} | 1 | 31.10
+{1,0.01439,60.00,2.930,0,0.4010,6.6040,18.80,6.2196,1,265.0,15.60,376.70,4.38} | 1 | 29.10
+{1,0.01381,80.00,0.460,0,0.4220,7.8750,32.00,5.6484,4,255.0,14.40,394.23,2.97} | 1 | 50.00
+{1,0.04011,80.00,1.520,0,0.4040,7.2870,34.10,7.3090,2,329.0,12.60,396.90,4.08} | 1 | 33.30
+{1,0.04666,80.00,1.520,0,0.4040,7.1070,36.60,7.3090,2,329.0,12.60,354.31,8.61} | 1 | 30.30
+{1,0.03768,80.00,1.520,0,0.4040,7.2740,38.30,7.3090,2,329.0,12.60,392.20,6.62} | 1 | 34.60
+{1,0.03150,95.00,1.470,0,0.4030,6.9750,15.30,7.6534,3,402.0,17.00,396.90,4.56} | 1 | 34.90
+{1,0.01778,95.00,1.470,0,0.4030,7.1350,13.90,7.6534,3,402.0,17.00,384.30,4.45} | 1 | 32.90
+{1,0.03445,82.50,2.030,0,0.4150,6.1620,38.40,6.2700,2,348.0,14.70,393.77,7.43} | 1 | 24.10
+{1,0.02177,82.50,2.030,0,0.4150,7.6100,15.70,6.2700,2,348.0,14.70,395.38,3.11} | 1 | 42.30
+{1,0.03510,95.00,2.680,0,0.4161,7.8530,33.20,5.1180,4,224.0,14.70,392.78,3.81} | 1 | 48.50
+{1,0.02009,95.00,2.680,0,0.4161,8.0340,31.90,5.1180,4,224.0,14.70,390.55,2.88} | 1 | 50.00
+{1,0.13642,0.00,10.590,0,0.4890,5.8910,22.30,3.9454,4,277.0,18.60,396.90,10.87} | 1 | 22.60
+{1,0.22969,0.00,10.590,0,0.4890,6.3260,52.50,4.3549,4,277.0,18.60,394.87,10.97} | 1 | 24.40
+{1,0.25199,0.00,10.590,0,0.4890,5.7830,72.70,4.3549,4,277.0,18.60,389.43,18.06} | 1 | 22.50
+{1,0.13587,0.00,10.590,1,0.4890,6.0640,59.10,4.2392,4,277.0,18.60,381.32,14.66} | 1 | 24.40
+{1,0.43571,0.00,10.590,1,0.4890,5.3440,100.00,3.8750,4,277.0,18.60,396.90,23.09} | 1 | 20.00
+{1,0.17446,0.00,10.590,1,0.4890,5.9600,92.10,3.8771,4,277.0,18.60,393.25,17.27} | 1 | 21.70
+{1,0.37578,0.00,10.590,1,0.4890,5.4040,88.60,3.6650,4,277.0,18.60,395.24,23.98} | 1 | 19.30
+{1,0.21719,0.00,10.590,1,0.4890,5.8070,53.80,3.6526,4,277.0,18.60,390.94,16.03} | 1 | 22.40
+{1,0.14052,0.00,10.590,0,0.4890,6.3750,32.30,3.9454,4,277.0,18.60,385.81,9.38} | 1 | 28.10
+{1,0.28955,0.00,10.590,0,0.4890,5.4120,9.80,3.5875,4,277.0,18.60,348.93,29.55} | 1 | 23.70
+{1,0.19802,0.00,10.590,0,0.4890,6.1820,42.40,3.9454,4,277.0,18.60,393.63,9.47} | 1 | 25.00
+{1,0.04560,0.00,13.890,1,0.5500,5.8880,56.00,3.1121,5,276.0,16.40,392.80,13.51} | 1 | 23.30
+{1,0.07013,0.00,13.890,0,0.5500,6.6420,85.10,3.4211,5,276.0,16.40,392.78,9.69} | 1 | 28.70
+{1,0.11069,0.00,13.890,1,0.5500,5.9510,93.80,2.8893,5,276.0,16.40,396.90,17.92} | 1 | 21.50
+{1,0.11425,0.00,13.890,1,0.5500,6.3730,92.40,3.3633,5,276.0,16.40,393.74,10.50} | 1 | 23.00
+{1,0.35809,0.00,6.200,1,0.5070,6.9510,88.50,2.8617,8,307.0,17.40,391.70,9.71} | 1 | 26.70
+{1,0.40771,0.00,6.200,1,0.5070,6.1640,91.30,3.0480,8,307.0,17.40,395.24,21.46} | 1 | 21.70
+{1,0.62356,0.00,6.200,1,0.5070,6.8790,77.70,3.2721,8,307.0,17.40,390.39,9.93} | 1 | 27.50
+{1,0.61470,0.00,6.200,0,0.5070,6.6180,80.80,3.2721,8,307.0,17.40,396.90,7.60} | 1 | 30.10
+{1,0.31533,0.00,6.200,0,0.5040,8.2660,78.30,2.8944,8,307.0,17.40,385.05,4.14} | 1 | 44.80
+{1,0.52693,0.00,6.200,0,0.5040,8.7250,83.00,2.8944,8,307.0,17.40,382.00,4.63} | 1 | 50.00
+{1,0.38214,0.00,6.200,0,0.5040,8.0400,86.50,3.2157,8,307.0,17.40,387.38,3.13} | 1 | 37.60
+{1,0.41238,0.00,6.200,0,0.5040,7.1630,79.90,3.2157,8,307.0,17.40,372.08,6.36} | 1 | 31.60
+{1,0.29819,0.00,6.200,0,0.5040,7.6860,17.00,3.3751,8,307.0,17.40,377.51,3.92} | 1 | 46.70
+{1,0.44178,0.00,6.200,0,0.5040,6.5520,21.40,3.3751,8,307.0,17.40,380.34,3.76} | 1 | 31.50
+{1,0.53700,0.00,6.200,0,0.5040,5.9810,68.10,3.6715,8,307.0,17.40,378.35,11.65} | 1 | 24.30
+{1,0.46296,0.00,6.200,0,0.5040,7.4120,76.90,3.6715,8,307.0,17.40,376.14,5.25} | 1 | 31.70
+{1,0.57529,0.00,6.200,0,0.5070,8.3370,73.30,3.8384,8,307.0,17.40,385.91,2.47} | 1 | 41.70
+{1,0.33147,0.00,6.200,0,0.5070,8.2470,70.40,3.6519,8,307.0,17.40,378.95,3.95} | 1 | 48.30
+{1,0.44791,0.00,6.200,1,0.5070,6.7260,66.50,3.6519,8,307.0,17.40,360.20,8.05} | 1 | 29.00
+{1,0.33045,0.00,6.200,0,0.5070,6.0860,61.50,3.6519,8,307.0,17.40,376.75,10.88} | 1 | 24.00
+{1,0.52058,0.00,6.200,1,0.5070,6.6310,76.50,4.1480,8,307.0,17.40,388.45,9.54} | 1 | 25.10
+{1,0.51183,0.00,6.200,0,0.5070,7.3580,71.60,4.1480,8,307.0,17.40,390.07,4.73} | 1 | 31.50
+{1,0.08244,30.00,4.930,0,0.4280,6.4810,18.50,6.1899,6,300.0,16.60,379.41,6.36} | 1 | 23.70
+{1,0.09252,30.00,4.930,0,0.4280,6.6060,42.20,6.1899,6,300.0,16.60,383.78,7.37} | 1 | 23.30
+{1,0.11329,30.00,4.930,0,0.4280,6.8970,54.30,6.3361,6,300.0,16.60,391.25,11.38} | 1 | 22.00
+{1,0.10612,30.00,4.930,0,0.4280,6.0950,65.10,6.3361,6,300.0,16.60,394.62,12.40} | 1 | 20.10
+{1,0.10290,30.00,4.930,0,0.4280,6.3580,52.90,7.0355,6,300.0,16.60,372.75,11.22} | 1 | 22.20
+{1,0.12757,30.00,4.930,0,0.4280,6.3930,7.80,7.0355,6,300.0,16.60,374.71,5.19} | 1 | 23.70
+{1,0.20608,22.00,5.860,0,0.4310,5.5930,76.50,7.9549,7,330.0,19.10,372.49,12.50} | 1 | 17.60
+{1,0.19133,22.00,5.860,0,0.4310,5.6050,70.20,7.9549,7,330.0,19.10,389.13,18.46} | 2 | 18.50
+{1,0.33983,22.00,5.860,0,0.4310,6.1080,34.90,8.0555,7,330.0,19.10,390.18,9.16} | 2 | 24.30
+{1,0.19657,22.00,5.860,0,0.4310,6.2260,79.20,8.0555,7,330.0,19.10,376.14,10.15} | 2 | 20.50
+{1,0.16439,22.00,5.860,0,0.4310,6.4330,49.10,7.8265,7,330.0,19.10,374.71,9.52} | 2 | 24.50
+{1,0.19073,22.00,5.860,0,0.4310,6.7180,17.50,7.8265,7,330.0,19.10,393.74,6.56} | 2 | 26.20
+{1,0.14030,22.00,5.860,0,0.4310,6.4870,13.00,7.3967,7,330.0,19.10,396.28,5.90} | 2 | 24.40
+{1,0.21409,22.00,5.860,0,0.4310,6.4380,8.90,7.3967,7,330.0,19.10,377.07,3.59} | 2 | 24.80
+{1,0.08221,22.00,5.860,0,0.4310,6.9570,6.80,8.9067,7,330.0,19.10,386.09,3.53} | 2 | 29.60
+{1,0.36894,22.00,5.860,0,0.4310,8.2590,8.40,8.9067,7,330.0,19.10,396.90,3.54} | 2 | 42.80
+{1,0.04819,80.00,3.640,0,0.3920,6.1080,32.00,9.2203,1,315.0,16.40,392.89,6.57} | 2 | 21.90
+{1,0.03548,80.00,3.640,0,0.3920,5.8760,19.10,9.2203,1,315.0,16.40,395.18,9.25} | 2 | 20.90
+{1,0.01538,90.00,3.750,0,0.3940,7.4540,34.20,6.3361,3,244.0,15.90,386.34,3.11} | 2 | 44.00
+{1,0.61154,20.00,3.970,0,0.6470,8.7040,86.90,1.8010,5,264.0,13.00,389.70,5.12} | 2 | 50.00
+{1,0.66351,20.00,3.970,0,0.6470,7.3330,100.00,1.8946,5,264.0,13.00,383.29,7.79} | 2 | 36.00
+{1,0.65665,20.00,3.970,0,0.6470,6.8420,100.00,2.0107,5,264.0,13.00,391.93,6.90} | 2 | 30.10
+{1,0.54011,20.00,3.970,0,0.6470,7.2030,81.80,2.1121,5,264.0,13.00,392.80,9.59} | 2 | 33.80
+{1,0.53412,20.00,3.970,0,0.6470,7.5200,89.40,2.1398,5,264.0,13.00,388.37,7.26} | 2 | 43.10
+{1,0.52014,20.00,3.970,0,0.6470,8.3980,91.50,2.2885,5,264.0,13.00,386.86,5.91} | 2 | 48.80
+{1,0.82526,20.00,3.970,0,0.6470,7.3270,94.50,2.0788,5,264.0,13.00,393.42,11.25} | 2 | 31.00
+{1,0.55007,20.00,3.970,0,0.6470,7.2060,91.60,1.9301,5,264.0,13.00,387.89,8.10} | 2 | 36.50
+{1,0.76162,20.00,3.970,0,0.6470,5.5600,62.80,1.9865,5,264.0,13.00,392.40,10.45} | 2 | 22.80
+{1,0.78570,20.00,3.970,0,0.6470,7.0140,84.60,2.1329,5,264.0,13.00,384.07,14.79} | 2 | 30.70
+{1,0.57834,20.00,3.970,0,0.5750,8.2970,67.00,2.4216,5,264.0,13.00,384.54,7.44} | 2 | 50.00
+{1,0.54050,20.00,3.970,0,0.5750,7.4700,52.60,2.8720,5,264.0,13.00,390.30,3.16} | 2 | 43.50
+{1,0.09065,20.00,6.960,1,0.4640,5.9200,61.50,3.9175,3,223.0,18.60,391.34,13.65} | 2 | 20.70
+{1,0.29916,20.00,6.960,0,0.4640,5.8560,42.10,4.4290,3,223.0,18.60,388.65,13.00} | 2 | 21.10
+{1,0.16211,20.00,6.960,0,0.4640,6.2400,16.30,4.4290,3,223.0,18.60,396.90,6.59} | 2 | 25.20
+{1,0.11460,20.00,6.960,0,0.4640,6.5380,58.70,3.9175,3,223.0,18.60,394.96,7.73} | 2 | 24.40
+{1,0.22188,20.00,6.960,1,0.4640,7.6910,51.80,4.3665,3,223.0,18.60,390.77,6.58} | 2 | 35.20
+{1,0.05644,40.00,6.410,1,0.4470,6.7580,32.90,4.0776,4,254.0,17.60,396.90,3.53} | 2 | 32.40
+{1,0.09604,40.00,6.410,0,0.4470,6.8540,42.80,4.2673,4,254.0,17.60,396.90,2.98} | 2 | 32.00
+{1,0.10469,40.00,6.410,1,0.4470,7.2670,49.00,4.7872,4,254.0,17.60,389.25,6.05} | 2 | 33.20
+{1,0.06127,40.00,6.410,1,0.4470,6.8260,27.60,4.8628,4,254.0,17.60,393.45,4.16} | 2 | 33.10
+{1,0.07978,40.00,6.410,0,0.4470,6.4820,32.10,4.1403,4,254.0,17.60,396.90,7.19} | 2 | 29.10
+{1,0.21038,20.00,3.330,0,0.4429,6.8120,32.20,4.1007,5,216.0,14.90,396.90,4.85} | 2 | 35.10
+{1,0.03578,20.00,3.330,0,0.4429,7.8200,64.50,4.6947,5,216.0,14.90,387.31,3.76} | 2 | 45.40
+{1,0.03705,20.00,3.330,0,0.4429,6.9680,37.20,5.2447,5,216.0,14.90,392.23,4.59} | 2 | 35.40
+{1,0.06129,20.00,3.330,1,0.4429,7.6450,49.70,5.2119,5,216.0,14.90,377.07,3.01} | 2 | 46.00
+{1,0.01501,90.00,1.210,1,0.4010,7.9230,24.80,5.8850,1,198.0,13.60,395.52,3.16} | 2 | 50.00
+{1,0.00906,90.00,2.970,0,0.4000,7.0880,20.80,7.3073,1,285.0,15.30,394.72,7.85} | 2 | 32.20
+{1,0.01096,55.00,2.250,0,0.3890,6.4530,31.90,7.3073,1,300.0,15.30,394.72,8.23} | 2 | 22.00
+{1,0.01965,80.00,1.760,0,0.3850,6.2300,31.50,9.0892,1,241.0,18.20,341.60,12.93} | 2 | 20.10
+{1,0.03871,52.50,5.320,0,0.4050,6.2090,31.30,7.3172,6,293.0,16.60,396.90,7.14} | 2 | 23.20
+{1,0.04590,52.50,5.320,0,0.4050,6.3150,45.60,7.3172,6,293.0,16.60,396.90,7.60} | 2 | 22.30
+{1,0.04297,52.50,5.320,0,0.4050,6.5650,22.90,7.3172,6,293.0,16.60,371.72,9.51} | 2 | 24.80
+{1,0.03502,80.00,4.950,0,0.4110,6.8610,27.90,5.1167,4,245.0,19.20,396.90,3.33} | 2 | 28.50
+{1,0.07886,80.00,4.950,0,0.4110,7.1480,27.70,5.1167,4,245.0,19.20,396.90,3.56} | 2 | 37.30
+{1,0.03615,80.00,4.950,0,0.4110,6.6300,23.40,5.1167,4,245.0,19.20,396.90,4.70} | 2 | 27.90
+{1,0.08265,0.00,13.920,0,0.4370,6.1270,18.40,5.5027,4,289.0,16.00,396.90,8.58} | 2 | 23.90
+{1,0.08199,0.00,13.920,0,0.4370,6.0090,42.30,5.5027,4,289.0,16.00,396.90,10.40} | 2 | 21.70
+{1,0.12932,0.00,13.920,0,0.4370,6.6780,31.10,5.9604,4,289.0,16.00,396.90,6.27} | 2 | 28.60
+{1,0.05372,0.00,13.920,0,0.4370,6.5490,51.00,5.9604,4,289.0,16.00,392.85,7.39} | 2 | 27.10
+{1,0.14103,0.00,13.920,0,0.4370,5.7900,58.00,6.3200,4,289.0,16.00,396.90,15.84} | 2 | 20.30
+{1,0.06466,70.00,2.240,0,0.4000,6.3450,20.10,7.8278,5,358.0,14.80,368.24,4.97} | 2 | 22.50
+{1,0.05561,70.00,2.240,0,0.4000,7.0410,10.00,7.8278,5,358.0,14.80,371.58,4.74} | 2 | 29.00
+{1,0.04417,70.00,2.240,0,0.4000,6.8710,47.40,7.8278,5,358.0,14.80,390.86,6.07} | 2 | 24.80
+{1,0.03537,34.00,6.090,0,0.4330,6.5900,40.40,5.4917,7,329.0,16.10,395.75,9.50} | 2 | 22.00
+{1,0.09266,34.00,6.090,0,0.4330,6.4950,18.40,5.4917,7,329.0,16.10,383.61,8.67} | 2 | 26.40
+{1,0.10000,34.00,6.090,0,0.4330,6.9820,17.70,5.4917,7,329.0,16.10,390.43,4.86} | 2 | 33.10
+{1,0.05515,33.00,2.180,0,0.4720,7.2360,41.10,4.0220,7,222.0,18.40,393.68,6.93} | 2 | 36.10
+{1,0.05479,33.00,2.180,0,0.4720,6.6160,58.10,3.3700,7,222.0,18.40,393.36,8.93} | 2 | 28.40
+{1,0.07503,33.00,2.180,0,0.4720,7.4200,71.90,3.0992,7,222.0,18.40,396.90,6.47} | 2 | 33.40
+{1,0.04932,33.00,2.180,0,0.4720,6.8490,70.30,3.1827,7,222.0,18.40,396.90,7.53} | 2 | 28.20
+{1,0.49298,0.00,9.900,0,0.5440,6.6350,82.50,3.3175,4,304.0,18.40,396.90,4.54} | 2 | 22.80
+{1,0.34940,0.00,9.900,0,0.5440,5.9720,76.70,3.1025,4,304.0,18.40,396.24,9.97} | 2 | 20.30
+{1,2.63548,0.00,9.900,0,0.5440,4.9730,37.80,2.5194,4,304.0,18.40,350.45,12.64} | 2 | 16.10
+{1,0.79041,0.00,9.900,0,0.5440,6.1220,52.80,2.6403,4,304.0,18.40,396.90,5.98} | 2 | 22.10
+{1,0.26169,0.00,9.900,0,0.5440,6.0230,90.40,2.8340,4,304.0,18.40,396.30,11.72} | 2 | 19.40
+{1,0.26938,0.00,9.900,0,0.5440,6.2660,82.80,3.2628,4,304.0,18.40,393.39,7.90} | 2 | 21.60
+{1,0.36920,0.00,9.900,0,0.5440,6.5670,87.30,3.6023,4,304.0,18.40,395.69,9.28} | 2 | 23.80
+{1,0.25356,0.00,9.900,0,0.5440,5.7050,77.70,3.9450,4,304.0,18.40,396.42,11.50} | 2 | 16.20
+{1,0.31827,0.00,9.900,0,0.5440,5.9140,83.20,3.9986,4,304.0,18.40,390.70,18.33} | 2 | 17.80
+{1,0.24522,0.00,9.900,0,0.5440,5.7820,71.70,4.0317,4,304.0,18.40,396.90,15.94} | 2 | 19.80
+{1,0.40202,0.00,9.900,0,0.5440,6.3820,67.20,3.5325,4,304.0,18.40,395.21,10.36} | 2 | 23.10
+{1,0.47547,0.00,9.900,0,0.5440,6.1130,58.80,4.0019,4,304.0,18.40,396.23,12.73} | 2 | 21.00
+{1,0.16760,0.00,7.380,0,0.4930,6.4260,52.30,4.5404,5,287.0,19.60,396.90,7.20} | 2 | 23.80
+{1,0.18159,0.00,7.380,0,0.4930,6.3760,54.30,4.5404,5,287.0,19.60,396.90,6.87} | 2 | 23.10
+{1,0.35114,0.00,7.380,0,0.4930,6.0410,49.90,4.7211,5,287.0,19.60,396.90,7.70} | 2 | 20.40
+{1,0.28392,0.00,7.380,0,0.4930,5.7080,74.30,4.7211,5,287.0,19.60,391.13,11.74} | 2 | 18.50
+{1,0.34109,0.00,7.380,0,0.4930,6.4150,40.10,4.7211,5,287.0,19.60,396.90,6.12} | 2 | 25.00
+{1,0.19186,0.00,7.380,0,0.4930,6.4310,14.70,5.4159,5,287.0,19.60,393.68,5.08} | 2 | 24.60
+{1,0.30347,0.00,7.380,0,0.4930,6.3120,28.90,5.4159,5,287.0,19.60,396.90,6.15} | 2 | 23.00
+{1,0.24103,0.00,7.380,0,0.4930,6.0830,43.70,5.4159,5,287.0,19.60,396.90,12.79} | 2 | 22.20
+{1,0.06617,0.00,3.240,0,0.4600,5.8680,25.80,5.2146,4,430.0,16.90,382.44,9.97} | 2 | 19.30
+{1,0.06724,0.00,3.240,0,0.4600,6.3330,17.20,5.2146,4,430.0,16.90,375.21,7.34} | 2 | 22.60
+{1,0.04544,0.00,3.240,0,0.4600,6.1440,32.20,5.8736,4,430.0,16.90,368.57,9.09} | 2 | 19.80
+{1,0.05023,35.00,6.060,0,0.4379,5.7060,28.40,6.6407,1,304.0,16.90,394.02,12.43} | 2 | 17.10
+{1,0.03466,35.00,6.060,0,0.4379,6.0310,23.30,6.6407,1,304.0,16.90,362.25,7.83} | 2 | 19.40
+{1,0.05083,0.00,5.190,0,0.5150,6.3160,38.10,6.4584,5,224.0,20.20,389.71,5.68} | 2 | 22.20
+{1,0.03738,0.00,5.190,0,0.5150,6.3100,38.50,6.4584,5,224.0,20.20,389.40,6.75} | 2 | 20.70
+{1,0.03961,0.00,5.190,0,0.5150,6.0370,34.50,5.9853,5,224.0,20.20,396.90,8.01} | 2 | 21.10
+{1,0.03427,0.00,5.190,0,0.5150,5.8690,46.30,5.2311,5,224.0,20.20,396.90,9.80} | 2 | 19.50
+{1,0.03041,0.00,5.190,0,0.5150,5.8950,59.60,5.6150,5,224.0,20.20,394.81,10.56} | 2 | 18.50
+{1,0.03306,0.00,5.190,0,0.5150,6.0590,37.30,4.8122,5,224.0,20.20,396.14,8.51} | 2 | 20.60
+{1,0.05497,0.00,5.190,0,0.5150,5.9850,45.40,4.8122,5,224.0,20.20,396.90,9.74} | 2 | 19.00
+{1,0.06151,0.00,5.190,0,0.5150,5.9680,58.50,4.8122,5,224.0,20.20,396.90,9.29} | 2 | 18.70
+{1,0.01301,35.00,1.520,0,0.4420,7.2410,49.30,7.0379,1,284.0,15.50,394.74,5.49} | 2 | 32.70
+{1,0.02498,0.00,1.890,0,0.5180,6.5400,59.70,6.2669,1,422.0,15.90,389.96,8.65} | 2 | 16.50
+{1,0.02543,55.00,3.780,0,0.4840,6.6960,56.40,5.7321,5,370.0,17.60,396.90,7.18} | 2 | 23.90
+{1,0.03049,55.00,3.780,0,0.4840,6.8740,28.10,6.4654,5,370.0,17.60,387.97,4.61} | 2 | 31.20
+{1,0.03113,0.00,4.390,0,0.4420,6.0140,48.50,8.0136,3,352.0,18.80,385.64,10.53} | 2 | 17.50
+{1,0.06162,0.00,4.390,0,0.4420,5.8980,52.30,8.0136,3,352.0,18.80,364.61,12.67} | 2 | 17.20
+{1,0.01870,85.00,4.150,0,0.4290,6.5160,27.70,8.5353,4,351.0,17.90,392.43,6.36} | 2 | 23.10
+{1,0.01501,80.00,2.010,0,0.4350,6.6350,29.70,8.3440,4,280.0,17.00,390.94,5.99} | 2 | 24.50
+{1,0.02899,40.00,1.250,0,0.4290,6.9390,34.50,8.7921,1,335.0,19.70,389.85,5.89} | 2 | 26.60
+{1,0.06211,40.00,1.250,0,0.4290,6.4900,44.40,8.7921,1,335.0,19.70,396.90,5.98} | 2 | 22.90
+{1,0.07950,60.00,1.690,0,0.4110,6.5790,35.90,10.7103,4,411.0,18.30,370.78,5.49} | 2 | 24.10
+{1,0.07244,60.00,1.690,0,0.4110,5.8840,18.50,10.7103,4,411.0,18.30,392.33,7.79} | 2 | 18.60
+{1,0.01709,90.00,2.020,0,0.4100,6.7280,36.10,12.1265,5,187.0,17.00,384.46,4.50} | 2 | 30.10
+{1,0.04301,80.00,1.910,0,0.4130,5.6630,21.90,10.5857,4,334.0,22.00,382.80,8.05} | 2 | 18.20
+{1,0.10659,80.00,1.910,0,0.4130,5.9360,19.50,10.5857,4,334.0,22.00,376.04,5.57} | 2 | 20.60
+{1,8.98296,0.00,18.100,1,0.7700,6.2120,97.40,2.1222,24,666.0,20.20,377.73,17.60} | 2 | 17.80
+{1,3.84970,0.00,18.100,1,0.7700,6.3950,91.00,2.5052,24,666.0,20.20,391.34,13.27} | 2 | 21.70
+{1,5.20177,0.00,18.100,1,0.7700,6.1270,83.40,2.7227,24,666.0,20.20,395.43,11.48} | 2 | 22.70
+{1,4.26131,0.00,18.100,0,0.7700,6.1120,81.30,2.5091,24,666.0,20.20,390.74,12.67} | 2 | 22.60
+{1,4.54192,0.00,18.100,0,0.7700,6.3980,88.00,2.5182,24,666.0,20.20,374.56,7.79} | 2 | 25.00
+{1,3.83684,0.00,18.100,0,0.7700,6.2510,91.10,2.2955,24,666.0,20.20,350.65,14.19} | 2 | 19.90
+{1,3.67822,0.00,18.100,0,0.7700,5.3620,96.20,2.1036,24,666.0,20.20,380.79,10.19} | 2 | 20.80
+{1,4.22239,0.00,18.100,1,0.7700,5.8030,89.00,1.9047,24,666.0,20.20,353.04,14.64} | 2 | 16.80
+{1,3.47428,0.00,18.100,1,0.7180,8.7800,82.90,1.9047,24,666.0,20.20,354.55,5.29} | 2 | 21.90
+{1,4.55587,0.00,18.100,0,0.7180,3.5610,87.90,1.6132,24,666.0,20.20,354.70,7.12} | 2 | 27.50
+{1,3.69695,0.00,18.100,0,0.7180,4.9630,91.40,1.7523,24,666.0,20.20,316.03,14.00} | 2 | 21.90
+{1,13.52220,0.00,18.100,0,0.6310,3.8630,100.00,1.5106,24,666.0,20.20,131.42,13.33} | 2 | 23.10
+{1,4.89822,0.00,18.100,0,0.6310,4.9700,100.00,1.3325,24,666.0,20.20,375.52,3.26} | 2 | 50.00
+{1,5.66998,0.00,18.100,1,0.6310,6.6830,96.80,1.3567,24,666.0,20.20,375.33,3.73} | 2 | 50.00
+{1,6.53876,0.00,18.100,1,0.6310,7.0160,97.50,1.2024,24,666.0,20.20,392.05,2.96} | 2 | 50.00
+{1,9.23230,0.00,18.100,0,0.6310,6.2160,100.00,1.1691,24,666.0,20.20,366.15,9.53} | 2 | 50.00
+{1,8.26725,0.00,18.100,1,0.6680,5.8750,89.60,1.1296,24,666.0,20.20,347.88,8.88} | 2 | 50.00
+{1,11.10810,0.00,18.100,0,0.6680,4.9060,100.00,1.1742,24,666.0,20.20,396.90,34.77} | 2 | 13.80
+{1,18.49820,0.00,18.100,0,0.6680,4.1380,100.00,1.1370,24,666.0,20.20,396.90,37.97} | 2 | 13.80
+{1,19.60910,0.00,18.100,0,0.6710,7.3130,97.90,1.3163,24,666.0,20.20,396.90,13.44} | 2 | 15.00
+{1,15.28800,0.00,18.100,0,0.6710,6.6490,93.30,1.3449,24,666.0,20.20,363.02,23.24} | 2 | 13.90
+{1,9.82349,0.00,18.100,0,0.6710,6.7940,98.80,1.3580,24,666.0,20.20,396.90,21.24} | 2 | 13.30
+{1,23.64820,0.00,18.100,0,0.6710,6.3800,96.20,1.3861,24,666.0,20.20,396.90,23.69} | 2 | 13.10
+{1,17.86670,0.00,18.100,0,0.6710,6.2230,100.00,1.3861,24,666.0,20.20,393.74,21.78} | 2 | 10.20
+{1,88.97620,0.00,18.100,0,0.6710,6.9680,91.90,1.4165,24,666.0,20.20,396.90,17.21} | 2 | 10.40
+{1,15.87440,0.00,18.100,0,0.6710,6.5450,99.10,1.5192,24,666.0,20.20,396.90,21.08} | 2 | 10.90
+{1,9.18702,0.00,18.100,0,0.7000,5.5360,100.00,1.5804,24,666.0,20.20,396.90,23.60} | 2 | 11.30
+{1,7.99248,0.00,18.100,0,0.7000,5.5200,100.00,1.5331,24,666.0,20.20,396.90,24.56} | 2 | 12.30
+{1,20.08490,0.00,18.100,0,0.7000,4.3680,91.20,1.4395,24,666.0,20.20,285.83,30.63} | 2 | 8.80
+{1,16.81180,0.00,18.100,0,0.7000,5.2770,98.10,1.4261,24,666.0,20.20,396.90,30.81} | 2 | 7.20
+{1,24.39380,0.00,18.100,0,0.7000,4.6520,100.00,1.4672,24,666.0,20.20,396.90,28.28} | 2 | 10.50
+{1,22.59710,0.00,18.100,0,0.7000,5.0000,89.50,1.5184,24,666.0,20.20,396.90,31.99} | 2 | 7.40
+{1,14.33370,0.00,18.100,0,0.7000,4.8800,100.00,1.5895,24,666.0,20.20,372.92,30.62} | 2 | 10.20
+{1,8.15174,0.00,18.100,0,0.7000,5.3900,98.90,1.7281,24,666.0,20.20,396.90,20.85} | 2 | 11.50
+{1,6.96215,0.00,18.100,0,0.7000,5.7130,97.00,1.9265,24,666.0,20.20,394.43,17.11} | 2 | 15.10
+{1,5.29305,0.00,18.100,0,0.7000,6.0510,82.50,2.1678,24,666.0,20.20,378.38,18.76} | 2 | 23.20
+{1,11.57790,0.00,18.100,0,0.7000,5.0360,97.00,1.7700,24,666.0,20.20,396.90,25.68} | 2 | 9.70
+{1,8.64476,0.00,18.100,0,0.6930,6.1930,92.60,1.7912,24,666.0,20.20,396.90,15.17} | 2 | 13.80
+{1,13.35980,0.00,18.100,0,0.6930,5.8870,94.70,1.7821,24,666.0,20.20,396.90,16.35} | 2 | 12.70
+{1,8.71675,0.00,18.100,0,0.6930,6.4710,98.80,1.7257,24,666.0,20.20,391.98,17.12} | 2 | 13.10
+{1,5.87205,0.00,18.100,0,0.6930,6.4050,96.00,1.6768,24,666.0,20.20,396.90,19.37} | 2 | 12.50
+{1,7.67202,0.00,18.100,0,0.6930,5.7470,98.90,1.6334,24,666.0,20.20,393.10,19.92} | 2 | 8.50
+{1,38.35180,0.00,18.100,0,0.6930,5.4530,100.00,1.4896,24,666.0,20.20,396.90,30.59} | 2 | 5.00
+{1,9.91655,0.00,18.100,0,0.6930,5.8520,77.80,1.5004,24,666.0,20.20,338.16,29.97} | 2 | 6.30
+{1,25.04610,0.00,18.100,0,0.6930,5.9870,100.00,1.5888,24,666.0,20.20,396.90,26.77} | 2 | 5.60
+{1,14.23620,0.00,18.100,0,0.6930,6.3430,100.00,1.5741,24,666.0,20.20,396.90,20.32} | 2 | 7.20
+{1,9.59571,0.00,18.100,0,0.6930,6.4040,100.00,1.6390,24,666.0,20.20,376.11,20.31} | 2 | 12.10
+{1,24.80170,0.00,18.100,0,0.6930,5.3490,96.00,1.7028,24,666.0,20.20,396.90,19.77} | 2 | 8.30
+{1,41.52920,0.00,18.100,0,0.6930,5.5310,85.40,1.6074,24,666.0,20.20,329.46,27.38} | 2 | 8.50
+{1,67.92080,0.00,18.100,0,0.6930,5.6830,100.00,1.4254,24,666.0,20.20,384.97,22.98} | 2 | 5.00
+{1,20.71620,0.00,18.100,0,0.6590,4.1380,100.00,1.1781,24,666.0,20.20,370.22,23.34} | 2 | 11.90
+{1,11.95110,0.00,18.100,0,0.6590,5.6080,100.00,1.2852,24,666.0,20.20,332.09,12.13} | 2 | 27.90
+{1,7.40389,0.00,18.100,0,0.5970,5.6170,97.90,1.4547,24,666.0,20.20,314.64,26.40} | 2 | 17.20
+{1,14.43830,0.00,18.100,0,0.5970,6.8520,100.00,1.4655,24,666.0,20.20,179.36,19.78} | 2 | 27.50
+{1,51.13580,0.00,18.100,0,0.5970,5.7570,100.00,1.4130,24,666.0,20.20,2.60,10.11} | 2 | 15.00
+{1,14.05070,0.00,18.100,0,0.5970,6.6570,100.00,1.5275,24,666.0,20.20,35.05,21.22} | 2 | 17.20
+{1,18.81100,0.00,18.100,0,0.5970,4.6280,100.00,1.5539,24,666.0,20.20,28.79,34.37} | 2 | 17.90
+{1,28.65580,0.00,18.100,0,0.5970,5.1550,100.00,1.5894,24,666.0,20.20,210.97,20.08} | 2 | 16.30
+{1,45.74610,0.00,18.100,0,0.6930,4.5190,100.00,1.6582,24,666.0,20.20,88.27,36.98} | 2 | 7.00
+{1,18.08460,0.00,18.100,0,0.6790,6.4340,100.00,1.8347,24,666.0,20.20,27.25,29.05} | 2 | 7.20
+{1,10.83420,0.00,18.100,0,0.6790,6.7820,90.80,1.8195,24,666.0,20.20,21.57,25.79} | 2 | 7.50
+{1,25.94060,0.00,18.100,0,0.6790,5.3040,89.10,1.6475,24,666.0,20.20,127.36,26.64} | 2 | 10.40
+{1,73.53410,0.00,18.100,0,0.6790,5.9570,100.00,1.8026,24,666.0,20.20,16.45,20.62} | 2 | 8.80
+{1,11.81230,0.00,18.100,0,0.7180,6.8240,76.50,1.7940,24,666.0,20.20,48.45,22.74} | 2 | 8.40
+{1,11.08740,0.00,18.100,0,0.7180,6.4110,100.00,1.8589,24,666.0,20.20,318.75,15.02} | 2 | 16.70
+{1,7.02259,0.00,18.100,0,0.7180,6.0060,95.30,1.8746,24,666.0,20.20,319.98,15.70} | 2 | 14.20
+{1,12.04820,0.00,18.100,0,0.6140,5.6480,87.60,1.9512,24,666.0,20.20,291.55,14.10} | 2 | 20.80
+{1,7.05042,0.00,18.100,0,0.6140,6.1030,85.10,2.0218,24,666.0,20.20,2.52,23.29} | 2 | 13.40
+{1,8.79212,0.00,18.100,0,0.5840,5.5650,70.60,2.0635,24,666.0,20.20,3.65,17.16} | 2 | 11.70
+{1,15.86030,0.00,18.100,0,0.6790,5.8960,95.40,1.9096,24,666.0,20.20,7.68,24.39} | 2 | 8.30
+{1,12.24720,0.00,18.100,0,0.5840,5.8370,59.70,1.9976,24,666.0,20.20,24.65,15.69} | 2 | 10.20
+{1,37.66190,0.00,18.100,0,0.6790,6.2020,78.70,1.8629,24,666.0,20.20,18.82,14.52} | 2 | 10.90
+{1,7.36711,0.00,18.100,0,0.6790,6.1930,78.10,1.9356,24,666.0,20.20,96.73,21.52} | 2 | 11.00
+{1,9.33889,0.00,18.100,0,0.6790,6.3800,95.60,1.9682,24,666.0,20.20,60.72,24.08} | 2 | 9.50
+{1,8.49213,0.00,18.100,0,0.5840,6.3480,86.10,2.0527,24,666.0,20.20,83.45,17.64} | 2 | 14.50
+{1,10.06230,0.00,18.100,0,0.5840,6.8330,94.30,2.0882,24,666.0,20.20,81.33,19.69} | 2 | 14.10
+{1,6.44405,0.00,18.100,0,0.5840,6.4250,74.80,2.2004,24,666.0,20.20,97.95,12.03} | 2 | 16.10
+{1,5.58107,0.00,18.100,0,0.7130,6.4360,87.90,2.3158,24,666.0,20.20,100.19,16.22} | 2 | 14.30
+{1,13.91340,0.00,18.100,0,0.7130,6.2080,95.00,2.2222,24,666.0,20.20,100.63,15.17} | 2 | 11.70
+{1,11.16040,0.00,18.100,0,0.7400,6.6290,94.60,2.1247,24,666.0,20.20,109.85,23.27} | 2 | 13.40
+{1,14.42080,0.00,18.100,0,0.7400,6.4610,93.30,2.0026,24,666.0,20.20,27.49,18.05} | 2 | 9.60
+{1,15.17720,0.00,18.100,0,0.7400,6.1520,100.00,1.9142,24,666.0,20.20,9.32,26.45} | 2 | 8.70
+{1,13.67810,0.00,18.100,0,0.7400,5.9350,87.90,1.8206,24,666.0,20.20,68.95,34.02} | 2 | 8.40
+{1,9.39063,0.00,18.100,0,0.7400,5.6270,93.90,1.8172,24,666.0,20.20,396.90,22.88} | 2 | 12.80
+{1,22.05110,0.00,18.100,0,0.7400,5.8180,92.40,1.8662,24,666.0,20.20,391.45,22.11} | 2 | 10.50
+{1,9.72418,0.00,18.100,0,0.7400,6.4060,97.20,2.0651,24,666.0,20.20,385.96,19.52} | 2 | 17.10
+{1,5.66637,0.00,18.100,0,0.7400,6.2190,100.00,2.0048,24,666.0,20.20,395.69,16.59} | 2 | 18.40
+{1,9.96654,0.00,18.100,0,0.7400,6.4850,100.00,1.9784,24,666.0,20.20,386.73,18.85} | 2 | 15.40
+{1,12.80230,0.00,18.100,0,0.7400,5.8540,96.60,1.8956,24,666.0,20.20,240.52,23.79} | 2 | 10.80
+{1,10.67180,0.00,18.100,0,0.7400,6.4590,94.80,1.9879,24,666.0,20.20,43.06,23.98} | 2 | 11.80
+{1,6.28807,0.00,18.100,0,0.7400,6.3410,96.40,2.0720,24,666.0,20.20,318.01,17.79} | 2 | 14.90
+{1,9.92485,0.00,18.100,0,0.7400,6.2510,96.60,2.1980,24,666.0,20.20,388.52,16.44} | 2 | 12.60
+{1,9.32909,0.00,18.100,0,0.7130,6.1850,98.70,2.2616,24,666.0,20.20,396.90,18.13} | 2 | 14.10
+{1,7.52601,0.00,18.100,0,0.7130,6.4170,98.30,2.1850,24,666.0,20.20,304.21,19.31} | 2 | 13.00
+{1,6.71772,0.00,18.100,0,0.7130,6.7490,92.60,2.3236,24,666.0,20.20,0.32,17.44} | 2 | 13.40
+{1,5.44114,0.00,18.100,0,0.7130,6.6550,98.20,2.3552,24,666.0,20.20,355.29,17.73} | 2 | 15.20
+{1,5.09017,0.00,18.100,0,0.7130,6.2970,91.80,2.3682,24,666.0,20.20,385.09,17.27} | 2 | 16.10
+{1,8.24809,0.00,18.100,0,0.7130,7.3930,99.30,2.4527,24,666.0,20.20,375.87,16.74} | 2 | 17.80
+{1,9.51363,0.00,18.100,0,0.7130,6.7280,94.10,2.4961,24,666.0,20.20,6.68,18.71} | 2 | 14.90
+{1,4.75237,0.00,18.100,0,0.7130,6.5250,86.50,2.4358,24,666.0,20.20,50.92,18.13} | 2 | 14.10
+{1,4.66883,0.00,18.100,0,0.7130,5.9760,87.90,2.5806,24,666.0,20.20,10.48,19.01} | 2 | 12.70
+{1,8.20058,0.00,18.100,0,0.7130,5.9360,80.30,2.7792,24,666.0,20.20,3.50,16.94} | 2 | 13.50
+{1,7.75223,0.00,18.100,0,0.7130,6.3010,83.70,2.7831,24,666.0,20.20,272.21,16.23} | 2 | 14.90
+{1,6.80117,0.00,18.100,0,0.7130,6.0810,84.40,2.7175,24,666.0,20.20,396.90,14.70} | 2 | 20.00
+{1,4.81213,0.00,18.100,0,0.7130,6.7010,90.00,2.5975,24,666.0,20.20,255.23,16.42} | 2 | 16.40
+{1,3.69311,0.00,18.100,0,0.7130,6.3760,88.40,2.5671,24,666.0,20.20,391.43,14.65} | 2 | 17.70
+{1,6.65492,0.00,18.100,0,0.7130,6.3170,83.00,2.7344,24,666.0,20.20,396.90,13.99} | 2 | 19.50
+{1,5.82115,0.00,18.100,0,0.7130,6.5130,89.90,2.8016,24,666.0,20.20,393.82,10.29} | 2 | 20.20
+{1,7.83932,0.00,18.100,0,0.6550,6.2090,65.40,2.9634,24,666.0,20.20,396.90,13.22} | 2 | 21.40
+{1,3.16360,0.00,18.100,0,0.6550,5.7590,48.20,3.0665,24,666.0,20.20,334.40,14.13} | 2 | 19.90
+{1,3.77498,0.00,18.100,0,0.6550,5.9520,84.70,2.8715,24,666.0,20.20,22.01,17.15} | 2 | 19.00
+{1,4.42228,0.00,18.100,0,0.5840,6.0030,94.50,2.5403,24,666.0,20.20,331.29,21.32} | 2 | 19.10
+{1,15.57570,0.00,18.100,0,0.5800,5.9260,71.00,2.9084,24,666.0,20.20,368.74,18.13} | 2 | 19.10
+{1,13.07510,0.00,18.100,0,0.5800,5.7130,56.70,2.8237,24,666.0,20.20,396.90,14.76} | 2 | 20.10
+{1,4.34879,0.00,18.100,0,0.5800,6.1670,84.00,3.0334,24,666.0,20.20,396.90,16.29} | 2 | 19.90
+{1,4.03841,0.00,18.100,0,0.5320,6.2290,90.70,3.0993,24,666.0,20.20,395.33,12.87} | 2 | 19.60
+{1,3.56868,0.00,18.100,0,0.5800,6.4370,75.00,2.8965,24,666.0,20.20,393.37,14.36} | 2 | 23.20
+{1,4.64689,0.00,18.100,0,0.6140,6.9800,67.60,2.5329,24,666.0,20.20,374.68,11.66} | 2 | 29.80
+{1,8.05579,0.00,18.100,0,0.5840,5.4270,95.40,2.4298,24,666.0,20.20,352.58,18.14} | 2 | 13.80
+{1,6.39312,0.00,18.100,0,0.5840,6.1620,97.40,2.2060,24,666.0,20.20,302.76,24.10} | 2 | 13.30
+{1,4.87141,0.00,18.100,0,0.6140,6.4840,93.60,2.3053,24,666.0,20.20,396.21,18.68} | 2 | 16.70
+{1,15.02340,0.00,18.100,0,0.6140,5.3040,97.30,2.1007,24,666.0,20.20,349.48,24.91} | 2 | 12.00
+{1,10.23300,0.00,18.100,0,0.6140,6.1850,96.70,2.1705,24,666.0,20.20,379.70,18.03} | 2 | 14.60
+{1,14.33370,0.00,18.100,0,0.6140,6.2290,88.00,1.9512,24,666.0,20.20,383.32,13.11} | 2 | 21.40
+{1,5.82401,0.00,18.100,0,0.5320,6.2420,64.70,3.4242,24,666.0,20.20,396.90,10.74} | 2 | 23.00
+{1,5.70818,0.00,18.100,0,0.5320,6.7500,74.90,3.3317,24,666.0,20.20,393.07,7.74} | 2 | 23.70
+{1,5.73116,0.00,18.100,0,0.5320,7.0610,77.00,3.4106,24,666.0,20.20,395.28,7.01} | 2 | 25.00
+{1,2.81838,0.00,18.100,0,0.5320,5.7620,40.30,4.0983,24,666.0,20.20,392.92,10.42} | 2 | 21.80
+{1,2.37857,0.00,18.100,0,0.5830,5.8710,41.90,3.7240,24,666.0,20.20,370.73,13.34} | 2 | 20.60
+{1,3.67367,0.00,18.100,0,0.5830,6.3120,51.90,3.9917,24,666.0,20.20,388.62,10.58} | 2 | 21.20
+{1,5.69175,0.00,18.100,0,0.5830,6.1140,79.80,3.5459,24,666.0,20.20,392.68,14.98} | 2 | 19.10
+{1,4.83567,0.00,18.100,0,0.5830,5.9050,53.20,3.1523,24,666.0,20.20,388.22,11.45} | 2 | 20.60
+{1,0.15086,0.00,27.740,0,0.6090,5.4540,92.70,1.8209,4,711.0,20.10,395.09,18.06} | 2 | 15.20
+{1,0.18337,0.00,27.740,0,0.6090,5.4140,98.30,1.7554,4,711.0,20.10,344.05,23.97} | 2 | 7.00
+{1,0.20746,0.00,27.740,0,0.6090,5.0930,98.00,1.8226,4,711.0,20.10,318.43,29.68} | 2 | 8.10
+{1,0.10574,0.00,27.740,0,0.6090,5.9830,98.80,1.8681,4,711.0,20.10,390.11,18.07} | 2 | 13.60
+{1,0.11132,0.00,27.740,0,0.6090,5.9830,83.50,2.1099,4,711.0,20.10,396.90,13.35} | 2 | 20.10
+{1,0.17331,0.00,9.690,0,0.5850,5.7070,54.00,2.3817,6,391.0,19.20,396.90,12.01} | 2 | 21.80
+{1,0.27957,0.00,9.690,0,0.5850,5.9260,42.60,2.3817,6,391.0,19.20,396.90,13.59} | 2 | 24.50
+{1,0.17899,0.00,9.690,0,0.5850,5.6700,28.80,2.7986,6,391.0,19.20,393.29,17.60} | 2 | 23.10
+{1,0.28960,0.00,9.690,0,0.5850,5.3900,72.90,2.7986,6,391.0,19.20,396.90,21.14} | 2 | 19.70
+{1,0.26838,0.00,9.690,0,0.5850,5.7940,70.60,2.8927,6,391.0,19.20,396.90,14.10} | 2 | 18.30
+{1,0.23912,0.00,9.690,0,0.5850,6.0190,65.30,2.4091,6,391.0,19.20,396.90,12.92} | 2 | 21.20
+{1,0.17783,0.00,9.690,0,0.5850,5.5690,73.50,2.3999,6,391.0,19.20,395.77,15.10} | 2 | 17.50
+{1,0.22438,0.00,9.690,0,0.5850,6.0270,79.70,2.4982,6,391.0,19.20,396.90,14.33} | 2 | 16.80
+{1,0.06263,0.00,11.930,0,0.5730,6.5930,69.10,2.4786,1,273.0,21.00,391.99,9.67} | 2 | 22.40
+{1,0.04527,0.00,11.930,0,0.5730,6.1200,76.70,2.2875,1,273.0,21.00,396.90,9.08} | 2 | 20.60
+{1,0.06076,0.00,11.930,0,0.5730,6.9760,91.00,2.1675,1,273.0,21.00,396.90,5.64} | 2 | 23.90
+{1,0.10959,0.00,11.930,0,0.5730,6.7940,89.30,2.3889,1,273.0,21.00,393.45,6.48} | 2 | 22.00
+{1,0.04741,0.00,11.930,0,0.5730,6.0300,80.80,2.5050,1,273.0,21.00,396.90,7.88} | 2 | 11.90
+\.
+
+-- Normalize the columns
+CREATE TEMPORARY TABLE maxs as(
+ SELECT
+ max(x[1]) m1,
+ max(x[2]) m2,
+ max(x[3]) m3,
+ max(x[4]) m4,
+ max(x[5]) m5,
+ max(x[6]) m6,
+ max(x[7]) m7,
+ max(x[8]) m8,
+ max(x[9]) m9,
+ max(x[10]) m10,
+ max(x[11]) m11,
+ max(x[12]) m12,
+ max(x[13]) m13,
+ max(x[14]) m14
+ from lin_housing_wi
+);
+
+CREATE TABLE lin_housing_wi_scaled AS
+SELECT ARRAY[
+ x[1]/(SELECT m1 from maxs),
+ x[2]/(SELECT m2 from maxs),
+ x[3]/(SELECT m3 from maxs),
+ x[4]/(SELECT m4 from maxs),
+ x[5]/(SELECT m5 from maxs),
+ x[6]/(SELECT m6 from maxs),
+ x[7]/(SELECT m7 from maxs),
+ x[8]/(SELECT m8 from maxs),
+ x[9]/(SELECT m9 from maxs),
+ x[10]/(SELECT m10 from maxs),
+ x[11]/(SELECT m11 from maxs),
+ x[12]/(SELECT m12 from maxs),
+ x[13]/(SELECT m13 from maxs),
+ x[14]/(SELECT m14 from maxs)] as x,
+ id,y
+FROM lin_housing_wi;
+
+DROP TABLE IF EXISTS maxs;
+DROP TABLE IF EXISTS mlp_regress;
+
+
+-- Convergence varies for regression
+
+SELECT setseed(0);
+SELECT mlp_regression(
+ 'lin_housing_wi_scaled', -- Source table
+ 'mlp_regress', -- Desination table
+ 'x', -- Input features
+ 'y', -- Dependent variable
+ ARRAY[5,5], -- Number of units per layer
+ 'step_size=0.007,
+ n_iterations=300,
+ tolerance=0',
+ 'sigmoid');
+
+
+SELECT assert(
+ loss < 5000, -- Corresponds to RMSE of sqrt(5000/506) = 3.14.
+ 'MLP: Loss is too high (> 0.1). Wrong result.'
+) FROM mlp_regress;
+
+DROP TABLE IF EXISTS lin_housing_wi_scaled;
--- End diff --
Might it make sense to cleanup other resources (tables) created/used during the test run? I manually ran the mlp tests and noticed the following resources are not removed.
```
madlibtest=# \d
List of relations
Schema | Name | Type | Owner
--------+-----------------------+----------+---------
public | iris_data | table | eespino
public | lin_housing_wi | table | eespino
public | lin_housing_wi_id_seq | sequence | eespino
public | mlp_class | table | eespino
public | mlp_class_summary | table | eespino
public | mlp_regress | table | eespino
public | mlp_regress_summary | table | eespino
(7 rows)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by edespino <gi...@git.apache.org>.
Github user edespino commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r127035041
--- Diff: src/ports/postgres/modules/convex/mlp.sql_in ---
@@ -0,0 +1,736 @@
+/* ----------------------------------------------------------------------- *//**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ *
+ * @file mlp.sql_in
+ *
+ * @brief SQL functions for multi-layer perceptron
+ * @date June 2012
+ *
+ *
+ *//* ----------------------------------------------------------------------- */
+
+m4_include(`SQLCommon.m4')
+
+/**
+@addtogroup grp_mlp
+
+<div class="toc"><b>Contents</b><ul>
+<li class="level1"><a href="#mlp_classification">Classification</a></li>
+<li class="level1"><a href="#mlp_regression">Regression</a></li>
+<li class="level1"><a href="#optimization_params">Optimization Parameters</a></li>
+<li class="level1"><a href="#predict">Prediction Functions/a></li>
+<li class="level1"><a href="#example">Examples</a></li>
+<li class="level1"><a href="#background">Technical Background</a></li>
+<li class="level1"><a href="#literature">Literature</a></li>
+<li class="level1"><a href="#related">Related Topics</a></li>
+</ul></div>
+
+Multi Layer Perceptron (MLP) is a model for regression and
+classification
+
+Also called "vanilla neural networks", they consist of several
+fully connected hidden layers with non-linear activation
+functions. In the case of classification, the final layer of the
+neural net has as many nodes as classes, and the output of the
+neural net can be interpreted as the probability of a given input
+feature belonging to a specific class.
+
+
+@brief Solves classification and regression problems with several
+fully connected layers and nonlinear activation functions.
+
+@anchor mlp_classification
+@par Classification Training Function
+The mlp classification training function has the following format:
+<pre class="syntax">
+mlp_classification(
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+\b Arguments
+<DL class="arglist">
+ <DT>source_table</DT>
+ <DD>TEXT. Name of the table containing the training data.</DD>
+
+ <DT>model_table</DT>
+ <DD>TEXT. Name of the output table containing the model. Details of the output
+ tables are provided below.
+ </DD>
+
+ <DT>independent_varname</DT>
+ <DD>TEXT. Expression list to evaluate for the
+ independent variables. An intercept variable should not be included as part
+ of this expression. Please note that expression should be able to be cast
+ to DOUBLE PRECISION[].
+ </DD>
+
+ <DT>dependent_varname</DT>
+ <DD> TEXT. Name of the dependent variable column. For classification, supported types are:
+ text, varchar, character varying, char, character
+ integer, smallint, bigint, and boolean. </DD>
+
+ <DT>hidden_layer_sizes</DT>
+ <DD>INTEGER[], default: ARRAY[].
+ The number of neurons in each hidden layer. The length of this array will
+ determine the number of hidden layers. Empty for no hidden layers.
+ </DD>
+
+
+ <DT>optimizer_params (optional)</DT>
+ <DD>TEXT, default: NULL.
+ Parameters for optimization in a comma-separated string
+ of key-value pairs. See the description below for details.
+ </DD>
+
+ <DT>activation</DT>
+ <DD>TEXT, default: 'sigmoid'.
+ Activation function. Currently three functions are supported: 'sigmoid' (default),
+ 'relu', and 'tanh'. The text can be any subset of the three
+ strings; for e.g., activation='s' will use the sigmoid activation.
+ </DD>
+</DL>
+
+<b>Output tables</b>
+<br>
+ The model table produced by mlp contains the following columns:
+ <table class="output">
+ <tr>
+ <th>coeffs</th>
+ <td>FLOAT8[]. Flat array containing the weights of the neural net</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>INTEGER. Number of iterations completed by stochastic gradient descent
+ algorithm. The algorithm either converged in this number of iterations
+ or hit the maximum number specified in the optimization parameters. </td>
+ </tr>
+ <tr>
+ <th>loss</th>
+ <td>FLOAT8. The cross entropy over the training data.
+ See Technical Background section below for more details.</td>
+ </tr>
+ </table>
+
+
+A summary table named \<model_table\>_summary is also created, which has the following columns:
+ <table class="output">
+ <tr>
+ <th>source_table</th>
+ <td>The source table.</td>
+ </tr>
+ <tr>
+ <th>dependent_varname</th>
+ <td>The dependent variable.</td>
+ </tr>
+ <tr>
+ <th>independent_varname</th>
+ <td>The independent variables.</td>
+ </tr>
+ <tr>
+ <th>tolerance</th>
+ <td>The tolerance as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>step_size</th>
+ <td>The step size as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>The number of iterations run</td>
+ </tr>
+ <tr>
+ <th>n_tries</th>
+ <td>The number of tries as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>layer_sizes</th>
+ <td>The number of units in each layer including the input and output layer.</td>
+ </tr>
+ <tr>
+ <th>activation_function</th>
+ <td>The activation function.</td>
+ </tr>
+ <tr>
+ <th>is_classification</th>
+ <td>True if the model was trained for classification, False if it was trained
+ for regression</td>
+ </tr>
+ <tr>
+ <th>classes</th>
+ <td>The classes which were trained against (empty for regression)</td>
+ </tr>
+
+ </table>
+
+
+@anchor mlp_regression
+@par Regression Training Function
+The mlp regression training function has the following format:
+<pre class="syntax">
+mlp_regression(source_table,
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+
+\b Arguments
+
+Specifications for regression are largely the same as for classification. In the
+model table, the loss will refer to mean square error instead of cross entropy. In the
+summary table, there is classes column. The following
+arguments have specifications which differ from mlp_classification:
+<DL class="arglist">
+<DT>dependent_varname</DT>
+ <DD>TEXT. Name of the dependent variable column.
+ For regression supported types are any numeric type, or array
+ or numeric types (for multiple regression).
+ </DD>
+</DL>
+
+
+@anchor optimizer_params
+@par Optimizer Parameters
+Parameters in this section are supplied in the \e optimizer_params argument as a string
+containing a comma-delimited list of name-value pairs. All of these named
+parameters are optional, and their order does not matter. You must use the
+format "<param_name> = <value>" to specify the value of a parameter, otherwise
+the parameter is ignored.
+
+
+<pre class="syntax">
+ 'step_size = <value>,
+ n_iterations = <value>,
+ n_tries = <value>,
+ tolerance = <value>'
+</pre>
+\b Optimizer Parameters
+<DL class="arglist">
+
+<DT>step_size</dt>
+<DD>Default: [0.001].
+Also known as the learning rate. A small value is usually desirable to
+ensure convergence, while a large value provides more room for progress during
+training. Since the best value depends on the condition number of the data, in
+practice one often tunes this parameter.
+</DD>
+
+
+<DT>n_iterations</dt>
+<DD>Default: [100]. The maximum number of iterations allowed.
+</DD>
+<DT>n_tries</dt>
+<DD>Default: [1]. Number of times to retrain the network with randomly initialized
+weights
+</DD>
+
+<DT>tolerance</dt>
+<DD>Default: 0.001. The criterion to end iterations. The training stops whenever
+<the difference between the training models of two consecutive iterations is
+<smaller than \e tolerance or the iteration number is larger than \e max_iter.
+</DD>
+
+</DL>
+
+@anchor predict
+@par Prediction Function
+Used to generate predictions given a previously trained model on novel data.
+The same syntax is used for classification, and regression.
+<pre class="syntax">
+mlp_predict(model_table,
+ data_table,
+ id_col_name,
+ output_table,
+ pred_type)
+</pre>
+
+\b Arguments
+<DL class="arglist">
+ <DT>model_table</DT>
+ <DD>TEXT. Model table produced by the training function.</DD>
+
+ <DT>data_table</DT>
+ <DD>TEXT. Name of the table containing the data for prediction. This table is expected
+ to contain the same input features that were used during training. The table should
+ also contain id_col_name used for identifying each row.</DD>
+
+ <DT>id_col_name</DT>
+ <DD>TEXT. The name of the id column in the input table.</DD>
+
+ <DT>output_table</DT>
+ <DD>TEXT. Name of the table where output predictions are written. If this
+table name is already in use, then an error is returned. Table contains:</DD>
+ <table class="output">
+ <tr>
+ <th>id</th>
+ <td>Gives the 'id' for each prediction, corresponding to each row from the data_table.</td>
+ </tr>
+ <tr>
+ <th>estimated_<COL_NAME></th>
+ <td>
+ (For pred_type='response') The estimated class
+ for classification or value for regression, where
+ <COL_NAME> is the name of the column to be
+ predicted from training data
+ </td>
+ </tr>
+ <tr>
+ <th>prob_<CLASS></th>
+ <td>
+ (For pred_type='prob' for classification) The
+ probability of a given class <CLASS> as given by
+ softmax. There will be one column for each class
+ in the training data.
+ </td>
+ </tr>
+
+
+ <DT>pred_type</DT>
+ <DD>TEXT.
+
+the type of output requested:
+'response' gives the actual prediction,
+'prob' gives the probability of each class.
+for regression, only type='response' is defined.
+The name of the id column in the input table.</DD>
+</DL>
+</table>
+
+@anchor example
+@par Examples
+-# Create an input data set.
+<pre class="example">
+CREATE TABLE iris_data(
+ id integer,
+ attributes numeric[],
+ class_text varchar,
+ class integer
+);
+INSERT INTO iris_data VALUES
+(1,ARRAY[5.1,3.5,1.4,0.2],'Iris-setosa',1),
+(2,ARRAY[4.9,3.0,1.4,0.2],'Iris-setosa',1),
+(3,ARRAY[4.7,3.2,1.3,0.2],'Iris-setosa',1),
+(4,ARRAY[4.6,3.1,1.5,0.2],'Iris-setosa',1),
+(5,ARRAY[5.0,3.6,1.4,0.2],'Iris-setosa',1),
+(6,ARRAY[5.4,3.9,1.7,0.4],'Iris-setosa',1),
+(7,ARRAY[4.6,3.4,1.4,0.3],'Iris-setosa',1),
+(8,ARRAY[5.0,3.4,1.5,0.2],'Iris-setosa',1),
+(9,ARRAY[4.4,2.9,1.4,0.2],'Iris-setosa',1),
+(10,ARRAY[4.9,3.1,1.5,0.1],'Iris-setosa',1),
+(11,ARRAY[7.0,3.2,4.7,1.4],'Iris-versicolor',2),
+(12,ARRAY[6.4,3.2,4.5,1.5],'Iris-versicolor',2),
+(13,ARRAY[6.9,3.1,4.9,1.5],'Iris-versicolor',2),
+(14,ARRAY[5.5,2.3,4.0,1.3],'Iris-versicolor',2),
+(15,ARRAY[6.5,2.8,4.6,1.5],'Iris-versicolor',2),
+(16,ARRAY[5.7,2.8,4.5,1.3],'Iris-versicolor',2),
+(17,ARRAY[6.3,3.3,4.7,1.6],'Iris-versicolor',2),
+(18,ARRAY[4.9,2.4,3.3,1.0],'Iris-versicolor',2),
+(19,ARRAY[6.6,2.9,4.6,1.3],'Iris-versicolor',2),
+(20,ARRAY[5.2,2.7,3.9,1.4],'Iris-versicolor',2);
+</pre>
+-# Generate a multilayer perceptron with a single hidden layer of 5 units.
+Use the attributes column as the independent variables, and use the class
+column as the classification. Set the tolerance to 0 so that 5000
+iterations will be run. Use a hyperbolic tangent activation function.
+The model will be written to mlp_result.
+<pre class="example">
+SELECT madlib.mlp_classification(
+ 'iris_data_2', -- Source table
+ 'mlp_model', -- Destination table
+ 'attributes', -- Input features
+ 'class_text', -- Label
+ ARRAY[5], -- Number of units per layer
+ 'step_size=0.003,
+ n_iterations=5000,
+ tolerance=0', -- Optimizer params
+ 'tanh'); -- Activation function
+</pre>
+-# View the result for the model.
+<pre class="example">
+-- Set extended display on for easier reading of output
+\\x ON
+-- Neural net Initialization is non-deterministic, so your results may vary
+SELECT * FROM mlp_model;
+</pre>
+Result:
+<pre class="result">
+-[ RECORD 1 ]--+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+coeff | {1,1,1,1,1,0.136374930803,0.188739676875,0.662387810001,-1.03381622734,-0.469961067046,0.0614006983397,0.0811504589436,0.299008228258,-0.47391918521,-0.215098143699,0.10519213944,0.145844617525,0.511683525606,-0.800215552382,-0.36417142683,0.120751709056,0.167531106521,0.587074895969,-0.916946198095,-0.417055067449,0.0539541885146,0.0694359704131,0.262598585854,-0.419234805076,-0.189915344282,1,1,1,1,1,1,0.105645702152,1.46247470474,0.484457903226,0.965962824478,1.19361986431,0.419805760087,-0.105696503487,-1.46245956666,-0.484427811691,-0.965730981426,-1.19365280555,-0.419973628863}
+loss | 0.0184092375519
+num_iterations | 5000
+</pre>
+-# Next train a regression example. First create some test data. This dataset
+contains housing prices data.
+<pre class="example">
+CREATE TABLE lin_housing (id serial, x float8[], grp_by_col int, y float8);
+COPY lin_housing (x, grp_by_col, y) FROM STDIN NULL '?' DELIMITER '|';
+{1,0.00632,18.00,2.310,0,0.5380,6.5750,65.20,4.0900,1,296.0,15.30,396.90,4.98} | 1 | 24.00
+{1,0.02731,0.00,7.070,0,0.4690,6.4210,78.90,4.9671,2,242.0,17.80,396.90,9.14} | 1 | 21.60
+{1,0.02729,0.00,7.070,0,0.4690,7.1850,61.10,4.9671,2,242.0,17.80,392.83,4.03} | 1 | 34.70
+{1,0.03237,0.00,2.180,0,0.4580,6.9980,45.80,6.0622,3,222.0,18.70,394.63,2.94} | 1 | 33.40
+{1,0.06905,0.00,2.180,0,0.4580,7.1470,54.20,6.0622,3,222.0,18.70,396.90,5.33} | 1 | 36.20
+{1,0.02985,0.00,2.180,0,0.4580,6.4300,58.70,6.0622,3,222.0,18.70,394.12,5.21} | 1 | 28.70
+{1,0.08829,12.50,7.870,0,0.5240,6.0120,66.60,5.5605,5,311.0,15.20,395.60,12.43} | 1 | 22.90
+{1,0.14455,12.50,7.870,0,0.5240,6.1720,96.10,5.9505,5,311.0,15.20,396.90,19.15} | 1 | 27.10
+{1,0.21124,12.50,7.870,0,0.5240,5.6310,100.00,6.0821,5,311.0,15.20,386.63,29.93} | 1 | 16.50
+{1,0.17004,12.50,7.870,0,0.5240,6.0040,85.90,6.5921,5,311.0,15.20,386.71,17.10} | 1 | 18.90
+{1,0.22489,12.50,7.870,0,0.5240,6.3770,94.30,6.3467,5,311.0,15.20,392.52,20.45} | 1 | 15.00
+{1,0.11747,12.50,7.870,0,0.5240,6.0090,82.90,6.2267,5,311.0,15.20,396.90,13.27} | 1 | 18.90
+{1,0.09378,12.50,7.870,0,0.5240,5.8890,39.00,5.4509,5,311.0,15.20,390.50,15.71} | 1 | 21.70
+{1,0.62976,0.00,8.140,0,0.5380,5.9490,61.80,4.7075,4,307.0,21.00,396.90,8.26} | 1 | 20.40
+{1,0.63796,0.00,8.140,0,0.5380,6.0960,84.50,4.4619,4,307.0,21.00,380.02,10.26} | 1 | 18.20
+{1,0.62739,0.00,8.140,0,0.5380,5.8340,56.50,4.4986,4,307.0,21.00,395.62,8.47} | 1 | 19.90
+{1,1.05393,0.00,8.140,0,0.5380,5.9350,29.30,4.4986,4,307.0,21.00,386.85,6.58} | 1 | 23.10
+{1,0.78420,0.00,8.140,0,0.5380,5.9900,81.70,4.2579,4,307.0,21.00,386.75,14.67} | 1 | 17.50
+{1,0.80271,0.00,8.140,0,0.5380,5.4560,36.60,3.7965,4,307.0,21.00,288.99,11.69} | 1 | 20.20
+{1,0.72580,0.00,8.140,0,0.5380,5.7270,69.50,3.7965,4,307.0,21.00,390.95,11.28} | 1 | 18.20
+\.
+</pre>
+-# Now train a regression model using a multilayer perceptron a single hidden layer of two nodes.
+<pre class="example">
+SELECT mlp_regression(
+ 'lin_housing', -- Source table
+ 'mlp_regress', -- Desination table
+ 'x', -- Input features
+ 'y', -- Dependent variable
+ ARRAY[5,5], -- Number of units per layer
+ 'step_size=0.000007,
+ n_iterations=5000,
+ tolerance=0',
+ 'relu');
+</pre>
+-# Check the results of the model
+<pre class="example">
+-- Set extended display on for easier reading of output
+\\x ON
+-- Neural net Initialization is non-deterministic, so your results may vary
+SELECT * FROM mlp_regress;
+</pre>
+Result:
+<pre class="result">
+-[ RECORD 1 ]--+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------
+coeff | {1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2.79506311399e-05,3.56715008915e-05,-6.09333559685e-05,0.000251228318768,-0.000224772841379,-3.71863030857e-05,-3.5757865148e-06,5.27936784854e-05,-2.48474166186e-05,6.19731184294e-05,3.07638968743e-05,6.8964698578e-06,0.000106016701083,-1.71484730318e-05,1.18691881812e-05,-0.000163975464208,0.000170026304906,3.11688265279e-05,0.000177050148787,-1.58265976603e-05,2.70144422657e-05,0.000112667883422,3.77575139073e-05,8.12474658795e-05,-7.90458917626e-05,0.000107566386158,-2.63771171506e-06,2.47996880915e-05,-0.00012642310887,0.000203827391081,0.000139315565565,4.86147243454e-05,-0.000176126471913,-6.47820782916e-05,-8.51592776447e-06,-6.60601176758e-05,2.91421874156e-05,6.3556873752e-05,0.000197557443129,0.000220531367259,0.000135036310289,0.000143735913975,-4.75034117786e-05,-0.000179547345838,-1.6919846786e-05,0.000162784312994,0.000268595819851,-0.000460066553287,8.69756071591e-05,-0.00311762727057,0.000126024763103,0.00020598824
2921,0.003463432426,-0.00729789075286,0.00151625867549,-0.000890852767597,-0.00525016037249,0.0031043106659,0.00798041103839,-0.00552693050079,0.0232180415786,0.0230489850143,-0.0437890272341,0.0165765426407,-0.248554261758,-7.81336427846e-05,0.00558145591752,0.283465844585,-0.571699956182,0.133474351994,-0.0785181945605,-0.419269930709,0.249547772912,0.631761009875,-0.431305975666,1,1,1,1,1,1,0.0158747497572,-9.02809160806e-05,0.00015574347618,4.10805373863e-06,0.00121532434965,0.101790351335,0.0647558401493,-0.00013654998677,-9.92872075948e-06,-5.5319694394e-05,0.00519320756484,0.412736586036,0.0011998026977,-1.53688189815e-05,1.94817888201e-05,-4.63111489966e-05,7.24547899029e-05,0.00880394144485,5.45309822095e-05,-0.000140943219275,-7.96211486227e-05,-1.04337307472e-05,0.000161936762028,0.00136273797767,-4.54737243585e-05,-3.4083840736e-05,3.69286883662e-05,9.9047243188e-08,3.75014011824e-06,-9.45366086368e-08,1,1,1,1,1,1,6.67488547054,0.102754199001,0.41668912471,0.008868672964
79,0.00136206007228,-9.88642499013e-05}
+loss | 144.965776158
+num_iterations | 10000
+</pre>
+-# Now let's look at the prediction functions. In the following examples we will
+use the training data set for prediction as well, which is not usual but serves to
+show the syntax. First we will test the classification exampel.
+The prediction is in the the estimated_class_text column with the
+actual value in the class_text column.
+<pre class="example">
+SELECT mlp_predict(
+ 'mlp_model', -- Model table
+ 'iris_data', -- Test data table
+ 'id', -- Id column in test table
+ 'mlp_prediction', -- Output table for predictions
+ 'response' -- Output classes, not probabilities
+ );
+SELECT * FROM mlp_prediction JOIN iris_data USING (id);
+</pre>
+Result:
+<pre class="result">
+ id | estimated_class_text | attributes | class_text | class
+----+----------------------+-------------------+-----------------+-------
+ 1 | Iris-setosa | {5.1,3.5,1.4,0.2} | Iris-setosa | 1
+ 2 | Iris-setosa | {4.9,3.0,1.4,0.2} | Iris-setosa | 1
+ 3 | Iris-setosa | {4.7,3.2,1.3,0.2} | Iris-setosa | 1
+ 4 | Iris-setosa | {4.6,3.1,1.5,0.2} | Iris-setosa | 1
+ 5 | Iris-setosa | {5.0,3.6,1.4,0.2} | Iris-setosa | 1
+ 6 | Iris-setosa | {5.4,3.9,1.7,0.4} | Iris-setosa | 1
+ 7 | Iris-setosa | {4.6,3.4,1.4,0.3} | Iris-setosa | 1
+ 8 | Iris-setosa | {5.0,3.4,1.5,0.2} | Iris-setosa | 1
+ 9 | Iris-setosa | {4.4,2.9,1.4,0.2} | Iris-setosa | 1
+ 10 | Iris-setosa | {4.9,3.1,1.5,0.1} | Iris-setosa | 1
+ 11 | Iris-versicolor | {7.0,3.2,4.7,1.4} | Iris-versicolor | 2
+ 12 | Iris-versicolor | {6.4,3.2,4.5,1.5} | Iris-versicolor | 2
+ 13 | Iris-versicolor | {6.9,3.1,4.9,1.5} | Iris-versicolor | 2
+ 14 | Iris-versicolor | {5.5,2.3,4.0,1.3} | Iris-versicolor | 2
+ 15 | Iris-versicolor | {6.5,2.8,4.6,1.5} | Iris-versicolor | 2
+ 16 | Iris-versicolor | {5.7,2.8,4.5,1.3} | Iris-versicolor | 2
+ 17 | Iris-versicolor | {6.3,3.3,4.7,1.6} | Iris-versicolor | 2
+ 18 | Iris-versicolor | {4.9,2.4,3.3,1.0} | Iris-versicolor | 2
+ 19 | Iris-versicolor | {6.6,2.9,4.6,1.3} | Iris-versicolor | 2
+ 20 | Iris-versicolor | {5.2,2.7,3.9,1.4} | Iris-versicolor | 2
+</pre>
+Prediction using the regression model:
+<pre class="example">
+SELECT mlp_predict(
+ 'mlp_regress', -- Model table
+ 'lin_housing', -- Test data table
+ 'id', -- Id column in test table
+ 'mlp_regress_prediction', -- Output table for predictions
+ 'response' -- Output values, not probabilities
+ );
+</pre>
+-# View results
+<pre class="example">
+SELECT madlib.mlp_predict('mlp_regress','lin_housing','id','mlp_regress_prediction','response');
--- End diff --
I believe this line should be removed as the call to `mlp_predict` is performed above.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib issue #149: MLP: Multilayer Perceptron
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/incubator-madlib/pull/149
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/madlib-pr-build/119/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by edespino <gi...@git.apache.org>.
Github user edespino commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r127309270
--- Diff: src/ports/postgres/modules/convex/mlp.sql_in ---
@@ -0,0 +1,744 @@
+/* ----------------------------------------------------------------------- *//**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ *
+ * @file mlp.sql_in
+ *
+ * @brief SQL functions for multilayer perceptron
+ * @date June 2012
+ *
+ *
+ *//* ----------------------------------------------------------------------- */
+
+m4_include(`SQLCommon.m4')
+
+/**
+@addtogroup grp_mlp
+
+<div class="toc"><b>Contents</b><ul>
+<li class="level1"><a href="#mlp_classification">Classification</a></li>
+<li class="level1"><a href="#mlp_regression">Regression</a></li>
+<li class="level1"><a href="#optimization_params">Optimizer Parameters</a></li>
+<li class="level1"><a href="#predict">Prediction Functions/a></li>
+<li class="level1"><a href="#example">Examples</a></li>
+<li class="level1"><a href="#background">Technical Background</a></li>
+<li class="level1"><a href="#literature">Literature</a></li>
+<li class="level1"><a href="#related">Related Topics</a></li>
+</ul></div>
+
+Multilayer Perceptron (MLP) is a model for regression and
+classification
+
+Also called "vanilla neural networks", they consist of several
+fully connected hidden layers with non-linear activation
+functions. In the case of classification, the final layer of the
+neural net has as many nodes as classes, and the output of the
+neural net can be interpreted as the probability of a given input
+feature belonging to a specific class.
+
+
+@brief Solves classification and regression problems with several
+fully connected layers and nonlinear activation functions.
+
+@anchor mlp_classification
+@par Classification Training Function
+The mlp classification training function has the following format:
+<pre class="syntax">
+mlp_classification(
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+\b Arguments
+<DL class="arglist">
+ <DT>source_table</DT>
+ <DD>TEXT. Name of the table containing the training data.</DD>
+
+ <DT>model_table</DT>
+ <DD>TEXT. Name of the output table containing the model. Details of the output
+ tables are provided below.
+ </DD>
+
+ <DT>independent_varname</DT>
+ <DD>TEXT. Expression list to evaluate for the
+ independent variables. An intercept variable should not be included as part
+ of this expression. Please note that expression should be able to be cast
+ to DOUBLE PRECISION[].
+ </DD>
+
+ <DT>dependent_varname</DT>
+ <DD> TEXT. Name of the dependent variable column. For classification, supported types are:
+ text, varchar, character varying, char, character
+ integer, smallint, bigint, and boolean. </DD>
+
+ <DT>hidden_layer_sizes</DT>
+ <DD>INTEGER[], default: ARRAY[].
+ The number of neurons in each hidden layer. The length of this array will
+ determine the number of hidden layers. Empty for no hidden layers.
+ </DD>
+
+
+ <DT>optimizer_params (optional)</DT>
+ <DD>TEXT, default: NULL.
+ Parameters for optimization in a comma-separated string
+ of key-value pairs. See the description below for details.
+ </DD>
+
+ <DT>activation</DT>
+ <DD>TEXT, default: 'sigmoid'.
+ Activation function. Currently three functions are supported: 'sigmoid' (default),
+ 'relu', and 'tanh'. The text can be any subset of the three
+ strings; for e.g., activation='s' will use the sigmoid activation.
+ </DD>
+</DL>
+
+<b>Output tables</b>
+<br>
+ The model table produced by mlp contains the following columns:
+ <table class="output">
+ <tr>
+ <th>coeffs</th>
+ <td>FLOAT8[]. Flat array containing the weights of the neural net</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>INTEGER. Number of iterations completed by stochastic gradient descent
+ algorithm. The algorithm either converged in this number of iterations
+ or hit the maximum number specified in the optimization parameters. </td>
+ </tr>
+ <tr>
+ <th>loss</th>
+ <td>FLOAT8. The cross entropy over the training data.
+ See Technical Background section below for more details.</td>
+ </tr>
+ </table>
+
+
+A summary table named \<model_table\>_summary is also created, which has the following columns:
+ <table class="output">
+ <tr>
+ <th>source_table</th>
+ <td>The source table.</td>
+ </tr>
+ <tr>
+ <th>dependent_varname</th>
+ <td>The dependent variable.</td>
+ </tr>
+ <tr>
+ <th>independent_varname</th>
+ <td>The independent variables.</td>
+ </tr>
+ <tr>
+ <th>tolerance</th>
+ <td>The tolerance as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>step_size</th>
+ <td>The step size as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>The number of iterations run</td>
+ </tr>
+ <tr>
+ <th>n_tries</th>
+ <td>The number of tries as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>layer_sizes</th>
+ <td>The number of units in each layer including the input and output layer.</td>
+ </tr>
+ <tr>
+ <th>activation_function</th>
+ <td>The activation function.</td>
+ </tr>
+ <tr>
+ <th>is_classification</th>
+ <td>True if the model was trained for classification, False if it was trained
+ for regression</td>
+ </tr>
+ <tr>
+ <th>classes</th>
+ <td>The classes which were trained against (empty for regression)</td>
+ </tr>
+
+ </table>
+
+
+@anchor mlp_regression
+@par Regression Training Function
+The mlp regression training function has the following format:
+<pre class="syntax">
+mlp_regression(source_table,
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+
+\b Arguments
+
+Specifications for regression are largely the same as for classification. In the
+model table, the loss will refer to mean square error instead of cross entropy. In the
+summary table, there is classes column. The following
+arguments have specifications which differ from mlp_classification:
+<DL class="arglist">
+<DT>dependent_varname</DT>
+ <DD>TEXT. Name of the dependent variable column.
+ For regression supported types are any numeric type, or array
+ or numeric types (for multiple regression).
+ </DD>
+</DL>
+
+
+@anchor optimizer_params
+@par Optimizer Parameters
+Parameters in this section are supplied in the \e optimizer_params argument as a string
+containing a comma-delimited list of name-value pairs. All of these named
+parameters are optional, and their order does not matter. You must use the
+format "<param_name> = <value>" to specify the value of a parameter, otherwise
+the parameter is ignored.
+
+
+<pre class="syntax">
+ 'step_size = <value>,
+ n_iterations = <value>,
+ n_tries = <value>,
+ tolerance = <value>'
+</pre>
+\b Optimizer Parameters
+<DL class="arglist">
+
+<DT>step_size</dt>
+<DD>Default: [0.001].
+Also known as the learning rate. A small value is usually desirable to
+ensure convergence, while a large value provides more room for progress during
+training. Since the best value depends on the condition number of the data, in
+practice one often tunes this parameter.
+</DD>
+
+
+<DT>n_iterations</dt>
+<DD>Default: [100]. The maximum number of iterations allowed.
+</DD>
+<DT>n_tries</dt>
+<DD>Default: [1]. Number of times to retrain the network with randomly initialized
+weights
+</DD>
+
+<DT>tolerance</dt>
+<DD>Default: 0.001. The criterion to end iterations. The training stops whenever
+<the difference between the training models of two consecutive iterations is
+<smaller than \e tolerance or the iteration number is larger than \e max_iter.
+</DD>
+
+</DL>
+
+@anchor predict
+@par Prediction Function
+Used to generate predictions given a previously trained model on novel data.
+The same syntax is used for classification, and regression.
+<pre class="syntax">
+mlp_predict(model_table,
+ data_table,
+ id_col_name,
+ output_table,
+ pred_type)
+</pre>
+
+\b Arguments
+<DL class="arglist">
+ <DT>model_table</DT>
+ <DD>TEXT. Model table produced by the training function.</DD>
+
+ <DT>data_table</DT>
+ <DD>TEXT. Name of the table containing the data for prediction. This table is expected
+ to contain the same input features that were used during training. The table should
+ also contain id_col_name used for identifying each row.</DD>
+
+ <DT>id_col_name</DT>
+ <DD>TEXT. The name of the id column in the input table.</DD>
+
+ <DT>output_table</DT>
+ <DD>TEXT. Name of the table where output predictions are written. If this
+table name is already in use, then an error is returned. Table contains:</DD>
+ <table class="output">
+ <tr>
+ <th>id</th>
+ <td>Gives the 'id' for each prediction, corresponding to each row from the data_table.</td>
+ </tr>
+ <tr>
+ <th>estimated_<COL_NAME></th>
+ <td>
+ (For pred_type='response') The estimated class
+ for classification or value for regression, where
+ <COL_NAME> is the name of the column to be
+ predicted from training data
+ </td>
+ </tr>
+ <tr>
+ <th>prob_<CLASS></th>
+ <td>
+ (For pred_type='prob' for classification) The
+ probability of a given class <CLASS> as given by
+ softmax. There will be one column for each class
+ in the training data.
+ </td>
+ </tr>
+
+
+ <DT>pred_type</DT>
+ <DD>TEXT.
+
+the type of output requested:
+'response' gives the actual prediction,
+'prob' gives the probability of each class.
+for regression, only type='response' is defined.
+The name of the id column in the input table.</DD>
+</DL>
+</table>
+
+@anchor example
+@par Examples
+-# Create an input data set.
+<pre class="example">
+CREATE TABLE iris_data(
+ id integer,
+ attributes numeric[],
+ class_text varchar,
+ class integer
+);
+INSERT INTO iris_data VALUES
+(1,ARRAY[5.1,3.5,1.4,0.2],'Iris-setosa',1),
+(2,ARRAY[4.9,3.0,1.4,0.2],'Iris-setosa',1),
+(3,ARRAY[4.7,3.2,1.3,0.2],'Iris-setosa',1),
+(4,ARRAY[4.6,3.1,1.5,0.2],'Iris-setosa',1),
+(5,ARRAY[5.0,3.6,1.4,0.2],'Iris-setosa',1),
+(6,ARRAY[5.4,3.9,1.7,0.4],'Iris-setosa',1),
+(7,ARRAY[4.6,3.4,1.4,0.3],'Iris-setosa',1),
+(8,ARRAY[5.0,3.4,1.5,0.2],'Iris-setosa',1),
+(9,ARRAY[4.4,2.9,1.4,0.2],'Iris-setosa',1),
+(10,ARRAY[4.9,3.1,1.5,0.1],'Iris-setosa',1),
+(11,ARRAY[7.0,3.2,4.7,1.4],'Iris-versicolor',2),
+(12,ARRAY[6.4,3.2,4.5,1.5],'Iris-versicolor',2),
+(13,ARRAY[6.9,3.1,4.9,1.5],'Iris-versicolor',2),
+(14,ARRAY[5.5,2.3,4.0,1.3],'Iris-versicolor',2),
+(15,ARRAY[6.5,2.8,4.6,1.5],'Iris-versicolor',2),
+(16,ARRAY[5.7,2.8,4.5,1.3],'Iris-versicolor',2),
+(17,ARRAY[6.3,3.3,4.7,1.6],'Iris-versicolor',2),
+(18,ARRAY[4.9,2.4,3.3,1.0],'Iris-versicolor',2),
+(19,ARRAY[6.6,2.9,4.6,1.3],'Iris-versicolor',2),
+(20,ARRAY[5.2,2.7,3.9,1.4],'Iris-versicolor',2);
+</pre>
+-# Generate a multilayer perceptron with a single hidden layer of 5 units.
+Use the attributes column as the independent variables, and use the class
+column as the classification. Set the tolerance to 0 so that 5000
+iterations will be run. Use a hyperbolic tangent activation function.
+The model will be written to mlp_model.
+<pre class="example">
+SELECT madlib.mlp_classification(
+ 'iris_data', -- Source table
+ 'mlp_model', -- Destination table
+ 'attributes', -- Input features
+ 'class_text', -- Label
+ ARRAY[5], -- Number of units per layer
+ 'step_size=0.003,
+ n_iterations=5000,
+ tolerance=0', -- Optimizer params
+ 'tanh'); -- Activation function
+</pre>
+-# View the result for the model.
+<pre class="example">
+-- Set extended display on for easier reading of output
+\\x ON
+-- Neural net Initialization is non-deterministic, so your results may vary
+SELECT * FROM mlp_model;
+</pre>
+Result:
+<pre class="result">
+-[ RECORD 1 ]--+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+coeff | {1,1,1,1,1,0.136374930803,0.188739676875,0.662387810001,-1.03381622734,-0.469961067046,0.0614006983397,0.0811504589436,0.299008228258,-0.47391918521,-0.215098143699,0.10519213944,0.145844617525,0.511683525606,-0.800215552382,-0.36417142683,0.120751709056,0.167531106521,0.587074895969,-0.916946198095,-0.417055067449,0.0539541885146,0.0694359704131,0.262598585854,-0.419234805076,-0.189915344282,1,1,1,1,1,1,0.105645702152,1.46247470474,0.484457903226,0.965962824478,1.19361986431,0.419805760087,-0.105696503487,-1.46245956666,-0.484427811691,-0.965730981426,-1.19365280555,-0.419973628863}
+loss | 0.0184092375519
+num_iterations | 5000
+</pre>
+-# Next train a regression example. First create some test data. This dataset
+contains housing prices data.
+<pre class="example">
+CREATE TABLE lin_housing (id serial, x float8[], grp_by_col int, y float8);
+COPY lin_housing (x, grp_by_col, y) FROM STDIN NULL '?' DELIMITER '|';
+{1,0.00632,18.00,2.310,0,0.5380,6.5750,65.20,4.0900,1,296.0,15.30,396.90,4.98} | 1 | 24.00
+{1,0.02731,0.00,7.070,0,0.4690,6.4210,78.90,4.9671,2,242.0,17.80,396.90,9.14} | 1 | 21.60
+{1,0.02729,0.00,7.070,0,0.4690,7.1850,61.10,4.9671,2,242.0,17.80,392.83,4.03} | 1 | 34.70
+{1,0.03237,0.00,2.180,0,0.4580,6.9980,45.80,6.0622,3,222.0,18.70,394.63,2.94} | 1 | 33.40
+{1,0.06905,0.00,2.180,0,0.4580,7.1470,54.20,6.0622,3,222.0,18.70,396.90,5.33} | 1 | 36.20
+{1,0.02985,0.00,2.180,0,0.4580,6.4300,58.70,6.0622,3,222.0,18.70,394.12,5.21} | 1 | 28.70
+{1,0.08829,12.50,7.870,0,0.5240,6.0120,66.60,5.5605,5,311.0,15.20,395.60,12.43} | 1 | 22.90
+{1,0.14455,12.50,7.870,0,0.5240,6.1720,96.10,5.9505,5,311.0,15.20,396.90,19.15} | 1 | 27.10
+{1,0.21124,12.50,7.870,0,0.5240,5.6310,100.00,6.0821,5,311.0,15.20,386.63,29.93} | 1 | 16.50
+{1,0.17004,12.50,7.870,0,0.5240,6.0040,85.90,6.5921,5,311.0,15.20,386.71,17.10} | 1 | 18.90
+{1,0.22489,12.50,7.870,0,0.5240,6.3770,94.30,6.3467,5,311.0,15.20,392.52,20.45} | 1 | 15.00
+{1,0.11747,12.50,7.870,0,0.5240,6.0090,82.90,6.2267,5,311.0,15.20,396.90,13.27} | 1 | 18.90
+{1,0.09378,12.50,7.870,0,0.5240,5.8890,39.00,5.4509,5,311.0,15.20,390.50,15.71} | 1 | 21.70
+{1,0.62976,0.00,8.140,0,0.5380,5.9490,61.80,4.7075,4,307.0,21.00,396.90,8.26} | 1 | 20.40
+{1,0.63796,0.00,8.140,0,0.5380,6.0960,84.50,4.4619,4,307.0,21.00,380.02,10.26} | 1 | 18.20
+{1,0.62739,0.00,8.140,0,0.5380,5.8340,56.50,4.4986,4,307.0,21.00,395.62,8.47} | 1 | 19.90
+{1,1.05393,0.00,8.140,0,0.5380,5.9350,29.30,4.4986,4,307.0,21.00,386.85,6.58} | 1 | 23.10
+{1,0.78420,0.00,8.140,0,0.5380,5.9900,81.70,4.2579,4,307.0,21.00,386.75,14.67} | 1 | 17.50
+{1,0.80271,0.00,8.140,0,0.5380,5.4560,36.60,3.7965,4,307.0,21.00,288.99,11.69} | 1 | 20.20
+{1,0.72580,0.00,8.140,0,0.5380,5.7270,69.50,3.7965,4,307.0,21.00,390.95,11.28} | 1 | 18.20
+\.
+</pre>
+-# Now train a regression model using a multilayer perceptron a single hidden layer of two nodes.
+<pre class="example">
+SELECT madlib.mlp_regression(
+ 'lin_housing', -- Source table
+ 'mlp_regress', -- Desination table
+ 'x', -- Input features
+ 'y', -- Dependent variable
+ ARRAY[5,5], -- Number of units per layer
+ 'step_size=0.000007,
+ n_iterations=10000,
+ tolerance=0',
+ 'relu');
+</pre>
+-# Check the results of the model
+<pre class="example">
+-- Set extended display on for easier reading of output
+\\x ON
+-- Neural net Initialization is non-deterministic, so your results may vary
+SELECT * FROM mlp_regress;
+</pre>
+Result:
+<pre class="result">
+-[ RECORD 1 ]--+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------
+coeff | {1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2.79506311399e-05,3.56715008915e-05,-6.09333559685e-05,0.000251228318768,-0.000224772841379,-3.71863030857e-05,-3.5757865148e-06,5.27936784854e-05,-2.48474166186e-05,6.19731184294e-05,3.07638968743e-05,6.8964698578e-06,0.000106016701083,-1.71484730318e-05,1.18691881812e-05,-0.000163975464208,0.000170026304906,3.11688265279e-05,0.000177050148787,-1.58265976603e-05,2.70144422657e-05,0.000112667883422,3.77575139073e-05,8.12474658795e-05,-7.90458917626e-05,0.000107566386158,-2.63771171506e-06,2.47996880915e-05,-0.00012642310887,0.000203827391081,0.000139315565565,4.86147243454e-05,-0.000176126471913,-6.47820782916e-05,-8.51592776447e-06,-6.60601176758e-05,2.91421874156e-05,6.3556873752e-05,0.000197557443129,0.000220531367259,0.000135036310289,0.000143735913975,-4.75034117786e-05,-0.000179547345838,-1.6919846786e-05,0.000162784312994,0.000268595819851,-0.000460066553287,8.69756071591e-05,-0.00311762727057,0.000126024763103,0.00020598824
2921,0.003463432426,-0.00729789075286,0.00151625867549,-0.000890852767597,-0.00525016037249,0.0031043106659,0.00798041103839,-0.00552693050079,0.0232180415786,0.0230489850143,-0.0437890272341,0.0165765426407,-0.248554261758,-7.81336427846e-05,0.00558145591752,0.283465844585,-0.571699956182,0.133474351994,-0.0785181945605,-0.419269930709,0.249547772912,0.631761009875,-0.431305975666,1,1,1,1,1,1,0.0158747497572,-9.02809160806e-05,0.00015574347618,4.10805373863e-06,0.00121532434965,0.101790351335,0.0647558401493,-0.00013654998677,-9.92872075948e-06,-5.5319694394e-05,0.00519320756484,0.412736586036,0.0011998026977,-1.53688189815e-05,1.94817888201e-05,-4.63111489966e-05,7.24547899029e-05,0.00880394144485,5.45309822095e-05,-0.000140943219275,-7.96211486227e-05,-1.04337307472e-05,0.000161936762028,0.00136273797767,-4.54737243585e-05,-3.4083840736e-05,3.69286883662e-05,9.9047243188e-08,3.75014011824e-06,-9.45366086368e-08,1,1,1,1,1,1,6.67488547054,0.102754199001,0.41668912471,0.008868672964
79,0.00136206007228,-9.88642499013e-05}
+loss | 144.965776158
+num_iterations | 10000
+</pre>
+-# Now let's look at the prediction functions. In the following examples we will
+use the training data set for prediction as well, which is not usual but serves to
+show the syntax. First we will test the classification exampel.
+The prediction is in the the estimated_class_text column with the
+actual value in the class_text column.
+<pre class="example">
+SELECT madlib.mlp_predict(
+ 'mlp_model', -- Model table
+ 'iris_data', -- Test data table
+ 'id', -- Id column in test table
+ 'mlp_prediction', -- Output table for predictions
+ 'response' -- Output classes, not probabilities
+ );
+SELECT * FROM mlp_prediction JOIN iris_data USING (id);
+</pre>
+Result:
+<pre class="result">
+ id | estimated_class_text | attributes | class_text | class
+----+----------------------+-------------------+-----------------+-------
+ 1 | Iris-setosa | {5.1,3.5,1.4,0.2} | Iris-setosa | 1
+ 2 | Iris-setosa | {4.9,3.0,1.4,0.2} | Iris-setosa | 1
+ 3 | Iris-setosa | {4.7,3.2,1.3,0.2} | Iris-setosa | 1
+ 4 | Iris-setosa | {4.6,3.1,1.5,0.2} | Iris-setosa | 1
+ 5 | Iris-setosa | {5.0,3.6,1.4,0.2} | Iris-setosa | 1
+ 6 | Iris-setosa | {5.4,3.9,1.7,0.4} | Iris-setosa | 1
+ 7 | Iris-setosa | {4.6,3.4,1.4,0.3} | Iris-setosa | 1
+ 8 | Iris-setosa | {5.0,3.4,1.5,0.2} | Iris-setosa | 1
+ 9 | Iris-setosa | {4.4,2.9,1.4,0.2} | Iris-setosa | 1
+ 10 | Iris-setosa | {4.9,3.1,1.5,0.1} | Iris-setosa | 1
+ 11 | Iris-versicolor | {7.0,3.2,4.7,1.4} | Iris-versicolor | 2
+ 12 | Iris-versicolor | {6.4,3.2,4.5,1.5} | Iris-versicolor | 2
+ 13 | Iris-versicolor | {6.9,3.1,4.9,1.5} | Iris-versicolor | 2
+ 14 | Iris-versicolor | {5.5,2.3,4.0,1.3} | Iris-versicolor | 2
+ 15 | Iris-versicolor | {6.5,2.8,4.6,1.5} | Iris-versicolor | 2
+ 16 | Iris-versicolor | {5.7,2.8,4.5,1.3} | Iris-versicolor | 2
+ 17 | Iris-versicolor | {6.3,3.3,4.7,1.6} | Iris-versicolor | 2
+ 18 | Iris-versicolor | {4.9,2.4,3.3,1.0} | Iris-versicolor | 2
+ 19 | Iris-versicolor | {6.6,2.9,4.6,1.3} | Iris-versicolor | 2
+ 20 | Iris-versicolor | {5.2,2.7,3.9,1.4} | Iris-versicolor | 2
+</pre>
+Prediction using the regression model:
+<pre class="example">
+SELECT madlib.mlp_predict(
+ 'mlp_regress', -- Model table
+ 'lin_housing', -- Test data table
+ 'id', -- Id column in test table
+ 'mlp_regress_prediction', -- Output table for predictions
+ 'response' -- Output values, not probabilities
+ );
+</pre>
+-# View results
+<pre class="example">
+SELECT * FROM lin_housing JOIN mlp_regress_prediction USING (id);
+</pre>
+Result for the regression model:
+<pre class="result">
+ id | x | grp_by_col | y | estimated_y
+----+-------------------------------------------------------------------------+------------+------+--------------------
+ 1 | {1,0.00632,18,2.31,0,0.538,6.575,65.2,4.09,1,296,15.3,396.9,4.98} | 1 | 24 | {23.2627062018087}
+ 2 | {1,0.02731,0,7.07,0,0.469,6.421,78.9,4.9671,2,242,17.8,396.9,9.14} | 1 | 21.6 | {25.7088419115781}
+ 3 | {1,0.02729,0,7.07,0,0.469,7.185,61.1,4.9671,2,242,17.8,392.83,4.03} | 1 | 34.7 | {27.5587003901404}
+ 4 | {1,0.03237,0,2.18,0,0.458,6.998,45.8,6.0622,3,222,18.7,394.63,2.94} | 1 | 33.4 | {31.1812237427816}
+ 5 | {1,0.06905,0,2.18,0,0.458,7.147,54.2,6.0622,3,222,18.7,396.9,5.33} | 1 | 36.2 | {30.3696873085477}
+ 6 | {1,0.02985,0,2.18,0,0.458,6.43,58.7,6.0622,3,222,18.7,394.12,5.21} | 1 | 28.7 | {29.5290259241882}
+ 7 | {1,0.08829,12.5,7.87,0,0.524,6.012,66.6,5.5605,5,311,15.2,395.6,12.43} | 1 | 22.9 | {21.1576051716888}
+ 8 | {1,0.14455,12.5,7.87,0,0.524,6.172,96.1,5.9505,5,311,15.2,396.9,19.15} | 1 | 27.1 | {17.6194200563055}
+ 9 | {1,0.21124,12.5,7.87,0,0.524,5.631,100,6.0821,5,311,15.2,386.63,29.93} | 1 | 16.5 | {15.1366297774139}
+10 | {1,0.17004,12.5,7.87,0,0.524,6.004,85.9,6.5921,5,311,15.2,386.71,17.1} | 1 | 18.9 | {17.6528662199369}
+11 | {1,0.22489,12.5,7.87,0,0.524,6.377,94.3,6.3467,5,311,15.2,392.52,20.45} | 1 | 15 | {17.2017487668181}
+12 | {1,0.11747,12.5,7.87,0,0.524,6.009,82.9,6.2267,5,311,15.2,396.9,13.27} | 1 | 18.9 | {19.4893860319992}
+13 | {1,0.09378,12.5,7.87,0,0.524,5.889,39,5.4509,5,311,15.2,390.5,15.71} | 1 | 21.7 | {23.2917226708039}
+14 | {1,0.62976,0,8.14,0,0.538,5.949,61.8,4.7075,4,307,21,396.9,8.26} | 1 | 20.4 | {22.8904812605193}
+15 | {1,0.63796,0,8.14,0,0.538,6.096,84.5,4.4619,4,307,21,380.02,10.26} | 1 | 18.2 | {18.2386754423677}
+16 | {1,0.62739,0,8.14,0,0.538,5.834,56.5,4.4986,4,307,21,395.62,8.47} | 1 | 19.9 | {23.28949550874}
+17 | {1,1.05393,0,8.14,0,0.538,5.935,29.3,4.4986,4,307,21,386.85,6.58} | 1 | 23.1 | {25.3288762085473}
+18 | {1,0.7842,0,8.14,0,0.538,5.99,81.7,4.2579,4,307,21,386.75,14.67} | 1 | 17.5 | {19.0203738118451}
+19 | {1,0.80271,0,8.14,0,0.538,5.456,36.6,3.7965,4,307,21,288.99,11.69} | 1 | 20.2 | {12.3162005347545}
+20 | {1,0.7258,0,8.14,0,0.538,5.727,69.5,3.7965,4,307,21,390.95,11.28} | 1 | 18.2 | {21.0902211848747}
+</pre>
+Note that the results you get for all examples may vary with the platform you are using.
+
+@anchor background
+@par Technical Background
--- End diff --
Observation: In relation to the Supervised Learnings Technical Background sections for the Regression Models (https://madlib.incubator.apache.org/docs/latest/group__grp__regml.html), it appears as if the MLP Technical Background could provide more details on the implementation. Of course one could review the source and external references. Providing implementation details might be beneficial to the data scientist taking advantage of the MLP model.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by edespino <gi...@git.apache.org>.
Github user edespino commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r126829841
--- Diff: src/ports/postgres/modules/convex/mlp.sql_in ---
@@ -0,0 +1,736 @@
+/* ----------------------------------------------------------------------- *//**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ *
+ * @file mlp.sql_in
+ *
+ * @brief SQL functions for multi-layer perceptron
+ * @date June 2012
+ *
+ *
+ *//* ----------------------------------------------------------------------- */
+
+m4_include(`SQLCommon.m4')
+
+/**
+@addtogroup grp_mlp
+
+<div class="toc"><b>Contents</b><ul>
+<li class="level1"><a href="#mlp_classification">Classification</a></li>
+<li class="level1"><a href="#mlp_regression">Regression</a></li>
+<li class="level1"><a href="#optimization_params">Optimization Parameters</a></li>
+<li class="level1"><a href="#predict">Prediction Functions/a></li>
+<li class="level1"><a href="#example">Examples</a></li>
+<li class="level1"><a href="#background">Technical Background</a></li>
+<li class="level1"><a href="#literature">Literature</a></li>
+<li class="level1"><a href="#related">Related Topics</a></li>
+</ul></div>
+
+Multi Layer Perceptron (MLP) is a model for regression and
+classification
+
+Also called "vanilla neural networks", they consist of several
+fully connected hidden layers with non-linear activation
+functions. In the case of classification, the final layer of the
+neural net has as many nodes as classes, and the output of the
+neural net can be interpreted as the probability of a given input
+feature belonging to a specific class.
+
+
+@brief Solves classification and regression problems with several
+fully connected layers and nonlinear activation functions.
+
+@anchor mlp_classification
+@par Classification Training Function
+The mlp classification training function has the following format:
+<pre class="syntax">
+mlp_classification(
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+\b Arguments
+<DL class="arglist">
+ <DT>source_table</DT>
+ <DD>TEXT. Name of the table containing the training data.</DD>
+
+ <DT>model_table</DT>
+ <DD>TEXT. Name of the output table containing the model. Details of the output
+ tables are provided below.
+ </DD>
+
+ <DT>independent_varname</DT>
+ <DD>TEXT. Expression list to evaluate for the
+ independent variables. An intercept variable should not be included as part
+ of this expression. Please note that expression should be able to be cast
+ to DOUBLE PRECISION[].
+ </DD>
+
+ <DT>dependent_varname</DT>
+ <DD> TEXT. Name of the dependent variable column. For classification, supported types are:
+ text, varchar, character varying, char, character
+ integer, smallint, bigint, and boolean. </DD>
+
+ <DT>hidden_layer_sizes</DT>
+ <DD>INTEGER[], default: ARRAY[].
+ The number of neurons in each hidden layer. The length of this array will
+ determine the number of hidden layers. Empty for no hidden layers.
+ </DD>
+
+
+ <DT>optimizer_params (optional)</DT>
+ <DD>TEXT, default: NULL.
+ Parameters for optimization in a comma-separated string
+ of key-value pairs. See the description below for details.
+ </DD>
+
+ <DT>activation</DT>
+ <DD>TEXT, default: 'sigmoid'.
+ Activation function. Currently three functions are supported: 'sigmoid' (default),
+ 'relu', and 'tanh'. The text can be any subset of the three
+ strings; for e.g., activation='s' will use the sigmoid activation.
+ </DD>
+</DL>
+
+<b>Output tables</b>
+<br>
+ The model table produced by mlp contains the following columns:
+ <table class="output">
+ <tr>
+ <th>coeffs</th>
+ <td>FLOAT8[]. Flat array containing the weights of the neural net</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>INTEGER. Number of iterations completed by stochastic gradient descent
+ algorithm. The algorithm either converged in this number of iterations
+ or hit the maximum number specified in the optimization parameters. </td>
+ </tr>
+ <tr>
+ <th>loss</th>
+ <td>FLOAT8. The cross entropy over the training data.
+ See Technical Background section below for more details.</td>
+ </tr>
+ </table>
+
+
+A summary table named \<model_table\>_summary is also created, which has the following columns:
+ <table class="output">
+ <tr>
+ <th>source_table</th>
+ <td>The source table.</td>
+ </tr>
+ <tr>
+ <th>dependent_varname</th>
+ <td>The dependent variable.</td>
+ </tr>
+ <tr>
+ <th>independent_varname</th>
+ <td>The independent variables.</td>
+ </tr>
+ <tr>
+ <th>tolerance</th>
+ <td>The tolerance as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>step_size</th>
+ <td>The step size as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>The number of iterations run</td>
+ </tr>
+ <tr>
+ <th>n_tries</th>
+ <td>The number of tries as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>layer_sizes</th>
+ <td>The number of units in each layer including the input and output layer.</td>
+ </tr>
+ <tr>
+ <th>activation_function</th>
+ <td>The activation function.</td>
+ </tr>
+ <tr>
+ <th>is_classification</th>
+ <td>True if the model was trained for classification, False if it was trained
+ for regression</td>
+ </tr>
+ <tr>
+ <th>classes</th>
+ <td>The classes which were trained against (empty for regression)</td>
+ </tr>
+
+ </table>
+
+
+@anchor mlp_regression
+@par Regression Training Function
+The mlp regression training function has the following format:
+<pre class="syntax">
+mlp_regression(source_table,
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+
+\b Arguments
+
+Specifications for regression are largely the same as for classification. In the
+model table, the loss will refer to mean square error instead of cross entropy. In the
+summary table, there is classes column. The following
+arguments have specifications which differ from mlp_classification:
+<DL class="arglist">
+<DT>dependent_varname</DT>
+ <DD>TEXT. Name of the dependent variable column.
+ For regression supported types are any numeric type, or array
+ or numeric types (for multiple regression).
+ </DD>
+</DL>
+
+
+@anchor optimizer_params
+@par Optimizer Parameters
+Parameters in this section are supplied in the \e optimizer_params argument as a string
+containing a comma-delimited list of name-value pairs. All of these named
+parameters are optional, and their order does not matter. You must use the
+format "<param_name> = <value>" to specify the value of a parameter, otherwise
+the parameter is ignored.
+
+
+<pre class="syntax">
+ 'step_size = <value>,
+ n_iterations = <value>,
+ n_tries = <value>,
+ tolerance = <value>'
+</pre>
+\b Optimizer Parameters
+<DL class="arglist">
+
+<DT>step_size</dt>
+<DD>Default: [0.001].
+Also known as the learning rate. A small value is usually desirable to
+ensure convergence, while a large value provides more room for progress during
+training. Since the best value depends on the condition number of the data, in
+practice one often tunes this parameter.
+</DD>
+
+
+<DT>n_iterations</dt>
+<DD>Default: [100]. The maximum number of iterations allowed.
+</DD>
+<DT>n_tries</dt>
+<DD>Default: [1]. Number of times to retrain the network with randomly initialized
+weights
+</DD>
+
+<DT>tolerance</dt>
+<DD>Default: 0.001. The criterion to end iterations. The training stops whenever
+<the difference between the training models of two consecutive iterations is
+<smaller than \e tolerance or the iteration number is larger than \e max_iter.
+</DD>
+
+</DL>
+
+@anchor predict
+@par Prediction Function
+Used to generate predictions given a previously trained model on novel data.
+The same syntax is used for classification, and regression.
+<pre class="syntax">
+mlp_predict(model_table,
+ data_table,
+ id_col_name,
+ output_table,
+ pred_type)
+</pre>
+
+\b Arguments
+<DL class="arglist">
+ <DT>model_table</DT>
+ <DD>TEXT. Model table produced by the training function.</DD>
+
+ <DT>data_table</DT>
+ <DD>TEXT. Name of the table containing the data for prediction. This table is expected
+ to contain the same input features that were used during training. The table should
+ also contain id_col_name used for identifying each row.</DD>
+
+ <DT>id_col_name</DT>
+ <DD>TEXT. The name of the id column in the input table.</DD>
+
+ <DT>output_table</DT>
+ <DD>TEXT. Name of the table where output predictions are written. If this
+table name is already in use, then an error is returned. Table contains:</DD>
+ <table class="output">
+ <tr>
+ <th>id</th>
+ <td>Gives the 'id' for each prediction, corresponding to each row from the data_table.</td>
+ </tr>
+ <tr>
+ <th>estimated_<COL_NAME></th>
+ <td>
+ (For pred_type='response') The estimated class
+ for classification or value for regression, where
+ <COL_NAME> is the name of the column to be
+ predicted from training data
+ </td>
+ </tr>
+ <tr>
+ <th>prob_<CLASS></th>
+ <td>
+ (For pred_type='prob' for classification) The
+ probability of a given class <CLASS> as given by
+ softmax. There will be one column for each class
+ in the training data.
+ </td>
+ </tr>
+
+
+ <DT>pred_type</DT>
+ <DD>TEXT.
+
+the type of output requested:
+'response' gives the actual prediction,
+'prob' gives the probability of each class.
+for regression, only type='response' is defined.
+The name of the id column in the input table.</DD>
+</DL>
+</table>
+
+@anchor example
+@par Examples
+-# Create an input data set.
+<pre class="example">
+CREATE TABLE iris_data(
+ id integer,
+ attributes numeric[],
+ class_text varchar,
+ class integer
+);
+INSERT INTO iris_data VALUES
+(1,ARRAY[5.1,3.5,1.4,0.2],'Iris-setosa',1),
+(2,ARRAY[4.9,3.0,1.4,0.2],'Iris-setosa',1),
+(3,ARRAY[4.7,3.2,1.3,0.2],'Iris-setosa',1),
+(4,ARRAY[4.6,3.1,1.5,0.2],'Iris-setosa',1),
+(5,ARRAY[5.0,3.6,1.4,0.2],'Iris-setosa',1),
+(6,ARRAY[5.4,3.9,1.7,0.4],'Iris-setosa',1),
+(7,ARRAY[4.6,3.4,1.4,0.3],'Iris-setosa',1),
+(8,ARRAY[5.0,3.4,1.5,0.2],'Iris-setosa',1),
+(9,ARRAY[4.4,2.9,1.4,0.2],'Iris-setosa',1),
+(10,ARRAY[4.9,3.1,1.5,0.1],'Iris-setosa',1),
+(11,ARRAY[7.0,3.2,4.7,1.4],'Iris-versicolor',2),
+(12,ARRAY[6.4,3.2,4.5,1.5],'Iris-versicolor',2),
+(13,ARRAY[6.9,3.1,4.9,1.5],'Iris-versicolor',2),
+(14,ARRAY[5.5,2.3,4.0,1.3],'Iris-versicolor',2),
+(15,ARRAY[6.5,2.8,4.6,1.5],'Iris-versicolor',2),
+(16,ARRAY[5.7,2.8,4.5,1.3],'Iris-versicolor',2),
+(17,ARRAY[6.3,3.3,4.7,1.6],'Iris-versicolor',2),
+(18,ARRAY[4.9,2.4,3.3,1.0],'Iris-versicolor',2),
+(19,ARRAY[6.6,2.9,4.6,1.3],'Iris-versicolor',2),
+(20,ARRAY[5.2,2.7,3.9,1.4],'Iris-versicolor',2);
+</pre>
+-# Generate a multilayer perceptron with a single hidden layer of 5 units.
+Use the attributes column as the independent variables, and use the class
+column as the classification. Set the tolerance to 0 so that 5000
+iterations will be run. Use a hyperbolic tangent activation function.
+The model will be written to mlp_result.
+<pre class="example">
+SELECT madlib.mlp_classification(
+ 'iris_data_2', -- Source table
+ 'mlp_model', -- Destination table
+ 'attributes', -- Input features
+ 'class_text', -- Label
+ ARRAY[5], -- Number of units per layer
+ 'step_size=0.003,
+ n_iterations=5000,
+ tolerance=0', -- Optimizer params
+ 'tanh'); -- Activation function
+</pre>
+-# View the result for the model.
+<pre class="example">
+-- Set extended display on for easier reading of output
+\\x ON
+-- Neural net Initialization is non-deterministic, so your results may vary
+SELECT * FROM mlp_model;
+</pre>
+Result:
+<pre class="result">
+-[ RECORD 1 ]--+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+coeff | {1,1,1,1,1,0.136374930803,0.188739676875,0.662387810001,-1.03381622734,-0.469961067046,0.0614006983397,0.0811504589436,0.299008228258,-0.47391918521,-0.215098143699,0.10519213944,0.145844617525,0.511683525606,-0.800215552382,-0.36417142683,0.120751709056,0.167531106521,0.587074895969,-0.916946198095,-0.417055067449,0.0539541885146,0.0694359704131,0.262598585854,-0.419234805076,-0.189915344282,1,1,1,1,1,1,0.105645702152,1.46247470474,0.484457903226,0.965962824478,1.19361986431,0.419805760087,-0.105696503487,-1.46245956666,-0.484427811691,-0.965730981426,-1.19365280555,-0.419973628863}
+loss | 0.0184092375519
+num_iterations | 5000
+</pre>
+-# Next train a regression example. First create some test data. This dataset
+contains housing prices data.
+<pre class="example">
+CREATE TABLE lin_housing (id serial, x float8[], grp_by_col int, y float8);
+COPY lin_housing (x, grp_by_col, y) FROM STDIN NULL '?' DELIMITER '|';
+{1,0.00632,18.00,2.310,0,0.5380,6.5750,65.20,4.0900,1,296.0,15.30,396.90,4.98} | 1 | 24.00
+{1,0.02731,0.00,7.070,0,0.4690,6.4210,78.90,4.9671,2,242.0,17.80,396.90,9.14} | 1 | 21.60
+{1,0.02729,0.00,7.070,0,0.4690,7.1850,61.10,4.9671,2,242.0,17.80,392.83,4.03} | 1 | 34.70
+{1,0.03237,0.00,2.180,0,0.4580,6.9980,45.80,6.0622,3,222.0,18.70,394.63,2.94} | 1 | 33.40
+{1,0.06905,0.00,2.180,0,0.4580,7.1470,54.20,6.0622,3,222.0,18.70,396.90,5.33} | 1 | 36.20
+{1,0.02985,0.00,2.180,0,0.4580,6.4300,58.70,6.0622,3,222.0,18.70,394.12,5.21} | 1 | 28.70
+{1,0.08829,12.50,7.870,0,0.5240,6.0120,66.60,5.5605,5,311.0,15.20,395.60,12.43} | 1 | 22.90
+{1,0.14455,12.50,7.870,0,0.5240,6.1720,96.10,5.9505,5,311.0,15.20,396.90,19.15} | 1 | 27.10
+{1,0.21124,12.50,7.870,0,0.5240,5.6310,100.00,6.0821,5,311.0,15.20,386.63,29.93} | 1 | 16.50
+{1,0.17004,12.50,7.870,0,0.5240,6.0040,85.90,6.5921,5,311.0,15.20,386.71,17.10} | 1 | 18.90
+{1,0.22489,12.50,7.870,0,0.5240,6.3770,94.30,6.3467,5,311.0,15.20,392.52,20.45} | 1 | 15.00
+{1,0.11747,12.50,7.870,0,0.5240,6.0090,82.90,6.2267,5,311.0,15.20,396.90,13.27} | 1 | 18.90
+{1,0.09378,12.50,7.870,0,0.5240,5.8890,39.00,5.4509,5,311.0,15.20,390.50,15.71} | 1 | 21.70
+{1,0.62976,0.00,8.140,0,0.5380,5.9490,61.80,4.7075,4,307.0,21.00,396.90,8.26} | 1 | 20.40
+{1,0.63796,0.00,8.140,0,0.5380,6.0960,84.50,4.4619,4,307.0,21.00,380.02,10.26} | 1 | 18.20
+{1,0.62739,0.00,8.140,0,0.5380,5.8340,56.50,4.4986,4,307.0,21.00,395.62,8.47} | 1 | 19.90
+{1,1.05393,0.00,8.140,0,0.5380,5.9350,29.30,4.4986,4,307.0,21.00,386.85,6.58} | 1 | 23.10
+{1,0.78420,0.00,8.140,0,0.5380,5.9900,81.70,4.2579,4,307.0,21.00,386.75,14.67} | 1 | 17.50
+{1,0.80271,0.00,8.140,0,0.5380,5.4560,36.60,3.7965,4,307.0,21.00,288.99,11.69} | 1 | 20.20
+{1,0.72580,0.00,8.140,0,0.5380,5.7270,69.50,3.7965,4,307.0,21.00,390.95,11.28} | 1 | 18.20
+\.
+</pre>
+-# Now train a regression model using a multilayer perceptron a single hidden layer of two nodes.
+<pre class="example">
+SELECT mlp_regression(
+ 'lin_housing', -- Source table
+ 'mlp_regress', -- Desination table
+ 'x', -- Input features
+ 'y', -- Dependent variable
+ ARRAY[5,5], -- Number of units per layer
+ 'step_size=0.000007,
+ n_iterations=5000,
+ tolerance=0',
+ 'relu');
+</pre>
+-# Check the results of the model
+<pre class="example">
+-- Set extended display on for easier reading of output
+\\x ON
+-- Neural net Initialization is non-deterministic, so your results may vary
+SELECT * FROM mlp_regress;
+</pre>
+Result:
+<pre class="result">
+-[ RECORD 1 ]--+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------
+coeff | {1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2.79506311399e-05,3.56715008915e-05,-6.09333559685e-05,0.000251228318768,-0.000224772841379,-3.71863030857e-05,-3.5757865148e-06,5.27936784854e-05,-2.48474166186e-05,6.19731184294e-05,3.07638968743e-05,6.8964698578e-06,0.000106016701083,-1.71484730318e-05,1.18691881812e-05,-0.000163975464208,0.000170026304906,3.11688265279e-05,0.000177050148787,-1.58265976603e-05,2.70144422657e-05,0.000112667883422,3.77575139073e-05,8.12474658795e-05,-7.90458917626e-05,0.000107566386158,-2.63771171506e-06,2.47996880915e-05,-0.00012642310887,0.000203827391081,0.000139315565565,4.86147243454e-05,-0.000176126471913,-6.47820782916e-05,-8.51592776447e-06,-6.60601176758e-05,2.91421874156e-05,6.3556873752e-05,0.000197557443129,0.000220531367259,0.000135036310289,0.000143735913975,-4.75034117786e-05,-0.000179547345838,-1.6919846786e-05,0.000162784312994,0.000268595819851,-0.000460066553287,8.69756071591e-05,-0.00311762727057,0.000126024763103,0.00020598824
2921,0.003463432426,-0.00729789075286,0.00151625867549,-0.000890852767597,-0.00525016037249,0.0031043106659,0.00798041103839,-0.00552693050079,0.0232180415786,0.0230489850143,-0.0437890272341,0.0165765426407,-0.248554261758,-7.81336427846e-05,0.00558145591752,0.283465844585,-0.571699956182,0.133474351994,-0.0785181945605,-0.419269930709,0.249547772912,0.631761009875,-0.431305975666,1,1,1,1,1,1,0.0158747497572,-9.02809160806e-05,0.00015574347618,4.10805373863e-06,0.00121532434965,0.101790351335,0.0647558401493,-0.00013654998677,-9.92872075948e-06,-5.5319694394e-05,0.00519320756484,0.412736586036,0.0011998026977,-1.53688189815e-05,1.94817888201e-05,-4.63111489966e-05,7.24547899029e-05,0.00880394144485,5.45309822095e-05,-0.000140943219275,-7.96211486227e-05,-1.04337307472e-05,0.000161936762028,0.00136273797767,-4.54737243585e-05,-3.4083840736e-05,3.69286883662e-05,9.9047243188e-08,3.75014011824e-06,-9.45366086368e-08,1,1,1,1,1,1,6.67488547054,0.102754199001,0.41668912471,0.008868672964
79,0.00136206007228,-9.88642499013e-05}
+loss | 144.965776158
+num_iterations | 10000
+</pre>
+-# Now let's look at the prediction functions. In the following examples we will
+use the training data set for prediction as well, which is not usual but serves to
+show the syntax. First we will test the classification exampel.
+The prediction is in the the estimated_class_text column with the
+actual value in the class_text column.
+<pre class="example">
+SELECT mlp_predict(
+ 'mlp_model', -- Model table
+ 'iris_data', -- Test data table
+ 'id', -- Id column in test table
+ 'mlp_prediction', -- Output table for predictions
+ 'response' -- Output classes, not probabilities
+ );
+SELECT * FROM mlp_prediction JOIN iris_data USING (id);
+</pre>
+Result:
+<pre class="result">
+ id | estimated_class_text | attributes | class_text | class
+----+----------------------+-------------------+-----------------+-------
+ 1 | Iris-setosa | {5.1,3.5,1.4,0.2} | Iris-setosa | 1
+ 2 | Iris-setosa | {4.9,3.0,1.4,0.2} | Iris-setosa | 1
+ 3 | Iris-setosa | {4.7,3.2,1.3,0.2} | Iris-setosa | 1
+ 4 | Iris-setosa | {4.6,3.1,1.5,0.2} | Iris-setosa | 1
+ 5 | Iris-setosa | {5.0,3.6,1.4,0.2} | Iris-setosa | 1
+ 6 | Iris-setosa | {5.4,3.9,1.7,0.4} | Iris-setosa | 1
+ 7 | Iris-setosa | {4.6,3.4,1.4,0.3} | Iris-setosa | 1
+ 8 | Iris-setosa | {5.0,3.4,1.5,0.2} | Iris-setosa | 1
+ 9 | Iris-setosa | {4.4,2.9,1.4,0.2} | Iris-setosa | 1
+ 10 | Iris-setosa | {4.9,3.1,1.5,0.1} | Iris-setosa | 1
+ 11 | Iris-versicolor | {7.0,3.2,4.7,1.4} | Iris-versicolor | 2
+ 12 | Iris-versicolor | {6.4,3.2,4.5,1.5} | Iris-versicolor | 2
+ 13 | Iris-versicolor | {6.9,3.1,4.9,1.5} | Iris-versicolor | 2
+ 14 | Iris-versicolor | {5.5,2.3,4.0,1.3} | Iris-versicolor | 2
+ 15 | Iris-versicolor | {6.5,2.8,4.6,1.5} | Iris-versicolor | 2
+ 16 | Iris-versicolor | {5.7,2.8,4.5,1.3} | Iris-versicolor | 2
+ 17 | Iris-versicolor | {6.3,3.3,4.7,1.6} | Iris-versicolor | 2
+ 18 | Iris-versicolor | {4.9,2.4,3.3,1.0} | Iris-versicolor | 2
+ 19 | Iris-versicolor | {6.6,2.9,4.6,1.3} | Iris-versicolor | 2
+ 20 | Iris-versicolor | {5.2,2.7,3.9,1.4} | Iris-versicolor | 2
+</pre>
+Prediction using the regression model:
+<pre class="example">
+SELECT mlp_predict(
+ 'mlp_regress', -- Model table
+ 'lin_housing', -- Test data table
+ 'id', -- Id column in test table
+ 'mlp_regress_prediction', -- Output table for predictions
+ 'response' -- Output values, not probabilities
+ );
+</pre>
+-# View results
+<pre class="example">
+SELECT madlib.mlp_predict('mlp_regress','lin_housing','id','mlp_regress_prediction','response');
+SELECT * FROM lin_housing JOIN mlp_regress_prediction USING (id);
+</pre>
+Result for the regression model:
+<pre class="result">
+ id | x | grp_by_col | y | estimated_y
+----+-------------------------------------------------------------------------+------------+------+--------------------
+ 1 | {1,0.00632,18,2.31,0,0.538,6.575,65.2,4.09,1,296,15.3,396.9,4.98} | 1 | 24 | {23.2627062018087}
+ 2 | {1,0.02731,0,7.07,0,0.469,6.421,78.9,4.9671,2,242,17.8,396.9,9.14} | 1 | 21.6 | {25.7088419115781}
+ 3 | {1,0.02729,0,7.07,0,0.469,7.185,61.1,4.9671,2,242,17.8,392.83,4.03} | 1 | 34.7 | {27.5587003901404}
+ 4 | {1,0.03237,0,2.18,0,0.458,6.998,45.8,6.0622,3,222,18.7,394.63,2.94} | 1 | 33.4 | {31.1812237427816}
+ 5 | {1,0.06905,0,2.18,0,0.458,7.147,54.2,6.0622,3,222,18.7,396.9,5.33} | 1 | 36.2 | {30.3696873085477}
+ 6 | {1,0.02985,0,2.18,0,0.458,6.43,58.7,6.0622,3,222,18.7,394.12,5.21} | 1 | 28.7 | {29.5290259241882}
+ 7 | {1,0.08829,12.5,7.87,0,0.524,6.012,66.6,5.5605,5,311,15.2,395.6,12.43} | 1 | 22.9 | {21.1576051716888}
+ 8 | {1,0.14455,12.5,7.87,0,0.524,6.172,96.1,5.9505,5,311,15.2,396.9,19.15} | 1 | 27.1 | {17.6194200563055}
+ 9 | {1,0.21124,12.5,7.87,0,0.524,5.631,100,6.0821,5,311,15.2,386.63,29.93} | 1 | 16.5 | {15.1366297774139}
+10 | {1,0.17004,12.5,7.87,0,0.524,6.004,85.9,6.5921,5,311,15.2,386.71,17.1} | 1 | 18.9 | {17.6528662199369}
+11 | {1,0.22489,12.5,7.87,0,0.524,6.377,94.3,6.3467,5,311,15.2,392.52,20.45} | 1 | 15 | {17.2017487668181}
+12 | {1,0.11747,12.5,7.87,0,0.524,6.009,82.9,6.2267,5,311,15.2,396.9,13.27} | 1 | 18.9 | {19.4893860319992}
+13 | {1,0.09378,12.5,7.87,0,0.524,5.889,39,5.4509,5,311,15.2,390.5,15.71} | 1 | 21.7 | {23.2917226708039}
+14 | {1,0.62976,0,8.14,0,0.538,5.949,61.8,4.7075,4,307,21,396.9,8.26} | 1 | 20.4 | {22.8904812605193}
+15 | {1,0.63796,0,8.14,0,0.538,6.096,84.5,4.4619,4,307,21,380.02,10.26} | 1 | 18.2 | {18.2386754423677}
+16 | {1,0.62739,0,8.14,0,0.538,5.834,56.5,4.4986,4,307,21,395.62,8.47} | 1 | 19.9 | {23.28949550874}
+17 | {1,1.05393,0,8.14,0,0.538,5.935,29.3,4.4986,4,307,21,386.85,6.58} | 1 | 23.1 | {25.3288762085473}
+18 | {1,0.7842,0,8.14,0,0.538,5.99,81.7,4.2579,4,307,21,386.75,14.67} | 1 | 17.5 | {19.0203738118451}
+19 | {1,0.80271,0,8.14,0,0.538,5.456,36.6,3.7965,4,307,21,288.99,11.69} | 1 | 20.2 | {12.3162005347545}
+20 | {1,0.7258,0,8.14,0,0.538,5.727,69.5,3.7965,4,307,21,390.95,11.28} | 1 | 18.2 | {21.0902211848747}
+</pre>
+Note that the results you get for all examples may vary with the platform you are using.
+
+@anchor background
+@par Technical Background
+
+To train a neural net, the respective loss function is minimized using stochastic gradient descent.
+In the case of classification, the loss function is cross entropy. For regression, mean square error
+is used. Weights in the neural net are updated via the backpropogation process, which uses dynamic
+programming to compute the partial derivative of each weight with respect to the overall loss. This
+partial derivative incorporates the respective activation function used, so this requires that the
+activation function be differentiable.
+
+@anchor literature
+@literature
+
+@anchor mlp-lit-1
+[1] Hastie, Trevor. Neural network. John Wiley & Sons, Ltd, 1998.
+
--- End diff --
Is it necessary to add three literature references that were added to doc/literature.bib? I noticed there were not added to the generated doc for the PR in the page titled "Multilayer Perceptron". The Literature section only lists the previous reference "[1] Hastie, Trevor. Neural network. John Wiley & Sons, Ltd, 1998."
```
+@misc{ann_wiki,
+ Url = {http://en.wikipedia.org/wiki/Artificial_neural_network},
+ Title = {Artificial neural network},
+ Author = {Wikipedia}
+}
+
+@misc{mlp_wiki,
+ Url = {http://en.wikipedia.org/wiki/Multilayer_perceptron},
+ Title = {Multilayer perceptron},
+ Author = {Wikipedia}
+}
+
+@misc{mlp_gradient_wisc,
+ Url = {http://homepages.cae.wisc.edu/~ece539/videocourse/notes/pdf/lec%2011%20MLP%20(3)%20BP.pdf},
+ Title = {{MLP(III): Back-Propagation}},
+ Author = {{Yu Hen Hu}}
+}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by njayaram2 <gi...@git.apache.org>.
Github user njayaram2 commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r128100838
--- Diff: src/ports/postgres/modules/convex/mlp_igd.py_in ---
@@ -0,0 +1,734 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+@file mlp_igd.py_in
+
+@brief Multilayer perceptron using IGD: Driver functions
+
+@namespace mlp_igd
+"""
+import plpy
+
+from utilities.control import MinWarning
+from utilities.utilities import add_postfix
+from utilities.utilities import py_list_to_sql_string
+from utilities.utilities import extract_keyvalue_params
+from utilities.utilities import _assert
+from utilities.utilities import unique_string
+from utilities.utilities import strip_end_quotes
+
+from utilities.validate_args import cols_in_tbl_valid
+from utilities.validate_args import input_tbl_valid
+from utilities.validate_args import is_var_valid
+from utilities.validate_args import output_tbl_valid
+from utilities.validate_args import get_expr_type
+from utilities.validate_args import array_col_has_same_dimension
+from utilities.validate_args import array_col_dimension
+
+
+def mlp(schema_madlib, source_table, output_table, independent_varname,
+ dependent_varname, hidden_layer_sizes,
+ optimizer_param_str, activation, is_classification, **kwargs):
+ """
+ Args:
+ @param schema_madlib
+ @param source_table
+ @param output_table
+ @param independent_varname
+ @param dependent_varname
+ @param hidden_layer_sizes
+ @param optimizer_param_str
+
+ Returns:
+ None
+ """
+ with MinWarning('info'):
--- End diff --
`info` -> `warning`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by edespino <gi...@git.apache.org>.
Github user edespino commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r127043254
--- Diff: src/ports/postgres/modules/convex/mlp.sql_in ---
@@ -0,0 +1,736 @@
+/* ----------------------------------------------------------------------- *//**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ *
+ * @file mlp.sql_in
+ *
+ * @brief SQL functions for multi-layer perceptron
+ * @date June 2012
+ *
+ *
+ *//* ----------------------------------------------------------------------- */
+
+m4_include(`SQLCommon.m4')
+
+/**
+@addtogroup grp_mlp
+
+<div class="toc"><b>Contents</b><ul>
+<li class="level1"><a href="#mlp_classification">Classification</a></li>
+<li class="level1"><a href="#mlp_regression">Regression</a></li>
+<li class="level1"><a href="#optimization_params">Optimization Parameters</a></li>
+<li class="level1"><a href="#predict">Prediction Functions/a></li>
+<li class="level1"><a href="#example">Examples</a></li>
+<li class="level1"><a href="#background">Technical Background</a></li>
+<li class="level1"><a href="#literature">Literature</a></li>
+<li class="level1"><a href="#related">Related Topics</a></li>
+</ul></div>
+
+Multi Layer Perceptron (MLP) is a model for regression and
+classification
+
+Also called "vanilla neural networks", they consist of several
+fully connected hidden layers with non-linear activation
+functions. In the case of classification, the final layer of the
+neural net has as many nodes as classes, and the output of the
+neural net can be interpreted as the probability of a given input
+feature belonging to a specific class.
+
+
+@brief Solves classification and regression problems with several
+fully connected layers and nonlinear activation functions.
+
+@anchor mlp_classification
+@par Classification Training Function
+The mlp classification training function has the following format:
+<pre class="syntax">
+mlp_classification(
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+\b Arguments
+<DL class="arglist">
+ <DT>source_table</DT>
+ <DD>TEXT. Name of the table containing the training data.</DD>
+
+ <DT>model_table</DT>
+ <DD>TEXT. Name of the output table containing the model. Details of the output
+ tables are provided below.
+ </DD>
+
+ <DT>independent_varname</DT>
+ <DD>TEXT. Expression list to evaluate for the
+ independent variables. An intercept variable should not be included as part
+ of this expression. Please note that expression should be able to be cast
+ to DOUBLE PRECISION[].
+ </DD>
+
+ <DT>dependent_varname</DT>
+ <DD> TEXT. Name of the dependent variable column. For classification, supported types are:
+ text, varchar, character varying, char, character
+ integer, smallint, bigint, and boolean. </DD>
+
+ <DT>hidden_layer_sizes</DT>
+ <DD>INTEGER[], default: ARRAY[].
+ The number of neurons in each hidden layer. The length of this array will
+ determine the number of hidden layers. Empty for no hidden layers.
+ </DD>
+
+
+ <DT>optimizer_params (optional)</DT>
+ <DD>TEXT, default: NULL.
+ Parameters for optimization in a comma-separated string
+ of key-value pairs. See the description below for details.
+ </DD>
+
+ <DT>activation</DT>
+ <DD>TEXT, default: 'sigmoid'.
+ Activation function. Currently three functions are supported: 'sigmoid' (default),
+ 'relu', and 'tanh'. The text can be any subset of the three
+ strings; for e.g., activation='s' will use the sigmoid activation.
+ </DD>
+</DL>
+
+<b>Output tables</b>
+<br>
+ The model table produced by mlp contains the following columns:
+ <table class="output">
+ <tr>
+ <th>coeffs</th>
+ <td>FLOAT8[]. Flat array containing the weights of the neural net</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>INTEGER. Number of iterations completed by stochastic gradient descent
+ algorithm. The algorithm either converged in this number of iterations
+ or hit the maximum number specified in the optimization parameters. </td>
+ </tr>
+ <tr>
+ <th>loss</th>
+ <td>FLOAT8. The cross entropy over the training data.
+ See Technical Background section below for more details.</td>
+ </tr>
+ </table>
+
+
+A summary table named \<model_table\>_summary is also created, which has the following columns:
+ <table class="output">
+ <tr>
+ <th>source_table</th>
+ <td>The source table.</td>
+ </tr>
+ <tr>
+ <th>dependent_varname</th>
+ <td>The dependent variable.</td>
+ </tr>
+ <tr>
+ <th>independent_varname</th>
+ <td>The independent variables.</td>
+ </tr>
+ <tr>
+ <th>tolerance</th>
+ <td>The tolerance as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>step_size</th>
+ <td>The step size as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>The number of iterations run</td>
+ </tr>
+ <tr>
+ <th>n_tries</th>
+ <td>The number of tries as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>layer_sizes</th>
+ <td>The number of units in each layer including the input and output layer.</td>
+ </tr>
+ <tr>
+ <th>activation_function</th>
+ <td>The activation function.</td>
+ </tr>
+ <tr>
+ <th>is_classification</th>
+ <td>True if the model was trained for classification, False if it was trained
+ for regression</td>
+ </tr>
+ <tr>
+ <th>classes</th>
+ <td>The classes which were trained against (empty for regression)</td>
+ </tr>
+
+ </table>
+
+
+@anchor mlp_regression
+@par Regression Training Function
+The mlp regression training function has the following format:
+<pre class="syntax">
+mlp_regression(source_table,
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+
+\b Arguments
+
+Specifications for regression are largely the same as for classification. In the
+model table, the loss will refer to mean square error instead of cross entropy. In the
+summary table, there is classes column. The following
+arguments have specifications which differ from mlp_classification:
+<DL class="arglist">
+<DT>dependent_varname</DT>
+ <DD>TEXT. Name of the dependent variable column.
+ For regression supported types are any numeric type, or array
+ or numeric types (for multiple regression).
+ </DD>
+</DL>
+
+
+@anchor optimizer_params
+@par Optimizer Parameters
+Parameters in this section are supplied in the \e optimizer_params argument as a string
+containing a comma-delimited list of name-value pairs. All of these named
+parameters are optional, and their order does not matter. You must use the
+format "<param_name> = <value>" to specify the value of a parameter, otherwise
+the parameter is ignored.
+
+
+<pre class="syntax">
+ 'step_size = <value>,
+ n_iterations = <value>,
+ n_tries = <value>,
+ tolerance = <value>'
+</pre>
+\b Optimizer Parameters
+<DL class="arglist">
+
+<DT>step_size</dt>
+<DD>Default: [0.001].
+Also known as the learning rate. A small value is usually desirable to
+ensure convergence, while a large value provides more room for progress during
+training. Since the best value depends on the condition number of the data, in
+practice one often tunes this parameter.
+</DD>
+
+
+<DT>n_iterations</dt>
+<DD>Default: [100]. The maximum number of iterations allowed.
+</DD>
+<DT>n_tries</dt>
+<DD>Default: [1]. Number of times to retrain the network with randomly initialized
+weights
+</DD>
+
+<DT>tolerance</dt>
+<DD>Default: 0.001. The criterion to end iterations. The training stops whenever
+<the difference between the training models of two consecutive iterations is
+<smaller than \e tolerance or the iteration number is larger than \e max_iter.
+</DD>
+
+</DL>
+
+@anchor predict
+@par Prediction Function
+Used to generate predictions given a previously trained model on novel data.
+The same syntax is used for classification, and regression.
+<pre class="syntax">
+mlp_predict(model_table,
+ data_table,
+ id_col_name,
+ output_table,
+ pred_type)
+</pre>
+
+\b Arguments
+<DL class="arglist">
+ <DT>model_table</DT>
+ <DD>TEXT. Model table produced by the training function.</DD>
+
+ <DT>data_table</DT>
+ <DD>TEXT. Name of the table containing the data for prediction. This table is expected
+ to contain the same input features that were used during training. The table should
+ also contain id_col_name used for identifying each row.</DD>
+
+ <DT>id_col_name</DT>
+ <DD>TEXT. The name of the id column in the input table.</DD>
+
+ <DT>output_table</DT>
+ <DD>TEXT. Name of the table where output predictions are written. If this
+table name is already in use, then an error is returned. Table contains:</DD>
+ <table class="output">
+ <tr>
+ <th>id</th>
+ <td>Gives the 'id' for each prediction, corresponding to each row from the data_table.</td>
+ </tr>
+ <tr>
+ <th>estimated_<COL_NAME></th>
+ <td>
+ (For pred_type='response') The estimated class
+ for classification or value for regression, where
+ <COL_NAME> is the name of the column to be
+ predicted from training data
+ </td>
+ </tr>
+ <tr>
+ <th>prob_<CLASS></th>
+ <td>
+ (For pred_type='prob' for classification) The
+ probability of a given class <CLASS> as given by
+ softmax. There will be one column for each class
+ in the training data.
+ </td>
+ </tr>
+
+
+ <DT>pred_type</DT>
+ <DD>TEXT.
+
+the type of output requested:
+'response' gives the actual prediction,
+'prob' gives the probability of each class.
+for regression, only type='response' is defined.
+The name of the id column in the input table.</DD>
+</DL>
+</table>
+
+@anchor example
+@par Examples
+-# Create an input data set.
+<pre class="example">
+CREATE TABLE iris_data(
+ id integer,
+ attributes numeric[],
+ class_text varchar,
+ class integer
+);
+INSERT INTO iris_data VALUES
+(1,ARRAY[5.1,3.5,1.4,0.2],'Iris-setosa',1),
+(2,ARRAY[4.9,3.0,1.4,0.2],'Iris-setosa',1),
+(3,ARRAY[4.7,3.2,1.3,0.2],'Iris-setosa',1),
+(4,ARRAY[4.6,3.1,1.5,0.2],'Iris-setosa',1),
+(5,ARRAY[5.0,3.6,1.4,0.2],'Iris-setosa',1),
+(6,ARRAY[5.4,3.9,1.7,0.4],'Iris-setosa',1),
+(7,ARRAY[4.6,3.4,1.4,0.3],'Iris-setosa',1),
+(8,ARRAY[5.0,3.4,1.5,0.2],'Iris-setosa',1),
+(9,ARRAY[4.4,2.9,1.4,0.2],'Iris-setosa',1),
+(10,ARRAY[4.9,3.1,1.5,0.1],'Iris-setosa',1),
+(11,ARRAY[7.0,3.2,4.7,1.4],'Iris-versicolor',2),
+(12,ARRAY[6.4,3.2,4.5,1.5],'Iris-versicolor',2),
+(13,ARRAY[6.9,3.1,4.9,1.5],'Iris-versicolor',2),
+(14,ARRAY[5.5,2.3,4.0,1.3],'Iris-versicolor',2),
+(15,ARRAY[6.5,2.8,4.6,1.5],'Iris-versicolor',2),
+(16,ARRAY[5.7,2.8,4.5,1.3],'Iris-versicolor',2),
+(17,ARRAY[6.3,3.3,4.7,1.6],'Iris-versicolor',2),
+(18,ARRAY[4.9,2.4,3.3,1.0],'Iris-versicolor',2),
+(19,ARRAY[6.6,2.9,4.6,1.3],'Iris-versicolor',2),
+(20,ARRAY[5.2,2.7,3.9,1.4],'Iris-versicolor',2);
+</pre>
+-# Generate a multilayer perceptron with a single hidden layer of 5 units.
+Use the attributes column as the independent variables, and use the class
+column as the classification. Set the tolerance to 0 so that 5000
+iterations will be run. Use a hyperbolic tangent activation function.
+The model will be written to mlp_result.
+<pre class="example">
+SELECT madlib.mlp_classification(
+ 'iris_data_2', -- Source table
+ 'mlp_model', -- Destination table
+ 'attributes', -- Input features
+ 'class_text', -- Label
+ ARRAY[5], -- Number of units per layer
+ 'step_size=0.003,
+ n_iterations=5000,
+ tolerance=0', -- Optimizer params
+ 'tanh'); -- Activation function
+</pre>
+-# View the result for the model.
+<pre class="example">
+-- Set extended display on for easier reading of output
+\\x ON
+-- Neural net Initialization is non-deterministic, so your results may vary
+SELECT * FROM mlp_model;
+</pre>
+Result:
+<pre class="result">
+-[ RECORD 1 ]--+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+coeff | {1,1,1,1,1,0.136374930803,0.188739676875,0.662387810001,-1.03381622734,-0.469961067046,0.0614006983397,0.0811504589436,0.299008228258,-0.47391918521,-0.215098143699,0.10519213944,0.145844617525,0.511683525606,-0.800215552382,-0.36417142683,0.120751709056,0.167531106521,0.587074895969,-0.916946198095,-0.417055067449,0.0539541885146,0.0694359704131,0.262598585854,-0.419234805076,-0.189915344282,1,1,1,1,1,1,0.105645702152,1.46247470474,0.484457903226,0.965962824478,1.19361986431,0.419805760087,-0.105696503487,-1.46245956666,-0.484427811691,-0.965730981426,-1.19365280555,-0.419973628863}
+loss | 0.0184092375519
+num_iterations | 5000
+</pre>
+-# Next train a regression example. First create some test data. This dataset
+contains housing prices data.
+<pre class="example">
+CREATE TABLE lin_housing (id serial, x float8[], grp_by_col int, y float8);
+COPY lin_housing (x, grp_by_col, y) FROM STDIN NULL '?' DELIMITER '|';
+{1,0.00632,18.00,2.310,0,0.5380,6.5750,65.20,4.0900,1,296.0,15.30,396.90,4.98} | 1 | 24.00
+{1,0.02731,0.00,7.070,0,0.4690,6.4210,78.90,4.9671,2,242.0,17.80,396.90,9.14} | 1 | 21.60
+{1,0.02729,0.00,7.070,0,0.4690,7.1850,61.10,4.9671,2,242.0,17.80,392.83,4.03} | 1 | 34.70
+{1,0.03237,0.00,2.180,0,0.4580,6.9980,45.80,6.0622,3,222.0,18.70,394.63,2.94} | 1 | 33.40
+{1,0.06905,0.00,2.180,0,0.4580,7.1470,54.20,6.0622,3,222.0,18.70,396.90,5.33} | 1 | 36.20
+{1,0.02985,0.00,2.180,0,0.4580,6.4300,58.70,6.0622,3,222.0,18.70,394.12,5.21} | 1 | 28.70
+{1,0.08829,12.50,7.870,0,0.5240,6.0120,66.60,5.5605,5,311.0,15.20,395.60,12.43} | 1 | 22.90
+{1,0.14455,12.50,7.870,0,0.5240,6.1720,96.10,5.9505,5,311.0,15.20,396.90,19.15} | 1 | 27.10
+{1,0.21124,12.50,7.870,0,0.5240,5.6310,100.00,6.0821,5,311.0,15.20,386.63,29.93} | 1 | 16.50
+{1,0.17004,12.50,7.870,0,0.5240,6.0040,85.90,6.5921,5,311.0,15.20,386.71,17.10} | 1 | 18.90
+{1,0.22489,12.50,7.870,0,0.5240,6.3770,94.30,6.3467,5,311.0,15.20,392.52,20.45} | 1 | 15.00
+{1,0.11747,12.50,7.870,0,0.5240,6.0090,82.90,6.2267,5,311.0,15.20,396.90,13.27} | 1 | 18.90
+{1,0.09378,12.50,7.870,0,0.5240,5.8890,39.00,5.4509,5,311.0,15.20,390.50,15.71} | 1 | 21.70
+{1,0.62976,0.00,8.140,0,0.5380,5.9490,61.80,4.7075,4,307.0,21.00,396.90,8.26} | 1 | 20.40
+{1,0.63796,0.00,8.140,0,0.5380,6.0960,84.50,4.4619,4,307.0,21.00,380.02,10.26} | 1 | 18.20
+{1,0.62739,0.00,8.140,0,0.5380,5.8340,56.50,4.4986,4,307.0,21.00,395.62,8.47} | 1 | 19.90
+{1,1.05393,0.00,8.140,0,0.5380,5.9350,29.30,4.4986,4,307.0,21.00,386.85,6.58} | 1 | 23.10
+{1,0.78420,0.00,8.140,0,0.5380,5.9900,81.70,4.2579,4,307.0,21.00,386.75,14.67} | 1 | 17.50
+{1,0.80271,0.00,8.140,0,0.5380,5.4560,36.60,3.7965,4,307.0,21.00,288.99,11.69} | 1 | 20.20
+{1,0.72580,0.00,8.140,0,0.5380,5.7270,69.50,3.7965,4,307.0,21.00,390.95,11.28} | 1 | 18.20
+\.
+</pre>
+-# Now train a regression model using a multilayer perceptron a single hidden layer of two nodes.
+<pre class="example">
+SELECT mlp_regression(
+ 'lin_housing', -- Source table
+ 'mlp_regress', -- Desination table
+ 'x', -- Input features
+ 'y', -- Dependent variable
+ ARRAY[5,5], -- Number of units per layer
+ 'step_size=0.000007,
+ n_iterations=5000,
+ tolerance=0',
+ 'relu');
+</pre>
+-# Check the results of the model
+<pre class="example">
+-- Set extended display on for easier reading of output
+\\x ON
+-- Neural net Initialization is non-deterministic, so your results may vary
+SELECT * FROM mlp_regress;
+</pre>
+Result:
+<pre class="result">
+-[ RECORD 1 ]--+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------
+coeff | {1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2.79506311399e-05,3.56715008915e-05,-6.09333559685e-05,0.000251228318768,-0.000224772841379,-3.71863030857e-05,-3.5757865148e-06,5.27936784854e-05,-2.48474166186e-05,6.19731184294e-05,3.07638968743e-05,6.8964698578e-06,0.000106016701083,-1.71484730318e-05,1.18691881812e-05,-0.000163975464208,0.000170026304906,3.11688265279e-05,0.000177050148787,-1.58265976603e-05,2.70144422657e-05,0.000112667883422,3.77575139073e-05,8.12474658795e-05,-7.90458917626e-05,0.000107566386158,-2.63771171506e-06,2.47996880915e-05,-0.00012642310887,0.000203827391081,0.000139315565565,4.86147243454e-05,-0.000176126471913,-6.47820782916e-05,-8.51592776447e-06,-6.60601176758e-05,2.91421874156e-05,6.3556873752e-05,0.000197557443129,0.000220531367259,0.000135036310289,0.000143735913975,-4.75034117786e-05,-0.000179547345838,-1.6919846786e-05,0.000162784312994,0.000268595819851,-0.000460066553287,8.69756071591e-05,-0.00311762727057,0.000126024763103,0.00020598824
2921,0.003463432426,-0.00729789075286,0.00151625867549,-0.000890852767597,-0.00525016037249,0.0031043106659,0.00798041103839,-0.00552693050079,0.0232180415786,0.0230489850143,-0.0437890272341,0.0165765426407,-0.248554261758,-7.81336427846e-05,0.00558145591752,0.283465844585,-0.571699956182,0.133474351994,-0.0785181945605,-0.419269930709,0.249547772912,0.631761009875,-0.431305975666,1,1,1,1,1,1,0.0158747497572,-9.02809160806e-05,0.00015574347618,4.10805373863e-06,0.00121532434965,0.101790351335,0.0647558401493,-0.00013654998677,-9.92872075948e-06,-5.5319694394e-05,0.00519320756484,0.412736586036,0.0011998026977,-1.53688189815e-05,1.94817888201e-05,-4.63111489966e-05,7.24547899029e-05,0.00880394144485,5.45309822095e-05,-0.000140943219275,-7.96211486227e-05,-1.04337307472e-05,0.000161936762028,0.00136273797767,-4.54737243585e-05,-3.4083840736e-05,3.69286883662e-05,9.9047243188e-08,3.75014011824e-06,-9.45366086368e-08,1,1,1,1,1,1,6.67488547054,0.102754199001,0.41668912471,0.008868672964
79,0.00136206007228,-9.88642499013e-05}
+loss | 144.965776158
+num_iterations | 10000
+</pre>
+-# Now let's look at the prediction functions. In the following examples we will
+use the training data set for prediction as well, which is not usual but serves to
+show the syntax. First we will test the classification exampel.
+The prediction is in the the estimated_class_text column with the
+actual value in the class_text column.
+<pre class="example">
+SELECT mlp_predict(
+ 'mlp_model', -- Model table
+ 'iris_data', -- Test data table
+ 'id', -- Id column in test table
+ 'mlp_prediction', -- Output table for predictions
+ 'response' -- Output classes, not probabilities
+ );
+SELECT * FROM mlp_prediction JOIN iris_data USING (id);
+</pre>
+Result:
+<pre class="result">
+ id | estimated_class_text | attributes | class_text | class
+----+----------------------+-------------------+-----------------+-------
+ 1 | Iris-setosa | {5.1,3.5,1.4,0.2} | Iris-setosa | 1
+ 2 | Iris-setosa | {4.9,3.0,1.4,0.2} | Iris-setosa | 1
+ 3 | Iris-setosa | {4.7,3.2,1.3,0.2} | Iris-setosa | 1
+ 4 | Iris-setosa | {4.6,3.1,1.5,0.2} | Iris-setosa | 1
+ 5 | Iris-setosa | {5.0,3.6,1.4,0.2} | Iris-setosa | 1
+ 6 | Iris-setosa | {5.4,3.9,1.7,0.4} | Iris-setosa | 1
+ 7 | Iris-setosa | {4.6,3.4,1.4,0.3} | Iris-setosa | 1
+ 8 | Iris-setosa | {5.0,3.4,1.5,0.2} | Iris-setosa | 1
+ 9 | Iris-setosa | {4.4,2.9,1.4,0.2} | Iris-setosa | 1
+ 10 | Iris-setosa | {4.9,3.1,1.5,0.1} | Iris-setosa | 1
+ 11 | Iris-versicolor | {7.0,3.2,4.7,1.4} | Iris-versicolor | 2
+ 12 | Iris-versicolor | {6.4,3.2,4.5,1.5} | Iris-versicolor | 2
+ 13 | Iris-versicolor | {6.9,3.1,4.9,1.5} | Iris-versicolor | 2
+ 14 | Iris-versicolor | {5.5,2.3,4.0,1.3} | Iris-versicolor | 2
+ 15 | Iris-versicolor | {6.5,2.8,4.6,1.5} | Iris-versicolor | 2
+ 16 | Iris-versicolor | {5.7,2.8,4.5,1.3} | Iris-versicolor | 2
+ 17 | Iris-versicolor | {6.3,3.3,4.7,1.6} | Iris-versicolor | 2
+ 18 | Iris-versicolor | {4.9,2.4,3.3,1.0} | Iris-versicolor | 2
+ 19 | Iris-versicolor | {6.6,2.9,4.6,1.3} | Iris-versicolor | 2
+ 20 | Iris-versicolor | {5.2,2.7,3.9,1.4} | Iris-versicolor | 2
+</pre>
+Prediction using the regression model:
+<pre class="example">
+SELECT mlp_predict(
+ 'mlp_regress', -- Model table
+ 'lin_housing', -- Test data table
+ 'id', -- Id column in test table
+ 'mlp_regress_prediction', -- Output table for predictions
+ 'response' -- Output values, not probabilities
+ );
+</pre>
+-# View results
+<pre class="example">
+SELECT madlib.mlp_predict('mlp_regress','lin_housing','id','mlp_regress_prediction','response');
+SELECT * FROM lin_housing JOIN mlp_regress_prediction USING (id);
+</pre>
+Result for the regression model:
+<pre class="result">
+ id | x | grp_by_col | y | estimated_y
+----+-------------------------------------------------------------------------+------------+------+--------------------
+ 1 | {1,0.00632,18,2.31,0,0.538,6.575,65.2,4.09,1,296,15.3,396.9,4.98} | 1 | 24 | {23.2627062018087}
+ 2 | {1,0.02731,0,7.07,0,0.469,6.421,78.9,4.9671,2,242,17.8,396.9,9.14} | 1 | 21.6 | {25.7088419115781}
+ 3 | {1,0.02729,0,7.07,0,0.469,7.185,61.1,4.9671,2,242,17.8,392.83,4.03} | 1 | 34.7 | {27.5587003901404}
+ 4 | {1,0.03237,0,2.18,0,0.458,6.998,45.8,6.0622,3,222,18.7,394.63,2.94} | 1 | 33.4 | {31.1812237427816}
+ 5 | {1,0.06905,0,2.18,0,0.458,7.147,54.2,6.0622,3,222,18.7,396.9,5.33} | 1 | 36.2 | {30.3696873085477}
+ 6 | {1,0.02985,0,2.18,0,0.458,6.43,58.7,6.0622,3,222,18.7,394.12,5.21} | 1 | 28.7 | {29.5290259241882}
+ 7 | {1,0.08829,12.5,7.87,0,0.524,6.012,66.6,5.5605,5,311,15.2,395.6,12.43} | 1 | 22.9 | {21.1576051716888}
+ 8 | {1,0.14455,12.5,7.87,0,0.524,6.172,96.1,5.9505,5,311,15.2,396.9,19.15} | 1 | 27.1 | {17.6194200563055}
+ 9 | {1,0.21124,12.5,7.87,0,0.524,5.631,100,6.0821,5,311,15.2,386.63,29.93} | 1 | 16.5 | {15.1366297774139}
+10 | {1,0.17004,12.5,7.87,0,0.524,6.004,85.9,6.5921,5,311,15.2,386.71,17.1} | 1 | 18.9 | {17.6528662199369}
+11 | {1,0.22489,12.5,7.87,0,0.524,6.377,94.3,6.3467,5,311,15.2,392.52,20.45} | 1 | 15 | {17.2017487668181}
+12 | {1,0.11747,12.5,7.87,0,0.524,6.009,82.9,6.2267,5,311,15.2,396.9,13.27} | 1 | 18.9 | {19.4893860319992}
+13 | {1,0.09378,12.5,7.87,0,0.524,5.889,39,5.4509,5,311,15.2,390.5,15.71} | 1 | 21.7 | {23.2917226708039}
+14 | {1,0.62976,0,8.14,0,0.538,5.949,61.8,4.7075,4,307,21,396.9,8.26} | 1 | 20.4 | {22.8904812605193}
+15 | {1,0.63796,0,8.14,0,0.538,6.096,84.5,4.4619,4,307,21,380.02,10.26} | 1 | 18.2 | {18.2386754423677}
+16 | {1,0.62739,0,8.14,0,0.538,5.834,56.5,4.4986,4,307,21,395.62,8.47} | 1 | 19.9 | {23.28949550874}
+17 | {1,1.05393,0,8.14,0,0.538,5.935,29.3,4.4986,4,307,21,386.85,6.58} | 1 | 23.1 | {25.3288762085473}
+18 | {1,0.7842,0,8.14,0,0.538,5.99,81.7,4.2579,4,307,21,386.75,14.67} | 1 | 17.5 | {19.0203738118451}
+19 | {1,0.80271,0,8.14,0,0.538,5.456,36.6,3.7965,4,307,21,288.99,11.69} | 1 | 20.2 | {12.3162005347545}
+20 | {1,0.7258,0,8.14,0,0.538,5.727,69.5,3.7965,4,307,21,390.95,11.28} | 1 | 18.2 | {21.0902211848747}
+</pre>
+Note that the results you get for all examples may vary with the platform you are using.
+
+@anchor background
+@par Technical Background
+
+To train a neural net, the respective loss function is minimized using stochastic gradient descent.
+In the case of classification, the loss function is cross entropy. For regression, mean square error
+is used. Weights in the neural net are updated via the backpropogation process, which uses dynamic
+programming to compute the partial derivative of each weight with respect to the overall loss. This
+partial derivative incorporates the respective activation function used, so this requires that the
+activation function be differentiable.
+
+@anchor literature
+@literature
+
+@anchor mlp-lit-1
+[1] Hastie, Trevor. Neural network. John Wiley & Sons, Ltd, 1998.
+
--- End diff --
I mentioned this as I noticed other modules listed similar references in the docs:
example:
http://madlib.incubator.apache.org/docs/latest/group__grp__linreg.html
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by njayaram2 <gi...@git.apache.org>.
Github user njayaram2 commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r128099747
--- Diff: src/ports/postgres/modules/convex/mlp_igd.py_in ---
@@ -0,0 +1,734 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+@file mlp_igd.py_in
+
+@brief Multilayer perceptron using IGD: Driver functions
+
+@namespace mlp_igd
+"""
+import plpy
+
+from utilities.control import MinWarning
+from utilities.utilities import add_postfix
+from utilities.utilities import py_list_to_sql_string
+from utilities.utilities import extract_keyvalue_params
+from utilities.utilities import _assert
+from utilities.utilities import unique_string
+from utilities.utilities import strip_end_quotes
+
+from utilities.validate_args import cols_in_tbl_valid
+from utilities.validate_args import input_tbl_valid
+from utilities.validate_args import is_var_valid
+from utilities.validate_args import output_tbl_valid
+from utilities.validate_args import get_expr_type
+from utilities.validate_args import array_col_has_same_dimension
+from utilities.validate_args import array_col_dimension
+
+
+def mlp(schema_madlib, source_table, output_table, independent_varname,
+ dependent_varname, hidden_layer_sizes,
+ optimizer_param_str, activation, is_classification, **kwargs):
+ """
+ Args:
+ @param schema_madlib
+ @param source_table
+ @param output_table
+ @param independent_varname
+ @param dependent_varname
+ @param hidden_layer_sizes
+ @param optimizer_param_str
+
+ Returns:
+ None
+ """
+ with MinWarning('info'):
+ optimizer_params = _get_optimizer_params(optimizer_param_str or "")
+ _validate_args(source_table, output_table, independent_varname,
+ dependent_varname, hidden_layer_sizes,
+ optimizer_params, is_classification)
+
+ current_iteration = 1
+ prev_state = None
+ tolerance = optimizer_params["tolerance"]
+ n_iterations = optimizer_params["n_iterations"]
+ step_size = optimizer_params["step_size"]
+ n_tries = optimizer_params["n_tries"]
+ activation_name = _get_activation_function_name(activation)
+ activation_index = _get_activation_index(activation_name)
+ num_input_nodes = array_col_dimension(source_table, independent_varname)
+ num_output_nodes = 0
+ classes = []
+ dependent_type = get_expr_type(dependent_varname, source_table)
+ original_dependent_varname = dependent_varname
+
+ if is_classification:
+ dependent_variable_sql = """
+ SELECT DISTINCT {dependent_varname}
+ FROM {source_table}
+ """.format(dependent_varname=dependent_varname,
+ source_table=source_table)
+ labels = plpy.execute(dependent_variable_sql)
+ one_hot_dependent_varname = 'ARRAY['
+ num_output_nodes = len(labels)
+ for label_obj in labels:
+ label = _format_label(label_obj[dependent_varname])
+ classes.append(label)
+ one_hot_dependent_varname += dependent_varname + "=" + str(label) + ","
+ # Remove the last comma
+ one_hot_dependent_varname = one_hot_dependent_varname[:-1]
+ one_hot_dependent_varname += ']::integer[]'
+ dependent_varname = one_hot_dependent_varname
+ if not is_classification:
+ if "[]" not in dependent_type:
+ dependent_varname = "ARRAY[" + dependent_varname + "]"
+ num_output_nodes = array_col_dimension(source_table, dependent_varname)
+ layer_sizes = [num_input_nodes] + hidden_layer_sizes + [num_output_nodes]
+
+ while True:
+ if prev_state:
+ prev_state_str = py_list_to_sql_string(prev_state, array_type="double precision")
+ else:
+ prev_state_str = "(NULL)::DOUBLE PRECISION[]"
+ train_sql = """
+ SELECT
+ {schema_madlib}.mlp_igd_step(
+ ({independent_varname})::DOUBLE PRECISION[],
+ ({dependent_varname})::DOUBLE PRECISION[],
+ {prev_state},
+ {layer_sizes},
+ ({step_size})::FLOAT8,
+ {activation},
+ {is_classification}) as curr_state
+ FROM {source_table} AS _src
+ """.format(schema_madlib=schema_madlib,
+ independent_varname=independent_varname,
+ dependent_varname=dependent_varname,
+ prev_state=prev_state_str,
+ # C++ uses double internally
+ layer_sizes=py_list_to_sql_string(layer_sizes,
+ array_type="double precision"),
+ step_size=step_size,
+ source_table=source_table,
+ activation=activation_index,
+ is_classification=int(is_classification))
+ curr_state = plpy.execute(train_sql)[0]["curr_state"]
+ dist_sql = """
+ SELECT {schema_madlib}.internal_mlp_igd_distance(
+ {prev_state},
+ {curr_state}) as state_dist
+ """.format(schema_madlib=schema_madlib,
+ prev_state=prev_state_str,
+ curr_state=py_list_to_sql_string(curr_state, "double precision"))
+ state_dist = plpy.execute(dist_sql)[0]["state_dist"]
+ if ((state_dist and state_dist < tolerance) or
+ current_iteration > n_iterations):
+ break
+ prev_state = curr_state
+ current_iteration += 1
+ _build_model_table(schema_madlib, output_table, curr_state, n_iterations)
+ layer_sizes_str = py_list_to_sql_string(layer_sizes, array_type="integer")
+ classes_str = py_list_to_sql_string([strip_end_quotes(cl, "'") for cl in classes],
+ array_type=dependent_type)
+ summary_table_creation_query = """
+ CREATE TABLE {output_table}_summary(
--- End diff --
Create the summary table name using `add_postfix()` function in `utilities.py_in`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib issue #149: MLP: Multilayer Perceptron
Posted by edespino <gi...@git.apache.org>.
Github user edespino commented on the issue:
https://github.com/apache/incubator-madlib/pull/149
Upon reviewing the MLP module code coverage metrics produced by running `madpack -s madlib -p postgres install-check -t convex/mlp`, the following is revealed:
- `mlp_igd_merge::run` is not tested
- The `relu` (rectified linear unit) activation is not tested
- The negative case testing for an invalid activation is not tested
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib issue #149: MLP: Multilayer Perceptron
Posted by edespino <gi...@git.apache.org>.
Github user edespino commented on the issue:
https://github.com/apache/incubator-madlib/pull/149
There are a few varying MLP naming references scattered about. I believe `Multilayer perceptron` is correct (no space or hyphen separating multi and layer). I have seen `Multi Layer perceptron` and `Multi-layer perceptron` also used. Can you please clean them up?
Here are those that I noticed:
```
-*- mode: ack; default-directory: "~/workspace/MADlib/incubator-madlib/src/" -*-
Ack started at Wed Jul 12 16:12:18
ack -i "layer perceptron"
modules/convex/mlp_igd.cpp
55: * @brief Perform the multi-layer perceptron transition step
173: * @brief Perform the multi-layer perceptron final step
modules/convex/mlp_igd.hpp
25: * @brief Multi-layer perceptron (incremental gradient): Transition function
30: * @brief Multi-layer perceptron (incremental gradient): State merge function
35: * @brief Multi-layer perceptron (incremental gradient): Final function
40: * @brief Multi-layer perceptron (incremental gradient): Difference in
46: * @brief Multi-layer perceptron (incremental gradient): Convert
52: * @brief Multi-layer perceptron (incremental gradient): Predict
modules/convex/type/state.hpp
547: * incremental gradient descent for multi-layer perceptron
ports/postgres/modules/convex/mlp.sql_in
23: * @brief SQL functions for multi-layer perceptron
45:Multi Layer Perceptron (MLP) is a model for regression and
358:-# Generate a multilayer perceptron with a single hidden layer of 5 units.
416:-# Now train a regression model using a multilayer perceptron a single hidden layer of two nodes.
ports/postgres/modules/convex/mlp_igd.py_in
23:@brief Multilayer perceptron using IGD: Driver functions
503: Multi Layer Perceptron (MLP) is a model for regression and
676: Multi Layer Perceptron (MLP) is a model for regression and
Ack finished at Wed Jul 12 16:12:18
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib issue #149: MLP: Multilayer Perceptron
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/incubator-madlib/pull/149
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/madlib-pr-build/110/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by edespino <gi...@git.apache.org>.
Github user edespino commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r127095011
--- Diff: src/ports/postgres/modules/convex/mlp_igd.py_in ---
@@ -0,0 +1,754 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+@file mlp_igd.py_in
+
+@brief Multilayer perceptron using IGD: Driver functions
+
+@namespace mlp_igd
+"""
+import plpy
+
+from utilities.control import MinWarning
+from utilities.utilities import add_postfix
+from utilities.utilities import py_list_to_sql_string
+from utilities.utilities import extract_keyvalue_params
+from utilities.utilities import _assert
+from utilities.utilities import unique_string
+from utilities.utilities import strip_end_quotes
+
+from utilities.validate_args import cols_in_tbl_valid
+from utilities.validate_args import input_tbl_valid
+from utilities.validate_args import is_var_valid
+from utilities.validate_args import output_tbl_valid
+from utilities.validate_args import get_expr_type
+from utilities.validate_args import array_col_has_same_dimension
+from utilities.validate_args import array_col_dimension
+
+
+def mlp(schema_madlib, source_table, output_table, independent_varname,
+ dependent_varname, hidden_layer_sizes,
+ optimizer_param_str, activation, is_classification, **kwargs):
+ """
+ Args:
+ @param schema_madlib
+ @param source_table
+ @param output_table
+ @param independent_varname
+ @param dependent_varname
+ @param hidden_layer_sizes
+ @param optimizer_param_str
+
+ Returns:
+ None
+ """
+ with MinWarning('info'):
+ optimizer_params = _get_optimizer_params(optimizer_param_str or "")
+ _validate_args(source_table, output_table, independent_varname,
+ dependent_varname, hidden_layer_sizes,
+ optimizer_params, is_classification)
+
+ current_iteration = 1
+ prev_state = None
+ tolerance = optimizer_params["tolerance"]
+ n_iterations = optimizer_params["n_iterations"]
+ step_size = optimizer_params["step_size"]
+ n_tries = optimizer_params["n_tries"]
+ activation_name = _get_activation_function_name(activation)
+ activation_index = _get_activation_index(activation_name)
+ num_input_nodes = array_col_dimension(source_table, independent_varname)
+ num_output_nodes = 0
+ classes = []
+ dependent_type = get_expr_type(dependent_varname, source_table)
+ original_dependent_varname = dependent_varname
+
+ if(is_classification):
+ dependent_variable_sql = """
+ SELECT DISTINCT {dependent_varname}
+ FROM {source_table}
+ """.format(dependent_varname=dependent_varname,
+ source_table=source_table)
+ labels = plpy.execute(dependent_variable_sql)
+ one_hot_dependent_varname = 'ARRAY['
+ num_output_nodes = len(labels)
+ for x in labels:
+ label = _format_label(x[dependent_varname])
+ classes.append(label)
+ one_hot_dependent_varname += dependent_varname + "=" + str(label) + ","
+ # Remove the last comma
+ one_hot_dependent_varname = one_hot_dependent_varname[:-1]
+ one_hot_dependent_varname += ']::integer[]'
+ dependent_varname = one_hot_dependent_varname
+ if(not is_classification):
+ if "[]" not in dependent_type:
+ dependent_varname = "ARRAY[" + dependent_varname + "]"
+ num_output_nodes = array_col_dimension(source_table, dependent_varname)
+ layer_sizes = [num_input_nodes] + hidden_layer_sizes + [num_output_nodes]
+
+ while True:
+ if prev_state:
+ prev_state_str = py_list_to_sql_string(prev_state, array_type="double precision")
+ else:
+ prev_state_str = "(NULL)::DOUBLE PRECISION[]"
+ train_sql = """
+ SELECT
+ {schema_madlib}.mlp_igd_step(
+ ({independent_varname})::DOUBLE PRECISION[],
+ ({dependent_varname})::DOUBLE PRECISION[],
+ {prev_state},
+ {layer_sizes},
+ ({step_size})::FLOAT8,
+ {activation},
+ {is_classification}) as curr_state
+ FROM {source_table} AS _src
+ """.format(schema_madlib=schema_madlib,
+ independent_varname=independent_varname,
+ dependent_varname=dependent_varname,
+ prev_state=prev_state_str,
+ # C++ uses double internally
+ layer_sizes=py_list_to_sql_string(layer_sizes,
+ array_type="double precision"),
+ step_size=step_size,
+ source_table=source_table,
+ activation=activation_index,
+ is_classification=int(is_classification))
+ curr_state = plpy.execute(train_sql)[0]["curr_state"]
+ dist_sql = """
+ SELECT {schema_madlib}.internal_mlp_igd_distance(
+ {prev_state},
+ {curr_state}) as state_dist
+ """.format(schema_madlib=schema_madlib,
+ prev_state=prev_state_str,
+ curr_state=py_list_to_sql_string(curr_state, "double precision"),
+ tolerance=tolerance)
+ state_dist = plpy.execute(dist_sql)[0]["state_dist"]
+ if ((state_dist and state_dist < tolerance) or
+ current_iteration > n_iterations):
+ break
+ prev_state = curr_state
+ # plpy.info("Iteration: "+str(current_iteration)+"/"+str(n_iterations))
+ current_iteration += 1
+ _build_model_table(schema_madlib, output_table, curr_state, n_iterations)
+ layer_sizes_str=py_list_to_sql_string(layer_sizes, array_type="integer")
+ classes_str=py_list_to_sql_string([strip_end_quotes(cl, "'") for cl in classes],
+ array_type=dependent_type)
+ # TODO validation for summary
+ summary_table_creation_query = """
+ CREATE TABLE {output_table}_summary(
+ source_table TEXT,
+ independent_varname TEXT,
+ dependent_varname TEXT,
+ tolerance FLOAT,
+ step_size FLOAT,
+ n_iterations INTEGER,
+ n_tries INTEGER,
+ layer_sizes INTEGER[],
+ activation_function TEXT,
+ is_classification BOOLEAN,
+ classes {dependent_type}[]
+ )""".format(output_table=output_table,
+ dependent_type=dependent_type)
+
+ summary_table_update_query = """
+ INSERT INTO {output_table}_summary VALUES(
+ '{source_table}',
+ '{independent_varname}',
+ '{original_dependent_varname}',
+ {tolerance},
+ {step_size},
+ {n_iterations},
+ {n_tries},
+ {layer_sizes_str},
+ '{activation_name}',
+ {is_classification},
+ {classes_str}
+ )
+ """.format(**locals())
+ plpy.execute(summary_table_creation_query)
+ plpy.execute(summary_table_update_query)
+# ----------------------------------------------------------------------
+
+
+def _build_model_table(schema_madlib, output_table, final_state, n_iterations):
+ final_state_str = py_list_to_sql_string(final_state, array_type="double precision")
+
+ model_table_query = """
+ CREATE TABLE {output_table} AS
+ SELECT
+ (result).coeff AS coeff,
+ (result).loss AS loss,
+ {n_iterations} AS num_iterations
+ -- (result).num_rows_processed AS num_rows_processed,
+ -- n_tuples_including_nulls - (result).num_rows_processed
+ FROM (
+ SELECT
+ {schema_madlib}.internal_mlp_igd_result(
+ {final_state_str}
+ ) AS result
+ ) rel_state_subq
+ """.format(**locals())
+ plpy.execute(model_table_query)
+# ----------------------------------------------------------------------
+
+
+def _get_optimizer_params(param_str):
+ params_defaults = {
+ "step_size": (0.001, float),
+ "n_iterations": (100, int),
+ "n_tries": (1, int),
+ "tolerance": (0.001, float),
+ }
+ param_defaults = dict([(k, v[0]) for k, v in params_defaults.items()])
+ param_types = dict([(k, v[1]) for k, v in params_defaults.items()])
+
+ if not param_str:
+ return param_defaults
+
+ name_value = extract_keyvalue_params(param_str, param_types, param_defaults,
+ ignore_invalid=True)
+ return name_value
+# ----------------------------------------------------------------------
+
+
+def _validate_args_classification(source_table, output_table, independent_varname,
+ dependent_varname, hidden_layer_sizes,
+ optimizer_params):
+ expr_type = get_expr_type(dependent_varname, source_table)
+ int_types = ['integer', 'smallint', 'bigint']
+ text_types = ['text', 'varchar', 'character varying', 'char', 'character']
+ boolean_types = ['boolean']
+ _assert("[]" in expr_type or expr_type in int_types + text_types + boolean_types,
+ "Dependent variable column should refer to an "
+ "integer, boolean, text, varchar, or character type.")
+# ----------------------------------------------------------------------
+
+
+def _validate_args_regression(source_table, output_table, independent_varname,
+ dependent_varname, hidden_layer_sizes,
+ optimizer_params):
+ expr_type = get_expr_type(dependent_varname, source_table)
+ int_types = ['integer', 'smallint', 'bigint']
+ float_types = ['double precision', 'real']
+ _assert("[]" in expr_type or expr_type in int_types + float_types,
+ "Dependent variable column should refer to an array or numeric type")
+ if("[]" in expr_type):
+ _assert(array_col_has_same_dimension(source_table, dependent_varname),
+ "Dependent variable column should refer to arrays of the same length")
+# ----------------------------------------------------------------------
+
+
+def _validate_args(source_table, output_table, independent_varname,
+ dependent_varname, hidden_layer_sizes,
+ optimizer_params, is_classification):
+ """
+ Args:
+ @param kwargs
+
+ Returns:
+
+ """
+ input_tbl_valid(source_table, "MLP")
+ output_tbl_valid(output_table, "MLP")
+ output_tbl_valid(output_table+"_summary", "MLP")
+ _assert(is_var_valid(source_table, independent_varname),
+ "MLP error: invalid independent_varname "
+ "('{independent_varname}') for source_table "
+ "({source_table})!".format(independent_varname=independent_varname,
+ source_table=source_table))
+
+ _assert(is_var_valid(source_table, dependent_varname),
+ "MLP error: invalid dependent_varname "
+ "('{dependent_varname}') for source_table "
+ "({source_table})!".format(dependent_varname=dependent_varname,
+ source_table=source_table))
+ _assert(hidden_layer_sizes is not None,
+ "hidden_layer_sizes may not be null")
+ _assert(type(hidden_layer_sizes) is list,
+ "hidden_layer_sizes must be an array of integers")
+ _assert(all(type(value) is int for value in hidden_layer_sizes),
+ "MLP error: Hidden layers sizes must be integers")
+ _assert(all(value >= 0 for value in hidden_layer_sizes),
+ "MLP error: Hidden layers sizes must be greater than 0.")
+ _assert(optimizer_params["tolerance"] >= 0,
+ "MLP error: Tolerance should be greater than or equal to 0.")
+ _assert(optimizer_params["n_tries"] >= 1,
+ "MLP error: Number of tries should be greater than or equal to 1")
+ _assert(optimizer_params["n_iterations"] >= 1,
+ "MLP error: Number of iterations should be greater than or equal to 1")
+ _assert(optimizer_params["step_size"] > 0,
+ "MLP error: Stepsize should be greater than 0.")
+ _assert("[]" in get_expr_type(independent_varname, source_table),
+ "Independent variable column should refer to an array")
+ _assert(array_col_has_same_dimension(source_table, independent_varname),
+ "Independent variable column should refer to arrays of the same length")
+
+ if is_classification:
+ _validate_args_classification(source_table, output_table, independent_varname,
+ dependent_varname, hidden_layer_sizes,
+ optimizer_params)
+ else:
+ _validate_args_regression(source_table, output_table, independent_varname,
+ dependent_varname, hidden_layer_sizes,
+ optimizer_params)
+# ----------------------------------------------------------------------
+
+
+def _get_activation_function_name(activation_function):
+ if not activation_function:
+ activation_function = 'sigmoid'
+ else:
+ # Add non-linear kernels below after implementing them.
+ supported_activation_function = ['sigmoid', 'tanh', 'relu']
+ try:
+ # allow user to specify a prefix substring of
+ # supported kernels. This works because the supported
+ # kernels have unique prefixes.
+ activation_function = next(x for x in supported_activation_function
+ if x.startswith(activation_function))
+ except StopIteration:
+ # next() returns a StopIteration if no element found
+ plpy.error("MLP Error: Invalid activation function: "
+ "{0}. Supported activation functions are ({1})"
+ .format(activation_function, ','.join(sorted(supported_activation_function))))
+ return activation_function
+# ------------------------------------------------------------------------------
+
+
+def _get_activation_index(activation_name):
+ table = {"relu": 0, "sigmoid": 1, "tanh": 2}
+ return table[activation_name]
+
+
+def _format_label(label):
+ if type(label) is str:
+ return "'" + label + "'"
+ return label
+# -------------------------------------------------------------------------
+
+
+def mlp_predict(schema_madlib, model_table, data_table,
+ id_col_name, output_table,
+ pred_type='response', **kwargs):
+ """ Score new observations using a trained neural network
+
+ @param schema_madlib Name of the schema where MADlib is installed
+ @param model_table Name of learned model
+ @param data_table Name of table/view containing the data
+ points to be scored
+ @param id_col_name Name of column in source_table containing
+ (integer) identifier for data point
+ @param output_table Name of table to store the results
+ @param pred_type: str, The type of output required:
+ 'response' gives the actual response values,
+ 'prob' gives the probability of the classes in a
+ For regression, only type='response' is defined.
+ """
+ # model table
+ input_tbl_valid(model_table, 'MLP')
+ cols_in_tbl_valid(model_table, ['coeff'], 'MLP')
+ # summary table
+ summary_table = add_postfix(model_table, "_summary")
+ input_tbl_valid(summary_table, 'MLP')
+ cols_in_tbl_valid(summary_table,
+ ['dependent_varname', 'independent_varname',
+ 'activation_function',
+ 'tolerance', 'step_size','n_iterations',
+ 'n_tries', 'classes', 'layer_sizes', 'source_table'],
+ 'MLP')
+
+ # read necessary info from summary
+ summary = plpy.execute("SELECT * FROM {0}".format(summary_table))[0]
+ coeff = py_list_to_sql_string(plpy.execute("SELECT * FROM {0}".format(model_table))[0]["coeff"])
+ dependent_varname = summary['dependent_varname']
+ independent_varname = summary['independent_varname']
+ source_table = summary['source_table']
+ activation_function = _get_activation_index(summary['activation_function'])
+ layer_sizes = py_list_to_sql_string(summary['layer_sizes'], array_type="DOUBLE PRECISION")
+ is_classification = int(summary["is_classification"])
+ is_response = int(pred_type=='response')
+
+ pred_name = ('"prob_{0}"' if pred_type == "prob" else
+ '"estimated_{0}"').format(dependent_varname.replace('"', '').strip())
+
+ input_tbl_valid(data_table, 'MLP')
+
+ _assert(is_var_valid(data_table, independent_varname),
+ "MLP Error: independent_varname ('{0}') is invalid for data_table ({1})".
+ format(independent_varname, data_table))
+ _assert(id_col_name is not None, "MLP Error: id_col_name is NULL")
+ _assert(is_var_valid(data_table, id_col_name),
+ "MLP Error: id_col_name ('{0}') is invalid for {1}".
+ format(id_col_name, data_table))
+ output_tbl_valid(output_table, 'MLP')
+ # optimizer_param_dict = _get_optimizer_params(optimizer_params)
+
+ with MinWarning("warning"):
+ header = "CREATE TABLE " + output_table + " AS "
+ # Regression
+ if not is_classification:
+ dependent_type = get_expr_type(dependent_varname, source_table)
+ unnest_if_not_array = ""
+ # Return the same type as the user provided. Internally we always use an array, but
+ # if they provided a scaler, unnest it for the user
+ if("[]" not in dependent_type):
+ unnest_if_not_array = "UNNEST"
+ sql = header + """
+ SELECT {id_col_name},
+ {unnest_if_not_array}({schema_madlib}.internal_predict_mlp(
+ {coeff},
+ {independent_varname}::DOUBLE PRECISION[],
+ {is_classification},
+ {activation_function},
+ {layer_sizes},
+ {is_response}
+ )) as {pred_name}
+ FROM {data_table}
+ """
+ else:
+ summary_query = """
+ SELECT classes FROM {0}_summary
+ """.format(model_table)
+ classes = plpy.execute(summary_query)[0]['classes']
+ if pred_type == "response":
+ # This join is to recover the class name from the summary table, as prediction just returns an index
+ classes_with_index_table = unique_string()
+ classes_table = unique_string()
+ sql = header + """
+ SELECT
+ q.{id_col_name}
+ ,(ARRAY{classes})[pred_idx[1]+1] as {pred_name}
+ FROM (
+ SELECT
+ {id_col_name},
+ {schema_madlib}.internal_predict_mlp(
+ {coeff}::DOUBLE PRECISION[],
+ {independent_varname}::DOUBLE PRECISION[],
+ {is_classification},
+ {activation_function},
+ {layer_sizes},
+ {is_response}
+ )
+ as pred_idx
+ FROM {data_table}
+ ) q
+ """
+ else:
+ # Incomplete
+ intermediate_col = unique_string()
+ score_format = ',\n'.join([
+ '{interim}[{j}] as "estimated_prob_{c}"'.
+ format(j=i + 1, c=c.strip(' "'), interim=intermediate_col)
+ for i, c in enumerate(classes)])
+ sql = header + """
+ SELECT
+ {id_col_name},
+ {score_format}
+ FROM (
+ SELECT {id_col_name},
+ {schema_madlib}.internal_predict_mlp_output(
+ {coeff}::DOUBLE PRECISION[],
+ {independent_varname}::DOUBLE PRECISION[],
+ {is_classification},
+ {activation_function},
+ {layer_sizes},
+ {is_response}
+ )::TEXT[]
+ AS {intermediate_col}
+ FROM {data_table}
+ ) q
+ """
+#TODO add join for validation
+# as s {join_str} {model} as m {using_str}
+ sql = sql.format(**locals())
+ with MinWarning('warning'):
+ plpy.execute(sql)
+
+ pass
+# ----------------------------------------------------------------------
+
+def mlp_help(schema_madlib, message, is_classification):
+ method = 'mlp_classification' if is_classification else 'mlp_regression'
+ # TODO
+ int_types = ['integer', 'smallint', 'bigint']
+ text_types = ['text', 'varchar', 'character varying', 'char', 'character']
+ boolean_types = ['boolean']
+ supported_types = " " * 33 + ", ".join(text_types) + "\n" + " " * 33 + ", ".join(int_types + boolean_types)
+ label_description_classification = "Name of a column which specifies label.\n" + " " * 33 + "Supported types are:\n" + supported_types
+ label_description_regression = "Dependent variable. May be an array for multiple\n" +\
+ " " * 33 + "regression or the name of a column which is any\n" + " " * 33 + "numeric type for single regression"
+ label_description = label_description_classification if is_classification else label_description_regression
+ args = dict(schema_madlib=schema_madlib, method=method, label_description=label_description)
+
+ summary = """
+ ----------------------------------------------------------------
+ SUMMARY
+ ----------------------------------------------------------------
+ Multi Layer Perceptron (MLP) is a model for regression and
+ classification
+
+ Also called "vanilla neural networks", they consist of several
+ fully connected hidden layers with non-linear activation
+ functions.
+
+ For more details on function usage:
+ SELECT {schema_madlib}.{method}('usage')
+
+ For a small example on using the function:
+ SELECT {schema_madlib}.{method}('example')""".format(**args)
+
+ usage = """
+ ---------------------------------------------------------------------------
+ USAGE
+ ---------------------------------------------------------------------------
+ SELECT {schema_madlib}.{method}(
+ source_table, -- name of input table
+ output_table, -- name of output model table
+ independent_varname, -- name of independent variable
+ dependent_varname, -- {label_description}
+ hidden_layer_sizes, -- Array of integers indicating the
+ number of hidden units per layer.
+ Length equal to the number of hidden layers.
+ optimizer_params, -- optional, default NULL
+ parameters for optimization in
+ a comma-separated string of key-value pairs.
+
+ step_size DOUBLE PRECISION, -- Default: 0.001
+ Learning rate
+ n_iterations INTEGER, -- Default: 100
+ Number of iterations per try
+ n_tries INTEGER, -- Default: 1
+ Total number of training cycles,
+ with random initializations to avoid
+ local minima.
+ tolerance DOUBLE PRECISION, -- Default: 0.001
+ If the distance in loss between
+ two iterations is less than the
+ tolerance training will stop, even if
+ n_iterations has not been reached
+
+ activation -- optional, default: 'sigmoid'.
+ supported activations: 'relu', 'sigmoid',
+ and 'tanh'
+ );
+
+
+ ---------------------------------------------------------------------------
+ OUTPUT
+ ---------------------------------------------------------------------------
+ The model table produced by MLP contains the following columns:
+
+ coeffs -- Flat array containing the weights of the neural net
+
+ loss -- The total loss over the training data. Cross entropy
+ for classification and MSE for regression
+
+ num_iterations -- The total number of training iterations
+
+ """.format(**args)
+
+ regression_example = """
+ - Create input table
+
+ CREATE TABLE lin_housing_wi (id serial, x float8[], grp_by_col int, y float8);
+ COPY lin_housing_wi (x, grp_by_col, y) FROM STDIN NULL '?' DELIMITER '|';
+ {1,0.00632,18.00,2.310,0,0.5380,6.5750,65.20,4.0900,1,296.0,15.30,396.90,4.98} | 1 | 24.00
+ {1,0.02731,0.00,7.070,0,0.4690,6.4210,78.90,4.9671,2,242.0,17.80,396.90,9.14} | 1 | 21.60
+ {1,0.02729,0.00,7.070,0,0.4690,7.1850,61.10,4.9671,2,242.0,17.80,392.83,4.03} | 1 | 34.70
+ {1,0.03237,0.00,2.180,0,0.4580,6.9980,45.80,6.0622,3,222.0,18.70,394.63,2.94} | 1 | 33.40
+ {1,0.06905,0.00,2.180,0,0.4580,7.1470,54.20,6.0622,3,222.0,18.70,396.90,5.33} | 1 | 36.20
+ {1,0.02985,0.00,2.180,0,0.4580,6.4300,58.70,6.0622,3,222.0,18.70,394.12,5.21} | 1 | 28.70
+ {1,0.08829,12.50,7.870,0,0.5240,6.0120,66.60,5.5605,5,311.0,15.20,395.60,12.43} | 1 | 22.90
+ {1,0.14455,12.50,7.870,0,0.5240,6.1720,96.10,5.9505,5,311.0,15.20,396.90,19.15} | 1 | 27.10
+ {1,0.21124,12.50,7.870,0,0.5240,5.6310,100.00,6.0821,5,311.0,15.20,386.63,29.93} | 1 | 16.50
+ {1,0.17004,12.50,7.870,0,0.5240,6.0040,85.90,6.5921,5,311.0,15.20,386.71,17.10} | 1 | 18.90
+ {1,0.22489,12.50,7.870,0,0.5240,6.3770,94.30,6.3467,5,311.0,15.20,392.52,20.45} | 1 | 15.00
+ {1,0.11747,12.50,7.870,0,0.5240,6.0090,82.90,6.2267,5,311.0,15.20,396.90,13.27} | 1 | 18.90
+ {1,0.09378,12.50,7.870,0,0.5240,5.8890,39.00,5.4509,5,311.0,15.20,390.50,15.71} | 1 | 21.70
+ \.
+
+ - Generate a multilayer perception with a two hidden layers of 5 units
+ each. Use the x column as the independent variables, and use the class
+ column as the classification. Set the tolerance to 0 so that 300
+ iterations will be run. Use a sigmoid activation function.
+ The model will be written to mlp_regress_result.
+
+ SELECT mlp_regression(
+ 'lin_housing_wi', -- Source table
+ 'mlp_regress_result', -- Desination table
+ 'x', -- Independent variable
+ 'y', -- Dependent variable
+ ARRAY[5,5], -- Number of hidden units per layer
+ 'step_size=0.007,
+ n_iterations=300,
+ tolerance=0',
+ 'sigmoid'); -- Activation
+
+ """
+
+ classification_example = """
+ -- Create input table
+
+ CREATE TABLE iris_data(
+ id integer,
+ attributes numeric[],
+ class_text varchar,
+ class integer
+ );
+
+ INSERT INTO iris_data VALUES
+ (1,ARRAY[5.1,3.5,1.4,0.2],'Iris-setosa',1),
+ (2,ARRAY[4.9,3.0,1.4,0.2],'Iris-setosa',1),
+ (3,ARRAY[4.7,3.2,1.3,0.2],'Iris-setosa',1),
+ (4,ARRAY[4.6,3.1,1.5,0.2],'Iris-setosa',1),
+ (5,ARRAY[5.0,3.6,1.4,0.2],'Iris-setosa',1),
+ (6,ARRAY[5.4,3.9,1.7,0.4],'Iris-setosa',1),
+ (7,ARRAY[4.6,3.4,1.4,0.3],'Iris-setosa',1),
+ (8,ARRAY[5.0,3.4,1.5,0.2],'Iris-setosa',1),
+ (9,ARRAY[4.4,2.9,1.4,0.2],'Iris-setosa',1),
+ (10,ARRAY[4.9,3.1,1.5,0.1],'Iris-setosa',1),
+ (11,ARRAY[7.0,3.2,4.7,1.4],'Iris-versicolor',2),
+ (12,ARRAY[6.4,3.2,4.5,1.5],'Iris-versicolor',2),
+ (13,ARRAY[6.9,3.1,4.9,1.5],'Iris-versicolor',2),
+ (14,ARRAY[5.5,2.3,4.0,1.3],'Iris-versicolor',2),
+ (15,ARRAY[6.5,2.8,4.6,1.5],'Iris-versicolor',2),
+ (16,ARRAY[5.7,2.8,4.5,1.3],'Iris-versicolor',2),
+ (17,ARRAY[6.3,3.3,4.7,1.6],'Iris-versicolor',2),
+ (18,ARRAY[4.9,2.4,3.3,1.0],'Iris-versicolor',2),
+ (19,ARRAY[6.6,2.9,4.6,1.3],'Iris-versicolor',2),
+ (20,ARRAY[5.2,2.7,3.9,1.4],'Iris-versicolor',2),
--- End diff --
In the INSERT example, the trailing comma (`,`) should be a semi-colon (`;`).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by edespino <gi...@git.apache.org>.
Github user edespino commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r127042431
--- Diff: src/ports/postgres/modules/convex/mlp_igd.py_in ---
@@ -0,0 +1,754 @@
+# coding=utf-8
--- End diff --
Pylint rating (src/ports/postgres/modules/convex/mlp_igd.py_in): `Your code has been rated at 6.37/10`
```
PYTHONPATH=/usr/local/madlib/Versions/1.12-dev/ports/postgres/9.6/modules pylint ../src/ports/postgres/modules/convex/mlp_igd.py_in
No config file found, using default configuration
************* Module mlp_igd
W:152, 0: TODO validation for summary (fixme)
W:477, 0: TODO add join for validation (fixme)
W:488, 0: TODO (fixme)
W:669, 0: TODO (fixme)
C: 82, 0: Unnecessary parens after u'if' keyword (superfluous-parens)
C: 99, 0: Unnecessary parens after u'if' keyword (superfluous-parens)
C:149, 0: Exactly one space required around assignment
layer_sizes_str=py_list_to_sql_string(layer_sizes, array_type="integer")
^ (bad-whitespace)
C:150, 0: Exactly one space required around assignment
classes_str=py_list_to_sql_string([strip_end_quotes(cl, "'") for cl in classes],
^ (bad-whitespace)
C:251, 0: Unnecessary parens after u'if' keyword (superfluous-parens)
C:329, 0: Line too long (101/100) (line-too-long)
C:372, 0: Exactly one space required after comma
'tolerance', 'step_size','n_iterations',
^ (bad-whitespace)
C:385, 0: Exactly one space required around comparison
is_response = int(pred_type=='response')
^^ (bad-whitespace)
C:410, 0: Unnecessary parens after u'if' keyword (superfluous-parens)
C:430, 0: Line too long (116/100) (line-too-long)
C:458, 0: Wrong hanging indentation (remove 4 spaces).
for i, c in enumerate(classes)])
| ^ (bad-continuation)
C:492, 0: Line too long (111/100) (line-too-long)
C:493, 0: Line too long (138/100) (line-too-long)
C:495, 0: Line too long (120/100) (line-too-long)
C:496, 0: Line too long (111/100) (line-too-long)
W:566, 0: Anomalous backslash in string: '\.'. String constant might be missing an r prefix. (anomalous-backslash-in-string)
E: 27, 0: Unable to import 'plpy' (import-error)
R: 46, 0: Too many arguments (9/5) (too-many-arguments)
R: 46, 0: Too many local variables (40/15) (too-many-locals)
C: 91,16: Invalid variable name "x" (invalid-name)
W:137,-1: Unused format argument 'tolerance' (unused-format-string-argument)
W: 46, 0: Unused argument 'kwargs' (unused-argument)
W:149, 8: Unused variable 'layer_sizes_str' (unused-variable)
W: 80, 8: Unused variable 'original_dependent_varname' (unused-variable)
W: 73, 8: Unused variable 'n_tries' (unused-variable)
W:150, 8: Unused variable 'classes_str' (unused-variable)
R: 46, 0: Too many statements (52/50) (too-many-statements)
W:189,65: Unused argument 'n_iterations' (unused-argument)
W:189,38: Unused argument 'output_table' (unused-argument)
W:190, 4: Unused variable 'final_state_str' (unused-variable)
W:189,23: Unused argument 'schema_madlib' (unused-argument)
R:230, 0: Too many arguments (6/5) (too-many-arguments)
W:230,48: Unused argument 'output_table' (unused-argument)
W:231,53: Unused argument 'hidden_layer_sizes' (unused-argument)
W:232,34: Unused argument 'optimizer_params' (unused-argument)
W:230,62: Unused argument 'independent_varname' (unused-argument)
R:243, 0: Too many arguments (6/5) (too-many-arguments)
W:243,44: Unused argument 'output_table' (unused-argument)
W:244,49: Unused argument 'hidden_layer_sizes' (unused-argument)
W:245,30: Unused argument 'optimizer_params' (unused-argument)
W:243,58: Unused argument 'independent_varname' (unused-argument)
R:257, 0: Too many arguments (7/5) (too-many-arguments)
C:283,12: Using type() instead of isinstance() for a typecheck. (unidiomatic-typecheck)
C:285,16: Using type() instead of isinstance() for a typecheck. (unidiomatic-typecheck)
C:340, 7: Using type() instead of isinstance() for a typecheck. (unidiomatic-typecheck)
R:346, 0: Too many arguments (6/5) (too-many-arguments)
R:346, 0: Too many local variables (30/15) (too-many-locals)
W:483, 4: Unnecessary pass statement (unnecessary-pass)
W:382, 4: Unused variable 'activation_function' (unused-variable)
W:432,16: Unused variable 'classes_table' (unused-variable)
W:455,16: Unused variable 'score_format' (unused-variable)
W:387, 4: Unused variable 'pred_name' (unused-variable)
W:346,16: Unused argument 'schema_madlib' (unused-argument)
W:431,16: Unused variable 'classes_with_index_table' (unused-variable)
W:385, 4: Unused variable 'is_response' (unused-variable)
W:378, 4: Unused variable 'coeff' (unused-variable)
W:383, 4: Unused variable 'layer_sizes' (unused-variable)
W:407,12: Unused variable 'unnest_if_not_array' (unused-variable)
W:346, 0: Unused argument 'kwargs' (unused-argument)
C:486, 0: Missing function docstring (missing-docstring)
R:486, 0: Too many local variables (17/15) (too-many-locals)
C:493, 4: Invalid variable name "label_description_classification" (invalid-name)
R:657, 4: Unnecessary "else" after "return" (no-else-return)
C:668, 0: Missing function docstring (missing-docstring)
R:745, 4: Unnecessary "else" after "return" (no-else-return)
W:668,45: Unused argument 'is_classification' (unused-argument)
-----------------------------------
Your code has been rated at 6.37/10
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by njayaram2 <gi...@git.apache.org>.
Github user njayaram2 commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r128072581
--- Diff: src/ports/postgres/modules/convex/mlp_igd.py_in ---
@@ -0,0 +1,734 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+@file mlp_igd.py_in
+
+@brief Multilayer perceptron using IGD: Driver functions
+
+@namespace mlp_igd
+"""
+import plpy
+
+from utilities.control import MinWarning
+from utilities.utilities import add_postfix
+from utilities.utilities import py_list_to_sql_string
+from utilities.utilities import extract_keyvalue_params
+from utilities.utilities import _assert
+from utilities.utilities import unique_string
+from utilities.utilities import strip_end_quotes
+
+from utilities.validate_args import cols_in_tbl_valid
+from utilities.validate_args import input_tbl_valid
+from utilities.validate_args import is_var_valid
+from utilities.validate_args import output_tbl_valid
+from utilities.validate_args import get_expr_type
+from utilities.validate_args import array_col_has_same_dimension
+from utilities.validate_args import array_col_dimension
+
+
+def mlp(schema_madlib, source_table, output_table, independent_varname,
+ dependent_varname, hidden_layer_sizes,
+ optimizer_param_str, activation, is_classification, **kwargs):
+ """
+ Args:
+ @param schema_madlib
+ @param source_table
+ @param output_table
+ @param independent_varname
+ @param dependent_varname
+ @param hidden_layer_sizes
+ @param optimizer_param_str
+
+ Returns:
+ None
+ """
+ with MinWarning('info'):
+ optimizer_params = _get_optimizer_params(optimizer_param_str or "")
+ _validate_args(source_table, output_table, independent_varname,
+ dependent_varname, hidden_layer_sizes,
+ optimizer_params, is_classification)
+
+ current_iteration = 1
+ prev_state = None
+ tolerance = optimizer_params["tolerance"]
+ n_iterations = optimizer_params["n_iterations"]
+ step_size = optimizer_params["step_size"]
+ n_tries = optimizer_params["n_tries"]
+ activation_name = _get_activation_function_name(activation)
+ activation_index = _get_activation_index(activation_name)
+ num_input_nodes = array_col_dimension(source_table, independent_varname)
+ num_output_nodes = 0
+ classes = []
+ dependent_type = get_expr_type(dependent_varname, source_table)
+ original_dependent_varname = dependent_varname
+
+ if is_classification:
+ dependent_variable_sql = """
+ SELECT DISTINCT {dependent_varname}
+ FROM {source_table}
+ """.format(dependent_varname=dependent_varname,
+ source_table=source_table)
+ labels = plpy.execute(dependent_variable_sql)
+ one_hot_dependent_varname = 'ARRAY['
+ num_output_nodes = len(labels)
+ for label_obj in labels:
+ label = _format_label(label_obj[dependent_varname])
+ classes.append(label)
+ one_hot_dependent_varname += dependent_varname + "=" + str(label) + ","
+ # Remove the last comma
+ one_hot_dependent_varname = one_hot_dependent_varname[:-1]
+ one_hot_dependent_varname += ']::integer[]'
+ dependent_varname = one_hot_dependent_varname
+ if not is_classification:
+ if "[]" not in dependent_type:
+ dependent_varname = "ARRAY[" + dependent_varname + "]"
+ num_output_nodes = array_col_dimension(source_table, dependent_varname)
+ layer_sizes = [num_input_nodes] + hidden_layer_sizes + [num_output_nodes]
+
+ while True:
+ if prev_state:
+ prev_state_str = py_list_to_sql_string(prev_state, array_type="double precision")
+ else:
+ prev_state_str = "(NULL)::DOUBLE PRECISION[]"
+ train_sql = """
+ SELECT
+ {schema_madlib}.mlp_igd_step(
+ ({independent_varname})::DOUBLE PRECISION[],
+ ({dependent_varname})::DOUBLE PRECISION[],
+ {prev_state},
+ {layer_sizes},
+ ({step_size})::FLOAT8,
+ {activation},
+ {is_classification}) as curr_state
+ FROM {source_table} AS _src
+ """.format(schema_madlib=schema_madlib,
+ independent_varname=independent_varname,
+ dependent_varname=dependent_varname,
+ prev_state=prev_state_str,
+ # C++ uses double internally
+ layer_sizes=py_list_to_sql_string(layer_sizes,
+ array_type="double precision"),
+ step_size=step_size,
+ source_table=source_table,
+ activation=activation_index,
+ is_classification=int(is_classification))
+ curr_state = plpy.execute(train_sql)[0]["curr_state"]
+ dist_sql = """
+ SELECT {schema_madlib}.internal_mlp_igd_distance(
+ {prev_state},
+ {curr_state}) as state_dist
+ """.format(schema_madlib=schema_madlib,
+ prev_state=prev_state_str,
+ curr_state=py_list_to_sql_string(curr_state, "double precision"))
+ state_dist = plpy.execute(dist_sql)[0]["state_dist"]
+ if ((state_dist and state_dist < tolerance) or
+ current_iteration > n_iterations):
+ break
+ prev_state = curr_state
+ current_iteration += 1
+ _build_model_table(schema_madlib, output_table, curr_state, n_iterations)
+ layer_sizes_str = py_list_to_sql_string(layer_sizes, array_type="integer")
+ classes_str = py_list_to_sql_string([strip_end_quotes(cl, "'") for cl in classes],
+ array_type=dependent_type)
+ summary_table_creation_query = """
+ CREATE TABLE {output_table}_summary(
+ source_table TEXT,
+ independent_varname TEXT,
+ dependent_varname TEXT,
+ tolerance FLOAT,
+ step_size FLOAT,
+ n_iterations INTEGER,
+ n_tries INTEGER,
+ layer_sizes INTEGER[],
+ activation_function TEXT,
+ is_classification BOOLEAN,
+ classes {dependent_type}[]
+ )""".format(output_table=output_table,
+ dependent_type=dependent_type)
+
+ summary_table_update_query = """
+ INSERT INTO {output_table}_summary VALUES(
+ '{source_table}',
+ '{independent_varname}',
+ '{original_dependent_varname}',
+ {tolerance},
+ {step_size},
+ {n_iterations},
+ {n_tries},
+ {layer_sizes_str},
+ '{activation_name}',
+ {is_classification},
+ {classes_str}
+ )
+ """.format(**locals())
+ plpy.execute(summary_table_creation_query)
+ plpy.execute(summary_table_update_query)
+# ----------------------------------------------------------------------
+
+
+def _build_model_table(schema_madlib, output_table, final_state, n_iterations):
+ final_state_str = py_list_to_sql_string(final_state, array_type="double precision")
+
+ model_table_query = """
+ CREATE TABLE {output_table} AS
+ SELECT
+ (result).coeff AS coeff,
+ (result).loss AS loss,
+ {n_iterations} AS num_iterations
+ -- (result).num_rows_processed AS num_rows_processed,
+ -- n_tuples_including_nulls - (result).num_rows_processed
+ FROM (
+ SELECT
+ {schema_madlib}.internal_mlp_igd_result(
+ {final_state_str}
+ ) AS result
+ ) rel_state_subq
+ """.format(**locals())
+ plpy.execute(model_table_query)
+# ----------------------------------------------------------------------
+
+
+def _get_optimizer_params(param_str):
+ params_defaults = {
+ "step_size": (0.001, float),
+ "n_iterations": (100, int),
+ "n_tries": (1, int),
+ "tolerance": (0.001, float),
+ }
+ param_defaults = dict([(k, v[0]) for k, v in params_defaults.items()])
+ param_types = dict([(k, v[1]) for k, v in params_defaults.items()])
+
+ if not param_str:
+ return param_defaults
+
+ name_value = extract_keyvalue_params(param_str, param_types, param_defaults,
+ ignore_invalid=False)
+ return name_value
+# ----------------------------------------------------------------------
+
+
+def _validate_args_classification(source_table, dependent_varname):
+ expr_type = get_expr_type(dependent_varname, source_table)
+ int_types = ['integer', 'smallint', 'bigint']
+ text_types = ['text', 'varchar', 'character varying', 'char', 'character']
+ boolean_types = ['boolean']
+ _assert("[]" in expr_type or expr_type in int_types + text_types + boolean_types,
+ "Dependent variable column should refer to an "
+ "integer, boolean, text, varchar, or character type.")
+# ----------------------------------------------------------------------
+
+
+def _validate_args_regression(source_table, dependent_varname):
+ expr_type = get_expr_type(dependent_varname, source_table)
+ int_types = ['integer', 'smallint', 'bigint']
+ float_types = ['double precision', 'real']
+ _assert("[]" in expr_type or expr_type in int_types + float_types,
+ "Dependent variable column should refer to an array or numeric type")
+ if "[]" in expr_type:
+ _assert(array_col_has_same_dimension(source_table, dependent_varname),
+ "Dependent variable column should refer to arrays of the same length")
+# ----------------------------------------------------------------------
+
+
+def _validate_args(source_table, output_table, independent_varname,
+ dependent_varname, hidden_layer_sizes,
+ optimizer_params, is_classification):
+ input_tbl_valid(source_table, "MLP")
+ output_tbl_valid(output_table, "MLP")
+ output_tbl_valid(output_table+"_summary", "MLP")
+ _assert(is_var_valid(source_table, independent_varname),
+ "MLP error: invalid independent_varname "
+ "('{independent_varname}') for source_table "
+ "({source_table})!".format(independent_varname=independent_varname,
+ source_table=source_table))
+
+ _assert(is_var_valid(source_table, dependent_varname),
+ "MLP error: invalid dependent_varname "
+ "('{dependent_varname}') for source_table "
+ "({source_table})!".format(dependent_varname=dependent_varname,
+ source_table=source_table))
+ _assert(hidden_layer_sizes is not None,
+ "hidden_layer_sizes may not be null")
+ _assert(isinstance(hidden_layer_sizes, list),
+ "hidden_layer_sizes must be an array of integers")
+ _assert(all(isinstance(value, int) for value in hidden_layer_sizes),
+ "MLP error: Hidden layers sizes must be integers")
+ _assert(all(value >= 0 for value in hidden_layer_sizes),
+ "MLP error: Hidden layers sizes must be greater than 0.")
+ _assert(optimizer_params["tolerance"] >= 0,
+ "MLP error: Tolerance should be greater than or equal to 0.")
+ _assert(optimizer_params["n_tries"] >= 1,
+ "MLP error: Number of tries should be greater than or equal to 1")
+ _assert(optimizer_params["n_iterations"] >= 1,
+ "MLP error: Number of iterations should be greater than or equal to 1")
+ _assert(optimizer_params["step_size"] > 0,
+ "MLP error: Stepsize should be greater than 0.")
+ _assert("[]" in get_expr_type(independent_varname, source_table),
+ "Independent variable column should refer to an array")
+ _assert(array_col_has_same_dimension(source_table, independent_varname),
+ "Independent variable column should refer to arrays of the same length")
+
+ if is_classification:
+ _validate_args_classification(source_table, dependent_varname)
+ else:
+ _validate_args_regression(source_table, dependent_varname)
+# ----------------------------------------------------------------------
+
+
+def _get_activation_function_name(activation_function):
+ if not activation_function:
+ activation_function = 'sigmoid'
+ else:
+ # Add non-linear kernels below after implementing them.
+ supported_activation_function = ['sigmoid', 'tanh', 'relu']
+ try:
+ # allow user to specify a prefix substring of
+ # supported kernels. This works because the supported
+ # kernels have unique prefixes.
+ activation_function = next(x for x in supported_activation_function
+ if x.startswith(activation_function))
+ except StopIteration:
+ # next() returns a StopIteration if no element found
+ plpy.error("MLP Error: Invalid activation function: "
+ "{0}. Supported activation functions are ({1})"
+ .format(activation_function, ','.join(
+ sorted(supported_activation_function))))
+ return activation_function
+# ------------------------------------------------------------------------------
+
+
+def _get_activation_index(activation_name):
+ table = {"relu": 0, "sigmoid": 1, "tanh": 2}
+ return table[activation_name]
+
+
+def _format_label(label):
+ if isinstance(label, str):
+ return "'" + label + "'"
+ return label
+# -------------------------------------------------------------------------
+
+
+def mlp_predict(schema_madlib, model_table, data_table,
+ id_col_name, output_table,
+ pred_type='response', **kwargs):
+ """ Score new observations using a trained neural network
+
+ @param schema_madlib Name of the schema where MADlib is installed
+ @param model_table Name of learned model
+ @param data_table Name of table/view containing the data
+ points to be scored
+ @param id_col_name Name of column in source_table containing
+ (integer) identifier for data point
+ @param output_table Name of table to store the results
+ @param pred_type: str, The type of output required:
+ 'response' gives the actual response values,
+ 'prob' gives the probability of the classes in a
+ For regression, only type='response' is defined.
+ """
+ # model table
+ input_tbl_valid(model_table, 'MLP')
+ cols_in_tbl_valid(model_table, ['coeff'], 'MLP')
+ # summary table
+ summary_table = add_postfix(model_table, "_summary")
+ input_tbl_valid(summary_table, 'MLP')
+ cols_in_tbl_valid(summary_table,
+ ['dependent_varname', 'independent_varname',
+ 'activation_function',
+ 'tolerance', 'step_size', 'n_iterations',
+ 'n_tries', 'classes', 'layer_sizes', 'source_table'],
+ 'MLP')
+
+ # read necessary info from summary
+ summary = plpy.execute("SELECT * FROM {0}".format(summary_table))[0]
+ coeff = py_list_to_sql_string(plpy.execute("SELECT * FROM {0}".format(model_table))[0]["coeff"])
+ dependent_varname = summary['dependent_varname']
+ independent_varname = summary['independent_varname']
+ source_table = summary['source_table']
+ activation_function = _get_activation_index(summary['activation_function'])
+ layer_sizes = py_list_to_sql_string(summary['layer_sizes'], array_type="DOUBLE PRECISION")
+ is_classification = int(summary["is_classification"])
+ is_response = int(pred_type == 'response')
+
+ pred_name = ('"prob_{0}"' if pred_type == "prob" else
+ '"estimated_{0}"').format(dependent_varname.replace('"', '').strip())
+
+ input_tbl_valid(data_table, 'MLP')
+
+ _assert(is_var_valid(data_table, independent_varname),
+ "MLP Error: independent_varname ('{0}') is invalid for data_table ({1})".
+ format(independent_varname, data_table))
+ _assert(id_col_name is not None, "MLP Error: id_col_name is NULL")
+ _assert(is_var_valid(data_table, id_col_name),
+ "MLP Error: id_col_name ('{0}') is invalid for {1}".
+ format(id_col_name, data_table))
+ output_tbl_valid(output_table, 'MLP')
+ # optimizer_param_dict = _get_optimizer_params(optimizer_params)
+
+ with MinWarning("warning"):
+ header = "CREATE TABLE " + output_table + " AS "
+ # Regression
+ if not is_classification:
+ dependent_type = get_expr_type(dependent_varname, source_table)
+ unnest_if_not_array = ""
+ # Return the same type as the user provided. Internally we always use an array, but
+ # if they provided a scaler, unnest it for the user
+ if "[]" not in dependent_type:
+ unnest_if_not_array = "UNNEST"
+ sql = header + """
+ SELECT {id_col_name},
+ {unnest_if_not_array}({schema_madlib}.internal_predict_mlp(
+ {coeff},
+ {independent_varname}::DOUBLE PRECISION[],
+ {is_classification},
+ {activation_function},
+ {layer_sizes},
+ {is_response}
+ )) as {pred_name}
+ FROM {data_table}
+ """
+ else:
+ summary_query = """
+ SELECT classes FROM {0}_summary
+ """.format(model_table)
+ classes = plpy.execute(summary_query)[0]['classes']
+ if pred_type == "response":
+ # This join is to recover the class name from the summary table,
+ # as prediction just returns an index
+ classes_with_index_table = unique_string()
+ classes_table = unique_string()
+ sql = header + """
+ SELECT
+ q.{id_col_name}
+ ,(ARRAY{classes})[pred_idx[1]+1] as {pred_name}
+ FROM (
+ SELECT
+ {id_col_name},
+ {schema_madlib}.internal_predict_mlp(
+ {coeff}::DOUBLE PRECISION[],
+ {independent_varname}::DOUBLE PRECISION[],
+ {is_classification},
+ {activation_function},
+ {layer_sizes},
+ {is_response}
+ )
+ as pred_idx
+ FROM {data_table}
+ ) q
+ """
+ else:
+ # Incomplete
+ intermediate_col = unique_string()
+ score_format = ',\n'.join([
+ 'CAST({interim}[{j}] as DOUBLE PRECISION) as "estimated_prob_{c_str}"'.
+ format(j=i + 1, c_str=str(c).strip(' "'), interim=intermediate_col)
+ for i, c in enumerate(classes)])
+ sql = header + """
+ SELECT
+ {id_col_name},
+ {score_format}
+ FROM (
+ SELECT {id_col_name},
+ {schema_madlib}.internal_predict_mlp(
+ {coeff}::DOUBLE PRECISION[],
+ {independent_varname}::DOUBLE PRECISION[],
+ {is_classification},
+ {activation_function},
+ {layer_sizes},
+ {is_response}
+ )::TEXT[]
+ AS {intermediate_col}
+ FROM {data_table}
+ ) q
+ """
+ sql = sql.format(**locals())
+ with MinWarning('warning'):
+ plpy.execute(sql)
+# ----------------------------------------------------------------------
+
+def mlp_help(schema_madlib, message, is_classification):
+ method = 'mlp_classification' if is_classification else 'mlp_regression'
+ int_types = ['integer', 'smallint', 'bigint']
+ text_types = ['text', 'varchar', 'character varying', 'char', 'character']
+ boolean_types = ['boolean']
+ supported_types = " " * 33 + ", ".join(text_types) + "\n" +\
+ " " * 33 + ", ".join(int_types + boolean_types)
+ label_description_classification = "Name of a column which specifies label.\n" +\
+ " " * 33 + "Supported types are:\n" + supported_types
+ label_description_regression = "Dependent variable. May be an array for multiple\n" +\
+ " " * 33 + "regression or the name of a column which is any\n" + " " * 33 +\
+ "numeric type for single regression"
+ label_description = label_description_classification if is_classification\
+ else label_description_regression
+ args = dict(schema_madlib=schema_madlib, method=method, label_description=label_description)
+
+ summary = """
+ ----------------------------------------------------------------
+ SUMMARY
+ ----------------------------------------------------------------
+ Multilayer Perceptron (MLP) is a model for regression and
+ classification
+
+ Also called "vanilla neural networks", they consist of several
+ fully connected hidden layers with non-linear activation
+ functions.
+
+ For more details on function usage:
+ SELECT {schema_madlib}.{method}('usage')
+
+ For a small example on using the function:
+ SELECT {schema_madlib}.{method}('example')""".format(**args)
+
+ usage = """
+ ---------------------------------------------------------------------------
+ USAGE
+ ---------------------------------------------------------------------------
+ SELECT {schema_madlib}.{method}(
+ source_table, -- name of input table
+ output_table, -- name of output model table
+ independent_varname, -- name of independent variable
+ dependent_varname, -- {label_description}
+ hidden_layer_sizes, -- Array of integers indicating the
+ number of hidden units per layer.
+ Length equal to the number of hidden layers.
+ optimizer_params, -- optional, default NULL
+ parameters for optimization in
+ a comma-separated string of key-value pairs.
+
+ step_size DOUBLE PRECISION, -- Default: 0.001
+ Learning rate
+ n_iterations INTEGER, -- Default: 100
+ Number of iterations per try
+ n_tries INTEGER, -- Default: 1
+ Total number of training cycles,
+ with random initializations to avoid
+ local minima.
+ tolerance DOUBLE PRECISION, -- Default: 0.001
+ If the distance in loss between
+ two iterations is less than the
+ tolerance training will stop, even if
+ n_iterations has not been reached
+
+ activation -- optional, default: 'sigmoid'.
+ supported activations: 'relu', 'sigmoid',
+ and 'tanh'
+ );
+
+
+ ---------------------------------------------------------------------------
+ OUTPUT
+ ---------------------------------------------------------------------------
+ The model table produced by MLP contains the following columns:
+
+ coeffs -- Flat array containing the weights of the neural net
+
+ loss -- The total loss over the training data. Cross entropy
+ for classification and MSE for regression
+
+ num_iterations -- The total number of training iterations
+
+ """.format(**args)
+
+ regression_example = """
+ - Create input table
+
+ CREATE TABLE lin_housing_wi (id serial, x float8[], grp_by_col int, y float8);
+ COPY lin_housing_wi (x, grp_by_col, y) FROM STDIN NULL '?' DELIMITER '|';
+ {1,0.00632,18.00,2.310,0,0.5380,6.5750,65.20,4.0900,1,296.0,15.30,396.90,4.98} | 1 | 24.00
+ {1,0.02731,0.00,7.070,0,0.4690,6.4210,78.90,4.9671,2,242.0,17.80,396.90,9.14} | 1 | 21.60
+ {1,0.02729,0.00,7.070,0,0.4690,7.1850,61.10,4.9671,2,242.0,17.80,392.83,4.03} | 1 | 34.70
+ {1,0.03237,0.00,2.180,0,0.4580,6.9980,45.80,6.0622,3,222.0,18.70,394.63,2.94} | 1 | 33.40
+ {1,0.06905,0.00,2.180,0,0.4580,7.1470,54.20,6.0622,3,222.0,18.70,396.90,5.33} | 1 | 36.20
+ {1,0.02985,0.00,2.180,0,0.4580,6.4300,58.70,6.0622,3,222.0,18.70,394.12,5.21} | 1 | 28.70
+ {1,0.08829,12.50,7.870,0,0.5240,6.0120,66.60,5.5605,5,311.0,15.20,395.60,12.43} | 1 | 22.90
+ {1,0.14455,12.50,7.870,0,0.5240,6.1720,96.10,5.9505,5,311.0,15.20,396.90,19.15} | 1 | 27.10
+ {1,0.21124,12.50,7.870,0,0.5240,5.6310,100.00,6.0821,5,311.0,15.20,386.63,29.93} | 1 | 16.50
+ {1,0.17004,12.50,7.870,0,0.5240,6.0040,85.90,6.5921,5,311.0,15.20,386.71,17.10} | 1 | 18.90
+ {1,0.22489,12.50,7.870,0,0.5240,6.3770,94.30,6.3467,5,311.0,15.20,392.52,20.45} | 1 | 15.00
+ {1,0.11747,12.50,7.870,0,0.5240,6.0090,82.90,6.2267,5,311.0,15.20,396.90,13.27} | 1 | 18.90
+ {1,0.09378,12.50,7.870,0,0.5240,5.8890,39.00,5.4509,5,311.0,15.20,390.50,15.71} | 1 | 21.70
+ \.
+
+ - Generate a multilayer perception with a two hidden layers of 5 units
+ each. Use the x column as the independent variables, and use the class
+ column as the classification. Set the tolerance to 0 so that 300
+ iterations will be run. Use a sigmoid activation function.
+ The model will be written to mlp_regress_result.
+
+ SELECT mlp_regression(
+ 'lin_housing_wi', -- Source table
+ 'mlp_regress_result', -- Desination table
+ 'x', -- Independent variable
+ 'y', -- Dependent variable
+ ARRAY[5,5], -- Number of hidden units per layer
+ 'step_size=0.007,
+ n_iterations=300,
+ tolerance=0',
+ 'sigmoid'); -- Activation
+
+ """
+
+ classification_example = """
+ -- Create input table
+
+ CREATE TABLE iris_data(
+ id integer,
+ attributes numeric[],
+ class_text varchar,
+ class integer
+ );
+
+ INSERT INTO iris_data VALUES
+ (1,ARRAY[5.1,3.5,1.4,0.2],'Iris-setosa',1),
+ (2,ARRAY[4.9,3.0,1.4,0.2],'Iris-setosa',1),
+ (3,ARRAY[4.7,3.2,1.3,0.2],'Iris-setosa',1),
+ (4,ARRAY[4.6,3.1,1.5,0.2],'Iris-setosa',1),
+ (5,ARRAY[5.0,3.6,1.4,0.2],'Iris-setosa',1),
+ (6,ARRAY[5.4,3.9,1.7,0.4],'Iris-setosa',1),
+ (7,ARRAY[4.6,3.4,1.4,0.3],'Iris-setosa',1),
+ (8,ARRAY[5.0,3.4,1.5,0.2],'Iris-setosa',1),
+ (9,ARRAY[4.4,2.9,1.4,0.2],'Iris-setosa',1),
+ (10,ARRAY[4.9,3.1,1.5,0.1],'Iris-setosa',1),
+ (11,ARRAY[7.0,3.2,4.7,1.4],'Iris-versicolor',2),
+ (12,ARRAY[6.4,3.2,4.5,1.5],'Iris-versicolor',2),
+ (13,ARRAY[6.9,3.1,4.9,1.5],'Iris-versicolor',2),
+ (14,ARRAY[5.5,2.3,4.0,1.3],'Iris-versicolor',2),
+ (15,ARRAY[6.5,2.8,4.6,1.5],'Iris-versicolor',2),
+ (16,ARRAY[5.7,2.8,4.5,1.3],'Iris-versicolor',2),
+ (17,ARRAY[6.3,3.3,4.7,1.6],'Iris-versicolor',2),
+ (18,ARRAY[4.9,2.4,3.3,1.0],'Iris-versicolor',2),
+ (19,ARRAY[6.6,2.9,4.6,1.3],'Iris-versicolor',2),
+ (20,ARRAY[5.2,2.7,3.9,1.4],'Iris-versicolor',2);
+
+
+ -- Generate a multilayer perception with a single hidden layer of 5 units.
+ Use the attributes column as the independent variables, and use the class
+ column as the classification. Set the tolerance to 0 so that 1000
+ iterations will be run. Use a hyperbolic tangent activation function.
+ The model will be written to mlp_result.
+
+ SELECT {schema_madlib}.mlp_classification(
+ 'iris_data', -- Source table
+ 'mlp_model', -- Destination table
+ 'attributes', -- Input features
+ 'class_text', -- Label
+ ARRAY[5], -- Number of units per layer
+ 'step_size=0.003,
+ n_iterations=5000,
+ tolerance=0', -- Optimizer params
+ 'tanh'); -- Activation function
+
+ """.format(**args)
+ example = classification_example if is_classification else regression_example
+ if not message:
+ return summary
+ elif message.lower() in ('usage', 'help', '?'):
+ return usage
+ elif message.lower() == 'example':
+ return example
+ return """
+ No such option. Use "SELECT {schema_madlib}.{method}()" for help.
+ """.format(**args)
+
+def mlp_predict_help(schema_madlib, message):
+ args = dict(schema_madlib=schema_madlib)
+
+ summary = """
+ ----------------------------------------------------------------
+ SUMMARY
+ ----------------------------------------------------------------
+ Multilayer Perceptron (MLP) is a model for regression and
+ classification
+
+ Also called "vanilla neural networks", they consist of several
+ fully connected hidden layers with non-linear activation
+ functions.
+
+ For more details on function usage:
+ SELECT {schema_madlib}.mlp_predict('usage')
+
+ For a small example on using the function:
+ SELECT {schema_madlib}.mlp_predict('example')""".format(**args)
+
+ usage = """
+ ---------------------------------------------------------------------------
+ USAGE
+ ---------------------------------------------------------------------------
+ SELECT {schema_madlib}.mlp_predict(
+ model_table, -- name of model table
+ data_table, -- name of data table
+ id_col_name, -- id column for data table
+ output_table, -- name of output table
+ pred_type -- the type of output requested:
+ -- 'response' gives the actual prediction,
+ -- 'prob' gives the probability of each class.
+ -- for regression, only type='response' is defined.
+ );
+
+
+ ---------------------------------------------------------------------------
+ OUTPUT
+ ---------------------------------------------------------------------------
+ The model table produced by mlp contains the following columns:
+
+ id -- The provided id for the given input vector
+
+ estimated_<COL_NAME> -- (For pred_type='response') The estimated class
+ for classification or value for regression, where
+ <COL_NAME> is the name of the column to be
+ predicted from training data
+
+ prob_<CLASS> -- (For pred_type='prob' for classification) The
+ probability of a given class <CLASS> as given by
+ softmax. There will be one column for each class
+ in the training data.
+
+ """.format(**args)
+
+ example = """
+ -- See {schema_madlib}.mlp_classification('example') for test
+ -- and model tables
+
+ -- Predict classes using
+ SELECT {schema_madlib}.mlp_predict(
+ 'mlp_model', -- Model table
+ 'iris_data', -- Test data table
+ 'id', -- Id column in test table
+ 'mlp_prediction', -- Output table for predictions
+ 'response' -- Output classes, not probabilities
+ );
+
+ SELECT * FROM mlp_prediction;
+
+ WITH total_count AS (SELECT count(*) AS c FROM iris_data)
+ SELECT count(*)/((SELECT c from total_count)::DOUBLE PRECISION) AS train_accuracy FROM
+ (select iris_data.class_text as actual_label, mlp_prediction.estimated_class_text as predicted_label from mlp_prediction INNER JOIN iris_data ON iris_data.id=mlp_prediction.id) q
+ WHERE q.actual_label=q.predicted_label;
+ """.format(**args)
--- End diff --
Indent this better and limit number of characters per line to less than 80 wherever possible. Convert sql keywords to caps: `from`->`FROM`. Example indentation:
```
WITH total_count AS (SELECT count(*) AS c FROM iris_data)
SELECT count(*)/((SELECT c FROM total_count)::DOUBLE PRECISION) AS train_accuracy
FROM
(
SELECT iris_data.class_text AS actual_label,
mlp_prediction.estimated_class_text AS predicted_label
FROM mlp_prediction
INNER JOIN iris_data ON iris_data.id=mlp_prediction.id
) q
WHERE q.actual_label=q.predicted_label;
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by edespino <gi...@git.apache.org>.
Github user edespino commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r127314914
--- Diff: src/ports/postgres/modules/convex/mlp.sql_in ---
@@ -0,0 +1,744 @@
+/* ----------------------------------------------------------------------- *//**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ *
+ * @file mlp.sql_in
+ *
+ * @brief SQL functions for multilayer perceptron
+ * @date June 2012
+ *
+ *
+ *//* ----------------------------------------------------------------------- */
+
+m4_include(`SQLCommon.m4')
+
+/**
+@addtogroup grp_mlp
+
+<div class="toc"><b>Contents</b><ul>
+<li class="level1"><a href="#mlp_classification">Classification</a></li>
+<li class="level1"><a href="#mlp_regression">Regression</a></li>
+<li class="level1"><a href="#optimization_params">Optimizer Parameters</a></li>
+<li class="level1"><a href="#predict">Prediction Functions/a></li>
+<li class="level1"><a href="#example">Examples</a></li>
+<li class="level1"><a href="#background">Technical Background</a></li>
+<li class="level1"><a href="#literature">Literature</a></li>
+<li class="level1"><a href="#related">Related Topics</a></li>
+</ul></div>
+
+Multilayer Perceptron (MLP) is a model for regression and
+classification
+
+Also called "vanilla neural networks", they consist of several
+fully connected hidden layers with non-linear activation
+functions. In the case of classification, the final layer of the
+neural net has as many nodes as classes, and the output of the
+neural net can be interpreted as the probability of a given input
+feature belonging to a specific class.
+
+
+@brief Solves classification and regression problems with several
+fully connected layers and nonlinear activation functions.
+
+@anchor mlp_classification
+@par Classification Training Function
+The mlp classification training function has the following format:
+<pre class="syntax">
+mlp_classification(
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+\b Arguments
+<DL class="arglist">
+ <DT>source_table</DT>
+ <DD>TEXT. Name of the table containing the training data.</DD>
+
+ <DT>model_table</DT>
+ <DD>TEXT. Name of the output table containing the model. Details of the output
+ tables are provided below.
+ </DD>
+
+ <DT>independent_varname</DT>
+ <DD>TEXT. Expression list to evaluate for the
+ independent variables. An intercept variable should not be included as part
+ of this expression. Please note that expression should be able to be cast
+ to DOUBLE PRECISION[].
+ </DD>
+
+ <DT>dependent_varname</DT>
+ <DD> TEXT. Name of the dependent variable column. For classification, supported types are:
+ text, varchar, character varying, char, character
+ integer, smallint, bigint, and boolean. </DD>
+
+ <DT>hidden_layer_sizes</DT>
+ <DD>INTEGER[], default: ARRAY[].
+ The number of neurons in each hidden layer. The length of this array will
+ determine the number of hidden layers. Empty for no hidden layers.
+ </DD>
+
+
+ <DT>optimizer_params (optional)</DT>
+ <DD>TEXT, default: NULL.
+ Parameters for optimization in a comma-separated string
+ of key-value pairs. See the description below for details.
+ </DD>
+
+ <DT>activation</DT>
+ <DD>TEXT, default: 'sigmoid'.
+ Activation function. Currently three functions are supported: 'sigmoid' (default),
+ 'relu', and 'tanh'. The text can be any subset of the three
+ strings; for e.g., activation='s' will use the sigmoid activation.
+ </DD>
+</DL>
+
+<b>Output tables</b>
+<br>
+ The model table produced by mlp contains the following columns:
+ <table class="output">
+ <tr>
+ <th>coeffs</th>
+ <td>FLOAT8[]. Flat array containing the weights of the neural net</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>INTEGER. Number of iterations completed by stochastic gradient descent
+ algorithm. The algorithm either converged in this number of iterations
+ or hit the maximum number specified in the optimization parameters. </td>
+ </tr>
+ <tr>
+ <th>loss</th>
+ <td>FLOAT8. The cross entropy over the training data.
+ See Technical Background section below for more details.</td>
+ </tr>
+ </table>
+
+
+A summary table named \<model_table\>_summary is also created, which has the following columns:
+ <table class="output">
+ <tr>
+ <th>source_table</th>
+ <td>The source table.</td>
+ </tr>
+ <tr>
+ <th>dependent_varname</th>
+ <td>The dependent variable.</td>
+ </tr>
+ <tr>
+ <th>independent_varname</th>
+ <td>The independent variables.</td>
+ </tr>
+ <tr>
+ <th>tolerance</th>
+ <td>The tolerance as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>step_size</th>
+ <td>The step size as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>The number of iterations run</td>
+ </tr>
+ <tr>
+ <th>n_tries</th>
+ <td>The number of tries as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>layer_sizes</th>
+ <td>The number of units in each layer including the input and output layer.</td>
+ </tr>
+ <tr>
+ <th>activation_function</th>
+ <td>The activation function.</td>
+ </tr>
+ <tr>
+ <th>is_classification</th>
+ <td>True if the model was trained for classification, False if it was trained
+ for regression</td>
+ </tr>
+ <tr>
+ <th>classes</th>
+ <td>The classes which were trained against (empty for regression)</td>
+ </tr>
+
+ </table>
+
+
+@anchor mlp_regression
+@par Regression Training Function
+The mlp regression training function has the following format:
+<pre class="syntax">
+mlp_regression(source_table,
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+
+\b Arguments
+
+Specifications for regression are largely the same as for classification. In the
+model table, the loss will refer to mean square error instead of cross entropy. In the
+summary table, there is classes column. The following
+arguments have specifications which differ from mlp_classification:
+<DL class="arglist">
+<DT>dependent_varname</DT>
+ <DD>TEXT. Name of the dependent variable column.
+ For regression supported types are any numeric type, or array
+ or numeric types (for multiple regression).
+ </DD>
+</DL>
+
+
+@anchor optimizer_params
+@par Optimizer Parameters
+Parameters in this section are supplied in the \e optimizer_params argument as a string
+containing a comma-delimited list of name-value pairs. All of these named
+parameters are optional, and their order does not matter. You must use the
+format "<param_name> = <value>" to specify the value of a parameter, otherwise
+the parameter is ignored.
+
+
+<pre class="syntax">
+ 'step_size = <value>,
+ n_iterations = <value>,
+ n_tries = <value>,
+ tolerance = <value>'
+</pre>
+\b Optimizer Parameters
+<DL class="arglist">
+
+<DT>step_size</dt>
+<DD>Default: [0.001].
+Also known as the learning rate. A small value is usually desirable to
+ensure convergence, while a large value provides more room for progress during
+training. Since the best value depends on the condition number of the data, in
+practice one often tunes this parameter.
+</DD>
+
+
+<DT>n_iterations</dt>
+<DD>Default: [100]. The maximum number of iterations allowed.
+</DD>
+<DT>n_tries</dt>
+<DD>Default: [1]. Number of times to retrain the network with randomly initialized
+weights
+</DD>
+
+<DT>tolerance</dt>
+<DD>Default: 0.001. The criterion to end iterations. The training stops whenever
+<the difference between the training models of two consecutive iterations is
+<smaller than \e tolerance or the iteration number is larger than \e max_iter.
+</DD>
+
+</DL>
+
+@anchor predict
+@par Prediction Function
+Used to generate predictions given a previously trained model on novel data.
+The same syntax is used for classification, and regression.
+<pre class="syntax">
+mlp_predict(model_table,
+ data_table,
+ id_col_name,
+ output_table,
+ pred_type)
+</pre>
+
+\b Arguments
+<DL class="arglist">
+ <DT>model_table</DT>
+ <DD>TEXT. Model table produced by the training function.</DD>
+
+ <DT>data_table</DT>
+ <DD>TEXT. Name of the table containing the data for prediction. This table is expected
+ to contain the same input features that were used during training. The table should
+ also contain id_col_name used for identifying each row.</DD>
+
+ <DT>id_col_name</DT>
+ <DD>TEXT. The name of the id column in the input table.</DD>
+
+ <DT>output_table</DT>
+ <DD>TEXT. Name of the table where output predictions are written. If this
+table name is already in use, then an error is returned. Table contains:</DD>
+ <table class="output">
+ <tr>
+ <th>id</th>
+ <td>Gives the 'id' for each prediction, corresponding to each row from the data_table.</td>
+ </tr>
+ <tr>
+ <th>estimated_<COL_NAME></th>
+ <td>
+ (For pred_type='response') The estimated class
+ for classification or value for regression, where
+ <COL_NAME> is the name of the column to be
+ predicted from training data
+ </td>
+ </tr>
+ <tr>
+ <th>prob_<CLASS></th>
+ <td>
+ (For pred_type='prob' for classification) The
+ probability of a given class <CLASS> as given by
+ softmax. There will be one column for each class
+ in the training data.
+ </td>
+ </tr>
+
+
+ <DT>pred_type</DT>
+ <DD>TEXT.
+
+the type of output requested:
+'response' gives the actual prediction,
+'prob' gives the probability of each class.
+for regression, only type='response' is defined.
+The name of the id column in the input table.</DD>
+</DL>
+</table>
+
+@anchor example
+@par Examples
+-# Create an input data set.
+<pre class="example">
+CREATE TABLE iris_data(
+ id integer,
+ attributes numeric[],
+ class_text varchar,
+ class integer
+);
+INSERT INTO iris_data VALUES
+(1,ARRAY[5.1,3.5,1.4,0.2],'Iris-setosa',1),
+(2,ARRAY[4.9,3.0,1.4,0.2],'Iris-setosa',1),
+(3,ARRAY[4.7,3.2,1.3,0.2],'Iris-setosa',1),
+(4,ARRAY[4.6,3.1,1.5,0.2],'Iris-setosa',1),
+(5,ARRAY[5.0,3.6,1.4,0.2],'Iris-setosa',1),
+(6,ARRAY[5.4,3.9,1.7,0.4],'Iris-setosa',1),
+(7,ARRAY[4.6,3.4,1.4,0.3],'Iris-setosa',1),
+(8,ARRAY[5.0,3.4,1.5,0.2],'Iris-setosa',1),
+(9,ARRAY[4.4,2.9,1.4,0.2],'Iris-setosa',1),
+(10,ARRAY[4.9,3.1,1.5,0.1],'Iris-setosa',1),
+(11,ARRAY[7.0,3.2,4.7,1.4],'Iris-versicolor',2),
+(12,ARRAY[6.4,3.2,4.5,1.5],'Iris-versicolor',2),
+(13,ARRAY[6.9,3.1,4.9,1.5],'Iris-versicolor',2),
+(14,ARRAY[5.5,2.3,4.0,1.3],'Iris-versicolor',2),
+(15,ARRAY[6.5,2.8,4.6,1.5],'Iris-versicolor',2),
+(16,ARRAY[5.7,2.8,4.5,1.3],'Iris-versicolor',2),
+(17,ARRAY[6.3,3.3,4.7,1.6],'Iris-versicolor',2),
+(18,ARRAY[4.9,2.4,3.3,1.0],'Iris-versicolor',2),
+(19,ARRAY[6.6,2.9,4.6,1.3],'Iris-versicolor',2),
+(20,ARRAY[5.2,2.7,3.9,1.4],'Iris-versicolor',2);
+</pre>
+-# Generate a multilayer perceptron with a single hidden layer of 5 units.
+Use the attributes column as the independent variables, and use the class
+column as the classification. Set the tolerance to 0 so that 5000
+iterations will be run. Use a hyperbolic tangent activation function.
+The model will be written to mlp_model.
+<pre class="example">
+SELECT madlib.mlp_classification(
+ 'iris_data', -- Source table
+ 'mlp_model', -- Destination table
+ 'attributes', -- Input features
+ 'class_text', -- Label
+ ARRAY[5], -- Number of units per layer
+ 'step_size=0.003,
+ n_iterations=5000,
+ tolerance=0', -- Optimizer params
+ 'tanh'); -- Activation function
+</pre>
+-# View the result for the model.
+<pre class="example">
+-- Set extended display on for easier reading of output
+\\x ON
+-- Neural net Initialization is non-deterministic, so your results may vary
+SELECT * FROM mlp_model;
+</pre>
+Result:
+<pre class="result">
+-[ RECORD 1 ]--+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+coeff | {1,1,1,1,1,0.136374930803,0.188739676875,0.662387810001,-1.03381622734,-0.469961067046,0.0614006983397,0.0811504589436,0.299008228258,-0.47391918521,-0.215098143699,0.10519213944,0.145844617525,0.511683525606,-0.800215552382,-0.36417142683,0.120751709056,0.167531106521,0.587074895969,-0.916946198095,-0.417055067449,0.0539541885146,0.0694359704131,0.262598585854,-0.419234805076,-0.189915344282,1,1,1,1,1,1,0.105645702152,1.46247470474,0.484457903226,0.965962824478,1.19361986431,0.419805760087,-0.105696503487,-1.46245956666,-0.484427811691,-0.965730981426,-1.19365280555,-0.419973628863}
+loss | 0.0184092375519
+num_iterations | 5000
+</pre>
+-# Next train a regression example. First create some test data. This dataset
+contains housing prices data.
+<pre class="example">
+CREATE TABLE lin_housing (id serial, x float8[], grp_by_col int, y float8);
+COPY lin_housing (x, grp_by_col, y) FROM STDIN NULL '?' DELIMITER '|';
+{1,0.00632,18.00,2.310,0,0.5380,6.5750,65.20,4.0900,1,296.0,15.30,396.90,4.98} | 1 | 24.00
+{1,0.02731,0.00,7.070,0,0.4690,6.4210,78.90,4.9671,2,242.0,17.80,396.90,9.14} | 1 | 21.60
+{1,0.02729,0.00,7.070,0,0.4690,7.1850,61.10,4.9671,2,242.0,17.80,392.83,4.03} | 1 | 34.70
+{1,0.03237,0.00,2.180,0,0.4580,6.9980,45.80,6.0622,3,222.0,18.70,394.63,2.94} | 1 | 33.40
+{1,0.06905,0.00,2.180,0,0.4580,7.1470,54.20,6.0622,3,222.0,18.70,396.90,5.33} | 1 | 36.20
+{1,0.02985,0.00,2.180,0,0.4580,6.4300,58.70,6.0622,3,222.0,18.70,394.12,5.21} | 1 | 28.70
+{1,0.08829,12.50,7.870,0,0.5240,6.0120,66.60,5.5605,5,311.0,15.20,395.60,12.43} | 1 | 22.90
+{1,0.14455,12.50,7.870,0,0.5240,6.1720,96.10,5.9505,5,311.0,15.20,396.90,19.15} | 1 | 27.10
+{1,0.21124,12.50,7.870,0,0.5240,5.6310,100.00,6.0821,5,311.0,15.20,386.63,29.93} | 1 | 16.50
+{1,0.17004,12.50,7.870,0,0.5240,6.0040,85.90,6.5921,5,311.0,15.20,386.71,17.10} | 1 | 18.90
+{1,0.22489,12.50,7.870,0,0.5240,6.3770,94.30,6.3467,5,311.0,15.20,392.52,20.45} | 1 | 15.00
+{1,0.11747,12.50,7.870,0,0.5240,6.0090,82.90,6.2267,5,311.0,15.20,396.90,13.27} | 1 | 18.90
+{1,0.09378,12.50,7.870,0,0.5240,5.8890,39.00,5.4509,5,311.0,15.20,390.50,15.71} | 1 | 21.70
+{1,0.62976,0.00,8.140,0,0.5380,5.9490,61.80,4.7075,4,307.0,21.00,396.90,8.26} | 1 | 20.40
+{1,0.63796,0.00,8.140,0,0.5380,6.0960,84.50,4.4619,4,307.0,21.00,380.02,10.26} | 1 | 18.20
+{1,0.62739,0.00,8.140,0,0.5380,5.8340,56.50,4.4986,4,307.0,21.00,395.62,8.47} | 1 | 19.90
+{1,1.05393,0.00,8.140,0,0.5380,5.9350,29.30,4.4986,4,307.0,21.00,386.85,6.58} | 1 | 23.10
+{1,0.78420,0.00,8.140,0,0.5380,5.9900,81.70,4.2579,4,307.0,21.00,386.75,14.67} | 1 | 17.50
+{1,0.80271,0.00,8.140,0,0.5380,5.4560,36.60,3.7965,4,307.0,21.00,288.99,11.69} | 1 | 20.20
+{1,0.72580,0.00,8.140,0,0.5380,5.7270,69.50,3.7965,4,307.0,21.00,390.95,11.28} | 1 | 18.20
+\.
+</pre>
+-# Now train a regression model using a multilayer perceptron a single hidden layer of two nodes.
+<pre class="example">
+SELECT madlib.mlp_regression(
--- End diff --
In case user attempts to run multiple times, precede `SELECT` statement with:
```
DROP TABLE IF EXISTS mlp_regress;
DROP TABLE IF EXISTS mlp_regress_summary;
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by iyerr3 <gi...@git.apache.org>.
Github user iyerr3 commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r128583465
--- Diff: doc/design/modules/neural-network.tex ---
@@ -0,0 +1,195 @@
+% Licensed to the Apache Software Foundation (ASF) under one
+% or more contributor license agreements. See the NOTICE file
+% distributed with this work for additional information
+% regarding copyright ownership. The ASF licenses this file
+% to you under the Apache License, Version 2.0 (the
+% "License"); you may not use this file except in compliance
+% with the License. You may obtain a copy of the License at
+
+% http://www.apache.org/licenses/LICENSE-2.0
+
+% Unless required by applicable law or agreed to in writing,
+% software distributed under the License is distributed on an
+% "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+% KIND, either express or implied. See the License for the
+% specific language governing permissions and limitations
+% under the License.
+
+% When using TeXShop on the Mac, let it know the root document.
+% The following must be one of the first 20 lines.
+% !TEX root = ../design.tex
+
+\chapter{Neural Network}
+
--- End diff --
Let's add `\item[Authors] {Xixuan Feng}` here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by iyerr3 <gi...@git.apache.org>.
Github user iyerr3 commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r128588139
--- Diff: src/ports/postgres/modules/utilities/validate_args.py_in ---
@@ -458,6 +458,22 @@ def scalar_col_has_no_null(tbl, col):
# -------------------------------------------------------------------------
+def array_col_dimension(tbl, col):
+ """
+ What is the dimension of this array column
+ """
+ if tbl is None or tbl.lower() == 'null':
+ plpy.error('Input error: Table name (NULL) is invalid')
+ if col is None or col.lower() == 'null':
--- End diff --
IMO if we shouldn't be checking for the string 'NULL'. There are multiple strings that are invalid table/column names - why make 'null' an exception?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by njayaram2 <gi...@git.apache.org>.
Github user njayaram2 commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r128095377
--- Diff: src/ports/postgres/modules/convex/mlp_igd.py_in ---
@@ -0,0 +1,734 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+@file mlp_igd.py_in
+
+@brief Multilayer perceptron using IGD: Driver functions
+
+@namespace mlp_igd
+"""
+import plpy
+
+from utilities.control import MinWarning
+from utilities.utilities import add_postfix
+from utilities.utilities import py_list_to_sql_string
+from utilities.utilities import extract_keyvalue_params
+from utilities.utilities import _assert
+from utilities.utilities import unique_string
+from utilities.utilities import strip_end_quotes
+
+from utilities.validate_args import cols_in_tbl_valid
+from utilities.validate_args import input_tbl_valid
+from utilities.validate_args import is_var_valid
+from utilities.validate_args import output_tbl_valid
+from utilities.validate_args import get_expr_type
+from utilities.validate_args import array_col_has_same_dimension
+from utilities.validate_args import array_col_dimension
+
+
+def mlp(schema_madlib, source_table, output_table, independent_varname,
+ dependent_varname, hidden_layer_sizes,
+ optimizer_param_str, activation, is_classification, **kwargs):
+ """
+ Args:
+ @param schema_madlib
+ @param source_table
+ @param output_table
+ @param independent_varname
+ @param dependent_varname
+ @param hidden_layer_sizes
+ @param optimizer_param_str
+
+ Returns:
+ None
+ """
+ with MinWarning('info'):
+ optimizer_params = _get_optimizer_params(optimizer_param_str or "")
+ _validate_args(source_table, output_table, independent_varname,
+ dependent_varname, hidden_layer_sizes,
+ optimizer_params, is_classification)
+
+ current_iteration = 1
+ prev_state = None
+ tolerance = optimizer_params["tolerance"]
+ n_iterations = optimizer_params["n_iterations"]
+ step_size = optimizer_params["step_size"]
+ n_tries = optimizer_params["n_tries"]
+ activation_name = _get_activation_function_name(activation)
+ activation_index = _get_activation_index(activation_name)
+ num_input_nodes = array_col_dimension(source_table, independent_varname)
+ num_output_nodes = 0
+ classes = []
+ dependent_type = get_expr_type(dependent_varname, source_table)
+ original_dependent_varname = dependent_varname
+
+ if is_classification:
+ dependent_variable_sql = """
+ SELECT DISTINCT {dependent_varname}
+ FROM {source_table}
+ """.format(dependent_varname=dependent_varname,
+ source_table=source_table)
+ labels = plpy.execute(dependent_variable_sql)
+ one_hot_dependent_varname = 'ARRAY['
+ num_output_nodes = len(labels)
+ for label_obj in labels:
+ label = _format_label(label_obj[dependent_varname])
+ classes.append(label)
+ one_hot_dependent_varname += dependent_varname + "=" + str(label) + ","
+ # Remove the last comma
+ one_hot_dependent_varname = one_hot_dependent_varname[:-1]
+ one_hot_dependent_varname += ']::integer[]'
+ dependent_varname = one_hot_dependent_varname
+ if not is_classification:
--- End diff --
`else`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by cooper-sloan <gi...@git.apache.org>.
Github user cooper-sloan commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r127038159
--- Diff: src/modules/convex/type/state.hpp ---
@@ -541,6 +541,161 @@ class GLMNewtonState {
} algo;
};
+
+/**
+ * @brief Inter- (Task State) and intra-iteration (Algo State) state of
+ * incremental gradient descent for multi-layer perceptron
+ *
+ * TransitionState encapsualtes the transition state during the
+ * aggregate function during an iteration. To the database, the state is
+ * exposed as a single DOUBLE PRECISION array, to the C++ code it is a proper
+ * object containing scalars and vectors.
+ *
+ * Note: We assume that the DOUBLE PRECISION array is initialized by the
+ * database with length at least 6, and at least first elemenet
+ * is 0 (exact values of other elements are ignored).
+ *
+ */
+template <class Handle>
+class MLPIGDState {
+ template <class OtherHandle>
+ friend class MLPIGDState;
+
+public:
+ MLPIGDState(const AnyType &inArray) : mStorage(inArray.getAs<Handle>()) {
+ rebind();
+ }
+
+ /**
+ * @brief Convert to backend representation
+ *
+ * We define this function so that we can use State in the
+ * argument list and as a return type.
+ */
+ inline operator AnyType() const {
+ return mStorage;
+ }
+
+ /**
+ * @brief Allocating the incremental gradient state.
+ */
+ inline void allocate(const Allocator &inAllocator,
+ const uint16_t &inNumberOfStages,
+ const double *inNumbersOfUnits) {
+ mStorage = inAllocator.allocateArray<double, dbal::AggregateContext,
+ dbal::DoZero, dbal::ThrowBadAlloc>(
+ arraySize(inNumberOfStages, inNumbersOfUnits));
+
+ // This rebind is totally for the following lines of code to take
--- End diff --
Thanks! Will fix.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib issue #149: MLP: Multilayer Perceptron
Posted by njayaram2 <gi...@git.apache.org>.
Github user njayaram2 commented on the issue:
https://github.com/apache/incubator-madlib/pull/149
Jenkins, ok to build.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib issue #149: MLP: Multilayer Perceptron
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/incubator-madlib/pull/149
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/madlib-pr-build/117/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by edespino <gi...@git.apache.org>.
Github user edespino commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r127315428
--- Diff: src/ports/postgres/modules/convex/mlp.sql_in ---
@@ -0,0 +1,744 @@
+/* ----------------------------------------------------------------------- *//**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ *
+ * @file mlp.sql_in
+ *
+ * @brief SQL functions for multilayer perceptron
+ * @date June 2012
+ *
+ *
+ *//* ----------------------------------------------------------------------- */
+
+m4_include(`SQLCommon.m4')
+
+/**
+@addtogroup grp_mlp
+
+<div class="toc"><b>Contents</b><ul>
+<li class="level1"><a href="#mlp_classification">Classification</a></li>
+<li class="level1"><a href="#mlp_regression">Regression</a></li>
+<li class="level1"><a href="#optimization_params">Optimizer Parameters</a></li>
+<li class="level1"><a href="#predict">Prediction Functions/a></li>
+<li class="level1"><a href="#example">Examples</a></li>
+<li class="level1"><a href="#background">Technical Background</a></li>
+<li class="level1"><a href="#literature">Literature</a></li>
+<li class="level1"><a href="#related">Related Topics</a></li>
+</ul></div>
+
+Multilayer Perceptron (MLP) is a model for regression and
+classification
+
+Also called "vanilla neural networks", they consist of several
+fully connected hidden layers with non-linear activation
+functions. In the case of classification, the final layer of the
+neural net has as many nodes as classes, and the output of the
+neural net can be interpreted as the probability of a given input
+feature belonging to a specific class.
+
+
+@brief Solves classification and regression problems with several
+fully connected layers and nonlinear activation functions.
+
+@anchor mlp_classification
+@par Classification Training Function
+The mlp classification training function has the following format:
+<pre class="syntax">
+mlp_classification(
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+\b Arguments
+<DL class="arglist">
+ <DT>source_table</DT>
+ <DD>TEXT. Name of the table containing the training data.</DD>
+
+ <DT>model_table</DT>
+ <DD>TEXT. Name of the output table containing the model. Details of the output
+ tables are provided below.
+ </DD>
+
+ <DT>independent_varname</DT>
+ <DD>TEXT. Expression list to evaluate for the
+ independent variables. An intercept variable should not be included as part
+ of this expression. Please note that expression should be able to be cast
+ to DOUBLE PRECISION[].
+ </DD>
+
+ <DT>dependent_varname</DT>
+ <DD> TEXT. Name of the dependent variable column. For classification, supported types are:
+ text, varchar, character varying, char, character
+ integer, smallint, bigint, and boolean. </DD>
+
+ <DT>hidden_layer_sizes</DT>
+ <DD>INTEGER[], default: ARRAY[].
+ The number of neurons in each hidden layer. The length of this array will
+ determine the number of hidden layers. Empty for no hidden layers.
+ </DD>
+
+
+ <DT>optimizer_params (optional)</DT>
+ <DD>TEXT, default: NULL.
+ Parameters for optimization in a comma-separated string
+ of key-value pairs. See the description below for details.
+ </DD>
+
+ <DT>activation</DT>
+ <DD>TEXT, default: 'sigmoid'.
+ Activation function. Currently three functions are supported: 'sigmoid' (default),
+ 'relu', and 'tanh'. The text can be any subset of the three
+ strings; for e.g., activation='s' will use the sigmoid activation.
+ </DD>
+</DL>
+
+<b>Output tables</b>
+<br>
+ The model table produced by mlp contains the following columns:
+ <table class="output">
+ <tr>
+ <th>coeffs</th>
+ <td>FLOAT8[]. Flat array containing the weights of the neural net</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>INTEGER. Number of iterations completed by stochastic gradient descent
+ algorithm. The algorithm either converged in this number of iterations
+ or hit the maximum number specified in the optimization parameters. </td>
+ </tr>
+ <tr>
+ <th>loss</th>
+ <td>FLOAT8. The cross entropy over the training data.
+ See Technical Background section below for more details.</td>
+ </tr>
+ </table>
+
+
+A summary table named \<model_table\>_summary is also created, which has the following columns:
+ <table class="output">
+ <tr>
+ <th>source_table</th>
+ <td>The source table.</td>
+ </tr>
+ <tr>
+ <th>dependent_varname</th>
+ <td>The dependent variable.</td>
+ </tr>
+ <tr>
+ <th>independent_varname</th>
+ <td>The independent variables.</td>
+ </tr>
+ <tr>
+ <th>tolerance</th>
+ <td>The tolerance as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>step_size</th>
+ <td>The step size as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>The number of iterations run</td>
+ </tr>
+ <tr>
+ <th>n_tries</th>
+ <td>The number of tries as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>layer_sizes</th>
+ <td>The number of units in each layer including the input and output layer.</td>
+ </tr>
+ <tr>
+ <th>activation_function</th>
+ <td>The activation function.</td>
+ </tr>
+ <tr>
+ <th>is_classification</th>
+ <td>True if the model was trained for classification, False if it was trained
+ for regression</td>
+ </tr>
+ <tr>
+ <th>classes</th>
+ <td>The classes which were trained against (empty for regression)</td>
+ </tr>
+
+ </table>
+
+
+@anchor mlp_regression
+@par Regression Training Function
+The mlp regression training function has the following format:
+<pre class="syntax">
+mlp_regression(source_table,
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+
+\b Arguments
+
+Specifications for regression are largely the same as for classification. In the
+model table, the loss will refer to mean square error instead of cross entropy. In the
+summary table, there is classes column. The following
+arguments have specifications which differ from mlp_classification:
+<DL class="arglist">
+<DT>dependent_varname</DT>
+ <DD>TEXT. Name of the dependent variable column.
+ For regression supported types are any numeric type, or array
+ or numeric types (for multiple regression).
+ </DD>
+</DL>
+
+
+@anchor optimizer_params
+@par Optimizer Parameters
+Parameters in this section are supplied in the \e optimizer_params argument as a string
+containing a comma-delimited list of name-value pairs. All of these named
+parameters are optional, and their order does not matter. You must use the
+format "<param_name> = <value>" to specify the value of a parameter, otherwise
+the parameter is ignored.
+
+
+<pre class="syntax">
+ 'step_size = <value>,
+ n_iterations = <value>,
+ n_tries = <value>,
+ tolerance = <value>'
+</pre>
+\b Optimizer Parameters
+<DL class="arglist">
+
+<DT>step_size</dt>
+<DD>Default: [0.001].
+Also known as the learning rate. A small value is usually desirable to
+ensure convergence, while a large value provides more room for progress during
+training. Since the best value depends on the condition number of the data, in
+practice one often tunes this parameter.
+</DD>
+
+
+<DT>n_iterations</dt>
+<DD>Default: [100]. The maximum number of iterations allowed.
+</DD>
+<DT>n_tries</dt>
+<DD>Default: [1]. Number of times to retrain the network with randomly initialized
+weights
+</DD>
+
+<DT>tolerance</dt>
+<DD>Default: 0.001. The criterion to end iterations. The training stops whenever
+<the difference between the training models of two consecutive iterations is
+<smaller than \e tolerance or the iteration number is larger than \e max_iter.
+</DD>
+
+</DL>
+
+@anchor predict
+@par Prediction Function
+Used to generate predictions given a previously trained model on novel data.
+The same syntax is used for classification, and regression.
+<pre class="syntax">
+mlp_predict(model_table,
+ data_table,
+ id_col_name,
+ output_table,
+ pred_type)
+</pre>
+
+\b Arguments
+<DL class="arglist">
+ <DT>model_table</DT>
+ <DD>TEXT. Model table produced by the training function.</DD>
+
+ <DT>data_table</DT>
+ <DD>TEXT. Name of the table containing the data for prediction. This table is expected
+ to contain the same input features that were used during training. The table should
+ also contain id_col_name used for identifying each row.</DD>
+
+ <DT>id_col_name</DT>
+ <DD>TEXT. The name of the id column in the input table.</DD>
+
+ <DT>output_table</DT>
+ <DD>TEXT. Name of the table where output predictions are written. If this
+table name is already in use, then an error is returned. Table contains:</DD>
+ <table class="output">
+ <tr>
+ <th>id</th>
+ <td>Gives the 'id' for each prediction, corresponding to each row from the data_table.</td>
+ </tr>
+ <tr>
+ <th>estimated_<COL_NAME></th>
+ <td>
+ (For pred_type='response') The estimated class
+ for classification or value for regression, where
+ <COL_NAME> is the name of the column to be
+ predicted from training data
+ </td>
+ </tr>
+ <tr>
+ <th>prob_<CLASS></th>
+ <td>
+ (For pred_type='prob' for classification) The
+ probability of a given class <CLASS> as given by
+ softmax. There will be one column for each class
+ in the training data.
+ </td>
+ </tr>
+
+
+ <DT>pred_type</DT>
+ <DD>TEXT.
+
+the type of output requested:
+'response' gives the actual prediction,
+'prob' gives the probability of each class.
+for regression, only type='response' is defined.
+The name of the id column in the input table.</DD>
+</DL>
+</table>
+
+@anchor example
+@par Examples
+-# Create an input data set.
+<pre class="example">
+CREATE TABLE iris_data(
+ id integer,
+ attributes numeric[],
+ class_text varchar,
+ class integer
+);
+INSERT INTO iris_data VALUES
+(1,ARRAY[5.1,3.5,1.4,0.2],'Iris-setosa',1),
+(2,ARRAY[4.9,3.0,1.4,0.2],'Iris-setosa',1),
+(3,ARRAY[4.7,3.2,1.3,0.2],'Iris-setosa',1),
+(4,ARRAY[4.6,3.1,1.5,0.2],'Iris-setosa',1),
+(5,ARRAY[5.0,3.6,1.4,0.2],'Iris-setosa',1),
+(6,ARRAY[5.4,3.9,1.7,0.4],'Iris-setosa',1),
+(7,ARRAY[4.6,3.4,1.4,0.3],'Iris-setosa',1),
+(8,ARRAY[5.0,3.4,1.5,0.2],'Iris-setosa',1),
+(9,ARRAY[4.4,2.9,1.4,0.2],'Iris-setosa',1),
+(10,ARRAY[4.9,3.1,1.5,0.1],'Iris-setosa',1),
+(11,ARRAY[7.0,3.2,4.7,1.4],'Iris-versicolor',2),
+(12,ARRAY[6.4,3.2,4.5,1.5],'Iris-versicolor',2),
+(13,ARRAY[6.9,3.1,4.9,1.5],'Iris-versicolor',2),
+(14,ARRAY[5.5,2.3,4.0,1.3],'Iris-versicolor',2),
+(15,ARRAY[6.5,2.8,4.6,1.5],'Iris-versicolor',2),
+(16,ARRAY[5.7,2.8,4.5,1.3],'Iris-versicolor',2),
+(17,ARRAY[6.3,3.3,4.7,1.6],'Iris-versicolor',2),
+(18,ARRAY[4.9,2.4,3.3,1.0],'Iris-versicolor',2),
+(19,ARRAY[6.6,2.9,4.6,1.3],'Iris-versicolor',2),
+(20,ARRAY[5.2,2.7,3.9,1.4],'Iris-versicolor',2);
+</pre>
+-# Generate a multilayer perceptron with a single hidden layer of 5 units.
+Use the attributes column as the independent variables, and use the class
+column as the classification. Set the tolerance to 0 so that 5000
+iterations will be run. Use a hyperbolic tangent activation function.
+The model will be written to mlp_model.
+<pre class="example">
+SELECT madlib.mlp_classification(
+ 'iris_data', -- Source table
+ 'mlp_model', -- Destination table
+ 'attributes', -- Input features
+ 'class_text', -- Label
+ ARRAY[5], -- Number of units per layer
+ 'step_size=0.003,
+ n_iterations=5000,
+ tolerance=0', -- Optimizer params
+ 'tanh'); -- Activation function
+</pre>
+-# View the result for the model.
+<pre class="example">
+-- Set extended display on for easier reading of output
+\\x ON
+-- Neural net Initialization is non-deterministic, so your results may vary
+SELECT * FROM mlp_model;
+</pre>
+Result:
+<pre class="result">
+-[ RECORD 1 ]--+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+coeff | {1,1,1,1,1,0.136374930803,0.188739676875,0.662387810001,-1.03381622734,-0.469961067046,0.0614006983397,0.0811504589436,0.299008228258,-0.47391918521,-0.215098143699,0.10519213944,0.145844617525,0.511683525606,-0.800215552382,-0.36417142683,0.120751709056,0.167531106521,0.587074895969,-0.916946198095,-0.417055067449,0.0539541885146,0.0694359704131,0.262598585854,-0.419234805076,-0.189915344282,1,1,1,1,1,1,0.105645702152,1.46247470474,0.484457903226,0.965962824478,1.19361986431,0.419805760087,-0.105696503487,-1.46245956666,-0.484427811691,-0.965730981426,-1.19365280555,-0.419973628863}
+loss | 0.0184092375519
+num_iterations | 5000
+</pre>
+-# Next train a regression example. First create some test data. This dataset
+contains housing prices data.
+<pre class="example">
+CREATE TABLE lin_housing (id serial, x float8[], grp_by_col int, y float8);
+COPY lin_housing (x, grp_by_col, y) FROM STDIN NULL '?' DELIMITER '|';
+{1,0.00632,18.00,2.310,0,0.5380,6.5750,65.20,4.0900,1,296.0,15.30,396.90,4.98} | 1 | 24.00
+{1,0.02731,0.00,7.070,0,0.4690,6.4210,78.90,4.9671,2,242.0,17.80,396.90,9.14} | 1 | 21.60
+{1,0.02729,0.00,7.070,0,0.4690,7.1850,61.10,4.9671,2,242.0,17.80,392.83,4.03} | 1 | 34.70
+{1,0.03237,0.00,2.180,0,0.4580,6.9980,45.80,6.0622,3,222.0,18.70,394.63,2.94} | 1 | 33.40
+{1,0.06905,0.00,2.180,0,0.4580,7.1470,54.20,6.0622,3,222.0,18.70,396.90,5.33} | 1 | 36.20
+{1,0.02985,0.00,2.180,0,0.4580,6.4300,58.70,6.0622,3,222.0,18.70,394.12,5.21} | 1 | 28.70
+{1,0.08829,12.50,7.870,0,0.5240,6.0120,66.60,5.5605,5,311.0,15.20,395.60,12.43} | 1 | 22.90
+{1,0.14455,12.50,7.870,0,0.5240,6.1720,96.10,5.9505,5,311.0,15.20,396.90,19.15} | 1 | 27.10
+{1,0.21124,12.50,7.870,0,0.5240,5.6310,100.00,6.0821,5,311.0,15.20,386.63,29.93} | 1 | 16.50
+{1,0.17004,12.50,7.870,0,0.5240,6.0040,85.90,6.5921,5,311.0,15.20,386.71,17.10} | 1 | 18.90
+{1,0.22489,12.50,7.870,0,0.5240,6.3770,94.30,6.3467,5,311.0,15.20,392.52,20.45} | 1 | 15.00
+{1,0.11747,12.50,7.870,0,0.5240,6.0090,82.90,6.2267,5,311.0,15.20,396.90,13.27} | 1 | 18.90
+{1,0.09378,12.50,7.870,0,0.5240,5.8890,39.00,5.4509,5,311.0,15.20,390.50,15.71} | 1 | 21.70
+{1,0.62976,0.00,8.140,0,0.5380,5.9490,61.80,4.7075,4,307.0,21.00,396.90,8.26} | 1 | 20.40
+{1,0.63796,0.00,8.140,0,0.5380,6.0960,84.50,4.4619,4,307.0,21.00,380.02,10.26} | 1 | 18.20
+{1,0.62739,0.00,8.140,0,0.5380,5.8340,56.50,4.4986,4,307.0,21.00,395.62,8.47} | 1 | 19.90
+{1,1.05393,0.00,8.140,0,0.5380,5.9350,29.30,4.4986,4,307.0,21.00,386.85,6.58} | 1 | 23.10
+{1,0.78420,0.00,8.140,0,0.5380,5.9900,81.70,4.2579,4,307.0,21.00,386.75,14.67} | 1 | 17.50
+{1,0.80271,0.00,8.140,0,0.5380,5.4560,36.60,3.7965,4,307.0,21.00,288.99,11.69} | 1 | 20.20
+{1,0.72580,0.00,8.140,0,0.5380,5.7270,69.50,3.7965,4,307.0,21.00,390.95,11.28} | 1 | 18.20
+\.
+</pre>
+-# Now train a regression model using a multilayer perceptron a single hidden layer of two nodes.
+<pre class="example">
+SELECT madlib.mlp_regression(
+ 'lin_housing', -- Source table
+ 'mlp_regress', -- Desination table
+ 'x', -- Input features
+ 'y', -- Dependent variable
+ ARRAY[5,5], -- Number of units per layer
+ 'step_size=0.000007,
+ n_iterations=10000,
+ tolerance=0',
+ 'relu');
+</pre>
+-# Check the results of the model
+<pre class="example">
+-- Set extended display on for easier reading of output
+\\x ON
+-- Neural net Initialization is non-deterministic, so your results may vary
+SELECT * FROM mlp_regress;
+</pre>
+Result:
+<pre class="result">
+-[ RECORD 1 ]--+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------
+coeff | {1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2.79506311399e-05,3.56715008915e-05,-6.09333559685e-05,0.000251228318768,-0.000224772841379,-3.71863030857e-05,-3.5757865148e-06,5.27936784854e-05,-2.48474166186e-05,6.19731184294e-05,3.07638968743e-05,6.8964698578e-06,0.000106016701083,-1.71484730318e-05,1.18691881812e-05,-0.000163975464208,0.000170026304906,3.11688265279e-05,0.000177050148787,-1.58265976603e-05,2.70144422657e-05,0.000112667883422,3.77575139073e-05,8.12474658795e-05,-7.90458917626e-05,0.000107566386158,-2.63771171506e-06,2.47996880915e-05,-0.00012642310887,0.000203827391081,0.000139315565565,4.86147243454e-05,-0.000176126471913,-6.47820782916e-05,-8.51592776447e-06,-6.60601176758e-05,2.91421874156e-05,6.3556873752e-05,0.000197557443129,0.000220531367259,0.000135036310289,0.000143735913975,-4.75034117786e-05,-0.000179547345838,-1.6919846786e-05,0.000162784312994,0.000268595819851,-0.000460066553287,8.69756071591e-05,-0.00311762727057,0.000126024763103,0.00020598824
2921,0.003463432426,-0.00729789075286,0.00151625867549,-0.000890852767597,-0.00525016037249,0.0031043106659,0.00798041103839,-0.00552693050079,0.0232180415786,0.0230489850143,-0.0437890272341,0.0165765426407,-0.248554261758,-7.81336427846e-05,0.00558145591752,0.283465844585,-0.571699956182,0.133474351994,-0.0785181945605,-0.419269930709,0.249547772912,0.631761009875,-0.431305975666,1,1,1,1,1,1,0.0158747497572,-9.02809160806e-05,0.00015574347618,4.10805373863e-06,0.00121532434965,0.101790351335,0.0647558401493,-0.00013654998677,-9.92872075948e-06,-5.5319694394e-05,0.00519320756484,0.412736586036,0.0011998026977,-1.53688189815e-05,1.94817888201e-05,-4.63111489966e-05,7.24547899029e-05,0.00880394144485,5.45309822095e-05,-0.000140943219275,-7.96211486227e-05,-1.04337307472e-05,0.000161936762028,0.00136273797767,-4.54737243585e-05,-3.4083840736e-05,3.69286883662e-05,9.9047243188e-08,3.75014011824e-06,-9.45366086368e-08,1,1,1,1,1,1,6.67488547054,0.102754199001,0.41668912471,0.008868672964
79,0.00136206007228,-9.88642499013e-05}
+loss | 144.965776158
+num_iterations | 10000
+</pre>
+-# Now let's look at the prediction functions. In the following examples we will
+use the training data set for prediction as well, which is not usual but serves to
+show the syntax. First we will test the classification exampel.
+The prediction is in the the estimated_class_text column with the
+actual value in the class_text column.
+<pre class="example">
+SELECT madlib.mlp_predict(
--- End diff --
In case user attempts to run multiple times, precede `SELECT` statement with:
```
DROP TABLE IF EXISTS mlp_prediction;
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib issue #149: MLP: Multilayer Perceptron
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/incubator-madlib/pull/149
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/madlib-pr-build/116/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by cooper-sloan <gi...@git.apache.org>.
Github user cooper-sloan commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r127080429
--- Diff: src/modules/convex/mlp_igd.cpp ---
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ * @file mlp_igd.cpp
+ *
+ * @brief Low-rank Matrix Factorization functions
+ *
+ *//* ----------------------------------------------------------------------- */
+#include <boost/lexical_cast.hpp>
+
+#include <dbconnector/dbconnector.hpp>
+#include <modules/shared/HandleTraits.hpp>
+
+#include "mlp_igd.hpp"
+
+#include "task/mlp.hpp"
+#include "algo/igd.hpp"
+#include "algo/loss.hpp"
+
+#include "type/tuple.hpp"
+#include "type/model.hpp"
+#include "type/state.hpp"
+
+namespace madlib {
+
+namespace modules {
+
+namespace convex {
+
+// These 2 classes contain public static methods that can be called
+typedef IGD<MLPIGDState<MutableArrayHandle<double> >, MLPIGDState<ArrayHandle<double> >,
+ MLP<MLPModel<MutableArrayHandle<double> >, MLPTuple > > MLPIGDAlgorithm;
+
+typedef Loss<MLPIGDState<MutableArrayHandle<double> >, MLPIGDState<ArrayHandle<double> >,
+ MLP<MLPModel<MutableArrayHandle<double> >, MLPTuple > > MLPLossAlgorithm;
+
+typedef MLP<MLPModel<MutableArrayHandle<double> >,MLPTuple> MLPTask;
+
+/**
+ * @brief Perform the multi-layer perceptron transition step
+ *
+ * Called for each tuple.
+ */
+AnyType
+mlp_igd_transition::run(AnyType &args) {
+ // For the first tuple: args[0] is nothing more than a marker that
+ // indicates that we should do some initial operations.
+ // For other tuples: args[0] holds the computation state until last tuple
+ MLPIGDState<MutableArrayHandle<double> > state = args[0];
+
+ // initilize the state if first tuple
+ if (state.algo.numRows == 0) {
+ if (!args[3].isNull()) {
+ MLPIGDState<ArrayHandle<double> > previousState = args[3];
+
+ state.allocate(*this, previousState.task.numberOfStages,
+ previousState.task.numbersOfUnits);
+ state = previousState;
+ } else {
+ // configuration parameters
+ ArrayHandle<double> numbersOfUnits = args[4].getAs<ArrayHandle<double> >();
+ if (numbersOfUnits.size() <= 1) {
+ throw std::runtime_error("Invalid parameter: numbers_of_units "
+ "has too few entries");
+ } else if (numbersOfUnits.size() >=
+ std::numeric_limits<uint16_t>::max()) {
+ throw std::runtime_error("Invalid parameter: numbers_of_units "
+ "has too many entries");
+ }
+ size_t k;
+ for (k = 0; k < numbersOfUnits.size(); k ++) {
+ if (numbersOfUnits[k] == 0) {
+ throw std::runtime_error("Invalid parameter: numbers_of_units "
+ "has zero entry");
+ }
+ }
+
+ double stepsize = args[5].getAs<double>();
+ if (stepsize <= 0.) {
+ throw std::runtime_error("Invalid parameter: stepsize <= 0.0");
--- End diff --
Good catch. I think I will remove this validation as it is done in python layer.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by cooper-sloan <gi...@git.apache.org>.
Github user cooper-sloan commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r127040170
--- Diff: src/ports/postgres/dbconnector/TypeTraits_impl.hpp ---
@@ -750,6 +784,30 @@ struct TypeTraits<dbal::eigen_integration::SparseColumnVector> {
);
};
+template <>
+struct TypeTraits<std::vector<uint16_t> > {
+ typedef std::vector<uint16_t> value_type;
+
+ WITH_OID( INT4ARRAYOID );
+ WITH_MUTABILITY( dbal::Mutable );
+ WITH_DEFAULT_EXTENDED_TRAITS;
+ // FIXME a quick hack
--- End diff --
We ended up not using this new functionality (using double ptr instead). I will remove this code, we can always add it back later. Does that seem reasonable?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by iyerr3 <gi...@git.apache.org>.
Github user iyerr3 commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r128586055
--- Diff: src/modules/convex/task/mlp.hpp ---
@@ -0,0 +1,334 @@
+/* ----------------------------------------------------------------------- *//**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ * @file mlp.hpp
+ *
+ * This file contains objective function related computation, which is called
+ * by classes in algo/, e.g., loss, gradient functions
+ *
+ *//* ----------------------------------------------------------------------- */
+
+#ifndef MADLIB_MODULES_CONVEX_TASK_MLP_HPP_
+#define MADLIB_MODULES_CONVEX_TASK_MLP_HPP_
+
+namespace madlib {
+
+namespace modules {
+
+namespace convex {
+
+// Use Eigen
+using namespace madlib::dbal::eigen_integration;
+
+template <class Model, class Tuple>
+class MLP {
+public:
+ typedef Model model_type;
+ typedef Tuple tuple_type;
+ typedef typename Tuple::independent_variables_type
+ independent_variables_type;
+ typedef typename Tuple::dependent_variable_type dependent_variable_type;
+
+ static void gradientInPlace(
+ model_type &model,
+ const independent_variables_type &y,
+ const dependent_variable_type &z,
+ const double &stepsize);
+
+ static double loss(
+ const model_type &model,
+ const independent_variables_type &y,
+ const dependent_variable_type &z);
+
+ static ColumnVector predict(
+ const model_type &model,
+ const independent_variables_type &y,
+ const bool get_class);
+
+ const static int RELU = 0;
+ const static int SIGMOID = 1;
+ const static int TANH = 2;
+
+ static double sigmoid(const double &xi) {
+ return 1. / (1. + std::exp(-xi));
+ }
+
+ static double relu(const double &xi) {
+ return xi*(xi>0);
+ }
+
+ static double tanh(const double &xi) {
+ return std::tanh(xi);
+ }
+
+
+private:
+
+ static double sigmoidDerivative(const double &xi) {
+ double value = sigmoid(xi);
+ return value * (1. - value);
+ }
+
+ static double reluDerivative(const double &xi) {
+ return xi>0;
+ }
+
+ static double tanhDerivative(const double &xi) {
+ double value = tanh(xi);
+ return 1-value*value;
+ }
+
+ static void feedForward(
+ const model_type &model,
+ const independent_variables_type &y,
+ std::vector<ColumnVector> &net,
+ std::vector<ColumnVector> &x);
+
+ static void endLayerDeltaError(
+ const std::vector<ColumnVector> &net,
+ const std::vector<ColumnVector> &x,
+ const dependent_variable_type &z,
+ ColumnVector &delta_N);
+
+ static void errorBackPropagation(
+ const ColumnVector &delta_N,
+ const std::vector<ColumnVector> &net,
+ const model_type &model,
+ std::vector<ColumnVector> &delta);
+};
+
+template <class Model, class Tuple>
+void
+MLP<Model, Tuple>::gradientInPlace(
+ model_type &model,
+ const independent_variables_type &y,
+ const dependent_variable_type &z,
+ const double &stepsize) {
+ (void) model;
+ (void) z;
+ (void) y;
+ (void) stepsize;
+ std::vector<ColumnVector> net;
+ std::vector<ColumnVector> x;
+ std::vector<ColumnVector> delta;
+ ColumnVector delta_N;
+
+ feedForward(model, y, net, x);
+ endLayerDeltaError(net, x, z, delta_N);
+ errorBackPropagation(delta_N, net, model, delta);
+
+ uint16_t N = model.u.size(); // assuming nu. of layers >= 1
+ uint16_t k, s, j;
+ // #TODO fix lambda
--- End diff --
Please add more details on the TODO
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib issue #149: MLP: Multilayer Perceptron
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/incubator-madlib/pull/149
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/madlib-pr-build/111/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by njayaram2 <gi...@git.apache.org>.
Github user njayaram2 commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r128065369
--- Diff: src/ports/postgres/modules/convex/mlp.sql_in ---
@@ -0,0 +1,752 @@
+/* ----------------------------------------------------------------------- *//**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ *
+ * @file mlp.sql_in
+ *
+ * @brief SQL functions for multilayer perceptron
+ * @date June 2012
+ *
+ *
+ *//* ----------------------------------------------------------------------- */
+
+m4_include(`SQLCommon.m4')
+
+/**
+@addtogroup grp_mlp
+
+<div class="toc"><b>Contents</b><ul>
+<li class="level1"><a href="#mlp_classification">Classification</a></li>
+<li class="level1"><a href="#mlp_regression">Regression</a></li>
+<li class="level1"><a href="#optimization_params">Optimizer Parameters</a></li>
+<li class="level1"><a href="#predict">Prediction Functions/a></li>
+<li class="level1"><a href="#example">Examples</a></li>
+<li class="level1"><a href="#background">Technical Background</a></li>
+<li class="level1"><a href="#literature">Literature</a></li>
+<li class="level1"><a href="#related">Related Topics</a></li>
+</ul></div>
+
+Multilayer Perceptron (MLP) is a model for regression and
+classification
--- End diff --
End this line with a `.`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib issue #149: MLP: Multilayer Perceptron
Posted by edespino <gi...@git.apache.org>.
Github user edespino commented on the issue:
https://github.com/apache/incubator-madlib/pull/149
Not sure what this is. While playing around with the module, I accidentally hit a `nan` condition. Here is the output of the session:
```
madlibtest=# CREATE TABLE iris_data(
madlibtest(# id integer,
madlibtest(# attributes numeric[],
madlibtest(# class_text varchar,
madlibtest(# class integer
madlibtest(# );
CREATE TABLE
madlibtest=# INSERT INTO iris_data VALUES
madlibtest-# (1,ARRAY[5.1,3.5,1.4,0.2],'Iris-setosa',1);
INSERT 0 1
madlibtest=# SELECT madlib.mlp_classification(
madlibtest(# 'iris_data', -- Source table
madlibtest(# 'mlp_model', -- Destination table
madlibtest(# 'attributes', -- Input features
madlibtest(# 'class_text', -- Label
madlibtest(# ARRAY[5], -- Number of units per layer
madlibtest(# 'step_size=0.003,
madlibtest'# n_iterations=5000,
madlibtest'# tolerance=0', -- Optimizer params
madlibtest(# 'tanh'); -- Activation function
WARNING: loss: nan
ERROR: spiexceptions.UndefinedColumn: column "nan" does not exist
LINE 4: ...7731e-05,-0.000170379656714,0.000110138191401,1.0,nan,1.0,1....
^
QUERY:
SELECT madlib.internal_mlp_igd_distance(
(NULL)::DOUBLE PRECISION[],
ARRAY[ 2.0,4.0,5.0,1.0,0.003,1.0,2.0,1.0,1.0,1.0,1.0,1.0,-0.000161677781273,2.34991541167e-05,-5.27816928274e-05,1.6207561377e-06,9.30167921009e-05,-2.2322484638e-06,-8.8803764219e-05,6.92299455941e-05,0.000174430298478,1.36067005437e-07,-0.000126402572755,0.00010846859258,-0.000131999662994,-0.000111939884066,0.000154233933791,9.62727960033e-05,7.92279598497e-05,3.55520197415e-05,4.28280053118e-05,-0.000136356574146,-3.07769363501e-05,-8.97131375537e-05,-0.000117176292437,-9.89281636713e-05,0.000234615355418,1.0,1.0,1.0,1.0,1.0,1.0,2.00514836476e-05,-9.37533114569e-05,7.58698692162e-05,-2.64470277731e-05,-0.000170379656714,0.000110138191401,1.0,nan,1.0,1.0,1.0,1.0,1.0,-0.000161677781273,2.34991541167e-05,-5.27816928274e-05,1.6207561377e-06,9.30167921009e-05,-2.2322484638e-06,-8.8803764219e-05,6.92299455941e-05,0.000174430298478,1.36067005437e-07,-0.000126402572755,0.00010846859258,-0.000131999662994,-0.000111939884066,0.000154233933791,9.62727960033e-05,7
.92279598497e-05,3.55520197415e-05,4.28280053118e-05,-0.000136356574146,-3.07769363501e-05,-8.97131375537e-05,-0.000117176292437,-9.89281636713e-05,0.000234615355418,1.0,1.0,1.0,1.0,1.0,1.0,2.00514836476e-05,-9.37533114569e-05,7.58698692162e-05,-2.64470277731e-05,-0.000170379656714,0.000110138191401,0.0,0.0 ]::double precision[]) as state_dist
CONTEXT: Traceback (most recent call last):
PL/Python function "mlp_classification", line 32, in <module>
True
PL/Python function "mlp_classification", line 140, in mlp
PL/Python function "mlp_classification"
madlibtest=#
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by njayaram2 <gi...@git.apache.org>.
Github user njayaram2 commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r128101328
--- Diff: src/ports/postgres/modules/convex/mlp_igd.py_in ---
@@ -0,0 +1,734 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+@file mlp_igd.py_in
+
+@brief Multilayer perceptron using IGD: Driver functions
+
+@namespace mlp_igd
+"""
+import plpy
+
+from utilities.control import MinWarning
+from utilities.utilities import add_postfix
+from utilities.utilities import py_list_to_sql_string
+from utilities.utilities import extract_keyvalue_params
+from utilities.utilities import _assert
+from utilities.utilities import unique_string
+from utilities.utilities import strip_end_quotes
+
+from utilities.validate_args import cols_in_tbl_valid
+from utilities.validate_args import input_tbl_valid
+from utilities.validate_args import is_var_valid
+from utilities.validate_args import output_tbl_valid
+from utilities.validate_args import get_expr_type
+from utilities.validate_args import array_col_has_same_dimension
+from utilities.validate_args import array_col_dimension
+
+
+def mlp(schema_madlib, source_table, output_table, independent_varname,
+ dependent_varname, hidden_layer_sizes,
+ optimizer_param_str, activation, is_classification, **kwargs):
+ """
+ Args:
+ @param schema_madlib
+ @param source_table
+ @param output_table
+ @param independent_varname
+ @param dependent_varname
+ @param hidden_layer_sizes
+ @param optimizer_param_str
+
+ Returns:
+ None
+ """
+ with MinWarning('info'):
+ optimizer_params = _get_optimizer_params(optimizer_param_str or "")
+ _validate_args(source_table, output_table, independent_varname,
+ dependent_varname, hidden_layer_sizes,
+ optimizer_params, is_classification)
+
+ current_iteration = 1
+ prev_state = None
+ tolerance = optimizer_params["tolerance"]
+ n_iterations = optimizer_params["n_iterations"]
+ step_size = optimizer_params["step_size"]
+ n_tries = optimizer_params["n_tries"]
+ activation_name = _get_activation_function_name(activation)
+ activation_index = _get_activation_index(activation_name)
+ num_input_nodes = array_col_dimension(source_table, independent_varname)
+ num_output_nodes = 0
+ classes = []
+ dependent_type = get_expr_type(dependent_varname, source_table)
+ original_dependent_varname = dependent_varname
+
+ if is_classification:
+ dependent_variable_sql = """
+ SELECT DISTINCT {dependent_varname}
+ FROM {source_table}
+ """.format(dependent_varname=dependent_varname,
+ source_table=source_table)
+ labels = plpy.execute(dependent_variable_sql)
+ one_hot_dependent_varname = 'ARRAY['
+ num_output_nodes = len(labels)
+ for label_obj in labels:
+ label = _format_label(label_obj[dependent_varname])
+ classes.append(label)
+ one_hot_dependent_varname += dependent_varname + "=" + str(label) + ","
+ # Remove the last comma
+ one_hot_dependent_varname = one_hot_dependent_varname[:-1]
+ one_hot_dependent_varname += ']::integer[]'
+ dependent_varname = one_hot_dependent_varname
+ if not is_classification:
+ if "[]" not in dependent_type:
+ dependent_varname = "ARRAY[" + dependent_varname + "]"
+ num_output_nodes = array_col_dimension(source_table, dependent_varname)
+ layer_sizes = [num_input_nodes] + hidden_layer_sizes + [num_output_nodes]
+
+ while True:
+ if prev_state:
+ prev_state_str = py_list_to_sql_string(prev_state, array_type="double precision")
+ else:
+ prev_state_str = "(NULL)::DOUBLE PRECISION[]"
+ train_sql = """
+ SELECT
+ {schema_madlib}.mlp_igd_step(
+ ({independent_varname})::DOUBLE PRECISION[],
+ ({dependent_varname})::DOUBLE PRECISION[],
+ {prev_state},
+ {layer_sizes},
+ ({step_size})::FLOAT8,
+ {activation},
+ {is_classification}) as curr_state
+ FROM {source_table} AS _src
+ """.format(schema_madlib=schema_madlib,
+ independent_varname=independent_varname,
+ dependent_varname=dependent_varname,
+ prev_state=prev_state_str,
+ # C++ uses double internally
+ layer_sizes=py_list_to_sql_string(layer_sizes,
+ array_type="double precision"),
+ step_size=step_size,
+ source_table=source_table,
+ activation=activation_index,
+ is_classification=int(is_classification))
+ curr_state = plpy.execute(train_sql)[0]["curr_state"]
+ dist_sql = """
+ SELECT {schema_madlib}.internal_mlp_igd_distance(
+ {prev_state},
+ {curr_state}) as state_dist
+ """.format(schema_madlib=schema_madlib,
+ prev_state=prev_state_str,
+ curr_state=py_list_to_sql_string(curr_state, "double precision"))
+ state_dist = plpy.execute(dist_sql)[0]["state_dist"]
+ if ((state_dist and state_dist < tolerance) or
+ current_iteration > n_iterations):
+ break
+ prev_state = curr_state
+ current_iteration += 1
+ _build_model_table(schema_madlib, output_table, curr_state, n_iterations)
+ layer_sizes_str = py_list_to_sql_string(layer_sizes, array_type="integer")
+ classes_str = py_list_to_sql_string([strip_end_quotes(cl, "'") for cl in classes],
+ array_type=dependent_type)
+ summary_table_creation_query = """
+ CREATE TABLE {output_table}_summary(
+ source_table TEXT,
+ independent_varname TEXT,
+ dependent_varname TEXT,
+ tolerance FLOAT,
+ step_size FLOAT,
+ n_iterations INTEGER,
+ n_tries INTEGER,
+ layer_sizes INTEGER[],
+ activation_function TEXT,
+ is_classification BOOLEAN,
+ classes {dependent_type}[]
+ )""".format(output_table=output_table,
+ dependent_type=dependent_type)
+
+ summary_table_update_query = """
+ INSERT INTO {output_table}_summary VALUES(
+ '{source_table}',
+ '{independent_varname}',
+ '{original_dependent_varname}',
+ {tolerance},
+ {step_size},
+ {n_iterations},
+ {n_tries},
+ {layer_sizes_str},
+ '{activation_name}',
+ {is_classification},
+ {classes_str}
+ )
+ """.format(**locals())
+ plpy.execute(summary_table_creation_query)
+ plpy.execute(summary_table_update_query)
+# ----------------------------------------------------------------------
+
+
+def _build_model_table(schema_madlib, output_table, final_state, n_iterations):
+ final_state_str = py_list_to_sql_string(final_state, array_type="double precision")
+
+ model_table_query = """
+ CREATE TABLE {output_table} AS
+ SELECT
+ (result).coeff AS coeff,
+ (result).loss AS loss,
+ {n_iterations} AS num_iterations
+ -- (result).num_rows_processed AS num_rows_processed,
+ -- n_tuples_including_nulls - (result).num_rows_processed
+ FROM (
+ SELECT
+ {schema_madlib}.internal_mlp_igd_result(
+ {final_state_str}
+ ) AS result
+ ) rel_state_subq
+ """.format(**locals())
+ plpy.execute(model_table_query)
+# ----------------------------------------------------------------------
+
+
+def _get_optimizer_params(param_str):
+ params_defaults = {
+ "step_size": (0.001, float),
+ "n_iterations": (100, int),
+ "n_tries": (1, int),
+ "tolerance": (0.001, float),
+ }
+ param_defaults = dict([(k, v[0]) for k, v in params_defaults.items()])
+ param_types = dict([(k, v[1]) for k, v in params_defaults.items()])
+
+ if not param_str:
+ return param_defaults
+
+ name_value = extract_keyvalue_params(param_str, param_types, param_defaults,
+ ignore_invalid=False)
+ return name_value
+# ----------------------------------------------------------------------
+
+
+def _validate_args_classification(source_table, dependent_varname):
+ expr_type = get_expr_type(dependent_varname, source_table)
+ int_types = ['integer', 'smallint', 'bigint']
+ text_types = ['text', 'varchar', 'character varying', 'char', 'character']
+ boolean_types = ['boolean']
+ _assert("[]" in expr_type or expr_type in int_types + text_types + boolean_types,
+ "Dependent variable column should refer to an "
+ "integer, boolean, text, varchar, or character type.")
+# ----------------------------------------------------------------------
+
+
+def _validate_args_regression(source_table, dependent_varname):
+ expr_type = get_expr_type(dependent_varname, source_table)
+ int_types = ['integer', 'smallint', 'bigint']
+ float_types = ['double precision', 'real']
+ _assert("[]" in expr_type or expr_type in int_types + float_types,
+ "Dependent variable column should refer to an array or numeric type")
+ if "[]" in expr_type:
+ _assert(array_col_has_same_dimension(source_table, dependent_varname),
+ "Dependent variable column should refer to arrays of the same length")
+# ----------------------------------------------------------------------
+
+
+def _validate_args(source_table, output_table, independent_varname,
+ dependent_varname, hidden_layer_sizes,
+ optimizer_params, is_classification):
+ input_tbl_valid(source_table, "MLP")
+ output_tbl_valid(output_table, "MLP")
+ output_tbl_valid(output_table+"_summary", "MLP")
+ _assert(is_var_valid(source_table, independent_varname),
+ "MLP error: invalid independent_varname "
+ "('{independent_varname}') for source_table "
+ "({source_table})!".format(independent_varname=independent_varname,
+ source_table=source_table))
+
+ _assert(is_var_valid(source_table, dependent_varname),
+ "MLP error: invalid dependent_varname "
+ "('{dependent_varname}') for source_table "
+ "({source_table})!".format(dependent_varname=dependent_varname,
+ source_table=source_table))
+ _assert(hidden_layer_sizes is not None,
+ "hidden_layer_sizes may not be null")
+ _assert(isinstance(hidden_layer_sizes, list),
+ "hidden_layer_sizes must be an array of integers")
+ _assert(all(isinstance(value, int) for value in hidden_layer_sizes),
+ "MLP error: Hidden layers sizes must be integers")
+ _assert(all(value >= 0 for value in hidden_layer_sizes),
+ "MLP error: Hidden layers sizes must be greater than 0.")
+ _assert(optimizer_params["tolerance"] >= 0,
+ "MLP error: Tolerance should be greater than or equal to 0.")
+ _assert(optimizer_params["n_tries"] >= 1,
+ "MLP error: Number of tries should be greater than or equal to 1")
+ _assert(optimizer_params["n_iterations"] >= 1,
+ "MLP error: Number of iterations should be greater than or equal to 1")
+ _assert(optimizer_params["step_size"] > 0,
+ "MLP error: Stepsize should be greater than 0.")
+ _assert("[]" in get_expr_type(independent_varname, source_table),
+ "Independent variable column should refer to an array")
+ _assert(array_col_has_same_dimension(source_table, independent_varname),
+ "Independent variable column should refer to arrays of the same length")
+
+ if is_classification:
+ _validate_args_classification(source_table, dependent_varname)
+ else:
+ _validate_args_regression(source_table, dependent_varname)
+# ----------------------------------------------------------------------
+
+
+def _get_activation_function_name(activation_function):
+ if not activation_function:
+ activation_function = 'sigmoid'
+ else:
+ # Add non-linear kernels below after implementing them.
+ supported_activation_function = ['sigmoid', 'tanh', 'relu']
+ try:
+ # allow user to specify a prefix substring of
+ # supported kernels. This works because the supported
+ # kernels have unique prefixes.
+ activation_function = next(x for x in supported_activation_function
+ if x.startswith(activation_function))
+ except StopIteration:
+ # next() returns a StopIteration if no element found
+ plpy.error("MLP Error: Invalid activation function: "
+ "{0}. Supported activation functions are ({1})"
+ .format(activation_function, ','.join(
+ sorted(supported_activation_function))))
+ return activation_function
+# ------------------------------------------------------------------------------
+
+
+def _get_activation_index(activation_name):
+ table = {"relu": 0, "sigmoid": 1, "tanh": 2}
+ return table[activation_name]
+
+
+def _format_label(label):
+ if isinstance(label, str):
+ return "'" + label + "'"
+ return label
+# -------------------------------------------------------------------------
+
+
+def mlp_predict(schema_madlib, model_table, data_table,
+ id_col_name, output_table,
+ pred_type='response', **kwargs):
+ """ Score new observations using a trained neural network
+
+ @param schema_madlib Name of the schema where MADlib is installed
+ @param model_table Name of learned model
+ @param data_table Name of table/view containing the data
+ points to be scored
+ @param id_col_name Name of column in source_table containing
+ (integer) identifier for data point
+ @param output_table Name of table to store the results
+ @param pred_type: str, The type of output required:
+ 'response' gives the actual response values,
+ 'prob' gives the probability of the classes in a
+ For regression, only type='response' is defined.
+ """
+ # model table
+ input_tbl_valid(model_table, 'MLP')
+ cols_in_tbl_valid(model_table, ['coeff'], 'MLP')
+ # summary table
+ summary_table = add_postfix(model_table, "_summary")
+ input_tbl_valid(summary_table, 'MLP')
+ cols_in_tbl_valid(summary_table,
+ ['dependent_varname', 'independent_varname',
+ 'activation_function',
+ 'tolerance', 'step_size', 'n_iterations',
+ 'n_tries', 'classes', 'layer_sizes', 'source_table'],
+ 'MLP')
+
+ # read necessary info from summary
+ summary = plpy.execute("SELECT * FROM {0}".format(summary_table))[0]
+ coeff = py_list_to_sql_string(plpy.execute("SELECT * FROM {0}".format(model_table))[0]["coeff"])
+ dependent_varname = summary['dependent_varname']
+ independent_varname = summary['independent_varname']
+ source_table = summary['source_table']
+ activation_function = _get_activation_index(summary['activation_function'])
+ layer_sizes = py_list_to_sql_string(summary['layer_sizes'], array_type="DOUBLE PRECISION")
+ is_classification = int(summary["is_classification"])
+ is_response = int(pred_type == 'response')
+
+ pred_name = ('"prob_{0}"' if pred_type == "prob" else
+ '"estimated_{0}"').format(dependent_varname.replace('"', '').strip())
+
+ input_tbl_valid(data_table, 'MLP')
+
+ _assert(is_var_valid(data_table, independent_varname),
+ "MLP Error: independent_varname ('{0}') is invalid for data_table ({1})".
+ format(independent_varname, data_table))
+ _assert(id_col_name is not None, "MLP Error: id_col_name is NULL")
+ _assert(is_var_valid(data_table, id_col_name),
+ "MLP Error: id_col_name ('{0}') is invalid for {1}".
+ format(id_col_name, data_table))
+ output_tbl_valid(output_table, 'MLP')
+ # optimizer_param_dict = _get_optimizer_params(optimizer_params)
+
+ with MinWarning("warning"):
+ header = "CREATE TABLE " + output_table + " AS "
+ # Regression
+ if not is_classification:
+ dependent_type = get_expr_type(dependent_varname, source_table)
+ unnest_if_not_array = ""
+ # Return the same type as the user provided. Internally we always use an array, but
+ # if they provided a scaler, unnest it for the user
+ if "[]" not in dependent_type:
+ unnest_if_not_array = "UNNEST"
+ sql = header + """
+ SELECT {id_col_name},
+ {unnest_if_not_array}({schema_madlib}.internal_predict_mlp(
+ {coeff},
+ {independent_varname}::DOUBLE PRECISION[],
+ {is_classification},
+ {activation_function},
+ {layer_sizes},
+ {is_response}
+ )) as {pred_name}
+ FROM {data_table}
+ """
+ else:
+ summary_query = """
+ SELECT classes FROM {0}_summary
+ """.format(model_table)
+ classes = plpy.execute(summary_query)[0]['classes']
+ if pred_type == "response":
+ # This join is to recover the class name from the summary table,
+ # as prediction just returns an index
+ classes_with_index_table = unique_string()
+ classes_table = unique_string()
+ sql = header + """
+ SELECT
+ q.{id_col_name}
+ ,(ARRAY{classes})[pred_idx[1]+1] as {pred_name}
+ FROM (
+ SELECT
+ {id_col_name},
+ {schema_madlib}.internal_predict_mlp(
+ {coeff}::DOUBLE PRECISION[],
+ {independent_varname}::DOUBLE PRECISION[],
+ {is_classification},
+ {activation_function},
+ {layer_sizes},
+ {is_response}
+ )
+ as pred_idx
+ FROM {data_table}
+ ) q
+ """
+ else:
+ # Incomplete
+ intermediate_col = unique_string()
+ score_format = ',\n'.join([
+ 'CAST({interim}[{j}] as DOUBLE PRECISION) as "estimated_prob_{c_str}"'.
+ format(j=i + 1, c_str=str(c).strip(' "'), interim=intermediate_col)
+ for i, c in enumerate(classes)])
+ sql = header + """
+ SELECT
+ {id_col_name},
+ {score_format}
+ FROM (
+ SELECT {id_col_name},
+ {schema_madlib}.internal_predict_mlp(
+ {coeff}::DOUBLE PRECISION[],
+ {independent_varname}::DOUBLE PRECISION[],
+ {is_classification},
+ {activation_function},
+ {layer_sizes},
+ {is_response}
+ )::TEXT[]
+ AS {intermediate_col}
+ FROM {data_table}
+ ) q
+ """
+ sql = sql.format(**locals())
+ with MinWarning('warning'):
--- End diff --
This might not be necessary, use the `MinWarning` that was defined before the `if` condition instead.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by njayaram2 <gi...@git.apache.org>.
Github user njayaram2 commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r128102125
--- Diff: src/modules/convex/mlp_igd.cpp ---
@@ -0,0 +1,236 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ * @file mlp_igd.cpp
+ *
+ * @brief Multilayer Perceptron functions
+ *
+ *//* ----------------------------------------------------------------------- */
+#include <boost/lexical_cast.hpp>
+
+#include <dbconnector/dbconnector.hpp>
+#include <modules/shared/HandleTraits.hpp>
+
+#include "mlp_igd.hpp"
+
+#include "task/mlp.hpp"
+#include "algo/igd.hpp"
+#include "algo/loss.hpp"
+
+#include "type/tuple.hpp"
+#include "type/model.hpp"
+#include "type/state.hpp"
+
+namespace madlib {
+
+namespace modules {
+
+namespace convex {
+
+// These 2 classes contain public static methods that can be called
+typedef IGD<MLPIGDState<MutableArrayHandle<double> >, MLPIGDState<ArrayHandle<double> >,
+ MLP<MLPModel<MutableArrayHandle<double> >, MLPTuple > > MLPIGDAlgorithm;
+
+typedef Loss<MLPIGDState<MutableArrayHandle<double> >, MLPIGDState<ArrayHandle<double> >,
+ MLP<MLPModel<MutableArrayHandle<double> >, MLPTuple > > MLPLossAlgorithm;
+
+typedef MLP<MLPModel<MutableArrayHandle<double> >,MLPTuple> MLPTask;
+
+/**
+ * @brief Perform the multilayer perceptron transition step
+ *
+ * Called for each tuple.
+ */
+AnyType
+mlp_igd_transition::run(AnyType &args) {
+ // For the first tuple: args[0] is nothing more than a marker that
+ // indicates that we should do some initial operations.
+ // For other tuples: args[0] holds the computation state until last tuple
+ MLPIGDState<MutableArrayHandle<double> > state = args[0];
+
+ // initilize the state if first tuple
+ if (state.algo.numRows == 0) {
+ if (!args[3].isNull()) {
+ MLPIGDState<ArrayHandle<double> > previousState = args[3];
+
+ state.allocate(*this, previousState.task.numberOfStages,
+ previousState.task.numbersOfUnits);
+ state = previousState;
+ } else {
+ // configuration parameters
+ ArrayHandle<double> numbersOfUnits = args[4].getAs<ArrayHandle<double> >();
+
+ double stepsize = args[5].getAs<double>();
+
+ state.allocate(*this, numbersOfUnits.size() - 1,
+ reinterpret_cast<const double *>(numbersOfUnits.ptr()));
+ state.task.stepsize = stepsize;
+
+
+ int activation = args[6].getAs<int>();
+
+ int is_classification = args[7].getAs<int>();
+ state.task.model.initialize(is_classification, activation);
+ }
+
+ // resetting in either case
+ state.reset();
+ }
+
+ // meta data
+ const uint16_t N = state.task.numberOfStages;
+ const double *n = state.task.numbersOfUnits;
+
+ // tuple
+ MappedColumnVector indVar;
+ MappedColumnVector depVar;
+ try {
+ // an exception is raised in the backend if args[2] contains nulls
+ MappedColumnVector x = args[1].getAs<MappedColumnVector>();
+ // x is a const reference, we can only rebind to change its pointer
+ indVar.rebind(x.memoryHandle(), x.size());
+ MappedColumnVector y = args[2].getAs<MappedColumnVector>();
+ depVar.rebind(y.memoryHandle(), y.size());
+
+ } catch (const ArrayWithNullException &e) {
+ return args[0];
+ }
+ MLPTuple tuple;
+ tuple.indVar.rebind(indVar.memoryHandle(), indVar.size());
+ tuple.depVar.rebind(depVar.memoryHandle(), depVar.size());
+
+ // Now do the transition step
+ MLPIGDAlgorithm::transition(state, tuple);
+ MLPLossAlgorithm::transition(state, tuple);
+ state.algo.numRows ++;
+
+ return state;
+}
+
+/**
+ * @brief Perform the perliminary aggregation function: Merge transition states
+ */
+AnyType
+mlp_igd_merge::run(AnyType &args) {
+ MLPIGDState<MutableArrayHandle<double> > stateLeft = args[0];
+ MLPIGDState<ArrayHandle<double> > stateRight = args[1];
+
+ // We first handle the trivial case where this function is called with one
+ // of the states being the initial state
+ if (stateLeft.algo.numRows == 0) { return stateRight; }
+ else if (stateRight.algo.numRows == 0) { return stateLeft; }
+
+ // Merge states together
+ MLPIGDAlgorithm::merge(stateLeft, stateRight);
+ MLPLossAlgorithm::merge(stateLeft, stateRight);
+ // The following numRows update, cannot be put above, because the model
+ // averaging depends on their original values
+ stateLeft.algo.numRows += stateRight.algo.numRows;
+
+ return stateLeft;
+}
+
+/**
+ * @brief Perform the multilayer perceptron final step
+ */
+AnyType
+mlp_igd_final::run(AnyType &args) {
+ // We request a mutable object. Depending on the backend, this might perform
+ // a deep copy.
+ MLPIGDState<MutableArrayHandle<double> > state = args[0];
+
+ // Aggregates that haven't seen any data just return Null.
+ if (state.algo.numRows == 0) { return Null(); }
+
+ // finalizing
+ MLPIGDAlgorithm::final(state);
+
+ // Return the mean loss
+ state.algo.loss = state.algo.loss/static_cast<double>(state.algo.numRows);
+
+ // for stepsize tuning
+ std::stringstream debug;
+ debug << "loss: " << state.algo.loss;
+ warning(debug.str());
--- End diff --
Remove debug statements, or change it to `debug` instead of `warning`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by edespino <gi...@git.apache.org>.
Github user edespino commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r126803370
--- Diff: src/modules/convex/type/state.hpp ---
@@ -541,6 +541,161 @@ class GLMNewtonState {
} algo;
};
+
+/**
+ * @brief Inter- (Task State) and intra-iteration (Algo State) state of
+ * incremental gradient descent for multi-layer perceptron
+ *
+ * TransitionState encapsualtes the transition state during the
+ * aggregate function during an iteration. To the database, the state is
+ * exposed as a single DOUBLE PRECISION array, to the C++ code it is a proper
+ * object containing scalars and vectors.
+ *
+ * Note: We assume that the DOUBLE PRECISION array is initialized by the
+ * database with length at least 6, and at least first elemenet
+ * is 0 (exact values of other elements are ignored).
+ *
+ */
+template <class Handle>
+class MLPIGDState {
+ template <class OtherHandle>
+ friend class MLPIGDState;
+
+public:
+ MLPIGDState(const AnyType &inArray) : mStorage(inArray.getAs<Handle>()) {
+ rebind();
+ }
+
+ /**
+ * @brief Convert to backend representation
+ *
+ * We define this function so that we can use State in the
+ * argument list and as a return type.
+ */
+ inline operator AnyType() const {
+ return mStorage;
+ }
+
+ /**
+ * @brief Allocating the incremental gradient state.
+ */
+ inline void allocate(const Allocator &inAllocator,
+ const uint16_t &inNumberOfStages,
+ const double *inNumbersOfUnits) {
+ mStorage = inAllocator.allocateArray<double, dbal::AggregateContext,
+ dbal::DoZero, dbal::ThrowBadAlloc>(
+ arraySize(inNumberOfStages, inNumbersOfUnits));
+
+ // This rebind is totally for the following lines of code to take
--- End diff --
I noticed you adjusted (removal of "totally") a similar comment in "src/modules/convex/type/state.hpp". Is the same update applicable here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by edespino <gi...@git.apache.org>.
Github user edespino commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r127024441
--- Diff: src/ports/postgres/modules/convex/mlp.sql_in ---
@@ -0,0 +1,736 @@
+/* ----------------------------------------------------------------------- *//**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ *
+ * @file mlp.sql_in
+ *
+ * @brief SQL functions for multi-layer perceptron
+ * @date June 2012
+ *
+ *
+ *//* ----------------------------------------------------------------------- */
+
+m4_include(`SQLCommon.m4')
+
+/**
+@addtogroup grp_mlp
+
+<div class="toc"><b>Contents</b><ul>
+<li class="level1"><a href="#mlp_classification">Classification</a></li>
+<li class="level1"><a href="#mlp_regression">Regression</a></li>
+<li class="level1"><a href="#optimization_params">Optimization Parameters</a></li>
+<li class="level1"><a href="#predict">Prediction Functions/a></li>
+<li class="level1"><a href="#example">Examples</a></li>
+<li class="level1"><a href="#background">Technical Background</a></li>
+<li class="level1"><a href="#literature">Literature</a></li>
+<li class="level1"><a href="#related">Related Topics</a></li>
+</ul></div>
+
+Multi Layer Perceptron (MLP) is a model for regression and
+classification
+
+Also called "vanilla neural networks", they consist of several
+fully connected hidden layers with non-linear activation
+functions. In the case of classification, the final layer of the
+neural net has as many nodes as classes, and the output of the
+neural net can be interpreted as the probability of a given input
+feature belonging to a specific class.
+
+
+@brief Solves classification and regression problems with several
+fully connected layers and nonlinear activation functions.
+
+@anchor mlp_classification
+@par Classification Training Function
+The mlp classification training function has the following format:
+<pre class="syntax">
+mlp_classification(
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+\b Arguments
+<DL class="arglist">
+ <DT>source_table</DT>
+ <DD>TEXT. Name of the table containing the training data.</DD>
+
+ <DT>model_table</DT>
+ <DD>TEXT. Name of the output table containing the model. Details of the output
+ tables are provided below.
+ </DD>
+
+ <DT>independent_varname</DT>
+ <DD>TEXT. Expression list to evaluate for the
+ independent variables. An intercept variable should not be included as part
+ of this expression. Please note that expression should be able to be cast
+ to DOUBLE PRECISION[].
+ </DD>
+
+ <DT>dependent_varname</DT>
+ <DD> TEXT. Name of the dependent variable column. For classification, supported types are:
+ text, varchar, character varying, char, character
+ integer, smallint, bigint, and boolean. </DD>
+
+ <DT>hidden_layer_sizes</DT>
+ <DD>INTEGER[], default: ARRAY[].
+ The number of neurons in each hidden layer. The length of this array will
+ determine the number of hidden layers. Empty for no hidden layers.
+ </DD>
+
+
+ <DT>optimizer_params (optional)</DT>
+ <DD>TEXT, default: NULL.
+ Parameters for optimization in a comma-separated string
+ of key-value pairs. See the description below for details.
+ </DD>
+
+ <DT>activation</DT>
+ <DD>TEXT, default: 'sigmoid'.
+ Activation function. Currently three functions are supported: 'sigmoid' (default),
+ 'relu', and 'tanh'. The text can be any subset of the three
+ strings; for e.g., activation='s' will use the sigmoid activation.
+ </DD>
+</DL>
+
+<b>Output tables</b>
+<br>
+ The model table produced by mlp contains the following columns:
+ <table class="output">
+ <tr>
+ <th>coeffs</th>
+ <td>FLOAT8[]. Flat array containing the weights of the neural net</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>INTEGER. Number of iterations completed by stochastic gradient descent
+ algorithm. The algorithm either converged in this number of iterations
+ or hit the maximum number specified in the optimization parameters. </td>
+ </tr>
+ <tr>
+ <th>loss</th>
+ <td>FLOAT8. The cross entropy over the training data.
+ See Technical Background section below for more details.</td>
+ </tr>
+ </table>
+
+
+A summary table named \<model_table\>_summary is also created, which has the following columns:
+ <table class="output">
+ <tr>
+ <th>source_table</th>
+ <td>The source table.</td>
+ </tr>
+ <tr>
+ <th>dependent_varname</th>
+ <td>The dependent variable.</td>
+ </tr>
+ <tr>
+ <th>independent_varname</th>
+ <td>The independent variables.</td>
+ </tr>
+ <tr>
+ <th>tolerance</th>
+ <td>The tolerance as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>step_size</th>
+ <td>The step size as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>The number of iterations run</td>
+ </tr>
+ <tr>
+ <th>n_tries</th>
+ <td>The number of tries as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>layer_sizes</th>
+ <td>The number of units in each layer including the input and output layer.</td>
+ </tr>
+ <tr>
+ <th>activation_function</th>
+ <td>The activation function.</td>
+ </tr>
+ <tr>
+ <th>is_classification</th>
+ <td>True if the model was trained for classification, False if it was trained
+ for regression</td>
+ </tr>
+ <tr>
+ <th>classes</th>
+ <td>The classes which were trained against (empty for regression)</td>
+ </tr>
+
+ </table>
+
+
+@anchor mlp_regression
+@par Regression Training Function
+The mlp regression training function has the following format:
+<pre class="syntax">
+mlp_regression(source_table,
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+
+\b Arguments
+
+Specifications for regression are largely the same as for classification. In the
+model table, the loss will refer to mean square error instead of cross entropy. In the
+summary table, there is classes column. The following
+arguments have specifications which differ from mlp_classification:
+<DL class="arglist">
+<DT>dependent_varname</DT>
+ <DD>TEXT. Name of the dependent variable column.
+ For regression supported types are any numeric type, or array
+ or numeric types (for multiple regression).
+ </DD>
+</DL>
+
+
+@anchor optimizer_params
+@par Optimizer Parameters
+Parameters in this section are supplied in the \e optimizer_params argument as a string
+containing a comma-delimited list of name-value pairs. All of these named
+parameters are optional, and their order does not matter. You must use the
+format "<param_name> = <value>" to specify the value of a parameter, otherwise
+the parameter is ignored.
+
+
+<pre class="syntax">
+ 'step_size = <value>,
+ n_iterations = <value>,
+ n_tries = <value>,
+ tolerance = <value>'
+</pre>
+\b Optimizer Parameters
+<DL class="arglist">
+
+<DT>step_size</dt>
+<DD>Default: [0.001].
+Also known as the learning rate. A small value is usually desirable to
+ensure convergence, while a large value provides more room for progress during
+training. Since the best value depends on the condition number of the data, in
+practice one often tunes this parameter.
+</DD>
+
+
+<DT>n_iterations</dt>
+<DD>Default: [100]. The maximum number of iterations allowed.
+</DD>
+<DT>n_tries</dt>
+<DD>Default: [1]. Number of times to retrain the network with randomly initialized
+weights
+</DD>
+
+<DT>tolerance</dt>
+<DD>Default: 0.001. The criterion to end iterations. The training stops whenever
+<the difference between the training models of two consecutive iterations is
+<smaller than \e tolerance or the iteration number is larger than \e max_iter.
+</DD>
+
+</DL>
+
+@anchor predict
+@par Prediction Function
+Used to generate predictions given a previously trained model on novel data.
+The same syntax is used for classification, and regression.
+<pre class="syntax">
+mlp_predict(model_table,
+ data_table,
+ id_col_name,
+ output_table,
+ pred_type)
+</pre>
+
+\b Arguments
+<DL class="arglist">
+ <DT>model_table</DT>
+ <DD>TEXT. Model table produced by the training function.</DD>
+
+ <DT>data_table</DT>
+ <DD>TEXT. Name of the table containing the data for prediction. This table is expected
+ to contain the same input features that were used during training. The table should
+ also contain id_col_name used for identifying each row.</DD>
+
+ <DT>id_col_name</DT>
+ <DD>TEXT. The name of the id column in the input table.</DD>
+
+ <DT>output_table</DT>
+ <DD>TEXT. Name of the table where output predictions are written. If this
+table name is already in use, then an error is returned. Table contains:</DD>
+ <table class="output">
+ <tr>
+ <th>id</th>
+ <td>Gives the 'id' for each prediction, corresponding to each row from the data_table.</td>
+ </tr>
+ <tr>
+ <th>estimated_<COL_NAME></th>
+ <td>
+ (For pred_type='response') The estimated class
+ for classification or value for regression, where
+ <COL_NAME> is the name of the column to be
+ predicted from training data
+ </td>
+ </tr>
+ <tr>
+ <th>prob_<CLASS></th>
+ <td>
+ (For pred_type='prob' for classification) The
+ probability of a given class <CLASS> as given by
+ softmax. There will be one column for each class
+ in the training data.
+ </td>
+ </tr>
+
+
+ <DT>pred_type</DT>
+ <DD>TEXT.
+
+the type of output requested:
+'response' gives the actual prediction,
+'prob' gives the probability of each class.
+for regression, only type='response' is defined.
+The name of the id column in the input table.</DD>
+</DL>
+</table>
+
+@anchor example
+@par Examples
+-# Create an input data set.
+<pre class="example">
+CREATE TABLE iris_data(
+ id integer,
+ attributes numeric[],
+ class_text varchar,
+ class integer
+);
+INSERT INTO iris_data VALUES
+(1,ARRAY[5.1,3.5,1.4,0.2],'Iris-setosa',1),
+(2,ARRAY[4.9,3.0,1.4,0.2],'Iris-setosa',1),
+(3,ARRAY[4.7,3.2,1.3,0.2],'Iris-setosa',1),
+(4,ARRAY[4.6,3.1,1.5,0.2],'Iris-setosa',1),
+(5,ARRAY[5.0,3.6,1.4,0.2],'Iris-setosa',1),
+(6,ARRAY[5.4,3.9,1.7,0.4],'Iris-setosa',1),
+(7,ARRAY[4.6,3.4,1.4,0.3],'Iris-setosa',1),
+(8,ARRAY[5.0,3.4,1.5,0.2],'Iris-setosa',1),
+(9,ARRAY[4.4,2.9,1.4,0.2],'Iris-setosa',1),
+(10,ARRAY[4.9,3.1,1.5,0.1],'Iris-setosa',1),
+(11,ARRAY[7.0,3.2,4.7,1.4],'Iris-versicolor',2),
+(12,ARRAY[6.4,3.2,4.5,1.5],'Iris-versicolor',2),
+(13,ARRAY[6.9,3.1,4.9,1.5],'Iris-versicolor',2),
+(14,ARRAY[5.5,2.3,4.0,1.3],'Iris-versicolor',2),
+(15,ARRAY[6.5,2.8,4.6,1.5],'Iris-versicolor',2),
+(16,ARRAY[5.7,2.8,4.5,1.3],'Iris-versicolor',2),
+(17,ARRAY[6.3,3.3,4.7,1.6],'Iris-versicolor',2),
+(18,ARRAY[4.9,2.4,3.3,1.0],'Iris-versicolor',2),
+(19,ARRAY[6.6,2.9,4.6,1.3],'Iris-versicolor',2),
+(20,ARRAY[5.2,2.7,3.9,1.4],'Iris-versicolor',2);
+</pre>
+-# Generate a multilayer perceptron with a single hidden layer of 5 units.
+Use the attributes column as the independent variables, and use the class
+column as the classification. Set the tolerance to 0 so that 5000
+iterations will be run. Use a hyperbolic tangent activation function.
+The model will be written to mlp_result.
+<pre class="example">
+SELECT madlib.mlp_classification(
+ 'iris_data_2', -- Source table
--- End diff --
Should the reference to `iris_data_2` (Source table) be `iris_data`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by haying <gi...@git.apache.org>.
Github user haying commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r127072514
--- Diff: src/modules/convex/mlp_igd.cpp ---
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ * @file mlp_igd.cpp
+ *
+ * @brief Low-rank Matrix Factorization functions
--- End diff --
not "Low-rank Matrix Factorization functions"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by edespino <gi...@git.apache.org>.
Github user edespino commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r127034699
--- Diff: src/ports/postgres/modules/convex/mlp.sql_in ---
@@ -0,0 +1,736 @@
+/* ----------------------------------------------------------------------- *//**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ *
+ * @file mlp.sql_in
+ *
+ * @brief SQL functions for multi-layer perceptron
+ * @date June 2012
+ *
+ *
+ *//* ----------------------------------------------------------------------- */
+
+m4_include(`SQLCommon.m4')
+
+/**
+@addtogroup grp_mlp
+
+<div class="toc"><b>Contents</b><ul>
+<li class="level1"><a href="#mlp_classification">Classification</a></li>
+<li class="level1"><a href="#mlp_regression">Regression</a></li>
+<li class="level1"><a href="#optimization_params">Optimization Parameters</a></li>
+<li class="level1"><a href="#predict">Prediction Functions/a></li>
+<li class="level1"><a href="#example">Examples</a></li>
+<li class="level1"><a href="#background">Technical Background</a></li>
+<li class="level1"><a href="#literature">Literature</a></li>
+<li class="level1"><a href="#related">Related Topics</a></li>
+</ul></div>
+
+Multi Layer Perceptron (MLP) is a model for regression and
+classification
+
+Also called "vanilla neural networks", they consist of several
+fully connected hidden layers with non-linear activation
+functions. In the case of classification, the final layer of the
+neural net has as many nodes as classes, and the output of the
+neural net can be interpreted as the probability of a given input
+feature belonging to a specific class.
+
+
+@brief Solves classification and regression problems with several
+fully connected layers and nonlinear activation functions.
+
+@anchor mlp_classification
+@par Classification Training Function
+The mlp classification training function has the following format:
+<pre class="syntax">
+mlp_classification(
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+\b Arguments
+<DL class="arglist">
+ <DT>source_table</DT>
+ <DD>TEXT. Name of the table containing the training data.</DD>
+
+ <DT>model_table</DT>
+ <DD>TEXT. Name of the output table containing the model. Details of the output
+ tables are provided below.
+ </DD>
+
+ <DT>independent_varname</DT>
+ <DD>TEXT. Expression list to evaluate for the
+ independent variables. An intercept variable should not be included as part
+ of this expression. Please note that expression should be able to be cast
+ to DOUBLE PRECISION[].
+ </DD>
+
+ <DT>dependent_varname</DT>
+ <DD> TEXT. Name of the dependent variable column. For classification, supported types are:
+ text, varchar, character varying, char, character
+ integer, smallint, bigint, and boolean. </DD>
+
+ <DT>hidden_layer_sizes</DT>
+ <DD>INTEGER[], default: ARRAY[].
+ The number of neurons in each hidden layer. The length of this array will
+ determine the number of hidden layers. Empty for no hidden layers.
+ </DD>
+
+
+ <DT>optimizer_params (optional)</DT>
+ <DD>TEXT, default: NULL.
+ Parameters for optimization in a comma-separated string
+ of key-value pairs. See the description below for details.
+ </DD>
+
+ <DT>activation</DT>
+ <DD>TEXT, default: 'sigmoid'.
+ Activation function. Currently three functions are supported: 'sigmoid' (default),
+ 'relu', and 'tanh'. The text can be any subset of the three
+ strings; for e.g., activation='s' will use the sigmoid activation.
+ </DD>
+</DL>
+
+<b>Output tables</b>
+<br>
+ The model table produced by mlp contains the following columns:
+ <table class="output">
+ <tr>
+ <th>coeffs</th>
+ <td>FLOAT8[]. Flat array containing the weights of the neural net</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>INTEGER. Number of iterations completed by stochastic gradient descent
+ algorithm. The algorithm either converged in this number of iterations
+ or hit the maximum number specified in the optimization parameters. </td>
+ </tr>
+ <tr>
+ <th>loss</th>
+ <td>FLOAT8. The cross entropy over the training data.
+ See Technical Background section below for more details.</td>
+ </tr>
+ </table>
+
+
+A summary table named \<model_table\>_summary is also created, which has the following columns:
+ <table class="output">
+ <tr>
+ <th>source_table</th>
+ <td>The source table.</td>
+ </tr>
+ <tr>
+ <th>dependent_varname</th>
+ <td>The dependent variable.</td>
+ </tr>
+ <tr>
+ <th>independent_varname</th>
+ <td>The independent variables.</td>
+ </tr>
+ <tr>
+ <th>tolerance</th>
+ <td>The tolerance as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>step_size</th>
+ <td>The step size as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>The number of iterations run</td>
+ </tr>
+ <tr>
+ <th>n_tries</th>
+ <td>The number of tries as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>layer_sizes</th>
+ <td>The number of units in each layer including the input and output layer.</td>
+ </tr>
+ <tr>
+ <th>activation_function</th>
+ <td>The activation function.</td>
+ </tr>
+ <tr>
+ <th>is_classification</th>
+ <td>True if the model was trained for classification, False if it was trained
+ for regression</td>
+ </tr>
+ <tr>
+ <th>classes</th>
+ <td>The classes which were trained against (empty for regression)</td>
+ </tr>
+
+ </table>
+
+
+@anchor mlp_regression
+@par Regression Training Function
+The mlp regression training function has the following format:
+<pre class="syntax">
+mlp_regression(source_table,
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+
+\b Arguments
+
+Specifications for regression are largely the same as for classification. In the
+model table, the loss will refer to mean square error instead of cross entropy. In the
+summary table, there is classes column. The following
+arguments have specifications which differ from mlp_classification:
+<DL class="arglist">
+<DT>dependent_varname</DT>
+ <DD>TEXT. Name of the dependent variable column.
+ For regression supported types are any numeric type, or array
+ or numeric types (for multiple regression).
+ </DD>
+</DL>
+
+
+@anchor optimizer_params
+@par Optimizer Parameters
+Parameters in this section are supplied in the \e optimizer_params argument as a string
+containing a comma-delimited list of name-value pairs. All of these named
+parameters are optional, and their order does not matter. You must use the
+format "<param_name> = <value>" to specify the value of a parameter, otherwise
+the parameter is ignored.
+
+
+<pre class="syntax">
+ 'step_size = <value>,
+ n_iterations = <value>,
+ n_tries = <value>,
+ tolerance = <value>'
+</pre>
+\b Optimizer Parameters
+<DL class="arglist">
+
+<DT>step_size</dt>
+<DD>Default: [0.001].
+Also known as the learning rate. A small value is usually desirable to
+ensure convergence, while a large value provides more room for progress during
+training. Since the best value depends on the condition number of the data, in
+practice one often tunes this parameter.
+</DD>
+
+
+<DT>n_iterations</dt>
+<DD>Default: [100]. The maximum number of iterations allowed.
+</DD>
+<DT>n_tries</dt>
+<DD>Default: [1]. Number of times to retrain the network with randomly initialized
+weights
+</DD>
+
+<DT>tolerance</dt>
+<DD>Default: 0.001. The criterion to end iterations. The training stops whenever
+<the difference between the training models of two consecutive iterations is
+<smaller than \e tolerance or the iteration number is larger than \e max_iter.
+</DD>
+
+</DL>
+
+@anchor predict
+@par Prediction Function
+Used to generate predictions given a previously trained model on novel data.
+The same syntax is used for classification, and regression.
+<pre class="syntax">
+mlp_predict(model_table,
+ data_table,
+ id_col_name,
+ output_table,
+ pred_type)
+</pre>
+
+\b Arguments
+<DL class="arglist">
+ <DT>model_table</DT>
+ <DD>TEXT. Model table produced by the training function.</DD>
+
+ <DT>data_table</DT>
+ <DD>TEXT. Name of the table containing the data for prediction. This table is expected
+ to contain the same input features that were used during training. The table should
+ also contain id_col_name used for identifying each row.</DD>
+
+ <DT>id_col_name</DT>
+ <DD>TEXT. The name of the id column in the input table.</DD>
+
+ <DT>output_table</DT>
+ <DD>TEXT. Name of the table where output predictions are written. If this
+table name is already in use, then an error is returned. Table contains:</DD>
+ <table class="output">
+ <tr>
+ <th>id</th>
+ <td>Gives the 'id' for each prediction, corresponding to each row from the data_table.</td>
+ </tr>
+ <tr>
+ <th>estimated_<COL_NAME></th>
+ <td>
+ (For pred_type='response') The estimated class
+ for classification or value for regression, where
+ <COL_NAME> is the name of the column to be
+ predicted from training data
+ </td>
+ </tr>
+ <tr>
+ <th>prob_<CLASS></th>
+ <td>
+ (For pred_type='prob' for classification) The
+ probability of a given class <CLASS> as given by
+ softmax. There will be one column for each class
+ in the training data.
+ </td>
+ </tr>
+
+
+ <DT>pred_type</DT>
+ <DD>TEXT.
+
+the type of output requested:
+'response' gives the actual prediction,
+'prob' gives the probability of each class.
+for regression, only type='response' is defined.
+The name of the id column in the input table.</DD>
+</DL>
+</table>
+
+@anchor example
+@par Examples
+-# Create an input data set.
+<pre class="example">
+CREATE TABLE iris_data(
+ id integer,
+ attributes numeric[],
+ class_text varchar,
+ class integer
+);
+INSERT INTO iris_data VALUES
+(1,ARRAY[5.1,3.5,1.4,0.2],'Iris-setosa',1),
+(2,ARRAY[4.9,3.0,1.4,0.2],'Iris-setosa',1),
+(3,ARRAY[4.7,3.2,1.3,0.2],'Iris-setosa',1),
+(4,ARRAY[4.6,3.1,1.5,0.2],'Iris-setosa',1),
+(5,ARRAY[5.0,3.6,1.4,0.2],'Iris-setosa',1),
+(6,ARRAY[5.4,3.9,1.7,0.4],'Iris-setosa',1),
+(7,ARRAY[4.6,3.4,1.4,0.3],'Iris-setosa',1),
+(8,ARRAY[5.0,3.4,1.5,0.2],'Iris-setosa',1),
+(9,ARRAY[4.4,2.9,1.4,0.2],'Iris-setosa',1),
+(10,ARRAY[4.9,3.1,1.5,0.1],'Iris-setosa',1),
+(11,ARRAY[7.0,3.2,4.7,1.4],'Iris-versicolor',2),
+(12,ARRAY[6.4,3.2,4.5,1.5],'Iris-versicolor',2),
+(13,ARRAY[6.9,3.1,4.9,1.5],'Iris-versicolor',2),
+(14,ARRAY[5.5,2.3,4.0,1.3],'Iris-versicolor',2),
+(15,ARRAY[6.5,2.8,4.6,1.5],'Iris-versicolor',2),
+(16,ARRAY[5.7,2.8,4.5,1.3],'Iris-versicolor',2),
+(17,ARRAY[6.3,3.3,4.7,1.6],'Iris-versicolor',2),
+(18,ARRAY[4.9,2.4,3.3,1.0],'Iris-versicolor',2),
+(19,ARRAY[6.6,2.9,4.6,1.3],'Iris-versicolor',2),
+(20,ARRAY[5.2,2.7,3.9,1.4],'Iris-versicolor',2);
+</pre>
+-# Generate a multilayer perceptron with a single hidden layer of 5 units.
+Use the attributes column as the independent variables, and use the class
+column as the classification. Set the tolerance to 0 so that 5000
+iterations will be run. Use a hyperbolic tangent activation function.
+The model will be written to mlp_result.
+<pre class="example">
+SELECT madlib.mlp_classification(
+ 'iris_data_2', -- Source table
+ 'mlp_model', -- Destination table
+ 'attributes', -- Input features
+ 'class_text', -- Label
+ ARRAY[5], -- Number of units per layer
+ 'step_size=0.003,
+ n_iterations=5000,
+ tolerance=0', -- Optimizer params
+ 'tanh'); -- Activation function
+</pre>
+-# View the result for the model.
+<pre class="example">
+-- Set extended display on for easier reading of output
+\\x ON
+-- Neural net Initialization is non-deterministic, so your results may vary
+SELECT * FROM mlp_model;
+</pre>
+Result:
+<pre class="result">
+-[ RECORD 1 ]--+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+coeff | {1,1,1,1,1,0.136374930803,0.188739676875,0.662387810001,-1.03381622734,-0.469961067046,0.0614006983397,0.0811504589436,0.299008228258,-0.47391918521,-0.215098143699,0.10519213944,0.145844617525,0.511683525606,-0.800215552382,-0.36417142683,0.120751709056,0.167531106521,0.587074895969,-0.916946198095,-0.417055067449,0.0539541885146,0.0694359704131,0.262598585854,-0.419234805076,-0.189915344282,1,1,1,1,1,1,0.105645702152,1.46247470474,0.484457903226,0.965962824478,1.19361986431,0.419805760087,-0.105696503487,-1.46245956666,-0.484427811691,-0.965730981426,-1.19365280555,-0.419973628863}
+loss | 0.0184092375519
+num_iterations | 5000
+</pre>
+-# Next train a regression example. First create some test data. This dataset
+contains housing prices data.
+<pre class="example">
+CREATE TABLE lin_housing (id serial, x float8[], grp_by_col int, y float8);
+COPY lin_housing (x, grp_by_col, y) FROM STDIN NULL '?' DELIMITER '|';
+{1,0.00632,18.00,2.310,0,0.5380,6.5750,65.20,4.0900,1,296.0,15.30,396.90,4.98} | 1 | 24.00
+{1,0.02731,0.00,7.070,0,0.4690,6.4210,78.90,4.9671,2,242.0,17.80,396.90,9.14} | 1 | 21.60
+{1,0.02729,0.00,7.070,0,0.4690,7.1850,61.10,4.9671,2,242.0,17.80,392.83,4.03} | 1 | 34.70
+{1,0.03237,0.00,2.180,0,0.4580,6.9980,45.80,6.0622,3,222.0,18.70,394.63,2.94} | 1 | 33.40
+{1,0.06905,0.00,2.180,0,0.4580,7.1470,54.20,6.0622,3,222.0,18.70,396.90,5.33} | 1 | 36.20
+{1,0.02985,0.00,2.180,0,0.4580,6.4300,58.70,6.0622,3,222.0,18.70,394.12,5.21} | 1 | 28.70
+{1,0.08829,12.50,7.870,0,0.5240,6.0120,66.60,5.5605,5,311.0,15.20,395.60,12.43} | 1 | 22.90
+{1,0.14455,12.50,7.870,0,0.5240,6.1720,96.10,5.9505,5,311.0,15.20,396.90,19.15} | 1 | 27.10
+{1,0.21124,12.50,7.870,0,0.5240,5.6310,100.00,6.0821,5,311.0,15.20,386.63,29.93} | 1 | 16.50
+{1,0.17004,12.50,7.870,0,0.5240,6.0040,85.90,6.5921,5,311.0,15.20,386.71,17.10} | 1 | 18.90
+{1,0.22489,12.50,7.870,0,0.5240,6.3770,94.30,6.3467,5,311.0,15.20,392.52,20.45} | 1 | 15.00
+{1,0.11747,12.50,7.870,0,0.5240,6.0090,82.90,6.2267,5,311.0,15.20,396.90,13.27} | 1 | 18.90
+{1,0.09378,12.50,7.870,0,0.5240,5.8890,39.00,5.4509,5,311.0,15.20,390.50,15.71} | 1 | 21.70
+{1,0.62976,0.00,8.140,0,0.5380,5.9490,61.80,4.7075,4,307.0,21.00,396.90,8.26} | 1 | 20.40
+{1,0.63796,0.00,8.140,0,0.5380,6.0960,84.50,4.4619,4,307.0,21.00,380.02,10.26} | 1 | 18.20
+{1,0.62739,0.00,8.140,0,0.5380,5.8340,56.50,4.4986,4,307.0,21.00,395.62,8.47} | 1 | 19.90
+{1,1.05393,0.00,8.140,0,0.5380,5.9350,29.30,4.4986,4,307.0,21.00,386.85,6.58} | 1 | 23.10
+{1,0.78420,0.00,8.140,0,0.5380,5.9900,81.70,4.2579,4,307.0,21.00,386.75,14.67} | 1 | 17.50
+{1,0.80271,0.00,8.140,0,0.5380,5.4560,36.60,3.7965,4,307.0,21.00,288.99,11.69} | 1 | 20.20
+{1,0.72580,0.00,8.140,0,0.5380,5.7270,69.50,3.7965,4,307.0,21.00,390.95,11.28} | 1 | 18.20
+\.
+</pre>
+-# Now train a regression model using a multilayer perceptron a single hidden layer of two nodes.
+<pre class="example">
+SELECT mlp_regression(
--- End diff --
For consistency sake, should the reference to `mlp_regression` be `madlib.mlp_regression`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by edespino <gi...@git.apache.org>.
Github user edespino commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r127035291
--- Diff: src/ports/postgres/modules/convex/mlp.sql_in ---
@@ -0,0 +1,736 @@
+/* ----------------------------------------------------------------------- *//**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ *
+ * @file mlp.sql_in
+ *
+ * @brief SQL functions for multi-layer perceptron
+ * @date June 2012
+ *
+ *
+ *//* ----------------------------------------------------------------------- */
+
+m4_include(`SQLCommon.m4')
+
+/**
+@addtogroup grp_mlp
+
+<div class="toc"><b>Contents</b><ul>
+<li class="level1"><a href="#mlp_classification">Classification</a></li>
+<li class="level1"><a href="#mlp_regression">Regression</a></li>
+<li class="level1"><a href="#optimization_params">Optimization Parameters</a></li>
+<li class="level1"><a href="#predict">Prediction Functions/a></li>
+<li class="level1"><a href="#example">Examples</a></li>
+<li class="level1"><a href="#background">Technical Background</a></li>
+<li class="level1"><a href="#literature">Literature</a></li>
+<li class="level1"><a href="#related">Related Topics</a></li>
+</ul></div>
+
+Multi Layer Perceptron (MLP) is a model for regression and
+classification
+
+Also called "vanilla neural networks", they consist of several
+fully connected hidden layers with non-linear activation
+functions. In the case of classification, the final layer of the
+neural net has as many nodes as classes, and the output of the
+neural net can be interpreted as the probability of a given input
+feature belonging to a specific class.
+
+
+@brief Solves classification and regression problems with several
+fully connected layers and nonlinear activation functions.
+
+@anchor mlp_classification
+@par Classification Training Function
+The mlp classification training function has the following format:
+<pre class="syntax">
+mlp_classification(
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+\b Arguments
+<DL class="arglist">
+ <DT>source_table</DT>
+ <DD>TEXT. Name of the table containing the training data.</DD>
+
+ <DT>model_table</DT>
+ <DD>TEXT. Name of the output table containing the model. Details of the output
+ tables are provided below.
+ </DD>
+
+ <DT>independent_varname</DT>
+ <DD>TEXT. Expression list to evaluate for the
+ independent variables. An intercept variable should not be included as part
+ of this expression. Please note that expression should be able to be cast
+ to DOUBLE PRECISION[].
+ </DD>
+
+ <DT>dependent_varname</DT>
+ <DD> TEXT. Name of the dependent variable column. For classification, supported types are:
+ text, varchar, character varying, char, character
+ integer, smallint, bigint, and boolean. </DD>
+
+ <DT>hidden_layer_sizes</DT>
+ <DD>INTEGER[], default: ARRAY[].
+ The number of neurons in each hidden layer. The length of this array will
+ determine the number of hidden layers. Empty for no hidden layers.
+ </DD>
+
+
+ <DT>optimizer_params (optional)</DT>
+ <DD>TEXT, default: NULL.
+ Parameters for optimization in a comma-separated string
+ of key-value pairs. See the description below for details.
+ </DD>
+
+ <DT>activation</DT>
+ <DD>TEXT, default: 'sigmoid'.
+ Activation function. Currently three functions are supported: 'sigmoid' (default),
+ 'relu', and 'tanh'. The text can be any subset of the three
+ strings; for e.g., activation='s' will use the sigmoid activation.
+ </DD>
+</DL>
+
+<b>Output tables</b>
+<br>
+ The model table produced by mlp contains the following columns:
+ <table class="output">
+ <tr>
+ <th>coeffs</th>
+ <td>FLOAT8[]. Flat array containing the weights of the neural net</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>INTEGER. Number of iterations completed by stochastic gradient descent
+ algorithm. The algorithm either converged in this number of iterations
+ or hit the maximum number specified in the optimization parameters. </td>
+ </tr>
+ <tr>
+ <th>loss</th>
+ <td>FLOAT8. The cross entropy over the training data.
+ See Technical Background section below for more details.</td>
+ </tr>
+ </table>
+
+
+A summary table named \<model_table\>_summary is also created, which has the following columns:
+ <table class="output">
+ <tr>
+ <th>source_table</th>
+ <td>The source table.</td>
+ </tr>
+ <tr>
+ <th>dependent_varname</th>
+ <td>The dependent variable.</td>
+ </tr>
+ <tr>
+ <th>independent_varname</th>
+ <td>The independent variables.</td>
+ </tr>
+ <tr>
+ <th>tolerance</th>
+ <td>The tolerance as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>step_size</th>
+ <td>The step size as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>The number of iterations run</td>
+ </tr>
+ <tr>
+ <th>n_tries</th>
+ <td>The number of tries as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>layer_sizes</th>
+ <td>The number of units in each layer including the input and output layer.</td>
+ </tr>
+ <tr>
+ <th>activation_function</th>
+ <td>The activation function.</td>
+ </tr>
+ <tr>
+ <th>is_classification</th>
+ <td>True if the model was trained for classification, False if it was trained
+ for regression</td>
+ </tr>
+ <tr>
+ <th>classes</th>
+ <td>The classes which were trained against (empty for regression)</td>
+ </tr>
+
+ </table>
+
+
+@anchor mlp_regression
+@par Regression Training Function
+The mlp regression training function has the following format:
+<pre class="syntax">
+mlp_regression(source_table,
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+
+\b Arguments
+
+Specifications for regression are largely the same as for classification. In the
+model table, the loss will refer to mean square error instead of cross entropy. In the
+summary table, there is classes column. The following
+arguments have specifications which differ from mlp_classification:
+<DL class="arglist">
+<DT>dependent_varname</DT>
+ <DD>TEXT. Name of the dependent variable column.
+ For regression supported types are any numeric type, or array
+ or numeric types (for multiple regression).
+ </DD>
+</DL>
+
+
+@anchor optimizer_params
+@par Optimizer Parameters
+Parameters in this section are supplied in the \e optimizer_params argument as a string
+containing a comma-delimited list of name-value pairs. All of these named
+parameters are optional, and their order does not matter. You must use the
+format "<param_name> = <value>" to specify the value of a parameter, otherwise
+the parameter is ignored.
+
+
+<pre class="syntax">
+ 'step_size = <value>,
+ n_iterations = <value>,
+ n_tries = <value>,
+ tolerance = <value>'
+</pre>
+\b Optimizer Parameters
+<DL class="arglist">
+
+<DT>step_size</dt>
+<DD>Default: [0.001].
+Also known as the learning rate. A small value is usually desirable to
+ensure convergence, while a large value provides more room for progress during
+training. Since the best value depends on the condition number of the data, in
+practice one often tunes this parameter.
+</DD>
+
+
+<DT>n_iterations</dt>
+<DD>Default: [100]. The maximum number of iterations allowed.
+</DD>
+<DT>n_tries</dt>
+<DD>Default: [1]. Number of times to retrain the network with randomly initialized
+weights
+</DD>
+
+<DT>tolerance</dt>
+<DD>Default: 0.001. The criterion to end iterations. The training stops whenever
+<the difference between the training models of two consecutive iterations is
+<smaller than \e tolerance or the iteration number is larger than \e max_iter.
+</DD>
+
+</DL>
+
+@anchor predict
+@par Prediction Function
+Used to generate predictions given a previously trained model on novel data.
+The same syntax is used for classification, and regression.
+<pre class="syntax">
+mlp_predict(model_table,
+ data_table,
+ id_col_name,
+ output_table,
+ pred_type)
+</pre>
+
+\b Arguments
+<DL class="arglist">
+ <DT>model_table</DT>
+ <DD>TEXT. Model table produced by the training function.</DD>
+
+ <DT>data_table</DT>
+ <DD>TEXT. Name of the table containing the data for prediction. This table is expected
+ to contain the same input features that were used during training. The table should
+ also contain id_col_name used for identifying each row.</DD>
+
+ <DT>id_col_name</DT>
+ <DD>TEXT. The name of the id column in the input table.</DD>
+
+ <DT>output_table</DT>
+ <DD>TEXT. Name of the table where output predictions are written. If this
+table name is already in use, then an error is returned. Table contains:</DD>
+ <table class="output">
+ <tr>
+ <th>id</th>
+ <td>Gives the 'id' for each prediction, corresponding to each row from the data_table.</td>
+ </tr>
+ <tr>
+ <th>estimated_<COL_NAME></th>
+ <td>
+ (For pred_type='response') The estimated class
+ for classification or value for regression, where
+ <COL_NAME> is the name of the column to be
+ predicted from training data
+ </td>
+ </tr>
+ <tr>
+ <th>prob_<CLASS></th>
+ <td>
+ (For pred_type='prob' for classification) The
+ probability of a given class <CLASS> as given by
+ softmax. There will be one column for each class
+ in the training data.
+ </td>
+ </tr>
+
+
+ <DT>pred_type</DT>
+ <DD>TEXT.
+
+the type of output requested:
+'response' gives the actual prediction,
+'prob' gives the probability of each class.
+for regression, only type='response' is defined.
+The name of the id column in the input table.</DD>
+</DL>
+</table>
+
+@anchor example
+@par Examples
+-# Create an input data set.
+<pre class="example">
+CREATE TABLE iris_data(
+ id integer,
+ attributes numeric[],
+ class_text varchar,
+ class integer
+);
+INSERT INTO iris_data VALUES
+(1,ARRAY[5.1,3.5,1.4,0.2],'Iris-setosa',1),
+(2,ARRAY[4.9,3.0,1.4,0.2],'Iris-setosa',1),
+(3,ARRAY[4.7,3.2,1.3,0.2],'Iris-setosa',1),
+(4,ARRAY[4.6,3.1,1.5,0.2],'Iris-setosa',1),
+(5,ARRAY[5.0,3.6,1.4,0.2],'Iris-setosa',1),
+(6,ARRAY[5.4,3.9,1.7,0.4],'Iris-setosa',1),
+(7,ARRAY[4.6,3.4,1.4,0.3],'Iris-setosa',1),
+(8,ARRAY[5.0,3.4,1.5,0.2],'Iris-setosa',1),
+(9,ARRAY[4.4,2.9,1.4,0.2],'Iris-setosa',1),
+(10,ARRAY[4.9,3.1,1.5,0.1],'Iris-setosa',1),
+(11,ARRAY[7.0,3.2,4.7,1.4],'Iris-versicolor',2),
+(12,ARRAY[6.4,3.2,4.5,1.5],'Iris-versicolor',2),
+(13,ARRAY[6.9,3.1,4.9,1.5],'Iris-versicolor',2),
+(14,ARRAY[5.5,2.3,4.0,1.3],'Iris-versicolor',2),
+(15,ARRAY[6.5,2.8,4.6,1.5],'Iris-versicolor',2),
+(16,ARRAY[5.7,2.8,4.5,1.3],'Iris-versicolor',2),
+(17,ARRAY[6.3,3.3,4.7,1.6],'Iris-versicolor',2),
+(18,ARRAY[4.9,2.4,3.3,1.0],'Iris-versicolor',2),
+(19,ARRAY[6.6,2.9,4.6,1.3],'Iris-versicolor',2),
+(20,ARRAY[5.2,2.7,3.9,1.4],'Iris-versicolor',2);
+</pre>
+-# Generate a multilayer perceptron with a single hidden layer of 5 units.
+Use the attributes column as the independent variables, and use the class
+column as the classification. Set the tolerance to 0 so that 5000
+iterations will be run. Use a hyperbolic tangent activation function.
+The model will be written to mlp_result.
+<pre class="example">
+SELECT madlib.mlp_classification(
+ 'iris_data_2', -- Source table
+ 'mlp_model', -- Destination table
+ 'attributes', -- Input features
+ 'class_text', -- Label
+ ARRAY[5], -- Number of units per layer
+ 'step_size=0.003,
+ n_iterations=5000,
+ tolerance=0', -- Optimizer params
+ 'tanh'); -- Activation function
+</pre>
+-# View the result for the model.
+<pre class="example">
+-- Set extended display on for easier reading of output
+\\x ON
+-- Neural net Initialization is non-deterministic, so your results may vary
+SELECT * FROM mlp_model;
+</pre>
+Result:
+<pre class="result">
+-[ RECORD 1 ]--+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+coeff | {1,1,1,1,1,0.136374930803,0.188739676875,0.662387810001,-1.03381622734,-0.469961067046,0.0614006983397,0.0811504589436,0.299008228258,-0.47391918521,-0.215098143699,0.10519213944,0.145844617525,0.511683525606,-0.800215552382,-0.36417142683,0.120751709056,0.167531106521,0.587074895969,-0.916946198095,-0.417055067449,0.0539541885146,0.0694359704131,0.262598585854,-0.419234805076,-0.189915344282,1,1,1,1,1,1,0.105645702152,1.46247470474,0.484457903226,0.965962824478,1.19361986431,0.419805760087,-0.105696503487,-1.46245956666,-0.484427811691,-0.965730981426,-1.19365280555,-0.419973628863}
+loss | 0.0184092375519
+num_iterations | 5000
+</pre>
+-# Next train a regression example. First create some test data. This dataset
+contains housing prices data.
+<pre class="example">
+CREATE TABLE lin_housing (id serial, x float8[], grp_by_col int, y float8);
+COPY lin_housing (x, grp_by_col, y) FROM STDIN NULL '?' DELIMITER '|';
+{1,0.00632,18.00,2.310,0,0.5380,6.5750,65.20,4.0900,1,296.0,15.30,396.90,4.98} | 1 | 24.00
+{1,0.02731,0.00,7.070,0,0.4690,6.4210,78.90,4.9671,2,242.0,17.80,396.90,9.14} | 1 | 21.60
+{1,0.02729,0.00,7.070,0,0.4690,7.1850,61.10,4.9671,2,242.0,17.80,392.83,4.03} | 1 | 34.70
+{1,0.03237,0.00,2.180,0,0.4580,6.9980,45.80,6.0622,3,222.0,18.70,394.63,2.94} | 1 | 33.40
+{1,0.06905,0.00,2.180,0,0.4580,7.1470,54.20,6.0622,3,222.0,18.70,396.90,5.33} | 1 | 36.20
+{1,0.02985,0.00,2.180,0,0.4580,6.4300,58.70,6.0622,3,222.0,18.70,394.12,5.21} | 1 | 28.70
+{1,0.08829,12.50,7.870,0,0.5240,6.0120,66.60,5.5605,5,311.0,15.20,395.60,12.43} | 1 | 22.90
+{1,0.14455,12.50,7.870,0,0.5240,6.1720,96.10,5.9505,5,311.0,15.20,396.90,19.15} | 1 | 27.10
+{1,0.21124,12.50,7.870,0,0.5240,5.6310,100.00,6.0821,5,311.0,15.20,386.63,29.93} | 1 | 16.50
+{1,0.17004,12.50,7.870,0,0.5240,6.0040,85.90,6.5921,5,311.0,15.20,386.71,17.10} | 1 | 18.90
+{1,0.22489,12.50,7.870,0,0.5240,6.3770,94.30,6.3467,5,311.0,15.20,392.52,20.45} | 1 | 15.00
+{1,0.11747,12.50,7.870,0,0.5240,6.0090,82.90,6.2267,5,311.0,15.20,396.90,13.27} | 1 | 18.90
+{1,0.09378,12.50,7.870,0,0.5240,5.8890,39.00,5.4509,5,311.0,15.20,390.50,15.71} | 1 | 21.70
+{1,0.62976,0.00,8.140,0,0.5380,5.9490,61.80,4.7075,4,307.0,21.00,396.90,8.26} | 1 | 20.40
+{1,0.63796,0.00,8.140,0,0.5380,6.0960,84.50,4.4619,4,307.0,21.00,380.02,10.26} | 1 | 18.20
+{1,0.62739,0.00,8.140,0,0.5380,5.8340,56.50,4.4986,4,307.0,21.00,395.62,8.47} | 1 | 19.90
+{1,1.05393,0.00,8.140,0,0.5380,5.9350,29.30,4.4986,4,307.0,21.00,386.85,6.58} | 1 | 23.10
+{1,0.78420,0.00,8.140,0,0.5380,5.9900,81.70,4.2579,4,307.0,21.00,386.75,14.67} | 1 | 17.50
+{1,0.80271,0.00,8.140,0,0.5380,5.4560,36.60,3.7965,4,307.0,21.00,288.99,11.69} | 1 | 20.20
+{1,0.72580,0.00,8.140,0,0.5380,5.7270,69.50,3.7965,4,307.0,21.00,390.95,11.28} | 1 | 18.20
+\.
+</pre>
+-# Now train a regression model using a multilayer perceptron a single hidden layer of two nodes.
+<pre class="example">
+SELECT mlp_regression(
+ 'lin_housing', -- Source table
+ 'mlp_regress', -- Desination table
+ 'x', -- Input features
+ 'y', -- Dependent variable
+ ARRAY[5,5], -- Number of units per layer
+ 'step_size=0.000007,
+ n_iterations=5000,
+ tolerance=0',
+ 'relu');
+</pre>
+-# Check the results of the model
+<pre class="example">
+-- Set extended display on for easier reading of output
+\\x ON
+-- Neural net Initialization is non-deterministic, so your results may vary
+SELECT * FROM mlp_regress;
+</pre>
+Result:
+<pre class="result">
+-[ RECORD 1 ]--+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------
+coeff | {1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2.79506311399e-05,3.56715008915e-05,-6.09333559685e-05,0.000251228318768,-0.000224772841379,-3.71863030857e-05,-3.5757865148e-06,5.27936784854e-05,-2.48474166186e-05,6.19731184294e-05,3.07638968743e-05,6.8964698578e-06,0.000106016701083,-1.71484730318e-05,1.18691881812e-05,-0.000163975464208,0.000170026304906,3.11688265279e-05,0.000177050148787,-1.58265976603e-05,2.70144422657e-05,0.000112667883422,3.77575139073e-05,8.12474658795e-05,-7.90458917626e-05,0.000107566386158,-2.63771171506e-06,2.47996880915e-05,-0.00012642310887,0.000203827391081,0.000139315565565,4.86147243454e-05,-0.000176126471913,-6.47820782916e-05,-8.51592776447e-06,-6.60601176758e-05,2.91421874156e-05,6.3556873752e-05,0.000197557443129,0.000220531367259,0.000135036310289,0.000143735913975,-4.75034117786e-05,-0.000179547345838,-1.6919846786e-05,0.000162784312994,0.000268595819851,-0.000460066553287,8.69756071591e-05,-0.00311762727057,0.000126024763103,0.00020598824
2921,0.003463432426,-0.00729789075286,0.00151625867549,-0.000890852767597,-0.00525016037249,0.0031043106659,0.00798041103839,-0.00552693050079,0.0232180415786,0.0230489850143,-0.0437890272341,0.0165765426407,-0.248554261758,-7.81336427846e-05,0.00558145591752,0.283465844585,-0.571699956182,0.133474351994,-0.0785181945605,-0.419269930709,0.249547772912,0.631761009875,-0.431305975666,1,1,1,1,1,1,0.0158747497572,-9.02809160806e-05,0.00015574347618,4.10805373863e-06,0.00121532434965,0.101790351335,0.0647558401493,-0.00013654998677,-9.92872075948e-06,-5.5319694394e-05,0.00519320756484,0.412736586036,0.0011998026977,-1.53688189815e-05,1.94817888201e-05,-4.63111489966e-05,7.24547899029e-05,0.00880394144485,5.45309822095e-05,-0.000140943219275,-7.96211486227e-05,-1.04337307472e-05,0.000161936762028,0.00136273797767,-4.54737243585e-05,-3.4083840736e-05,3.69286883662e-05,9.9047243188e-08,3.75014011824e-06,-9.45366086368e-08,1,1,1,1,1,1,6.67488547054,0.102754199001,0.41668912471,0.008868672964
79,0.00136206007228,-9.88642499013e-05}
+loss | 144.965776158
+num_iterations | 10000
+</pre>
+-# Now let's look at the prediction functions. In the following examples we will
+use the training data set for prediction as well, which is not usual but serves to
+show the syntax. First we will test the classification exampel.
+The prediction is in the the estimated_class_text column with the
+actual value in the class_text column.
+<pre class="example">
+SELECT mlp_predict(
+ 'mlp_model', -- Model table
+ 'iris_data', -- Test data table
+ 'id', -- Id column in test table
+ 'mlp_prediction', -- Output table for predictions
+ 'response' -- Output classes, not probabilities
+ );
+SELECT * FROM mlp_prediction JOIN iris_data USING (id);
+</pre>
+Result:
+<pre class="result">
+ id | estimated_class_text | attributes | class_text | class
+----+----------------------+-------------------+-----------------+-------
+ 1 | Iris-setosa | {5.1,3.5,1.4,0.2} | Iris-setosa | 1
+ 2 | Iris-setosa | {4.9,3.0,1.4,0.2} | Iris-setosa | 1
+ 3 | Iris-setosa | {4.7,3.2,1.3,0.2} | Iris-setosa | 1
+ 4 | Iris-setosa | {4.6,3.1,1.5,0.2} | Iris-setosa | 1
+ 5 | Iris-setosa | {5.0,3.6,1.4,0.2} | Iris-setosa | 1
+ 6 | Iris-setosa | {5.4,3.9,1.7,0.4} | Iris-setosa | 1
+ 7 | Iris-setosa | {4.6,3.4,1.4,0.3} | Iris-setosa | 1
+ 8 | Iris-setosa | {5.0,3.4,1.5,0.2} | Iris-setosa | 1
+ 9 | Iris-setosa | {4.4,2.9,1.4,0.2} | Iris-setosa | 1
+ 10 | Iris-setosa | {4.9,3.1,1.5,0.1} | Iris-setosa | 1
+ 11 | Iris-versicolor | {7.0,3.2,4.7,1.4} | Iris-versicolor | 2
+ 12 | Iris-versicolor | {6.4,3.2,4.5,1.5} | Iris-versicolor | 2
+ 13 | Iris-versicolor | {6.9,3.1,4.9,1.5} | Iris-versicolor | 2
+ 14 | Iris-versicolor | {5.5,2.3,4.0,1.3} | Iris-versicolor | 2
+ 15 | Iris-versicolor | {6.5,2.8,4.6,1.5} | Iris-versicolor | 2
+ 16 | Iris-versicolor | {5.7,2.8,4.5,1.3} | Iris-versicolor | 2
+ 17 | Iris-versicolor | {6.3,3.3,4.7,1.6} | Iris-versicolor | 2
+ 18 | Iris-versicolor | {4.9,2.4,3.3,1.0} | Iris-versicolor | 2
+ 19 | Iris-versicolor | {6.6,2.9,4.6,1.3} | Iris-versicolor | 2
+ 20 | Iris-versicolor | {5.2,2.7,3.9,1.4} | Iris-versicolor | 2
+</pre>
+Prediction using the regression model:
+<pre class="example">
+SELECT mlp_predict(
--- End diff --
For consistency sake, should the reference to `mlp_predict` be `madlib.mlp_predict`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by njayaram2 <gi...@git.apache.org>.
Github user njayaram2 commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r128100566
--- Diff: src/ports/postgres/modules/convex/mlp_igd.py_in ---
@@ -0,0 +1,734 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+@file mlp_igd.py_in
+
+@brief Multilayer perceptron using IGD: Driver functions
+
+@namespace mlp_igd
+"""
+import plpy
+
+from utilities.control import MinWarning
+from utilities.utilities import add_postfix
+from utilities.utilities import py_list_to_sql_string
+from utilities.utilities import extract_keyvalue_params
+from utilities.utilities import _assert
+from utilities.utilities import unique_string
+from utilities.utilities import strip_end_quotes
+
+from utilities.validate_args import cols_in_tbl_valid
+from utilities.validate_args import input_tbl_valid
+from utilities.validate_args import is_var_valid
+from utilities.validate_args import output_tbl_valid
+from utilities.validate_args import get_expr_type
+from utilities.validate_args import array_col_has_same_dimension
+from utilities.validate_args import array_col_dimension
+
+
+def mlp(schema_madlib, source_table, output_table, independent_varname,
+ dependent_varname, hidden_layer_sizes,
+ optimizer_param_str, activation, is_classification, **kwargs):
+ """
+ Args:
+ @param schema_madlib
+ @param source_table
+ @param output_table
+ @param independent_varname
+ @param dependent_varname
+ @param hidden_layer_sizes
+ @param optimizer_param_str
+
+ Returns:
+ None
+ """
+ with MinWarning('info'):
+ optimizer_params = _get_optimizer_params(optimizer_param_str or "")
+ _validate_args(source_table, output_table, independent_varname,
+ dependent_varname, hidden_layer_sizes,
+ optimizer_params, is_classification)
+
+ current_iteration = 1
+ prev_state = None
+ tolerance = optimizer_params["tolerance"]
+ n_iterations = optimizer_params["n_iterations"]
+ step_size = optimizer_params["step_size"]
+ n_tries = optimizer_params["n_tries"]
+ activation_name = _get_activation_function_name(activation)
+ activation_index = _get_activation_index(activation_name)
+ num_input_nodes = array_col_dimension(source_table, independent_varname)
+ num_output_nodes = 0
+ classes = []
+ dependent_type = get_expr_type(dependent_varname, source_table)
+ original_dependent_varname = dependent_varname
+
+ if is_classification:
+ dependent_variable_sql = """
+ SELECT DISTINCT {dependent_varname}
+ FROM {source_table}
+ """.format(dependent_varname=dependent_varname,
+ source_table=source_table)
+ labels = plpy.execute(dependent_variable_sql)
+ one_hot_dependent_varname = 'ARRAY['
+ num_output_nodes = len(labels)
+ for label_obj in labels:
+ label = _format_label(label_obj[dependent_varname])
+ classes.append(label)
+ one_hot_dependent_varname += dependent_varname + "=" + str(label) + ","
+ # Remove the last comma
+ one_hot_dependent_varname = one_hot_dependent_varname[:-1]
+ one_hot_dependent_varname += ']::integer[]'
+ dependent_varname = one_hot_dependent_varname
+ if not is_classification:
+ if "[]" not in dependent_type:
+ dependent_varname = "ARRAY[" + dependent_varname + "]"
+ num_output_nodes = array_col_dimension(source_table, dependent_varname)
+ layer_sizes = [num_input_nodes] + hidden_layer_sizes + [num_output_nodes]
+
+ while True:
+ if prev_state:
+ prev_state_str = py_list_to_sql_string(prev_state, array_type="double precision")
+ else:
+ prev_state_str = "(NULL)::DOUBLE PRECISION[]"
+ train_sql = """
+ SELECT
+ {schema_madlib}.mlp_igd_step(
+ ({independent_varname})::DOUBLE PRECISION[],
+ ({dependent_varname})::DOUBLE PRECISION[],
+ {prev_state},
+ {layer_sizes},
+ ({step_size})::FLOAT8,
+ {activation},
+ {is_classification}) as curr_state
+ FROM {source_table} AS _src
+ """.format(schema_madlib=schema_madlib,
+ independent_varname=independent_varname,
+ dependent_varname=dependent_varname,
+ prev_state=prev_state_str,
+ # C++ uses double internally
+ layer_sizes=py_list_to_sql_string(layer_sizes,
+ array_type="double precision"),
+ step_size=step_size,
+ source_table=source_table,
+ activation=activation_index,
+ is_classification=int(is_classification))
+ curr_state = plpy.execute(train_sql)[0]["curr_state"]
+ dist_sql = """
+ SELECT {schema_madlib}.internal_mlp_igd_distance(
+ {prev_state},
+ {curr_state}) as state_dist
+ """.format(schema_madlib=schema_madlib,
+ prev_state=prev_state_str,
+ curr_state=py_list_to_sql_string(curr_state, "double precision"))
+ state_dist = plpy.execute(dist_sql)[0]["state_dist"]
+ if ((state_dist and state_dist < tolerance) or
+ current_iteration > n_iterations):
+ break
+ prev_state = curr_state
+ current_iteration += 1
+ _build_model_table(schema_madlib, output_table, curr_state, n_iterations)
+ layer_sizes_str = py_list_to_sql_string(layer_sizes, array_type="integer")
+ classes_str = py_list_to_sql_string([strip_end_quotes(cl, "'") for cl in classes],
+ array_type=dependent_type)
+ summary_table_creation_query = """
+ CREATE TABLE {output_table}_summary(
+ source_table TEXT,
+ independent_varname TEXT,
+ dependent_varname TEXT,
+ tolerance FLOAT,
+ step_size FLOAT,
+ n_iterations INTEGER,
+ n_tries INTEGER,
+ layer_sizes INTEGER[],
+ activation_function TEXT,
+ is_classification BOOLEAN,
+ classes {dependent_type}[]
+ )""".format(output_table=output_table,
+ dependent_type=dependent_type)
+
+ summary_table_update_query = """
+ INSERT INTO {output_table}_summary VALUES(
+ '{source_table}',
+ '{independent_varname}',
+ '{original_dependent_varname}',
+ {tolerance},
+ {step_size},
+ {n_iterations},
+ {n_tries},
+ {layer_sizes_str},
+ '{activation_name}',
+ {is_classification},
+ {classes_str}
+ )
+ """.format(**locals())
+ plpy.execute(summary_table_creation_query)
+ plpy.execute(summary_table_update_query)
+# ----------------------------------------------------------------------
+
+
+def _build_model_table(schema_madlib, output_table, final_state, n_iterations):
+ final_state_str = py_list_to_sql_string(final_state, array_type="double precision")
+
+ model_table_query = """
+ CREATE TABLE {output_table} AS
+ SELECT
+ (result).coeff AS coeff,
+ (result).loss AS loss,
+ {n_iterations} AS num_iterations
+ -- (result).num_rows_processed AS num_rows_processed,
+ -- n_tuples_including_nulls - (result).num_rows_processed
+ FROM (
+ SELECT
+ {schema_madlib}.internal_mlp_igd_result(
+ {final_state_str}
+ ) AS result
+ ) rel_state_subq
+ """.format(**locals())
+ plpy.execute(model_table_query)
+# ----------------------------------------------------------------------
+
+
+def _get_optimizer_params(param_str):
+ params_defaults = {
+ "step_size": (0.001, float),
+ "n_iterations": (100, int),
+ "n_tries": (1, int),
+ "tolerance": (0.001, float),
+ }
+ param_defaults = dict([(k, v[0]) for k, v in params_defaults.items()])
+ param_types = dict([(k, v[1]) for k, v in params_defaults.items()])
+
+ if not param_str:
+ return param_defaults
+
+ name_value = extract_keyvalue_params(param_str, param_types, param_defaults,
+ ignore_invalid=False)
+ return name_value
+# ----------------------------------------------------------------------
+
+
+def _validate_args_classification(source_table, dependent_varname):
+ expr_type = get_expr_type(dependent_varname, source_table)
+ int_types = ['integer', 'smallint', 'bigint']
+ text_types = ['text', 'varchar', 'character varying', 'char', 'character']
+ boolean_types = ['boolean']
+ _assert("[]" in expr_type or expr_type in int_types + text_types + boolean_types,
+ "Dependent variable column should refer to an "
+ "integer, boolean, text, varchar, or character type.")
+# ----------------------------------------------------------------------
+
+
+def _validate_args_regression(source_table, dependent_varname):
+ expr_type = get_expr_type(dependent_varname, source_table)
+ int_types = ['integer', 'smallint', 'bigint']
+ float_types = ['double precision', 'real']
+ _assert("[]" in expr_type or expr_type in int_types + float_types,
+ "Dependent variable column should refer to an array or numeric type")
+ if "[]" in expr_type:
+ _assert(array_col_has_same_dimension(source_table, dependent_varname),
+ "Dependent variable column should refer to arrays of the same length")
+# ----------------------------------------------------------------------
+
+
+def _validate_args(source_table, output_table, independent_varname,
+ dependent_varname, hidden_layer_sizes,
+ optimizer_params, is_classification):
+ input_tbl_valid(source_table, "MLP")
+ output_tbl_valid(output_table, "MLP")
+ output_tbl_valid(output_table+"_summary", "MLP")
--- End diff --
Fetch the summary table name using `add_postfix()` function in `utilities.py_in`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by edespino <gi...@git.apache.org>.
Github user edespino commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r127317330
--- Diff: src/ports/postgres/modules/convex/mlp.sql_in ---
@@ -0,0 +1,744 @@
+/* ----------------------------------------------------------------------- *//**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ *
+ * @file mlp.sql_in
+ *
+ * @brief SQL functions for multilayer perceptron
+ * @date June 2012
+ *
+ *
+ *//* ----------------------------------------------------------------------- */
+
+m4_include(`SQLCommon.m4')
+
+/**
+@addtogroup grp_mlp
+
+<div class="toc"><b>Contents</b><ul>
+<li class="level1"><a href="#mlp_classification">Classification</a></li>
+<li class="level1"><a href="#mlp_regression">Regression</a></li>
+<li class="level1"><a href="#optimization_params">Optimizer Parameters</a></li>
+<li class="level1"><a href="#predict">Prediction Functions/a></li>
+<li class="level1"><a href="#example">Examples</a></li>
+<li class="level1"><a href="#background">Technical Background</a></li>
+<li class="level1"><a href="#literature">Literature</a></li>
+<li class="level1"><a href="#related">Related Topics</a></li>
+</ul></div>
+
+Multilayer Perceptron (MLP) is a model for regression and
+classification
+
+Also called "vanilla neural networks", they consist of several
+fully connected hidden layers with non-linear activation
+functions. In the case of classification, the final layer of the
+neural net has as many nodes as classes, and the output of the
+neural net can be interpreted as the probability of a given input
+feature belonging to a specific class.
+
+
+@brief Solves classification and regression problems with several
+fully connected layers and nonlinear activation functions.
+
+@anchor mlp_classification
+@par Classification Training Function
+The mlp classification training function has the following format:
+<pre class="syntax">
+mlp_classification(
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+\b Arguments
+<DL class="arglist">
+ <DT>source_table</DT>
+ <DD>TEXT. Name of the table containing the training data.</DD>
+
+ <DT>model_table</DT>
+ <DD>TEXT. Name of the output table containing the model. Details of the output
+ tables are provided below.
+ </DD>
+
+ <DT>independent_varname</DT>
+ <DD>TEXT. Expression list to evaluate for the
+ independent variables. An intercept variable should not be included as part
+ of this expression. Please note that expression should be able to be cast
+ to DOUBLE PRECISION[].
+ </DD>
+
+ <DT>dependent_varname</DT>
+ <DD> TEXT. Name of the dependent variable column. For classification, supported types are:
+ text, varchar, character varying, char, character
+ integer, smallint, bigint, and boolean. </DD>
+
+ <DT>hidden_layer_sizes</DT>
+ <DD>INTEGER[], default: ARRAY[].
+ The number of neurons in each hidden layer. The length of this array will
+ determine the number of hidden layers. Empty for no hidden layers.
+ </DD>
+
+
+ <DT>optimizer_params (optional)</DT>
+ <DD>TEXT, default: NULL.
+ Parameters for optimization in a comma-separated string
+ of key-value pairs. See the description below for details.
+ </DD>
+
+ <DT>activation</DT>
+ <DD>TEXT, default: 'sigmoid'.
+ Activation function. Currently three functions are supported: 'sigmoid' (default),
+ 'relu', and 'tanh'. The text can be any subset of the three
+ strings; for e.g., activation='s' will use the sigmoid activation.
+ </DD>
+</DL>
+
+<b>Output tables</b>
+<br>
+ The model table produced by mlp contains the following columns:
+ <table class="output">
+ <tr>
+ <th>coeffs</th>
+ <td>FLOAT8[]. Flat array containing the weights of the neural net</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>INTEGER. Number of iterations completed by stochastic gradient descent
+ algorithm. The algorithm either converged in this number of iterations
+ or hit the maximum number specified in the optimization parameters. </td>
+ </tr>
+ <tr>
+ <th>loss</th>
+ <td>FLOAT8. The cross entropy over the training data.
+ See Technical Background section below for more details.</td>
+ </tr>
+ </table>
+
+
+A summary table named \<model_table\>_summary is also created, which has the following columns:
+ <table class="output">
+ <tr>
+ <th>source_table</th>
+ <td>The source table.</td>
+ </tr>
+ <tr>
+ <th>dependent_varname</th>
+ <td>The dependent variable.</td>
+ </tr>
+ <tr>
+ <th>independent_varname</th>
+ <td>The independent variables.</td>
+ </tr>
+ <tr>
+ <th>tolerance</th>
+ <td>The tolerance as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>step_size</th>
+ <td>The step size as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>The number of iterations run</td>
+ </tr>
+ <tr>
+ <th>n_tries</th>
+ <td>The number of tries as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>layer_sizes</th>
+ <td>The number of units in each layer including the input and output layer.</td>
+ </tr>
+ <tr>
+ <th>activation_function</th>
+ <td>The activation function.</td>
+ </tr>
+ <tr>
+ <th>is_classification</th>
+ <td>True if the model was trained for classification, False if it was trained
+ for regression</td>
+ </tr>
+ <tr>
+ <th>classes</th>
+ <td>The classes which were trained against (empty for regression)</td>
+ </tr>
+
+ </table>
+
+
+@anchor mlp_regression
+@par Regression Training Function
+The mlp regression training function has the following format:
+<pre class="syntax">
+mlp_regression(source_table,
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+
+\b Arguments
+
+Specifications for regression are largely the same as for classification. In the
+model table, the loss will refer to mean square error instead of cross entropy. In the
+summary table, there is classes column. The following
+arguments have specifications which differ from mlp_classification:
+<DL class="arglist">
+<DT>dependent_varname</DT>
+ <DD>TEXT. Name of the dependent variable column.
+ For regression supported types are any numeric type, or array
+ or numeric types (for multiple regression).
+ </DD>
+</DL>
+
+
+@anchor optimizer_params
+@par Optimizer Parameters
+Parameters in this section are supplied in the \e optimizer_params argument as a string
+containing a comma-delimited list of name-value pairs. All of these named
+parameters are optional, and their order does not matter. You must use the
+format "<param_name> = <value>" to specify the value of a parameter, otherwise
+the parameter is ignored.
+
+
+<pre class="syntax">
+ 'step_size = <value>,
+ n_iterations = <value>,
+ n_tries = <value>,
+ tolerance = <value>'
+</pre>
+\b Optimizer Parameters
+<DL class="arglist">
+
+<DT>step_size</dt>
+<DD>Default: [0.001].
+Also known as the learning rate. A small value is usually desirable to
+ensure convergence, while a large value provides more room for progress during
+training. Since the best value depends on the condition number of the data, in
+practice one often tunes this parameter.
+</DD>
+
+
+<DT>n_iterations</dt>
+<DD>Default: [100]. The maximum number of iterations allowed.
+</DD>
+<DT>n_tries</dt>
+<DD>Default: [1]. Number of times to retrain the network with randomly initialized
+weights
+</DD>
+
+<DT>tolerance</dt>
+<DD>Default: 0.001. The criterion to end iterations. The training stops whenever
+<the difference between the training models of two consecutive iterations is
+<smaller than \e tolerance or the iteration number is larger than \e max_iter.
+</DD>
+
+</DL>
+
+@anchor predict
+@par Prediction Function
+Used to generate predictions given a previously trained model on novel data.
+The same syntax is used for classification, and regression.
+<pre class="syntax">
+mlp_predict(model_table,
+ data_table,
+ id_col_name,
+ output_table,
+ pred_type)
+</pre>
+
+\b Arguments
+<DL class="arglist">
+ <DT>model_table</DT>
+ <DD>TEXT. Model table produced by the training function.</DD>
+
+ <DT>data_table</DT>
+ <DD>TEXT. Name of the table containing the data for prediction. This table is expected
+ to contain the same input features that were used during training. The table should
+ also contain id_col_name used for identifying each row.</DD>
+
+ <DT>id_col_name</DT>
+ <DD>TEXT. The name of the id column in the input table.</DD>
+
+ <DT>output_table</DT>
+ <DD>TEXT. Name of the table where output predictions are written. If this
+table name is already in use, then an error is returned. Table contains:</DD>
+ <table class="output">
+ <tr>
+ <th>id</th>
+ <td>Gives the 'id' for each prediction, corresponding to each row from the data_table.</td>
+ </tr>
+ <tr>
+ <th>estimated_<COL_NAME></th>
+ <td>
+ (For pred_type='response') The estimated class
+ for classification or value for regression, where
+ <COL_NAME> is the name of the column to be
+ predicted from training data
+ </td>
+ </tr>
+ <tr>
+ <th>prob_<CLASS></th>
+ <td>
+ (For pred_type='prob' for classification) The
+ probability of a given class <CLASS> as given by
+ softmax. There will be one column for each class
+ in the training data.
+ </td>
+ </tr>
+
+
+ <DT>pred_type</DT>
+ <DD>TEXT.
+
+the type of output requested:
+'response' gives the actual prediction,
+'prob' gives the probability of each class.
+for regression, only type='response' is defined.
+The name of the id column in the input table.</DD>
+</DL>
+</table>
+
+@anchor example
+@par Examples
+-# Create an input data set.
+<pre class="example">
+CREATE TABLE iris_data(
+ id integer,
+ attributes numeric[],
+ class_text varchar,
+ class integer
+);
+INSERT INTO iris_data VALUES
+(1,ARRAY[5.1,3.5,1.4,0.2],'Iris-setosa',1),
+(2,ARRAY[4.9,3.0,1.4,0.2],'Iris-setosa',1),
+(3,ARRAY[4.7,3.2,1.3,0.2],'Iris-setosa',1),
+(4,ARRAY[4.6,3.1,1.5,0.2],'Iris-setosa',1),
+(5,ARRAY[5.0,3.6,1.4,0.2],'Iris-setosa',1),
+(6,ARRAY[5.4,3.9,1.7,0.4],'Iris-setosa',1),
+(7,ARRAY[4.6,3.4,1.4,0.3],'Iris-setosa',1),
+(8,ARRAY[5.0,3.4,1.5,0.2],'Iris-setosa',1),
+(9,ARRAY[4.4,2.9,1.4,0.2],'Iris-setosa',1),
+(10,ARRAY[4.9,3.1,1.5,0.1],'Iris-setosa',1),
+(11,ARRAY[7.0,3.2,4.7,1.4],'Iris-versicolor',2),
+(12,ARRAY[6.4,3.2,4.5,1.5],'Iris-versicolor',2),
+(13,ARRAY[6.9,3.1,4.9,1.5],'Iris-versicolor',2),
+(14,ARRAY[5.5,2.3,4.0,1.3],'Iris-versicolor',2),
+(15,ARRAY[6.5,2.8,4.6,1.5],'Iris-versicolor',2),
+(16,ARRAY[5.7,2.8,4.5,1.3],'Iris-versicolor',2),
+(17,ARRAY[6.3,3.3,4.7,1.6],'Iris-versicolor',2),
+(18,ARRAY[4.9,2.4,3.3,1.0],'Iris-versicolor',2),
+(19,ARRAY[6.6,2.9,4.6,1.3],'Iris-versicolor',2),
+(20,ARRAY[5.2,2.7,3.9,1.4],'Iris-versicolor',2);
+</pre>
+-# Generate a multilayer perceptron with a single hidden layer of 5 units.
+Use the attributes column as the independent variables, and use the class
+column as the classification. Set the tolerance to 0 so that 5000
+iterations will be run. Use a hyperbolic tangent activation function.
+The model will be written to mlp_model.
+<pre class="example">
+SELECT madlib.mlp_classification(
+ 'iris_data', -- Source table
+ 'mlp_model', -- Destination table
+ 'attributes', -- Input features
+ 'class_text', -- Label
+ ARRAY[5], -- Number of units per layer
+ 'step_size=0.003,
+ n_iterations=5000,
+ tolerance=0', -- Optimizer params
+ 'tanh'); -- Activation function
+</pre>
+-# View the result for the model.
+<pre class="example">
+-- Set extended display on for easier reading of output
+\\x ON
+-- Neural net Initialization is non-deterministic, so your results may vary
+SELECT * FROM mlp_model;
+</pre>
+Result:
+<pre class="result">
+-[ RECORD 1 ]--+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+coeff | {1,1,1,1,1,0.136374930803,0.188739676875,0.662387810001,-1.03381622734,-0.469961067046,0.0614006983397,0.0811504589436,0.299008228258,-0.47391918521,-0.215098143699,0.10519213944,0.145844617525,0.511683525606,-0.800215552382,-0.36417142683,0.120751709056,0.167531106521,0.587074895969,-0.916946198095,-0.417055067449,0.0539541885146,0.0694359704131,0.262598585854,-0.419234805076,-0.189915344282,1,1,1,1,1,1,0.105645702152,1.46247470474,0.484457903226,0.965962824478,1.19361986431,0.419805760087,-0.105696503487,-1.46245956666,-0.484427811691,-0.965730981426,-1.19365280555,-0.419973628863}
+loss | 0.0184092375519
+num_iterations | 5000
+</pre>
+-# Next train a regression example. First create some test data. This dataset
+contains housing prices data.
+<pre class="example">
+CREATE TABLE lin_housing (id serial, x float8[], grp_by_col int, y float8);
+COPY lin_housing (x, grp_by_col, y) FROM STDIN NULL '?' DELIMITER '|';
+{1,0.00632,18.00,2.310,0,0.5380,6.5750,65.20,4.0900,1,296.0,15.30,396.90,4.98} | 1 | 24.00
+{1,0.02731,0.00,7.070,0,0.4690,6.4210,78.90,4.9671,2,242.0,17.80,396.90,9.14} | 1 | 21.60
+{1,0.02729,0.00,7.070,0,0.4690,7.1850,61.10,4.9671,2,242.0,17.80,392.83,4.03} | 1 | 34.70
+{1,0.03237,0.00,2.180,0,0.4580,6.9980,45.80,6.0622,3,222.0,18.70,394.63,2.94} | 1 | 33.40
+{1,0.06905,0.00,2.180,0,0.4580,7.1470,54.20,6.0622,3,222.0,18.70,396.90,5.33} | 1 | 36.20
+{1,0.02985,0.00,2.180,0,0.4580,6.4300,58.70,6.0622,3,222.0,18.70,394.12,5.21} | 1 | 28.70
+{1,0.08829,12.50,7.870,0,0.5240,6.0120,66.60,5.5605,5,311.0,15.20,395.60,12.43} | 1 | 22.90
+{1,0.14455,12.50,7.870,0,0.5240,6.1720,96.10,5.9505,5,311.0,15.20,396.90,19.15} | 1 | 27.10
+{1,0.21124,12.50,7.870,0,0.5240,5.6310,100.00,6.0821,5,311.0,15.20,386.63,29.93} | 1 | 16.50
+{1,0.17004,12.50,7.870,0,0.5240,6.0040,85.90,6.5921,5,311.0,15.20,386.71,17.10} | 1 | 18.90
+{1,0.22489,12.50,7.870,0,0.5240,6.3770,94.30,6.3467,5,311.0,15.20,392.52,20.45} | 1 | 15.00
+{1,0.11747,12.50,7.870,0,0.5240,6.0090,82.90,6.2267,5,311.0,15.20,396.90,13.27} | 1 | 18.90
+{1,0.09378,12.50,7.870,0,0.5240,5.8890,39.00,5.4509,5,311.0,15.20,390.50,15.71} | 1 | 21.70
+{1,0.62976,0.00,8.140,0,0.5380,5.9490,61.80,4.7075,4,307.0,21.00,396.90,8.26} | 1 | 20.40
+{1,0.63796,0.00,8.140,0,0.5380,6.0960,84.50,4.4619,4,307.0,21.00,380.02,10.26} | 1 | 18.20
+{1,0.62739,0.00,8.140,0,0.5380,5.8340,56.50,4.4986,4,307.0,21.00,395.62,8.47} | 1 | 19.90
+{1,1.05393,0.00,8.140,0,0.5380,5.9350,29.30,4.4986,4,307.0,21.00,386.85,6.58} | 1 | 23.10
+{1,0.78420,0.00,8.140,0,0.5380,5.9900,81.70,4.2579,4,307.0,21.00,386.75,14.67} | 1 | 17.50
+{1,0.80271,0.00,8.140,0,0.5380,5.4560,36.60,3.7965,4,307.0,21.00,288.99,11.69} | 1 | 20.20
+{1,0.72580,0.00,8.140,0,0.5380,5.7270,69.50,3.7965,4,307.0,21.00,390.95,11.28} | 1 | 18.20
+\.
+</pre>
+-# Now train a regression model using a multilayer perceptron a single hidden layer of two nodes.
+<pre class="example">
+SELECT madlib.mlp_regression(
+ 'lin_housing', -- Source table
+ 'mlp_regress', -- Desination table
+ 'x', -- Input features
+ 'y', -- Dependent variable
+ ARRAY[5,5], -- Number of units per layer
+ 'step_size=0.000007,
+ n_iterations=10000,
+ tolerance=0',
+ 'relu');
+</pre>
+-# Check the results of the model
+<pre class="example">
+-- Set extended display on for easier reading of output
+\\x ON
+-- Neural net Initialization is non-deterministic, so your results may vary
+SELECT * FROM mlp_regress;
+</pre>
+Result:
+<pre class="result">
+-[ RECORD 1 ]--+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------
+coeff | {1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2.79506311399e-05,3.56715008915e-05,-6.09333559685e-05,0.000251228318768,-0.000224772841379,-3.71863030857e-05,-3.5757865148e-06,5.27936784854e-05,-2.48474166186e-05,6.19731184294e-05,3.07638968743e-05,6.8964698578e-06,0.000106016701083,-1.71484730318e-05,1.18691881812e-05,-0.000163975464208,0.000170026304906,3.11688265279e-05,0.000177050148787,-1.58265976603e-05,2.70144422657e-05,0.000112667883422,3.77575139073e-05,8.12474658795e-05,-7.90458917626e-05,0.000107566386158,-2.63771171506e-06,2.47996880915e-05,-0.00012642310887,0.000203827391081,0.000139315565565,4.86147243454e-05,-0.000176126471913,-6.47820782916e-05,-8.51592776447e-06,-6.60601176758e-05,2.91421874156e-05,6.3556873752e-05,0.000197557443129,0.000220531367259,0.000135036310289,0.000143735913975,-4.75034117786e-05,-0.000179547345838,-1.6919846786e-05,0.000162784312994,0.000268595819851,-0.000460066553287,8.69756071591e-05,-0.00311762727057,0.000126024763103,0.00020598824
2921,0.003463432426,-0.00729789075286,0.00151625867549,-0.000890852767597,-0.00525016037249,0.0031043106659,0.00798041103839,-0.00552693050079,0.0232180415786,0.0230489850143,-0.0437890272341,0.0165765426407,-0.248554261758,-7.81336427846e-05,0.00558145591752,0.283465844585,-0.571699956182,0.133474351994,-0.0785181945605,-0.419269930709,0.249547772912,0.631761009875,-0.431305975666,1,1,1,1,1,1,0.0158747497572,-9.02809160806e-05,0.00015574347618,4.10805373863e-06,0.00121532434965,0.101790351335,0.0647558401493,-0.00013654998677,-9.92872075948e-06,-5.5319694394e-05,0.00519320756484,0.412736586036,0.0011998026977,-1.53688189815e-05,1.94817888201e-05,-4.63111489966e-05,7.24547899029e-05,0.00880394144485,5.45309822095e-05,-0.000140943219275,-7.96211486227e-05,-1.04337307472e-05,0.000161936762028,0.00136273797767,-4.54737243585e-05,-3.4083840736e-05,3.69286883662e-05,9.9047243188e-08,3.75014011824e-06,-9.45366086368e-08,1,1,1,1,1,1,6.67488547054,0.102754199001,0.41668912471,0.008868672964
79,0.00136206007228,-9.88642499013e-05}
+loss | 144.965776158
+num_iterations | 10000
+</pre>
+-# Now let's look at the prediction functions. In the following examples we will
+use the training data set for prediction as well, which is not usual but serves to
+show the syntax. First we will test the classification exampel.
--- End diff --
Typo: `exampel` should be `example`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib issue #149: MLP: Multilayer Perceptron
Posted by cooper-sloan <gi...@git.apache.org>.
Github user cooper-sloan commented on the issue:
https://github.com/apache/incubator-madlib/pull/149
In regards to 'nan' issue. We have run across this in the past. It has to do with regularization, which is scheduled for phase 2. Once we implement regularization this issue should go away. However I will try recreate this bug and verify that is the true issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by edespino <gi...@git.apache.org>.
Github user edespino commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r127049501
--- Diff: src/ports/postgres/modules/convex/mlp.sql_in ---
@@ -0,0 +1,736 @@
+/* ----------------------------------------------------------------------- *//**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ *
+ * @file mlp.sql_in
+ *
+ * @brief SQL functions for multi-layer perceptron
+ * @date June 2012
+ *
+ *
+ *//* ----------------------------------------------------------------------- */
+
+m4_include(`SQLCommon.m4')
+
+/**
+@addtogroup grp_mlp
+
+<div class="toc"><b>Contents</b><ul>
+<li class="level1"><a href="#mlp_classification">Classification</a></li>
+<li class="level1"><a href="#mlp_regression">Regression</a></li>
+<li class="level1"><a href="#optimization_params">Optimization Parameters</a></li>
+<li class="level1"><a href="#predict">Prediction Functions/a></li>
+<li class="level1"><a href="#example">Examples</a></li>
+<li class="level1"><a href="#background">Technical Background</a></li>
+<li class="level1"><a href="#literature">Literature</a></li>
+<li class="level1"><a href="#related">Related Topics</a></li>
+</ul></div>
+
+Multi Layer Perceptron (MLP) is a model for regression and
+classification
+
+Also called "vanilla neural networks", they consist of several
+fully connected hidden layers with non-linear activation
+functions. In the case of classification, the final layer of the
+neural net has as many nodes as classes, and the output of the
+neural net can be interpreted as the probability of a given input
+feature belonging to a specific class.
+
+
+@brief Solves classification and regression problems with several
+fully connected layers and nonlinear activation functions.
+
+@anchor mlp_classification
+@par Classification Training Function
+The mlp classification training function has the following format:
+<pre class="syntax">
+mlp_classification(
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+\b Arguments
+<DL class="arglist">
+ <DT>source_table</DT>
+ <DD>TEXT. Name of the table containing the training data.</DD>
+
+ <DT>model_table</DT>
+ <DD>TEXT. Name of the output table containing the model. Details of the output
+ tables are provided below.
+ </DD>
+
+ <DT>independent_varname</DT>
+ <DD>TEXT. Expression list to evaluate for the
+ independent variables. An intercept variable should not be included as part
+ of this expression. Please note that expression should be able to be cast
+ to DOUBLE PRECISION[].
+ </DD>
+
+ <DT>dependent_varname</DT>
+ <DD> TEXT. Name of the dependent variable column. For classification, supported types are:
+ text, varchar, character varying, char, character
+ integer, smallint, bigint, and boolean. </DD>
+
+ <DT>hidden_layer_sizes</DT>
+ <DD>INTEGER[], default: ARRAY[].
+ The number of neurons in each hidden layer. The length of this array will
+ determine the number of hidden layers. Empty for no hidden layers.
+ </DD>
+
+
+ <DT>optimizer_params (optional)</DT>
+ <DD>TEXT, default: NULL.
+ Parameters for optimization in a comma-separated string
+ of key-value pairs. See the description below for details.
+ </DD>
+
+ <DT>activation</DT>
+ <DD>TEXT, default: 'sigmoid'.
+ Activation function. Currently three functions are supported: 'sigmoid' (default),
+ 'relu', and 'tanh'. The text can be any subset of the three
+ strings; for e.g., activation='s' will use the sigmoid activation.
+ </DD>
+</DL>
+
+<b>Output tables</b>
+<br>
+ The model table produced by mlp contains the following columns:
+ <table class="output">
+ <tr>
+ <th>coeffs</th>
+ <td>FLOAT8[]. Flat array containing the weights of the neural net</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>INTEGER. Number of iterations completed by stochastic gradient descent
+ algorithm. The algorithm either converged in this number of iterations
+ or hit the maximum number specified in the optimization parameters. </td>
+ </tr>
+ <tr>
+ <th>loss</th>
+ <td>FLOAT8. The cross entropy over the training data.
+ See Technical Background section below for more details.</td>
+ </tr>
+ </table>
+
+
+A summary table named \<model_table\>_summary is also created, which has the following columns:
+ <table class="output">
+ <tr>
+ <th>source_table</th>
+ <td>The source table.</td>
+ </tr>
+ <tr>
+ <th>dependent_varname</th>
+ <td>The dependent variable.</td>
+ </tr>
+ <tr>
+ <th>independent_varname</th>
+ <td>The independent variables.</td>
+ </tr>
+ <tr>
+ <th>tolerance</th>
+ <td>The tolerance as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>step_size</th>
+ <td>The step size as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>The number of iterations run</td>
+ </tr>
+ <tr>
+ <th>n_tries</th>
+ <td>The number of tries as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>layer_sizes</th>
+ <td>The number of units in each layer including the input and output layer.</td>
+ </tr>
+ <tr>
+ <th>activation_function</th>
+ <td>The activation function.</td>
+ </tr>
+ <tr>
+ <th>is_classification</th>
+ <td>True if the model was trained for classification, False if it was trained
+ for regression</td>
+ </tr>
+ <tr>
+ <th>classes</th>
+ <td>The classes which were trained against (empty for regression)</td>
+ </tr>
+
+ </table>
+
+
+@anchor mlp_regression
+@par Regression Training Function
+The mlp regression training function has the following format:
+<pre class="syntax">
+mlp_regression(source_table,
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+
+\b Arguments
+
+Specifications for regression are largely the same as for classification. In the
+model table, the loss will refer to mean square error instead of cross entropy. In the
+summary table, there is classes column. The following
+arguments have specifications which differ from mlp_classification:
+<DL class="arglist">
+<DT>dependent_varname</DT>
+ <DD>TEXT. Name of the dependent variable column.
+ For regression supported types are any numeric type, or array
+ or numeric types (for multiple regression).
+ </DD>
+</DL>
+
+
+@anchor optimizer_params
+@par Optimizer Parameters
+Parameters in this section are supplied in the \e optimizer_params argument as a string
+containing a comma-delimited list of name-value pairs. All of these named
+parameters are optional, and their order does not matter. You must use the
+format "<param_name> = <value>" to specify the value of a parameter, otherwise
+the parameter is ignored.
+
+
+<pre class="syntax">
+ 'step_size = <value>,
+ n_iterations = <value>,
+ n_tries = <value>,
+ tolerance = <value>'
+</pre>
+\b Optimizer Parameters
+<DL class="arglist">
+
+<DT>step_size</dt>
+<DD>Default: [0.001].
+Also known as the learning rate. A small value is usually desirable to
+ensure convergence, while a large value provides more room for progress during
+training. Since the best value depends on the condition number of the data, in
+practice one often tunes this parameter.
+</DD>
+
+
+<DT>n_iterations</dt>
+<DD>Default: [100]. The maximum number of iterations allowed.
+</DD>
+<DT>n_tries</dt>
+<DD>Default: [1]. Number of times to retrain the network with randomly initialized
+weights
+</DD>
+
+<DT>tolerance</dt>
+<DD>Default: 0.001. The criterion to end iterations. The training stops whenever
+<the difference between the training models of two consecutive iterations is
+<smaller than \e tolerance or the iteration number is larger than \e max_iter.
+</DD>
+
+</DL>
+
+@anchor predict
+@par Prediction Function
+Used to generate predictions given a previously trained model on novel data.
+The same syntax is used for classification, and regression.
+<pre class="syntax">
+mlp_predict(model_table,
+ data_table,
+ id_col_name,
+ output_table,
+ pred_type)
+</pre>
+
+\b Arguments
+<DL class="arglist">
+ <DT>model_table</DT>
+ <DD>TEXT. Model table produced by the training function.</DD>
+
+ <DT>data_table</DT>
+ <DD>TEXT. Name of the table containing the data for prediction. This table is expected
+ to contain the same input features that were used during training. The table should
+ also contain id_col_name used for identifying each row.</DD>
+
+ <DT>id_col_name</DT>
+ <DD>TEXT. The name of the id column in the input table.</DD>
+
+ <DT>output_table</DT>
+ <DD>TEXT. Name of the table where output predictions are written. If this
+table name is already in use, then an error is returned. Table contains:</DD>
+ <table class="output">
+ <tr>
+ <th>id</th>
+ <td>Gives the 'id' for each prediction, corresponding to each row from the data_table.</td>
+ </tr>
+ <tr>
+ <th>estimated_<COL_NAME></th>
+ <td>
+ (For pred_type='response') The estimated class
+ for classification or value for regression, where
+ <COL_NAME> is the name of the column to be
+ predicted from training data
+ </td>
+ </tr>
+ <tr>
+ <th>prob_<CLASS></th>
+ <td>
+ (For pred_type='prob' for classification) The
+ probability of a given class <CLASS> as given by
+ softmax. There will be one column for each class
+ in the training data.
+ </td>
+ </tr>
+
+
+ <DT>pred_type</DT>
+ <DD>TEXT.
+
+the type of output requested:
+'response' gives the actual prediction,
+'prob' gives the probability of each class.
+for regression, only type='response' is defined.
+The name of the id column in the input table.</DD>
+</DL>
+</table>
+
+@anchor example
+@par Examples
+-# Create an input data set.
+<pre class="example">
+CREATE TABLE iris_data(
+ id integer,
+ attributes numeric[],
+ class_text varchar,
+ class integer
+);
+INSERT INTO iris_data VALUES
+(1,ARRAY[5.1,3.5,1.4,0.2],'Iris-setosa',1),
+(2,ARRAY[4.9,3.0,1.4,0.2],'Iris-setosa',1),
+(3,ARRAY[4.7,3.2,1.3,0.2],'Iris-setosa',1),
+(4,ARRAY[4.6,3.1,1.5,0.2],'Iris-setosa',1),
+(5,ARRAY[5.0,3.6,1.4,0.2],'Iris-setosa',1),
+(6,ARRAY[5.4,3.9,1.7,0.4],'Iris-setosa',1),
+(7,ARRAY[4.6,3.4,1.4,0.3],'Iris-setosa',1),
+(8,ARRAY[5.0,3.4,1.5,0.2],'Iris-setosa',1),
+(9,ARRAY[4.4,2.9,1.4,0.2],'Iris-setosa',1),
+(10,ARRAY[4.9,3.1,1.5,0.1],'Iris-setosa',1),
+(11,ARRAY[7.0,3.2,4.7,1.4],'Iris-versicolor',2),
+(12,ARRAY[6.4,3.2,4.5,1.5],'Iris-versicolor',2),
+(13,ARRAY[6.9,3.1,4.9,1.5],'Iris-versicolor',2),
+(14,ARRAY[5.5,2.3,4.0,1.3],'Iris-versicolor',2),
+(15,ARRAY[6.5,2.8,4.6,1.5],'Iris-versicolor',2),
+(16,ARRAY[5.7,2.8,4.5,1.3],'Iris-versicolor',2),
+(17,ARRAY[6.3,3.3,4.7,1.6],'Iris-versicolor',2),
+(18,ARRAY[4.9,2.4,3.3,1.0],'Iris-versicolor',2),
+(19,ARRAY[6.6,2.9,4.6,1.3],'Iris-versicolor',2),
+(20,ARRAY[5.2,2.7,3.9,1.4],'Iris-versicolor',2);
+</pre>
+-# Generate a multilayer perceptron with a single hidden layer of 5 units.
+Use the attributes column as the independent variables, and use the class
+column as the classification. Set the tolerance to 0 so that 5000
+iterations will be run. Use a hyperbolic tangent activation function.
+The model will be written to mlp_result.
+<pre class="example">
+SELECT madlib.mlp_classification(
+ 'iris_data_2', -- Source table
+ 'mlp_model', -- Destination table
+ 'attributes', -- Input features
+ 'class_text', -- Label
+ ARRAY[5], -- Number of units per layer
+ 'step_size=0.003,
+ n_iterations=5000,
+ tolerance=0', -- Optimizer params
+ 'tanh'); -- Activation function
+</pre>
+-# View the result for the model.
+<pre class="example">
+-- Set extended display on for easier reading of output
+\\x ON
+-- Neural net Initialization is non-deterministic, so your results may vary
+SELECT * FROM mlp_model;
+</pre>
+Result:
+<pre class="result">
+-[ RECORD 1 ]--+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+coeff | {1,1,1,1,1,0.136374930803,0.188739676875,0.662387810001,-1.03381622734,-0.469961067046,0.0614006983397,0.0811504589436,0.299008228258,-0.47391918521,-0.215098143699,0.10519213944,0.145844617525,0.511683525606,-0.800215552382,-0.36417142683,0.120751709056,0.167531106521,0.587074895969,-0.916946198095,-0.417055067449,0.0539541885146,0.0694359704131,0.262598585854,-0.419234805076,-0.189915344282,1,1,1,1,1,1,0.105645702152,1.46247470474,0.484457903226,0.965962824478,1.19361986431,0.419805760087,-0.105696503487,-1.46245956666,-0.484427811691,-0.965730981426,-1.19365280555,-0.419973628863}
+loss | 0.0184092375519
+num_iterations | 5000
+</pre>
+-# Next train a regression example. First create some test data. This dataset
+contains housing prices data.
+<pre class="example">
+CREATE TABLE lin_housing (id serial, x float8[], grp_by_col int, y float8);
+COPY lin_housing (x, grp_by_col, y) FROM STDIN NULL '?' DELIMITER '|';
+{1,0.00632,18.00,2.310,0,0.5380,6.5750,65.20,4.0900,1,296.0,15.30,396.90,4.98} | 1 | 24.00
+{1,0.02731,0.00,7.070,0,0.4690,6.4210,78.90,4.9671,2,242.0,17.80,396.90,9.14} | 1 | 21.60
+{1,0.02729,0.00,7.070,0,0.4690,7.1850,61.10,4.9671,2,242.0,17.80,392.83,4.03} | 1 | 34.70
+{1,0.03237,0.00,2.180,0,0.4580,6.9980,45.80,6.0622,3,222.0,18.70,394.63,2.94} | 1 | 33.40
+{1,0.06905,0.00,2.180,0,0.4580,7.1470,54.20,6.0622,3,222.0,18.70,396.90,5.33} | 1 | 36.20
+{1,0.02985,0.00,2.180,0,0.4580,6.4300,58.70,6.0622,3,222.0,18.70,394.12,5.21} | 1 | 28.70
+{1,0.08829,12.50,7.870,0,0.5240,6.0120,66.60,5.5605,5,311.0,15.20,395.60,12.43} | 1 | 22.90
+{1,0.14455,12.50,7.870,0,0.5240,6.1720,96.10,5.9505,5,311.0,15.20,396.90,19.15} | 1 | 27.10
+{1,0.21124,12.50,7.870,0,0.5240,5.6310,100.00,6.0821,5,311.0,15.20,386.63,29.93} | 1 | 16.50
+{1,0.17004,12.50,7.870,0,0.5240,6.0040,85.90,6.5921,5,311.0,15.20,386.71,17.10} | 1 | 18.90
+{1,0.22489,12.50,7.870,0,0.5240,6.3770,94.30,6.3467,5,311.0,15.20,392.52,20.45} | 1 | 15.00
+{1,0.11747,12.50,7.870,0,0.5240,6.0090,82.90,6.2267,5,311.0,15.20,396.90,13.27} | 1 | 18.90
+{1,0.09378,12.50,7.870,0,0.5240,5.8890,39.00,5.4509,5,311.0,15.20,390.50,15.71} | 1 | 21.70
+{1,0.62976,0.00,8.140,0,0.5380,5.9490,61.80,4.7075,4,307.0,21.00,396.90,8.26} | 1 | 20.40
+{1,0.63796,0.00,8.140,0,0.5380,6.0960,84.50,4.4619,4,307.0,21.00,380.02,10.26} | 1 | 18.20
+{1,0.62739,0.00,8.140,0,0.5380,5.8340,56.50,4.4986,4,307.0,21.00,395.62,8.47} | 1 | 19.90
+{1,1.05393,0.00,8.140,0,0.5380,5.9350,29.30,4.4986,4,307.0,21.00,386.85,6.58} | 1 | 23.10
+{1,0.78420,0.00,8.140,0,0.5380,5.9900,81.70,4.2579,4,307.0,21.00,386.75,14.67} | 1 | 17.50
+{1,0.80271,0.00,8.140,0,0.5380,5.4560,36.60,3.7965,4,307.0,21.00,288.99,11.69} | 1 | 20.20
+{1,0.72580,0.00,8.140,0,0.5380,5.7270,69.50,3.7965,4,307.0,21.00,390.95,11.28} | 1 | 18.20
+\.
+</pre>
+-# Now train a regression model using a multilayer perceptron a single hidden layer of two nodes.
+<pre class="example">
+SELECT mlp_regression(
+ 'lin_housing', -- Source table
+ 'mlp_regress', -- Desination table
+ 'x', -- Input features
+ 'y', -- Dependent variable
+ ARRAY[5,5], -- Number of units per layer
+ 'step_size=0.000007,
+ n_iterations=5000,
+ tolerance=0',
+ 'relu');
+</pre>
+-# Check the results of the model
+<pre class="example">
+-- Set extended display on for easier reading of output
+\\x ON
+-- Neural net Initialization is non-deterministic, so your results may vary
+SELECT * FROM mlp_regress;
+</pre>
+Result:
+<pre class="result">
+-[ RECORD 1 ]--+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------
+coeff | {1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2.79506311399e-05,3.56715008915e-05,-6.09333559685e-05,0.000251228318768,-0.000224772841379,-3.71863030857e-05,-3.5757865148e-06,5.27936784854e-05,-2.48474166186e-05,6.19731184294e-05,3.07638968743e-05,6.8964698578e-06,0.000106016701083,-1.71484730318e-05,1.18691881812e-05,-0.000163975464208,0.000170026304906,3.11688265279e-05,0.000177050148787,-1.58265976603e-05,2.70144422657e-05,0.000112667883422,3.77575139073e-05,8.12474658795e-05,-7.90458917626e-05,0.000107566386158,-2.63771171506e-06,2.47996880915e-05,-0.00012642310887,0.000203827391081,0.000139315565565,4.86147243454e-05,-0.000176126471913,-6.47820782916e-05,-8.51592776447e-06,-6.60601176758e-05,2.91421874156e-05,6.3556873752e-05,0.000197557443129,0.000220531367259,0.000135036310289,0.000143735913975,-4.75034117786e-05,-0.000179547345838,-1.6919846786e-05,0.000162784312994,0.000268595819851,-0.000460066553287,8.69756071591e-05,-0.00311762727057,0.000126024763103,0.00020598824
2921,0.003463432426,-0.00729789075286,0.00151625867549,-0.000890852767597,-0.00525016037249,0.0031043106659,0.00798041103839,-0.00552693050079,0.0232180415786,0.0230489850143,-0.0437890272341,0.0165765426407,-0.248554261758,-7.81336427846e-05,0.00558145591752,0.283465844585,-0.571699956182,0.133474351994,-0.0785181945605,-0.419269930709,0.249547772912,0.631761009875,-0.431305975666,1,1,1,1,1,1,0.0158747497572,-9.02809160806e-05,0.00015574347618,4.10805373863e-06,0.00121532434965,0.101790351335,0.0647558401493,-0.00013654998677,-9.92872075948e-06,-5.5319694394e-05,0.00519320756484,0.412736586036,0.0011998026977,-1.53688189815e-05,1.94817888201e-05,-4.63111489966e-05,7.24547899029e-05,0.00880394144485,5.45309822095e-05,-0.000140943219275,-7.96211486227e-05,-1.04337307472e-05,0.000161936762028,0.00136273797767,-4.54737243585e-05,-3.4083840736e-05,3.69286883662e-05,9.9047243188e-08,3.75014011824e-06,-9.45366086368e-08,1,1,1,1,1,1,6.67488547054,0.102754199001,0.41668912471,0.008868672964
79,0.00136206007228,-9.88642499013e-05}
+loss | 144.965776158
+num_iterations | 10000
--- End diff --
This example output is not consistent with the number of iterations (5000 instead of 10000) used. I am validating this PR with the `PostgreSQL 9.6.3`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by cooper-sloan <gi...@git.apache.org>.
Github user cooper-sloan commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r128630568
--- Diff: doc/design/modules/neural-network.tex ---
@@ -0,0 +1,195 @@
+% Licensed to the Apache Software Foundation (ASF) under one
+% or more contributor license agreements. See the NOTICE file
+% distributed with this work for additional information
+% regarding copyright ownership. The ASF licenses this file
+% to you under the Apache License, Version 2.0 (the
+% "License"); you may not use this file except in compliance
+% with the License. You may obtain a copy of the License at
+
+% http://www.apache.org/licenses/LICENSE-2.0
+
+% Unless required by applicable law or agreed to in writing,
+% software distributed under the License is distributed on an
+% "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+% KIND, either express or implied. See the License for the
+% specific language governing permissions and limitations
+% under the License.
+
+% When using TeXShop on the Mac, let it know the root document.
+% The following must be one of the first 20 lines.
+% !TEX root = ../design.tex
+
+\chapter{Neural Network}
+
--- End diff --
Will do
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by cooper-sloan <gi...@git.apache.org>.
Github user cooper-sloan commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r127076187
--- Diff: src/ports/postgres/modules/convex/mlp.sql_in ---
@@ -0,0 +1,736 @@
+/* ----------------------------------------------------------------------- *//**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ *
+ * @file mlp.sql_in
+ *
+ * @brief SQL functions for multi-layer perceptron
+ * @date June 2012
+ *
+ *
+ *//* ----------------------------------------------------------------------- */
+
+m4_include(`SQLCommon.m4')
+
+/**
+@addtogroup grp_mlp
+
+<div class="toc"><b>Contents</b><ul>
+<li class="level1"><a href="#mlp_classification">Classification</a></li>
+<li class="level1"><a href="#mlp_regression">Regression</a></li>
+<li class="level1"><a href="#optimization_params">Optimization Parameters</a></li>
+<li class="level1"><a href="#predict">Prediction Functions/a></li>
+<li class="level1"><a href="#example">Examples</a></li>
+<li class="level1"><a href="#background">Technical Background</a></li>
+<li class="level1"><a href="#literature">Literature</a></li>
+<li class="level1"><a href="#related">Related Topics</a></li>
+</ul></div>
+
+Multi Layer Perceptron (MLP) is a model for regression and
+classification
+
+Also called "vanilla neural networks", they consist of several
+fully connected hidden layers with non-linear activation
+functions. In the case of classification, the final layer of the
+neural net has as many nodes as classes, and the output of the
+neural net can be interpreted as the probability of a given input
+feature belonging to a specific class.
+
+
+@brief Solves classification and regression problems with several
+fully connected layers and nonlinear activation functions.
+
+@anchor mlp_classification
+@par Classification Training Function
+The mlp classification training function has the following format:
+<pre class="syntax">
+mlp_classification(
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+\b Arguments
+<DL class="arglist">
+ <DT>source_table</DT>
+ <DD>TEXT. Name of the table containing the training data.</DD>
+
+ <DT>model_table</DT>
+ <DD>TEXT. Name of the output table containing the model. Details of the output
+ tables are provided below.
+ </DD>
+
+ <DT>independent_varname</DT>
+ <DD>TEXT. Expression list to evaluate for the
+ independent variables. An intercept variable should not be included as part
+ of this expression. Please note that expression should be able to be cast
+ to DOUBLE PRECISION[].
+ </DD>
+
+ <DT>dependent_varname</DT>
+ <DD> TEXT. Name of the dependent variable column. For classification, supported types are:
+ text, varchar, character varying, char, character
+ integer, smallint, bigint, and boolean. </DD>
+
+ <DT>hidden_layer_sizes</DT>
+ <DD>INTEGER[], default: ARRAY[].
+ The number of neurons in each hidden layer. The length of this array will
+ determine the number of hidden layers. Empty for no hidden layers.
+ </DD>
+
+
+ <DT>optimizer_params (optional)</DT>
+ <DD>TEXT, default: NULL.
+ Parameters for optimization in a comma-separated string
+ of key-value pairs. See the description below for details.
+ </DD>
+
+ <DT>activation</DT>
+ <DD>TEXT, default: 'sigmoid'.
+ Activation function. Currently three functions are supported: 'sigmoid' (default),
+ 'relu', and 'tanh'. The text can be any subset of the three
+ strings; for e.g., activation='s' will use the sigmoid activation.
+ </DD>
+</DL>
+
+<b>Output tables</b>
+<br>
+ The model table produced by mlp contains the following columns:
+ <table class="output">
+ <tr>
+ <th>coeffs</th>
+ <td>FLOAT8[]. Flat array containing the weights of the neural net</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>INTEGER. Number of iterations completed by stochastic gradient descent
+ algorithm. The algorithm either converged in this number of iterations
+ or hit the maximum number specified in the optimization parameters. </td>
+ </tr>
+ <tr>
+ <th>loss</th>
+ <td>FLOAT8. The cross entropy over the training data.
+ See Technical Background section below for more details.</td>
+ </tr>
+ </table>
+
+
+A summary table named \<model_table\>_summary is also created, which has the following columns:
+ <table class="output">
+ <tr>
+ <th>source_table</th>
+ <td>The source table.</td>
+ </tr>
+ <tr>
+ <th>dependent_varname</th>
+ <td>The dependent variable.</td>
+ </tr>
+ <tr>
+ <th>independent_varname</th>
+ <td>The independent variables.</td>
+ </tr>
+ <tr>
+ <th>tolerance</th>
+ <td>The tolerance as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>step_size</th>
+ <td>The step size as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>The number of iterations run</td>
+ </tr>
+ <tr>
+ <th>n_tries</th>
+ <td>The number of tries as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>layer_sizes</th>
+ <td>The number of units in each layer including the input and output layer.</td>
+ </tr>
+ <tr>
+ <th>activation_function</th>
+ <td>The activation function.</td>
+ </tr>
+ <tr>
+ <th>is_classification</th>
+ <td>True if the model was trained for classification, False if it was trained
+ for regression</td>
+ </tr>
+ <tr>
+ <th>classes</th>
+ <td>The classes which were trained against (empty for regression)</td>
+ </tr>
+
+ </table>
+
+
+@anchor mlp_regression
+@par Regression Training Function
+The mlp regression training function has the following format:
+<pre class="syntax">
+mlp_regression(source_table,
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+
+\b Arguments
+
+Specifications for regression are largely the same as for classification. In the
+model table, the loss will refer to mean square error instead of cross entropy. In the
+summary table, there is classes column. The following
+arguments have specifications which differ from mlp_classification:
+<DL class="arglist">
+<DT>dependent_varname</DT>
+ <DD>TEXT. Name of the dependent variable column.
+ For regression supported types are any numeric type, or array
+ or numeric types (for multiple regression).
+ </DD>
+</DL>
+
+
+@anchor optimizer_params
+@par Optimizer Parameters
+Parameters in this section are supplied in the \e optimizer_params argument as a string
+containing a comma-delimited list of name-value pairs. All of these named
+parameters are optional, and their order does not matter. You must use the
+format "<param_name> = <value>" to specify the value of a parameter, otherwise
+the parameter is ignored.
+
+
+<pre class="syntax">
+ 'step_size = <value>,
+ n_iterations = <value>,
+ n_tries = <value>,
+ tolerance = <value>'
+</pre>
+\b Optimizer Parameters
+<DL class="arglist">
+
+<DT>step_size</dt>
+<DD>Default: [0.001].
+Also known as the learning rate. A small value is usually desirable to
+ensure convergence, while a large value provides more room for progress during
+training. Since the best value depends on the condition number of the data, in
+practice one often tunes this parameter.
+</DD>
+
+
+<DT>n_iterations</dt>
+<DD>Default: [100]. The maximum number of iterations allowed.
+</DD>
+<DT>n_tries</dt>
+<DD>Default: [1]. Number of times to retrain the network with randomly initialized
+weights
+</DD>
+
+<DT>tolerance</dt>
+<DD>Default: 0.001. The criterion to end iterations. The training stops whenever
+<the difference between the training models of two consecutive iterations is
+<smaller than \e tolerance or the iteration number is larger than \e max_iter.
+</DD>
+
+</DL>
+
+@anchor predict
+@par Prediction Function
+Used to generate predictions given a previously trained model on novel data.
+The same syntax is used for classification, and regression.
+<pre class="syntax">
+mlp_predict(model_table,
+ data_table,
+ id_col_name,
+ output_table,
+ pred_type)
+</pre>
+
+\b Arguments
+<DL class="arglist">
+ <DT>model_table</DT>
+ <DD>TEXT. Model table produced by the training function.</DD>
+
+ <DT>data_table</DT>
+ <DD>TEXT. Name of the table containing the data for prediction. This table is expected
+ to contain the same input features that were used during training. The table should
+ also contain id_col_name used for identifying each row.</DD>
+
+ <DT>id_col_name</DT>
+ <DD>TEXT. The name of the id column in the input table.</DD>
+
+ <DT>output_table</DT>
+ <DD>TEXT. Name of the table where output predictions are written. If this
+table name is already in use, then an error is returned. Table contains:</DD>
+ <table class="output">
+ <tr>
+ <th>id</th>
+ <td>Gives the 'id' for each prediction, corresponding to each row from the data_table.</td>
+ </tr>
+ <tr>
+ <th>estimated_<COL_NAME></th>
+ <td>
+ (For pred_type='response') The estimated class
+ for classification or value for regression, where
+ <COL_NAME> is the name of the column to be
+ predicted from training data
+ </td>
+ </tr>
+ <tr>
+ <th>prob_<CLASS></th>
+ <td>
+ (For pred_type='prob' for classification) The
+ probability of a given class <CLASS> as given by
+ softmax. There will be one column for each class
+ in the training data.
+ </td>
+ </tr>
+
+
+ <DT>pred_type</DT>
+ <DD>TEXT.
+
+the type of output requested:
+'response' gives the actual prediction,
+'prob' gives the probability of each class.
+for regression, only type='response' is defined.
+The name of the id column in the input table.</DD>
+</DL>
+</table>
+
+@anchor example
+@par Examples
+-# Create an input data set.
+<pre class="example">
+CREATE TABLE iris_data(
+ id integer,
+ attributes numeric[],
+ class_text varchar,
+ class integer
+);
+INSERT INTO iris_data VALUES
+(1,ARRAY[5.1,3.5,1.4,0.2],'Iris-setosa',1),
+(2,ARRAY[4.9,3.0,1.4,0.2],'Iris-setosa',1),
+(3,ARRAY[4.7,3.2,1.3,0.2],'Iris-setosa',1),
+(4,ARRAY[4.6,3.1,1.5,0.2],'Iris-setosa',1),
+(5,ARRAY[5.0,3.6,1.4,0.2],'Iris-setosa',1),
+(6,ARRAY[5.4,3.9,1.7,0.4],'Iris-setosa',1),
+(7,ARRAY[4.6,3.4,1.4,0.3],'Iris-setosa',1),
+(8,ARRAY[5.0,3.4,1.5,0.2],'Iris-setosa',1),
+(9,ARRAY[4.4,2.9,1.4,0.2],'Iris-setosa',1),
+(10,ARRAY[4.9,3.1,1.5,0.1],'Iris-setosa',1),
+(11,ARRAY[7.0,3.2,4.7,1.4],'Iris-versicolor',2),
+(12,ARRAY[6.4,3.2,4.5,1.5],'Iris-versicolor',2),
+(13,ARRAY[6.9,3.1,4.9,1.5],'Iris-versicolor',2),
+(14,ARRAY[5.5,2.3,4.0,1.3],'Iris-versicolor',2),
+(15,ARRAY[6.5,2.8,4.6,1.5],'Iris-versicolor',2),
+(16,ARRAY[5.7,2.8,4.5,1.3],'Iris-versicolor',2),
+(17,ARRAY[6.3,3.3,4.7,1.6],'Iris-versicolor',2),
+(18,ARRAY[4.9,2.4,3.3,1.0],'Iris-versicolor',2),
+(19,ARRAY[6.6,2.9,4.6,1.3],'Iris-versicolor',2),
+(20,ARRAY[5.2,2.7,3.9,1.4],'Iris-versicolor',2);
+</pre>
+-# Generate a multilayer perceptron with a single hidden layer of 5 units.
+Use the attributes column as the independent variables, and use the class
+column as the classification. Set the tolerance to 0 so that 5000
+iterations will be run. Use a hyperbolic tangent activation function.
+The model will be written to mlp_result.
+<pre class="example">
+SELECT madlib.mlp_classification(
+ 'iris_data_2', -- Source table
+ 'mlp_model', -- Destination table
+ 'attributes', -- Input features
+ 'class_text', -- Label
+ ARRAY[5], -- Number of units per layer
+ 'step_size=0.003,
+ n_iterations=5000,
+ tolerance=0', -- Optimizer params
+ 'tanh'); -- Activation function
+</pre>
+-# View the result for the model.
+<pre class="example">
+-- Set extended display on for easier reading of output
+\\x ON
+-- Neural net Initialization is non-deterministic, so your results may vary
+SELECT * FROM mlp_model;
+</pre>
+Result:
+<pre class="result">
+-[ RECORD 1 ]--+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+coeff | {1,1,1,1,1,0.136374930803,0.188739676875,0.662387810001,-1.03381622734,-0.469961067046,0.0614006983397,0.0811504589436,0.299008228258,-0.47391918521,-0.215098143699,0.10519213944,0.145844617525,0.511683525606,-0.800215552382,-0.36417142683,0.120751709056,0.167531106521,0.587074895969,-0.916946198095,-0.417055067449,0.0539541885146,0.0694359704131,0.262598585854,-0.419234805076,-0.189915344282,1,1,1,1,1,1,0.105645702152,1.46247470474,0.484457903226,0.965962824478,1.19361986431,0.419805760087,-0.105696503487,-1.46245956666,-0.484427811691,-0.965730981426,-1.19365280555,-0.419973628863}
+loss | 0.0184092375519
+num_iterations | 5000
+</pre>
+-# Next train a regression example. First create some test data. This dataset
+contains housing prices data.
+<pre class="example">
+CREATE TABLE lin_housing (id serial, x float8[], grp_by_col int, y float8);
+COPY lin_housing (x, grp_by_col, y) FROM STDIN NULL '?' DELIMITER '|';
+{1,0.00632,18.00,2.310,0,0.5380,6.5750,65.20,4.0900,1,296.0,15.30,396.90,4.98} | 1 | 24.00
+{1,0.02731,0.00,7.070,0,0.4690,6.4210,78.90,4.9671,2,242.0,17.80,396.90,9.14} | 1 | 21.60
+{1,0.02729,0.00,7.070,0,0.4690,7.1850,61.10,4.9671,2,242.0,17.80,392.83,4.03} | 1 | 34.70
+{1,0.03237,0.00,2.180,0,0.4580,6.9980,45.80,6.0622,3,222.0,18.70,394.63,2.94} | 1 | 33.40
+{1,0.06905,0.00,2.180,0,0.4580,7.1470,54.20,6.0622,3,222.0,18.70,396.90,5.33} | 1 | 36.20
+{1,0.02985,0.00,2.180,0,0.4580,6.4300,58.70,6.0622,3,222.0,18.70,394.12,5.21} | 1 | 28.70
+{1,0.08829,12.50,7.870,0,0.5240,6.0120,66.60,5.5605,5,311.0,15.20,395.60,12.43} | 1 | 22.90
+{1,0.14455,12.50,7.870,0,0.5240,6.1720,96.10,5.9505,5,311.0,15.20,396.90,19.15} | 1 | 27.10
+{1,0.21124,12.50,7.870,0,0.5240,5.6310,100.00,6.0821,5,311.0,15.20,386.63,29.93} | 1 | 16.50
+{1,0.17004,12.50,7.870,0,0.5240,6.0040,85.90,6.5921,5,311.0,15.20,386.71,17.10} | 1 | 18.90
+{1,0.22489,12.50,7.870,0,0.5240,6.3770,94.30,6.3467,5,311.0,15.20,392.52,20.45} | 1 | 15.00
+{1,0.11747,12.50,7.870,0,0.5240,6.0090,82.90,6.2267,5,311.0,15.20,396.90,13.27} | 1 | 18.90
+{1,0.09378,12.50,7.870,0,0.5240,5.8890,39.00,5.4509,5,311.0,15.20,390.50,15.71} | 1 | 21.70
+{1,0.62976,0.00,8.140,0,0.5380,5.9490,61.80,4.7075,4,307.0,21.00,396.90,8.26} | 1 | 20.40
+{1,0.63796,0.00,8.140,0,0.5380,6.0960,84.50,4.4619,4,307.0,21.00,380.02,10.26} | 1 | 18.20
+{1,0.62739,0.00,8.140,0,0.5380,5.8340,56.50,4.4986,4,307.0,21.00,395.62,8.47} | 1 | 19.90
+{1,1.05393,0.00,8.140,0,0.5380,5.9350,29.30,4.4986,4,307.0,21.00,386.85,6.58} | 1 | 23.10
+{1,0.78420,0.00,8.140,0,0.5380,5.9900,81.70,4.2579,4,307.0,21.00,386.75,14.67} | 1 | 17.50
+{1,0.80271,0.00,8.140,0,0.5380,5.4560,36.60,3.7965,4,307.0,21.00,288.99,11.69} | 1 | 20.20
+{1,0.72580,0.00,8.140,0,0.5380,5.7270,69.50,3.7965,4,307.0,21.00,390.95,11.28} | 1 | 18.20
+\.
+</pre>
+-# Now train a regression model using a multilayer perceptron a single hidden layer of two nodes.
+<pre class="example">
+SELECT mlp_regression(
+ 'lin_housing', -- Source table
+ 'mlp_regress', -- Desination table
+ 'x', -- Input features
+ 'y', -- Dependent variable
+ ARRAY[5,5], -- Number of units per layer
+ 'step_size=0.000007,
+ n_iterations=5000,
+ tolerance=0',
+ 'relu');
+</pre>
+-# Check the results of the model
+<pre class="example">
+-- Set extended display on for easier reading of output
+\\x ON
+-- Neural net Initialization is non-deterministic, so your results may vary
+SELECT * FROM mlp_regress;
+</pre>
+Result:
+<pre class="result">
+-[ RECORD 1 ]--+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------
+coeff | {1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2.79506311399e-05,3.56715008915e-05,-6.09333559685e-05,0.000251228318768,-0.000224772841379,-3.71863030857e-05,-3.5757865148e-06,5.27936784854e-05,-2.48474166186e-05,6.19731184294e-05,3.07638968743e-05,6.8964698578e-06,0.000106016701083,-1.71484730318e-05,1.18691881812e-05,-0.000163975464208,0.000170026304906,3.11688265279e-05,0.000177050148787,-1.58265976603e-05,2.70144422657e-05,0.000112667883422,3.77575139073e-05,8.12474658795e-05,-7.90458917626e-05,0.000107566386158,-2.63771171506e-06,2.47996880915e-05,-0.00012642310887,0.000203827391081,0.000139315565565,4.86147243454e-05,-0.000176126471913,-6.47820782916e-05,-8.51592776447e-06,-6.60601176758e-05,2.91421874156e-05,6.3556873752e-05,0.000197557443129,0.000220531367259,0.000135036310289,0.000143735913975,-4.75034117786e-05,-0.000179547345838,-1.6919846786e-05,0.000162784312994,0.000268595819851,-0.000460066553287,8.69756071591e-05,-0.00311762727057,0.000126024763103,0.00020598824
2921,0.003463432426,-0.00729789075286,0.00151625867549,-0.000890852767597,-0.00525016037249,0.0031043106659,0.00798041103839,-0.00552693050079,0.0232180415786,0.0230489850143,-0.0437890272341,0.0165765426407,-0.248554261758,-7.81336427846e-05,0.00558145591752,0.283465844585,-0.571699956182,0.133474351994,-0.0785181945605,-0.419269930709,0.249547772912,0.631761009875,-0.431305975666,1,1,1,1,1,1,0.0158747497572,-9.02809160806e-05,0.00015574347618,4.10805373863e-06,0.00121532434965,0.101790351335,0.0647558401493,-0.00013654998677,-9.92872075948e-06,-5.5319694394e-05,0.00519320756484,0.412736586036,0.0011998026977,-1.53688189815e-05,1.94817888201e-05,-4.63111489966e-05,7.24547899029e-05,0.00880394144485,5.45309822095e-05,-0.000140943219275,-7.96211486227e-05,-1.04337307472e-05,0.000161936762028,0.00136273797767,-4.54737243585e-05,-3.4083840736e-05,3.69286883662e-05,9.9047243188e-08,3.75014011824e-06,-9.45366086368e-08,1,1,1,1,1,1,6.67488547054,0.102754199001,0.41668912471,0.008868672964
79,0.00136206007228,-9.88642499013e-05}
+loss | 144.965776158
+num_iterations | 10000
+</pre>
+-# Now let's look at the prediction functions. In the following examples we will
+use the training data set for prediction as well, which is not usual but serves to
+show the syntax. First we will test the classification exampel.
+The prediction is in the the estimated_class_text column with the
+actual value in the class_text column.
+<pre class="example">
+SELECT mlp_predict(
+ 'mlp_model', -- Model table
+ 'iris_data', -- Test data table
+ 'id', -- Id column in test table
+ 'mlp_prediction', -- Output table for predictions
+ 'response' -- Output classes, not probabilities
+ );
+SELECT * FROM mlp_prediction JOIN iris_data USING (id);
+</pre>
+Result:
+<pre class="result">
+ id | estimated_class_text | attributes | class_text | class
+----+----------------------+-------------------+-----------------+-------
+ 1 | Iris-setosa | {5.1,3.5,1.4,0.2} | Iris-setosa | 1
+ 2 | Iris-setosa | {4.9,3.0,1.4,0.2} | Iris-setosa | 1
+ 3 | Iris-setosa | {4.7,3.2,1.3,0.2} | Iris-setosa | 1
+ 4 | Iris-setosa | {4.6,3.1,1.5,0.2} | Iris-setosa | 1
+ 5 | Iris-setosa | {5.0,3.6,1.4,0.2} | Iris-setosa | 1
+ 6 | Iris-setosa | {5.4,3.9,1.7,0.4} | Iris-setosa | 1
+ 7 | Iris-setosa | {4.6,3.4,1.4,0.3} | Iris-setosa | 1
+ 8 | Iris-setosa | {5.0,3.4,1.5,0.2} | Iris-setosa | 1
+ 9 | Iris-setosa | {4.4,2.9,1.4,0.2} | Iris-setosa | 1
+ 10 | Iris-setosa | {4.9,3.1,1.5,0.1} | Iris-setosa | 1
+ 11 | Iris-versicolor | {7.0,3.2,4.7,1.4} | Iris-versicolor | 2
+ 12 | Iris-versicolor | {6.4,3.2,4.5,1.5} | Iris-versicolor | 2
+ 13 | Iris-versicolor | {6.9,3.1,4.9,1.5} | Iris-versicolor | 2
+ 14 | Iris-versicolor | {5.5,2.3,4.0,1.3} | Iris-versicolor | 2
+ 15 | Iris-versicolor | {6.5,2.8,4.6,1.5} | Iris-versicolor | 2
+ 16 | Iris-versicolor | {5.7,2.8,4.5,1.3} | Iris-versicolor | 2
+ 17 | Iris-versicolor | {6.3,3.3,4.7,1.6} | Iris-versicolor | 2
+ 18 | Iris-versicolor | {4.9,2.4,3.3,1.0} | Iris-versicolor | 2
+ 19 | Iris-versicolor | {6.6,2.9,4.6,1.3} | Iris-versicolor | 2
+ 20 | Iris-versicolor | {5.2,2.7,3.9,1.4} | Iris-versicolor | 2
+</pre>
+Prediction using the regression model:
+<pre class="example">
+SELECT mlp_predict(
+ 'mlp_regress', -- Model table
+ 'lin_housing', -- Test data table
+ 'id', -- Id column in test table
+ 'mlp_regress_prediction', -- Output table for predictions
+ 'response' -- Output values, not probabilities
+ );
+</pre>
+-# View results
+<pre class="example">
+SELECT madlib.mlp_predict('mlp_regress','lin_housing','id','mlp_regress_prediction','response');
+SELECT * FROM lin_housing JOIN mlp_regress_prediction USING (id);
+</pre>
+Result for the regression model:
+<pre class="result">
+ id | x | grp_by_col | y | estimated_y
+----+-------------------------------------------------------------------------+------------+------+--------------------
+ 1 | {1,0.00632,18,2.31,0,0.538,6.575,65.2,4.09,1,296,15.3,396.9,4.98} | 1 | 24 | {23.2627062018087}
+ 2 | {1,0.02731,0,7.07,0,0.469,6.421,78.9,4.9671,2,242,17.8,396.9,9.14} | 1 | 21.6 | {25.7088419115781}
+ 3 | {1,0.02729,0,7.07,0,0.469,7.185,61.1,4.9671,2,242,17.8,392.83,4.03} | 1 | 34.7 | {27.5587003901404}
+ 4 | {1,0.03237,0,2.18,0,0.458,6.998,45.8,6.0622,3,222,18.7,394.63,2.94} | 1 | 33.4 | {31.1812237427816}
+ 5 | {1,0.06905,0,2.18,0,0.458,7.147,54.2,6.0622,3,222,18.7,396.9,5.33} | 1 | 36.2 | {30.3696873085477}
+ 6 | {1,0.02985,0,2.18,0,0.458,6.43,58.7,6.0622,3,222,18.7,394.12,5.21} | 1 | 28.7 | {29.5290259241882}
+ 7 | {1,0.08829,12.5,7.87,0,0.524,6.012,66.6,5.5605,5,311,15.2,395.6,12.43} | 1 | 22.9 | {21.1576051716888}
+ 8 | {1,0.14455,12.5,7.87,0,0.524,6.172,96.1,5.9505,5,311,15.2,396.9,19.15} | 1 | 27.1 | {17.6194200563055}
+ 9 | {1,0.21124,12.5,7.87,0,0.524,5.631,100,6.0821,5,311,15.2,386.63,29.93} | 1 | 16.5 | {15.1366297774139}
+10 | {1,0.17004,12.5,7.87,0,0.524,6.004,85.9,6.5921,5,311,15.2,386.71,17.1} | 1 | 18.9 | {17.6528662199369}
+11 | {1,0.22489,12.5,7.87,0,0.524,6.377,94.3,6.3467,5,311,15.2,392.52,20.45} | 1 | 15 | {17.2017487668181}
+12 | {1,0.11747,12.5,7.87,0,0.524,6.009,82.9,6.2267,5,311,15.2,396.9,13.27} | 1 | 18.9 | {19.4893860319992}
+13 | {1,0.09378,12.5,7.87,0,0.524,5.889,39,5.4509,5,311,15.2,390.5,15.71} | 1 | 21.7 | {23.2917226708039}
+14 | {1,0.62976,0,8.14,0,0.538,5.949,61.8,4.7075,4,307,21,396.9,8.26} | 1 | 20.4 | {22.8904812605193}
+15 | {1,0.63796,0,8.14,0,0.538,6.096,84.5,4.4619,4,307,21,380.02,10.26} | 1 | 18.2 | {18.2386754423677}
+16 | {1,0.62739,0,8.14,0,0.538,5.834,56.5,4.4986,4,307,21,395.62,8.47} | 1 | 19.9 | {23.28949550874}
+17 | {1,1.05393,0,8.14,0,0.538,5.935,29.3,4.4986,4,307,21,386.85,6.58} | 1 | 23.1 | {25.3288762085473}
+18 | {1,0.7842,0,8.14,0,0.538,5.99,81.7,4.2579,4,307,21,386.75,14.67} | 1 | 17.5 | {19.0203738118451}
+19 | {1,0.80271,0,8.14,0,0.538,5.456,36.6,3.7965,4,307,21,288.99,11.69} | 1 | 20.2 | {12.3162005347545}
+20 | {1,0.7258,0,8.14,0,0.538,5.727,69.5,3.7965,4,307,21,390.95,11.28} | 1 | 18.2 | {21.0902211848747}
+</pre>
+Note that the results you get for all examples may vary with the platform you are using.
+
+@anchor background
+@par Technical Background
+
+To train a neural net, the respective loss function is minimized using stochastic gradient descent.
+In the case of classification, the loss function is cross entropy. For regression, mean square error
+is used. Weights in the neural net are updated via the backpropogation process, which uses dynamic
+programming to compute the partial derivative of each weight with respect to the overall loss. This
+partial derivative incorporates the respective activation function used, so this requires that the
+activation function be differentiable.
+
+@anchor literature
+@literature
+
+@anchor mlp-lit-1
+[1] Hastie, Trevor. Neural network. John Wiley & Sons, Ltd, 1998.
+
--- End diff --
I see. Sorry I misunderstood. I will go ahead and add them!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by edespino <gi...@git.apache.org>.
Github user edespino commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r127313158
--- Diff: src/ports/postgres/modules/convex/mlp.sql_in ---
@@ -0,0 +1,744 @@
+/* ----------------------------------------------------------------------- *//**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ *
+ * @file mlp.sql_in
+ *
+ * @brief SQL functions for multilayer perceptron
+ * @date June 2012
+ *
+ *
+ *//* ----------------------------------------------------------------------- */
+
+m4_include(`SQLCommon.m4')
+
+/**
+@addtogroup grp_mlp
+
+<div class="toc"><b>Contents</b><ul>
+<li class="level1"><a href="#mlp_classification">Classification</a></li>
+<li class="level1"><a href="#mlp_regression">Regression</a></li>
+<li class="level1"><a href="#optimization_params">Optimizer Parameters</a></li>
+<li class="level1"><a href="#predict">Prediction Functions/a></li>
+<li class="level1"><a href="#example">Examples</a></li>
+<li class="level1"><a href="#background">Technical Background</a></li>
+<li class="level1"><a href="#literature">Literature</a></li>
+<li class="level1"><a href="#related">Related Topics</a></li>
+</ul></div>
+
+Multilayer Perceptron (MLP) is a model for regression and
+classification
+
+Also called "vanilla neural networks", they consist of several
+fully connected hidden layers with non-linear activation
+functions. In the case of classification, the final layer of the
+neural net has as many nodes as classes, and the output of the
+neural net can be interpreted as the probability of a given input
+feature belonging to a specific class.
+
+
+@brief Solves classification and regression problems with several
+fully connected layers and nonlinear activation functions.
+
+@anchor mlp_classification
+@par Classification Training Function
+The mlp classification training function has the following format:
+<pre class="syntax">
+mlp_classification(
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+\b Arguments
+<DL class="arglist">
+ <DT>source_table</DT>
+ <DD>TEXT. Name of the table containing the training data.</DD>
+
+ <DT>model_table</DT>
+ <DD>TEXT. Name of the output table containing the model. Details of the output
+ tables are provided below.
+ </DD>
+
+ <DT>independent_varname</DT>
+ <DD>TEXT. Expression list to evaluate for the
+ independent variables. An intercept variable should not be included as part
+ of this expression. Please note that expression should be able to be cast
+ to DOUBLE PRECISION[].
+ </DD>
+
+ <DT>dependent_varname</DT>
+ <DD> TEXT. Name of the dependent variable column. For classification, supported types are:
+ text, varchar, character varying, char, character
+ integer, smallint, bigint, and boolean. </DD>
+
+ <DT>hidden_layer_sizes</DT>
+ <DD>INTEGER[], default: ARRAY[].
+ The number of neurons in each hidden layer. The length of this array will
+ determine the number of hidden layers. Empty for no hidden layers.
+ </DD>
+
+
+ <DT>optimizer_params (optional)</DT>
+ <DD>TEXT, default: NULL.
+ Parameters for optimization in a comma-separated string
+ of key-value pairs. See the description below for details.
+ </DD>
+
+ <DT>activation</DT>
+ <DD>TEXT, default: 'sigmoid'.
+ Activation function. Currently three functions are supported: 'sigmoid' (default),
+ 'relu', and 'tanh'. The text can be any subset of the three
+ strings; for e.g., activation='s' will use the sigmoid activation.
+ </DD>
+</DL>
+
+<b>Output tables</b>
+<br>
+ The model table produced by mlp contains the following columns:
+ <table class="output">
+ <tr>
+ <th>coeffs</th>
+ <td>FLOAT8[]. Flat array containing the weights of the neural net</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>INTEGER. Number of iterations completed by stochastic gradient descent
+ algorithm. The algorithm either converged in this number of iterations
+ or hit the maximum number specified in the optimization parameters. </td>
+ </tr>
+ <tr>
+ <th>loss</th>
+ <td>FLOAT8. The cross entropy over the training data.
+ See Technical Background section below for more details.</td>
+ </tr>
+ </table>
+
+
+A summary table named \<model_table\>_summary is also created, which has the following columns:
+ <table class="output">
+ <tr>
+ <th>source_table</th>
+ <td>The source table.</td>
+ </tr>
+ <tr>
+ <th>dependent_varname</th>
+ <td>The dependent variable.</td>
+ </tr>
+ <tr>
+ <th>independent_varname</th>
+ <td>The independent variables.</td>
+ </tr>
+ <tr>
+ <th>tolerance</th>
+ <td>The tolerance as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>step_size</th>
+ <td>The step size as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>The number of iterations run</td>
+ </tr>
+ <tr>
+ <th>n_tries</th>
+ <td>The number of tries as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>layer_sizes</th>
+ <td>The number of units in each layer including the input and output layer.</td>
+ </tr>
+ <tr>
+ <th>activation_function</th>
+ <td>The activation function.</td>
+ </tr>
+ <tr>
+ <th>is_classification</th>
+ <td>True if the model was trained for classification, False if it was trained
+ for regression</td>
+ </tr>
+ <tr>
+ <th>classes</th>
+ <td>The classes which were trained against (empty for regression)</td>
+ </tr>
+
+ </table>
+
+
+@anchor mlp_regression
+@par Regression Training Function
+The mlp regression training function has the following format:
+<pre class="syntax">
+mlp_regression(source_table,
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+
+\b Arguments
+
+Specifications for regression are largely the same as for classification. In the
+model table, the loss will refer to mean square error instead of cross entropy. In the
+summary table, there is classes column. The following
+arguments have specifications which differ from mlp_classification:
+<DL class="arglist">
+<DT>dependent_varname</DT>
+ <DD>TEXT. Name of the dependent variable column.
+ For regression supported types are any numeric type, or array
+ or numeric types (for multiple regression).
+ </DD>
+</DL>
+
+
+@anchor optimizer_params
+@par Optimizer Parameters
+Parameters in this section are supplied in the \e optimizer_params argument as a string
+containing a comma-delimited list of name-value pairs. All of these named
+parameters are optional, and their order does not matter. You must use the
+format "<param_name> = <value>" to specify the value of a parameter, otherwise
+the parameter is ignored.
+
+
+<pre class="syntax">
+ 'step_size = <value>,
+ n_iterations = <value>,
+ n_tries = <value>,
+ tolerance = <value>'
+</pre>
+\b Optimizer Parameters
+<DL class="arglist">
+
+<DT>step_size</dt>
+<DD>Default: [0.001].
+Also known as the learning rate. A small value is usually desirable to
+ensure convergence, while a large value provides more room for progress during
+training. Since the best value depends on the condition number of the data, in
+practice one often tunes this parameter.
+</DD>
+
+
+<DT>n_iterations</dt>
+<DD>Default: [100]. The maximum number of iterations allowed.
+</DD>
+<DT>n_tries</dt>
+<DD>Default: [1]. Number of times to retrain the network with randomly initialized
+weights
+</DD>
+
+<DT>tolerance</dt>
+<DD>Default: 0.001. The criterion to end iterations. The training stops whenever
+<the difference between the training models of two consecutive iterations is
+<smaller than \e tolerance or the iteration number is larger than \e max_iter.
+</DD>
+
+</DL>
+
+@anchor predict
+@par Prediction Function
+Used to generate predictions given a previously trained model on novel data.
+The same syntax is used for classification, and regression.
+<pre class="syntax">
+mlp_predict(model_table,
+ data_table,
+ id_col_name,
+ output_table,
+ pred_type)
+</pre>
+
+\b Arguments
+<DL class="arglist">
+ <DT>model_table</DT>
+ <DD>TEXT. Model table produced by the training function.</DD>
+
+ <DT>data_table</DT>
+ <DD>TEXT. Name of the table containing the data for prediction. This table is expected
+ to contain the same input features that were used during training. The table should
+ also contain id_col_name used for identifying each row.</DD>
+
+ <DT>id_col_name</DT>
+ <DD>TEXT. The name of the id column in the input table.</DD>
+
+ <DT>output_table</DT>
+ <DD>TEXT. Name of the table where output predictions are written. If this
+table name is already in use, then an error is returned. Table contains:</DD>
+ <table class="output">
+ <tr>
+ <th>id</th>
+ <td>Gives the 'id' for each prediction, corresponding to each row from the data_table.</td>
+ </tr>
+ <tr>
+ <th>estimated_<COL_NAME></th>
+ <td>
+ (For pred_type='response') The estimated class
+ for classification or value for regression, where
+ <COL_NAME> is the name of the column to be
+ predicted from training data
+ </td>
+ </tr>
+ <tr>
+ <th>prob_<CLASS></th>
+ <td>
+ (For pred_type='prob' for classification) The
+ probability of a given class <CLASS> as given by
+ softmax. There will be one column for each class
+ in the training data.
+ </td>
+ </tr>
+
+
+ <DT>pred_type</DT>
+ <DD>TEXT.
+
+the type of output requested:
+'response' gives the actual prediction,
+'prob' gives the probability of each class.
+for regression, only type='response' is defined.
+The name of the id column in the input table.</DD>
+</DL>
+</table>
+
+@anchor example
+@par Examples
+-# Create an input data set.
+<pre class="example">
+CREATE TABLE iris_data(
+ id integer,
+ attributes numeric[],
+ class_text varchar,
+ class integer
+);
+INSERT INTO iris_data VALUES
+(1,ARRAY[5.1,3.5,1.4,0.2],'Iris-setosa',1),
+(2,ARRAY[4.9,3.0,1.4,0.2],'Iris-setosa',1),
+(3,ARRAY[4.7,3.2,1.3,0.2],'Iris-setosa',1),
+(4,ARRAY[4.6,3.1,1.5,0.2],'Iris-setosa',1),
+(5,ARRAY[5.0,3.6,1.4,0.2],'Iris-setosa',1),
+(6,ARRAY[5.4,3.9,1.7,0.4],'Iris-setosa',1),
+(7,ARRAY[4.6,3.4,1.4,0.3],'Iris-setosa',1),
+(8,ARRAY[5.0,3.4,1.5,0.2],'Iris-setosa',1),
+(9,ARRAY[4.4,2.9,1.4,0.2],'Iris-setosa',1),
+(10,ARRAY[4.9,3.1,1.5,0.1],'Iris-setosa',1),
+(11,ARRAY[7.0,3.2,4.7,1.4],'Iris-versicolor',2),
+(12,ARRAY[6.4,3.2,4.5,1.5],'Iris-versicolor',2),
+(13,ARRAY[6.9,3.1,4.9,1.5],'Iris-versicolor',2),
+(14,ARRAY[5.5,2.3,4.0,1.3],'Iris-versicolor',2),
+(15,ARRAY[6.5,2.8,4.6,1.5],'Iris-versicolor',2),
+(16,ARRAY[5.7,2.8,4.5,1.3],'Iris-versicolor',2),
+(17,ARRAY[6.3,3.3,4.7,1.6],'Iris-versicolor',2),
+(18,ARRAY[4.9,2.4,3.3,1.0],'Iris-versicolor',2),
+(19,ARRAY[6.6,2.9,4.6,1.3],'Iris-versicolor',2),
+(20,ARRAY[5.2,2.7,3.9,1.4],'Iris-versicolor',2);
+</pre>
+-# Generate a multilayer perceptron with a single hidden layer of 5 units.
+Use the attributes column as the independent variables, and use the class
+column as the classification. Set the tolerance to 0 so that 5000
+iterations will be run. Use a hyperbolic tangent activation function.
+The model will be written to mlp_model.
+<pre class="example">
+SELECT madlib.mlp_classification(
--- End diff --
In case user attempts to run multiple times, precede `SELECT` statement with:
```
DROP TABLE IF EXISTS mlp_model;
DROP TABLE IF EXISTS mlp_model_summary;
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib issue #149: MLP: Multilayer Perceptron
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/incubator-madlib/pull/149
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/madlib-pr-build/122/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by haying <gi...@git.apache.org>.
Github user haying commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r126795536
--- Diff: src/ports/postgres/dbconnector/TypeTraits_impl.hpp ---
@@ -750,6 +784,30 @@ struct TypeTraits<dbal::eigen_integration::SparseColumnVector> {
);
};
+template <>
+struct TypeTraits<std::vector<uint16_t> > {
+ typedef std::vector<uint16_t> value_type;
+
+ WITH_OID( INT4ARRAYOID );
+ WITH_MUTABILITY( dbal::Mutable );
+ WITH_DEFAULT_EXTENDED_TRAITS;
+ // FIXME a quick hack
--- End diff --
Could we add a bit more details in the comment saying the cons of this hack and a potential direction for a future fix?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib issue #149: MLP: Multilayer Perceptron
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/incubator-madlib/pull/149
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/madlib-pr-build/118/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by njayaram2 <gi...@git.apache.org>.
Github user njayaram2 commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r128065611
--- Diff: src/ports/postgres/modules/convex/mlp.sql_in ---
@@ -0,0 +1,752 @@
+/* ----------------------------------------------------------------------- *//**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ *
+ * @file mlp.sql_in
+ *
+ * @brief SQL functions for multilayer perceptron
+ * @date June 2012
+ *
+ *
+ *//* ----------------------------------------------------------------------- */
+
+m4_include(`SQLCommon.m4')
+
+/**
+@addtogroup grp_mlp
+
+<div class="toc"><b>Contents</b><ul>
+<li class="level1"><a href="#mlp_classification">Classification</a></li>
+<li class="level1"><a href="#mlp_regression">Regression</a></li>
+<li class="level1"><a href="#optimization_params">Optimizer Parameters</a></li>
+<li class="level1"><a href="#predict">Prediction Functions/a></li>
+<li class="level1"><a href="#example">Examples</a></li>
+<li class="level1"><a href="#background">Technical Background</a></li>
+<li class="level1"><a href="#literature">Literature</a></li>
+<li class="level1"><a href="#related">Related Topics</a></li>
+</ul></div>
+
+Multilayer Perceptron (MLP) is a model for regression and
+classification
+
+Also called "vanilla neural networks", they consist of several
+fully connected hidden layers with non-linear activation
+functions. In the case of classification, the final layer of the
+neural net has as many nodes as classes, and the output of the
+neural net can be interpreted as the probability of a given input
+feature belonging to a specific class.
+
+
+@brief Solves classification and regression problems with several
+fully connected layers and nonlinear activation functions.
+
+@anchor mlp_classification
+@par Classification Training Function
+The mlp classification training function has the following format:
+<pre class="syntax">
+mlp_classification(
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+\b Arguments
+<DL class="arglist">
+ <DT>source_table</DT>
+ <DD>TEXT. Name of the table containing the training data.</DD>
+
+ <DT>model_table</DT>
+ <DD>TEXT. Name of the output table containing the model. Details of the output
+ tables are provided below.
+ </DD>
+
+ <DT>independent_varname</DT>
+ <DD>TEXT. Expression list to evaluate for the
+ independent variables. An intercept variable should not be included as part
+ of this expression. Please note that expression should be able to be cast
+ to DOUBLE PRECISION[].
+ </DD>
+
+ <DT>dependent_varname</DT>
+ <DD> TEXT. Name of the dependent variable column. For classification, supported types are:
+ text, varchar, character varying, char, character
+ integer, smallint, bigint, and boolean. </DD>
+
+ <DT>hidden_layer_sizes</DT>
--- End diff --
Is this is also an optional param, since we have a default value? If yes, please add `(optional)` after the parameter label.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by njayaram2 <gi...@git.apache.org>.
Github user njayaram2 commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r128065248
--- Diff: src/ports/postgres/modules/convex/mlp.sql_in ---
@@ -0,0 +1,752 @@
+/* ----------------------------------------------------------------------- *//**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ *
+ * @file mlp.sql_in
+ *
+ * @brief SQL functions for multilayer perceptron
+ * @date June 2012
+ *
+ *
+ *//* ----------------------------------------------------------------------- */
+
+m4_include(`SQLCommon.m4')
+
+/**
+@addtogroup grp_mlp
+
+<div class="toc"><b>Contents</b><ul>
+<li class="level1"><a href="#mlp_classification">Classification</a></li>
+<li class="level1"><a href="#mlp_regression">Regression</a></li>
+<li class="level1"><a href="#optimization_params">Optimizer Parameters</a></li>
+<li class="level1"><a href="#predict">Prediction Functions/a></li>
+<li class="level1"><a href="#example">Examples</a></li>
+<li class="level1"><a href="#background">Technical Background</a></li>
+<li class="level1"><a href="#literature">Literature</a></li>
+<li class="level1"><a href="#related">Related Topics</a></li>
+</ul></div>
+
+Multilayer Perceptron (MLP) is a model for regression and
+classification
+
+Also called "vanilla neural networks", they consist of several
+fully connected hidden layers with non-linear activation
+functions. In the case of classification, the final layer of the
+neural net has as many nodes as classes, and the output of the
+neural net can be interpreted as the probability of a given input
+feature belonging to a specific class.
+
+
+@brief Solves classification and regression problems with several
+fully connected layers and nonlinear activation functions.
+
+@anchor mlp_classification
+@par Classification Training Function
+The mlp classification training function has the following format:
+<pre class="syntax">
+mlp_classification(
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+\b Arguments
+<DL class="arglist">
+ <DT>source_table</DT>
+ <DD>TEXT. Name of the table containing the training data.</DD>
+
+ <DT>model_table</DT>
--- End diff --
It is `model_table` here but `output_table` above.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by edespino <gi...@git.apache.org>.
Github user edespino commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r127045518
--- Diff: src/ports/postgres/modules/convex/mlp.sql_in ---
@@ -0,0 +1,736 @@
+/* ----------------------------------------------------------------------- *//**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ *
+ * @file mlp.sql_in
+ *
+ * @brief SQL functions for multi-layer perceptron
+ * @date June 2012
+ *
+ *
+ *//* ----------------------------------------------------------------------- */
+
+m4_include(`SQLCommon.m4')
+
+/**
+@addtogroup grp_mlp
+
+<div class="toc"><b>Contents</b><ul>
+<li class="level1"><a href="#mlp_classification">Classification</a></li>
+<li class="level1"><a href="#mlp_regression">Regression</a></li>
+<li class="level1"><a href="#optimization_params">Optimization Parameters</a></li>
--- End diff --
The references (2) to `[oO]ptimization` should be `[oO]ptimizer`. This is resulting in a link which does not resolve.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by cooper-sloan <gi...@git.apache.org>.
Github user cooper-sloan commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r127039918
--- Diff: src/ports/postgres/modules/convex/mlp.sql_in ---
@@ -0,0 +1,736 @@
+/* ----------------------------------------------------------------------- *//**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ *
+ * @file mlp.sql_in
+ *
+ * @brief SQL functions for multi-layer perceptron
+ * @date June 2012
+ *
+ *
+ *//* ----------------------------------------------------------------------- */
+
+m4_include(`SQLCommon.m4')
+
+/**
+@addtogroup grp_mlp
+
+<div class="toc"><b>Contents</b><ul>
+<li class="level1"><a href="#mlp_classification">Classification</a></li>
+<li class="level1"><a href="#mlp_regression">Regression</a></li>
+<li class="level1"><a href="#optimization_params">Optimization Parameters</a></li>
+<li class="level1"><a href="#predict">Prediction Functions/a></li>
+<li class="level1"><a href="#example">Examples</a></li>
+<li class="level1"><a href="#background">Technical Background</a></li>
+<li class="level1"><a href="#literature">Literature</a></li>
+<li class="level1"><a href="#related">Related Topics</a></li>
+</ul></div>
+
+Multi Layer Perceptron (MLP) is a model for regression and
+classification
+
+Also called "vanilla neural networks", they consist of several
+fully connected hidden layers with non-linear activation
+functions. In the case of classification, the final layer of the
+neural net has as many nodes as classes, and the output of the
+neural net can be interpreted as the probability of a given input
+feature belonging to a specific class.
+
+
+@brief Solves classification and regression problems with several
+fully connected layers and nonlinear activation functions.
+
+@anchor mlp_classification
+@par Classification Training Function
+The mlp classification training function has the following format:
+<pre class="syntax">
+mlp_classification(
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+\b Arguments
+<DL class="arglist">
+ <DT>source_table</DT>
+ <DD>TEXT. Name of the table containing the training data.</DD>
+
+ <DT>model_table</DT>
+ <DD>TEXT. Name of the output table containing the model. Details of the output
+ tables are provided below.
+ </DD>
+
+ <DT>independent_varname</DT>
+ <DD>TEXT. Expression list to evaluate for the
+ independent variables. An intercept variable should not be included as part
+ of this expression. Please note that expression should be able to be cast
+ to DOUBLE PRECISION[].
+ </DD>
+
+ <DT>dependent_varname</DT>
+ <DD> TEXT. Name of the dependent variable column. For classification, supported types are:
+ text, varchar, character varying, char, character
+ integer, smallint, bigint, and boolean. </DD>
+
+ <DT>hidden_layer_sizes</DT>
+ <DD>INTEGER[], default: ARRAY[].
+ The number of neurons in each hidden layer. The length of this array will
+ determine the number of hidden layers. Empty for no hidden layers.
+ </DD>
+
+
+ <DT>optimizer_params (optional)</DT>
+ <DD>TEXT, default: NULL.
+ Parameters for optimization in a comma-separated string
+ of key-value pairs. See the description below for details.
+ </DD>
+
+ <DT>activation</DT>
+ <DD>TEXT, default: 'sigmoid'.
+ Activation function. Currently three functions are supported: 'sigmoid' (default),
+ 'relu', and 'tanh'. The text can be any subset of the three
+ strings; for e.g., activation='s' will use the sigmoid activation.
+ </DD>
+</DL>
+
+<b>Output tables</b>
+<br>
+ The model table produced by mlp contains the following columns:
+ <table class="output">
+ <tr>
+ <th>coeffs</th>
+ <td>FLOAT8[]. Flat array containing the weights of the neural net</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>INTEGER. Number of iterations completed by stochastic gradient descent
+ algorithm. The algorithm either converged in this number of iterations
+ or hit the maximum number specified in the optimization parameters. </td>
+ </tr>
+ <tr>
+ <th>loss</th>
+ <td>FLOAT8. The cross entropy over the training data.
+ See Technical Background section below for more details.</td>
+ </tr>
+ </table>
+
+
+A summary table named \<model_table\>_summary is also created, which has the following columns:
+ <table class="output">
+ <tr>
+ <th>source_table</th>
+ <td>The source table.</td>
+ </tr>
+ <tr>
+ <th>dependent_varname</th>
+ <td>The dependent variable.</td>
+ </tr>
+ <tr>
+ <th>independent_varname</th>
+ <td>The independent variables.</td>
+ </tr>
+ <tr>
+ <th>tolerance</th>
+ <td>The tolerance as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>step_size</th>
+ <td>The step size as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>The number of iterations run</td>
+ </tr>
+ <tr>
+ <th>n_tries</th>
+ <td>The number of tries as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>layer_sizes</th>
+ <td>The number of units in each layer including the input and output layer.</td>
+ </tr>
+ <tr>
+ <th>activation_function</th>
+ <td>The activation function.</td>
+ </tr>
+ <tr>
+ <th>is_classification</th>
+ <td>True if the model was trained for classification, False if it was trained
+ for regression</td>
+ </tr>
+ <tr>
+ <th>classes</th>
+ <td>The classes which were trained against (empty for regression)</td>
+ </tr>
+
+ </table>
+
+
+@anchor mlp_regression
+@par Regression Training Function
+The mlp regression training function has the following format:
+<pre class="syntax">
+mlp_regression(source_table,
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+
+\b Arguments
+
+Specifications for regression are largely the same as for classification. In the
+model table, the loss will refer to mean square error instead of cross entropy. In the
+summary table, there is classes column. The following
+arguments have specifications which differ from mlp_classification:
+<DL class="arglist">
+<DT>dependent_varname</DT>
+ <DD>TEXT. Name of the dependent variable column.
+ For regression supported types are any numeric type, or array
+ or numeric types (for multiple regression).
+ </DD>
+</DL>
+
+
+@anchor optimizer_params
+@par Optimizer Parameters
+Parameters in this section are supplied in the \e optimizer_params argument as a string
+containing a comma-delimited list of name-value pairs. All of these named
+parameters are optional, and their order does not matter. You must use the
+format "<param_name> = <value>" to specify the value of a parameter, otherwise
+the parameter is ignored.
+
+
+<pre class="syntax">
+ 'step_size = <value>,
+ n_iterations = <value>,
+ n_tries = <value>,
+ tolerance = <value>'
+</pre>
+\b Optimizer Parameters
+<DL class="arglist">
+
+<DT>step_size</dt>
+<DD>Default: [0.001].
+Also known as the learning rate. A small value is usually desirable to
+ensure convergence, while a large value provides more room for progress during
+training. Since the best value depends on the condition number of the data, in
+practice one often tunes this parameter.
+</DD>
+
+
+<DT>n_iterations</dt>
+<DD>Default: [100]. The maximum number of iterations allowed.
+</DD>
+<DT>n_tries</dt>
+<DD>Default: [1]. Number of times to retrain the network with randomly initialized
+weights
+</DD>
+
+<DT>tolerance</dt>
+<DD>Default: 0.001. The criterion to end iterations. The training stops whenever
+<the difference between the training models of two consecutive iterations is
+<smaller than \e tolerance or the iteration number is larger than \e max_iter.
+</DD>
+
+</DL>
+
+@anchor predict
+@par Prediction Function
+Used to generate predictions given a previously trained model on novel data.
+The same syntax is used for classification, and regression.
+<pre class="syntax">
+mlp_predict(model_table,
+ data_table,
+ id_col_name,
+ output_table,
+ pred_type)
+</pre>
+
+\b Arguments
+<DL class="arglist">
+ <DT>model_table</DT>
+ <DD>TEXT. Model table produced by the training function.</DD>
+
+ <DT>data_table</DT>
+ <DD>TEXT. Name of the table containing the data for prediction. This table is expected
+ to contain the same input features that were used during training. The table should
+ also contain id_col_name used for identifying each row.</DD>
+
+ <DT>id_col_name</DT>
+ <DD>TEXT. The name of the id column in the input table.</DD>
+
+ <DT>output_table</DT>
+ <DD>TEXT. Name of the table where output predictions are written. If this
+table name is already in use, then an error is returned. Table contains:</DD>
+ <table class="output">
+ <tr>
+ <th>id</th>
+ <td>Gives the 'id' for each prediction, corresponding to each row from the data_table.</td>
+ </tr>
+ <tr>
+ <th>estimated_<COL_NAME></th>
+ <td>
+ (For pred_type='response') The estimated class
+ for classification or value for regression, where
+ <COL_NAME> is the name of the column to be
+ predicted from training data
+ </td>
+ </tr>
+ <tr>
+ <th>prob_<CLASS></th>
+ <td>
+ (For pred_type='prob' for classification) The
+ probability of a given class <CLASS> as given by
+ softmax. There will be one column for each class
+ in the training data.
+ </td>
+ </tr>
+
+
+ <DT>pred_type</DT>
+ <DD>TEXT.
+
+the type of output requested:
+'response' gives the actual prediction,
+'prob' gives the probability of each class.
+for regression, only type='response' is defined.
+The name of the id column in the input table.</DD>
+</DL>
+</table>
+
+@anchor example
+@par Examples
+-# Create an input data set.
+<pre class="example">
+CREATE TABLE iris_data(
+ id integer,
+ attributes numeric[],
+ class_text varchar,
+ class integer
+);
+INSERT INTO iris_data VALUES
+(1,ARRAY[5.1,3.5,1.4,0.2],'Iris-setosa',1),
+(2,ARRAY[4.9,3.0,1.4,0.2],'Iris-setosa',1),
+(3,ARRAY[4.7,3.2,1.3,0.2],'Iris-setosa',1),
+(4,ARRAY[4.6,3.1,1.5,0.2],'Iris-setosa',1),
+(5,ARRAY[5.0,3.6,1.4,0.2],'Iris-setosa',1),
+(6,ARRAY[5.4,3.9,1.7,0.4],'Iris-setosa',1),
+(7,ARRAY[4.6,3.4,1.4,0.3],'Iris-setosa',1),
+(8,ARRAY[5.0,3.4,1.5,0.2],'Iris-setosa',1),
+(9,ARRAY[4.4,2.9,1.4,0.2],'Iris-setosa',1),
+(10,ARRAY[4.9,3.1,1.5,0.1],'Iris-setosa',1),
+(11,ARRAY[7.0,3.2,4.7,1.4],'Iris-versicolor',2),
+(12,ARRAY[6.4,3.2,4.5,1.5],'Iris-versicolor',2),
+(13,ARRAY[6.9,3.1,4.9,1.5],'Iris-versicolor',2),
+(14,ARRAY[5.5,2.3,4.0,1.3],'Iris-versicolor',2),
+(15,ARRAY[6.5,2.8,4.6,1.5],'Iris-versicolor',2),
+(16,ARRAY[5.7,2.8,4.5,1.3],'Iris-versicolor',2),
+(17,ARRAY[6.3,3.3,4.7,1.6],'Iris-versicolor',2),
+(18,ARRAY[4.9,2.4,3.3,1.0],'Iris-versicolor',2),
+(19,ARRAY[6.6,2.9,4.6,1.3],'Iris-versicolor',2),
+(20,ARRAY[5.2,2.7,3.9,1.4],'Iris-versicolor',2);
+</pre>
+-# Generate a multilayer perceptron with a single hidden layer of 5 units.
+Use the attributes column as the independent variables, and use the class
+column as the classification. Set the tolerance to 0 so that 5000
+iterations will be run. Use a hyperbolic tangent activation function.
+The model will be written to mlp_result.
+<pre class="example">
+SELECT madlib.mlp_classification(
+ 'iris_data_2', -- Source table
+ 'mlp_model', -- Destination table
+ 'attributes', -- Input features
+ 'class_text', -- Label
+ ARRAY[5], -- Number of units per layer
+ 'step_size=0.003,
+ n_iterations=5000,
+ tolerance=0', -- Optimizer params
+ 'tanh'); -- Activation function
+</pre>
+-# View the result for the model.
+<pre class="example">
+-- Set extended display on for easier reading of output
+\\x ON
+-- Neural net Initialization is non-deterministic, so your results may vary
+SELECT * FROM mlp_model;
+</pre>
+Result:
+<pre class="result">
+-[ RECORD 1 ]--+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+coeff | {1,1,1,1,1,0.136374930803,0.188739676875,0.662387810001,-1.03381622734,-0.469961067046,0.0614006983397,0.0811504589436,0.299008228258,-0.47391918521,-0.215098143699,0.10519213944,0.145844617525,0.511683525606,-0.800215552382,-0.36417142683,0.120751709056,0.167531106521,0.587074895969,-0.916946198095,-0.417055067449,0.0539541885146,0.0694359704131,0.262598585854,-0.419234805076,-0.189915344282,1,1,1,1,1,1,0.105645702152,1.46247470474,0.484457903226,0.965962824478,1.19361986431,0.419805760087,-0.105696503487,-1.46245956666,-0.484427811691,-0.965730981426,-1.19365280555,-0.419973628863}
+loss | 0.0184092375519
+num_iterations | 5000
+</pre>
+-# Next train a regression example. First create some test data. This dataset
+contains housing prices data.
+<pre class="example">
+CREATE TABLE lin_housing (id serial, x float8[], grp_by_col int, y float8);
+COPY lin_housing (x, grp_by_col, y) FROM STDIN NULL '?' DELIMITER '|';
+{1,0.00632,18.00,2.310,0,0.5380,6.5750,65.20,4.0900,1,296.0,15.30,396.90,4.98} | 1 | 24.00
+{1,0.02731,0.00,7.070,0,0.4690,6.4210,78.90,4.9671,2,242.0,17.80,396.90,9.14} | 1 | 21.60
+{1,0.02729,0.00,7.070,0,0.4690,7.1850,61.10,4.9671,2,242.0,17.80,392.83,4.03} | 1 | 34.70
+{1,0.03237,0.00,2.180,0,0.4580,6.9980,45.80,6.0622,3,222.0,18.70,394.63,2.94} | 1 | 33.40
+{1,0.06905,0.00,2.180,0,0.4580,7.1470,54.20,6.0622,3,222.0,18.70,396.90,5.33} | 1 | 36.20
+{1,0.02985,0.00,2.180,0,0.4580,6.4300,58.70,6.0622,3,222.0,18.70,394.12,5.21} | 1 | 28.70
+{1,0.08829,12.50,7.870,0,0.5240,6.0120,66.60,5.5605,5,311.0,15.20,395.60,12.43} | 1 | 22.90
+{1,0.14455,12.50,7.870,0,0.5240,6.1720,96.10,5.9505,5,311.0,15.20,396.90,19.15} | 1 | 27.10
+{1,0.21124,12.50,7.870,0,0.5240,5.6310,100.00,6.0821,5,311.0,15.20,386.63,29.93} | 1 | 16.50
+{1,0.17004,12.50,7.870,0,0.5240,6.0040,85.90,6.5921,5,311.0,15.20,386.71,17.10} | 1 | 18.90
+{1,0.22489,12.50,7.870,0,0.5240,6.3770,94.30,6.3467,5,311.0,15.20,392.52,20.45} | 1 | 15.00
+{1,0.11747,12.50,7.870,0,0.5240,6.0090,82.90,6.2267,5,311.0,15.20,396.90,13.27} | 1 | 18.90
+{1,0.09378,12.50,7.870,0,0.5240,5.8890,39.00,5.4509,5,311.0,15.20,390.50,15.71} | 1 | 21.70
+{1,0.62976,0.00,8.140,0,0.5380,5.9490,61.80,4.7075,4,307.0,21.00,396.90,8.26} | 1 | 20.40
+{1,0.63796,0.00,8.140,0,0.5380,6.0960,84.50,4.4619,4,307.0,21.00,380.02,10.26} | 1 | 18.20
+{1,0.62739,0.00,8.140,0,0.5380,5.8340,56.50,4.4986,4,307.0,21.00,395.62,8.47} | 1 | 19.90
+{1,1.05393,0.00,8.140,0,0.5380,5.9350,29.30,4.4986,4,307.0,21.00,386.85,6.58} | 1 | 23.10
+{1,0.78420,0.00,8.140,0,0.5380,5.9900,81.70,4.2579,4,307.0,21.00,386.75,14.67} | 1 | 17.50
+{1,0.80271,0.00,8.140,0,0.5380,5.4560,36.60,3.7965,4,307.0,21.00,288.99,11.69} | 1 | 20.20
+{1,0.72580,0.00,8.140,0,0.5380,5.7270,69.50,3.7965,4,307.0,21.00,390.95,11.28} | 1 | 18.20
+\.
+</pre>
+-# Now train a regression model using a multilayer perceptron a single hidden layer of two nodes.
+<pre class="example">
+SELECT mlp_regression(
+ 'lin_housing', -- Source table
+ 'mlp_regress', -- Desination table
+ 'x', -- Input features
+ 'y', -- Dependent variable
+ ARRAY[5,5], -- Number of units per layer
+ 'step_size=0.000007,
+ n_iterations=5000,
+ tolerance=0',
+ 'relu');
+</pre>
+-# Check the results of the model
+<pre class="example">
+-- Set extended display on for easier reading of output
+\\x ON
+-- Neural net Initialization is non-deterministic, so your results may vary
+SELECT * FROM mlp_regress;
+</pre>
+Result:
+<pre class="result">
+-[ RECORD 1 ]--+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------
+coeff | {1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2.79506311399e-05,3.56715008915e-05,-6.09333559685e-05,0.000251228318768,-0.000224772841379,-3.71863030857e-05,-3.5757865148e-06,5.27936784854e-05,-2.48474166186e-05,6.19731184294e-05,3.07638968743e-05,6.8964698578e-06,0.000106016701083,-1.71484730318e-05,1.18691881812e-05,-0.000163975464208,0.000170026304906,3.11688265279e-05,0.000177050148787,-1.58265976603e-05,2.70144422657e-05,0.000112667883422,3.77575139073e-05,8.12474658795e-05,-7.90458917626e-05,0.000107566386158,-2.63771171506e-06,2.47996880915e-05,-0.00012642310887,0.000203827391081,0.000139315565565,4.86147243454e-05,-0.000176126471913,-6.47820782916e-05,-8.51592776447e-06,-6.60601176758e-05,2.91421874156e-05,6.3556873752e-05,0.000197557443129,0.000220531367259,0.000135036310289,0.000143735913975,-4.75034117786e-05,-0.000179547345838,-1.6919846786e-05,0.000162784312994,0.000268595819851,-0.000460066553287,8.69756071591e-05,-0.00311762727057,0.000126024763103,0.00020598824
2921,0.003463432426,-0.00729789075286,0.00151625867549,-0.000890852767597,-0.00525016037249,0.0031043106659,0.00798041103839,-0.00552693050079,0.0232180415786,0.0230489850143,-0.0437890272341,0.0165765426407,-0.248554261758,-7.81336427846e-05,0.00558145591752,0.283465844585,-0.571699956182,0.133474351994,-0.0785181945605,-0.419269930709,0.249547772912,0.631761009875,-0.431305975666,1,1,1,1,1,1,0.0158747497572,-9.02809160806e-05,0.00015574347618,4.10805373863e-06,0.00121532434965,0.101790351335,0.0647558401493,-0.00013654998677,-9.92872075948e-06,-5.5319694394e-05,0.00519320756484,0.412736586036,0.0011998026977,-1.53688189815e-05,1.94817888201e-05,-4.63111489966e-05,7.24547899029e-05,0.00880394144485,5.45309822095e-05,-0.000140943219275,-7.96211486227e-05,-1.04337307472e-05,0.000161936762028,0.00136273797767,-4.54737243585e-05,-3.4083840736e-05,3.69286883662e-05,9.9047243188e-08,3.75014011824e-06,-9.45366086368e-08,1,1,1,1,1,1,6.67488547054,0.102754199001,0.41668912471,0.008868672964
79,0.00136206007228,-9.88642499013e-05}
+loss | 144.965776158
+num_iterations | 10000
+</pre>
+-# Now let's look at the prediction functions. In the following examples we will
+use the training data set for prediction as well, which is not usual but serves to
+show the syntax. First we will test the classification exampel.
+The prediction is in the the estimated_class_text column with the
+actual value in the class_text column.
+<pre class="example">
+SELECT mlp_predict(
+ 'mlp_model', -- Model table
+ 'iris_data', -- Test data table
+ 'id', -- Id column in test table
+ 'mlp_prediction', -- Output table for predictions
+ 'response' -- Output classes, not probabilities
+ );
+SELECT * FROM mlp_prediction JOIN iris_data USING (id);
+</pre>
+Result:
+<pre class="result">
+ id | estimated_class_text | attributes | class_text | class
+----+----------------------+-------------------+-----------------+-------
+ 1 | Iris-setosa | {5.1,3.5,1.4,0.2} | Iris-setosa | 1
+ 2 | Iris-setosa | {4.9,3.0,1.4,0.2} | Iris-setosa | 1
+ 3 | Iris-setosa | {4.7,3.2,1.3,0.2} | Iris-setosa | 1
+ 4 | Iris-setosa | {4.6,3.1,1.5,0.2} | Iris-setosa | 1
+ 5 | Iris-setosa | {5.0,3.6,1.4,0.2} | Iris-setosa | 1
+ 6 | Iris-setosa | {5.4,3.9,1.7,0.4} | Iris-setosa | 1
+ 7 | Iris-setosa | {4.6,3.4,1.4,0.3} | Iris-setosa | 1
+ 8 | Iris-setosa | {5.0,3.4,1.5,0.2} | Iris-setosa | 1
+ 9 | Iris-setosa | {4.4,2.9,1.4,0.2} | Iris-setosa | 1
+ 10 | Iris-setosa | {4.9,3.1,1.5,0.1} | Iris-setosa | 1
+ 11 | Iris-versicolor | {7.0,3.2,4.7,1.4} | Iris-versicolor | 2
+ 12 | Iris-versicolor | {6.4,3.2,4.5,1.5} | Iris-versicolor | 2
+ 13 | Iris-versicolor | {6.9,3.1,4.9,1.5} | Iris-versicolor | 2
+ 14 | Iris-versicolor | {5.5,2.3,4.0,1.3} | Iris-versicolor | 2
+ 15 | Iris-versicolor | {6.5,2.8,4.6,1.5} | Iris-versicolor | 2
+ 16 | Iris-versicolor | {5.7,2.8,4.5,1.3} | Iris-versicolor | 2
+ 17 | Iris-versicolor | {6.3,3.3,4.7,1.6} | Iris-versicolor | 2
+ 18 | Iris-versicolor | {4.9,2.4,3.3,1.0} | Iris-versicolor | 2
+ 19 | Iris-versicolor | {6.6,2.9,4.6,1.3} | Iris-versicolor | 2
+ 20 | Iris-versicolor | {5.2,2.7,3.9,1.4} | Iris-versicolor | 2
+</pre>
+Prediction using the regression model:
+<pre class="example">
+SELECT mlp_predict(
+ 'mlp_regress', -- Model table
+ 'lin_housing', -- Test data table
+ 'id', -- Id column in test table
+ 'mlp_regress_prediction', -- Output table for predictions
+ 'response' -- Output values, not probabilities
+ );
+</pre>
+-# View results
+<pre class="example">
+SELECT madlib.mlp_predict('mlp_regress','lin_housing','id','mlp_regress_prediction','response');
+SELECT * FROM lin_housing JOIN mlp_regress_prediction USING (id);
+</pre>
+Result for the regression model:
+<pre class="result">
+ id | x | grp_by_col | y | estimated_y
+----+-------------------------------------------------------------------------+------------+------+--------------------
+ 1 | {1,0.00632,18,2.31,0,0.538,6.575,65.2,4.09,1,296,15.3,396.9,4.98} | 1 | 24 | {23.2627062018087}
+ 2 | {1,0.02731,0,7.07,0,0.469,6.421,78.9,4.9671,2,242,17.8,396.9,9.14} | 1 | 21.6 | {25.7088419115781}
+ 3 | {1,0.02729,0,7.07,0,0.469,7.185,61.1,4.9671,2,242,17.8,392.83,4.03} | 1 | 34.7 | {27.5587003901404}
+ 4 | {1,0.03237,0,2.18,0,0.458,6.998,45.8,6.0622,3,222,18.7,394.63,2.94} | 1 | 33.4 | {31.1812237427816}
+ 5 | {1,0.06905,0,2.18,0,0.458,7.147,54.2,6.0622,3,222,18.7,396.9,5.33} | 1 | 36.2 | {30.3696873085477}
+ 6 | {1,0.02985,0,2.18,0,0.458,6.43,58.7,6.0622,3,222,18.7,394.12,5.21} | 1 | 28.7 | {29.5290259241882}
+ 7 | {1,0.08829,12.5,7.87,0,0.524,6.012,66.6,5.5605,5,311,15.2,395.6,12.43} | 1 | 22.9 | {21.1576051716888}
+ 8 | {1,0.14455,12.5,7.87,0,0.524,6.172,96.1,5.9505,5,311,15.2,396.9,19.15} | 1 | 27.1 | {17.6194200563055}
+ 9 | {1,0.21124,12.5,7.87,0,0.524,5.631,100,6.0821,5,311,15.2,386.63,29.93} | 1 | 16.5 | {15.1366297774139}
+10 | {1,0.17004,12.5,7.87,0,0.524,6.004,85.9,6.5921,5,311,15.2,386.71,17.1} | 1 | 18.9 | {17.6528662199369}
+11 | {1,0.22489,12.5,7.87,0,0.524,6.377,94.3,6.3467,5,311,15.2,392.52,20.45} | 1 | 15 | {17.2017487668181}
+12 | {1,0.11747,12.5,7.87,0,0.524,6.009,82.9,6.2267,5,311,15.2,396.9,13.27} | 1 | 18.9 | {19.4893860319992}
+13 | {1,0.09378,12.5,7.87,0,0.524,5.889,39,5.4509,5,311,15.2,390.5,15.71} | 1 | 21.7 | {23.2917226708039}
+14 | {1,0.62976,0,8.14,0,0.538,5.949,61.8,4.7075,4,307,21,396.9,8.26} | 1 | 20.4 | {22.8904812605193}
+15 | {1,0.63796,0,8.14,0,0.538,6.096,84.5,4.4619,4,307,21,380.02,10.26} | 1 | 18.2 | {18.2386754423677}
+16 | {1,0.62739,0,8.14,0,0.538,5.834,56.5,4.4986,4,307,21,395.62,8.47} | 1 | 19.9 | {23.28949550874}
+17 | {1,1.05393,0,8.14,0,0.538,5.935,29.3,4.4986,4,307,21,386.85,6.58} | 1 | 23.1 | {25.3288762085473}
+18 | {1,0.7842,0,8.14,0,0.538,5.99,81.7,4.2579,4,307,21,386.75,14.67} | 1 | 17.5 | {19.0203738118451}
+19 | {1,0.80271,0,8.14,0,0.538,5.456,36.6,3.7965,4,307,21,288.99,11.69} | 1 | 20.2 | {12.3162005347545}
+20 | {1,0.7258,0,8.14,0,0.538,5.727,69.5,3.7965,4,307,21,390.95,11.28} | 1 | 18.2 | {21.0902211848747}
+</pre>
+Note that the results you get for all examples may vary with the platform you are using.
+
+@anchor background
+@par Technical Background
+
+To train a neural net, the respective loss function is minimized using stochastic gradient descent.
+In the case of classification, the loss function is cross entropy. For regression, mean square error
+is used. Weights in the neural net are updated via the backpropogation process, which uses dynamic
+programming to compute the partial derivative of each weight with respect to the overall loss. This
+partial derivative incorporates the respective activation function used, so this requires that the
+activation function be differentiable.
+
+@anchor literature
+@literature
+
+@anchor mlp-lit-1
+[1] Hastie, Trevor. Neural network. John Wiley & Sons, Ltd, 1998.
+
--- End diff --
These articles are referenced within the design doc.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib issue #149: MLP: Multilayer Perceptron
Posted by njayaram2 <gi...@git.apache.org>.
Github user njayaram2 commented on the issue:
https://github.com/apache/incubator-madlib/pull/149
jenkins ok to build
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-madlib/pull/149
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by edespino <gi...@git.apache.org>.
Github user edespino commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r127034752
--- Diff: src/ports/postgres/modules/convex/mlp.sql_in ---
@@ -0,0 +1,736 @@
+/* ----------------------------------------------------------------------- *//**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ *
+ * @file mlp.sql_in
+ *
+ * @brief SQL functions for multi-layer perceptron
+ * @date June 2012
+ *
+ *
+ *//* ----------------------------------------------------------------------- */
+
+m4_include(`SQLCommon.m4')
+
+/**
+@addtogroup grp_mlp
+
+<div class="toc"><b>Contents</b><ul>
+<li class="level1"><a href="#mlp_classification">Classification</a></li>
+<li class="level1"><a href="#mlp_regression">Regression</a></li>
+<li class="level1"><a href="#optimization_params">Optimization Parameters</a></li>
+<li class="level1"><a href="#predict">Prediction Functions/a></li>
+<li class="level1"><a href="#example">Examples</a></li>
+<li class="level1"><a href="#background">Technical Background</a></li>
+<li class="level1"><a href="#literature">Literature</a></li>
+<li class="level1"><a href="#related">Related Topics</a></li>
+</ul></div>
+
+Multi Layer Perceptron (MLP) is a model for regression and
+classification
+
+Also called "vanilla neural networks", they consist of several
+fully connected hidden layers with non-linear activation
+functions. In the case of classification, the final layer of the
+neural net has as many nodes as classes, and the output of the
+neural net can be interpreted as the probability of a given input
+feature belonging to a specific class.
+
+
+@brief Solves classification and regression problems with several
+fully connected layers and nonlinear activation functions.
+
+@anchor mlp_classification
+@par Classification Training Function
+The mlp classification training function has the following format:
+<pre class="syntax">
+mlp_classification(
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+\b Arguments
+<DL class="arglist">
+ <DT>source_table</DT>
+ <DD>TEXT. Name of the table containing the training data.</DD>
+
+ <DT>model_table</DT>
+ <DD>TEXT. Name of the output table containing the model. Details of the output
+ tables are provided below.
+ </DD>
+
+ <DT>independent_varname</DT>
+ <DD>TEXT. Expression list to evaluate for the
+ independent variables. An intercept variable should not be included as part
+ of this expression. Please note that expression should be able to be cast
+ to DOUBLE PRECISION[].
+ </DD>
+
+ <DT>dependent_varname</DT>
+ <DD> TEXT. Name of the dependent variable column. For classification, supported types are:
+ text, varchar, character varying, char, character
+ integer, smallint, bigint, and boolean. </DD>
+
+ <DT>hidden_layer_sizes</DT>
+ <DD>INTEGER[], default: ARRAY[].
+ The number of neurons in each hidden layer. The length of this array will
+ determine the number of hidden layers. Empty for no hidden layers.
+ </DD>
+
+
+ <DT>optimizer_params (optional)</DT>
+ <DD>TEXT, default: NULL.
+ Parameters for optimization in a comma-separated string
+ of key-value pairs. See the description below for details.
+ </DD>
+
+ <DT>activation</DT>
+ <DD>TEXT, default: 'sigmoid'.
+ Activation function. Currently three functions are supported: 'sigmoid' (default),
+ 'relu', and 'tanh'. The text can be any subset of the three
+ strings; for e.g., activation='s' will use the sigmoid activation.
+ </DD>
+</DL>
+
+<b>Output tables</b>
+<br>
+ The model table produced by mlp contains the following columns:
+ <table class="output">
+ <tr>
+ <th>coeffs</th>
+ <td>FLOAT8[]. Flat array containing the weights of the neural net</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>INTEGER. Number of iterations completed by stochastic gradient descent
+ algorithm. The algorithm either converged in this number of iterations
+ or hit the maximum number specified in the optimization parameters. </td>
+ </tr>
+ <tr>
+ <th>loss</th>
+ <td>FLOAT8. The cross entropy over the training data.
+ See Technical Background section below for more details.</td>
+ </tr>
+ </table>
+
+
+A summary table named \<model_table\>_summary is also created, which has the following columns:
+ <table class="output">
+ <tr>
+ <th>source_table</th>
+ <td>The source table.</td>
+ </tr>
+ <tr>
+ <th>dependent_varname</th>
+ <td>The dependent variable.</td>
+ </tr>
+ <tr>
+ <th>independent_varname</th>
+ <td>The independent variables.</td>
+ </tr>
+ <tr>
+ <th>tolerance</th>
+ <td>The tolerance as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>step_size</th>
+ <td>The step size as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>The number of iterations run</td>
+ </tr>
+ <tr>
+ <th>n_tries</th>
+ <td>The number of tries as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>layer_sizes</th>
+ <td>The number of units in each layer including the input and output layer.</td>
+ </tr>
+ <tr>
+ <th>activation_function</th>
+ <td>The activation function.</td>
+ </tr>
+ <tr>
+ <th>is_classification</th>
+ <td>True if the model was trained for classification, False if it was trained
+ for regression</td>
+ </tr>
+ <tr>
+ <th>classes</th>
+ <td>The classes which were trained against (empty for regression)</td>
+ </tr>
+
+ </table>
+
+
+@anchor mlp_regression
+@par Regression Training Function
+The mlp regression training function has the following format:
+<pre class="syntax">
+mlp_regression(source_table,
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+
+\b Arguments
+
+Specifications for regression are largely the same as for classification. In the
+model table, the loss will refer to mean square error instead of cross entropy. In the
+summary table, there is classes column. The following
+arguments have specifications which differ from mlp_classification:
+<DL class="arglist">
+<DT>dependent_varname</DT>
+ <DD>TEXT. Name of the dependent variable column.
+ For regression supported types are any numeric type, or array
+ or numeric types (for multiple regression).
+ </DD>
+</DL>
+
+
+@anchor optimizer_params
+@par Optimizer Parameters
+Parameters in this section are supplied in the \e optimizer_params argument as a string
+containing a comma-delimited list of name-value pairs. All of these named
+parameters are optional, and their order does not matter. You must use the
+format "<param_name> = <value>" to specify the value of a parameter, otherwise
+the parameter is ignored.
+
+
+<pre class="syntax">
+ 'step_size = <value>,
+ n_iterations = <value>,
+ n_tries = <value>,
+ tolerance = <value>'
+</pre>
+\b Optimizer Parameters
+<DL class="arglist">
+
+<DT>step_size</dt>
+<DD>Default: [0.001].
+Also known as the learning rate. A small value is usually desirable to
+ensure convergence, while a large value provides more room for progress during
+training. Since the best value depends on the condition number of the data, in
+practice one often tunes this parameter.
+</DD>
+
+
+<DT>n_iterations</dt>
+<DD>Default: [100]. The maximum number of iterations allowed.
+</DD>
+<DT>n_tries</dt>
+<DD>Default: [1]. Number of times to retrain the network with randomly initialized
+weights
+</DD>
+
+<DT>tolerance</dt>
+<DD>Default: 0.001. The criterion to end iterations. The training stops whenever
+<the difference between the training models of two consecutive iterations is
+<smaller than \e tolerance or the iteration number is larger than \e max_iter.
+</DD>
+
+</DL>
+
+@anchor predict
+@par Prediction Function
+Used to generate predictions given a previously trained model on novel data.
+The same syntax is used for classification, and regression.
+<pre class="syntax">
+mlp_predict(model_table,
+ data_table,
+ id_col_name,
+ output_table,
+ pred_type)
+</pre>
+
+\b Arguments
+<DL class="arglist">
+ <DT>model_table</DT>
+ <DD>TEXT. Model table produced by the training function.</DD>
+
+ <DT>data_table</DT>
+ <DD>TEXT. Name of the table containing the data for prediction. This table is expected
+ to contain the same input features that were used during training. The table should
+ also contain id_col_name used for identifying each row.</DD>
+
+ <DT>id_col_name</DT>
+ <DD>TEXT. The name of the id column in the input table.</DD>
+
+ <DT>output_table</DT>
+ <DD>TEXT. Name of the table where output predictions are written. If this
+table name is already in use, then an error is returned. Table contains:</DD>
+ <table class="output">
+ <tr>
+ <th>id</th>
+ <td>Gives the 'id' for each prediction, corresponding to each row from the data_table.</td>
+ </tr>
+ <tr>
+ <th>estimated_<COL_NAME></th>
+ <td>
+ (For pred_type='response') The estimated class
+ for classification or value for regression, where
+ <COL_NAME> is the name of the column to be
+ predicted from training data
+ </td>
+ </tr>
+ <tr>
+ <th>prob_<CLASS></th>
+ <td>
+ (For pred_type='prob' for classification) The
+ probability of a given class <CLASS> as given by
+ softmax. There will be one column for each class
+ in the training data.
+ </td>
+ </tr>
+
+
+ <DT>pred_type</DT>
+ <DD>TEXT.
+
+the type of output requested:
+'response' gives the actual prediction,
+'prob' gives the probability of each class.
+for regression, only type='response' is defined.
+The name of the id column in the input table.</DD>
+</DL>
+</table>
+
+@anchor example
+@par Examples
+-# Create an input data set.
+<pre class="example">
+CREATE TABLE iris_data(
+ id integer,
+ attributes numeric[],
+ class_text varchar,
+ class integer
+);
+INSERT INTO iris_data VALUES
+(1,ARRAY[5.1,3.5,1.4,0.2],'Iris-setosa',1),
+(2,ARRAY[4.9,3.0,1.4,0.2],'Iris-setosa',1),
+(3,ARRAY[4.7,3.2,1.3,0.2],'Iris-setosa',1),
+(4,ARRAY[4.6,3.1,1.5,0.2],'Iris-setosa',1),
+(5,ARRAY[5.0,3.6,1.4,0.2],'Iris-setosa',1),
+(6,ARRAY[5.4,3.9,1.7,0.4],'Iris-setosa',1),
+(7,ARRAY[4.6,3.4,1.4,0.3],'Iris-setosa',1),
+(8,ARRAY[5.0,3.4,1.5,0.2],'Iris-setosa',1),
+(9,ARRAY[4.4,2.9,1.4,0.2],'Iris-setosa',1),
+(10,ARRAY[4.9,3.1,1.5,0.1],'Iris-setosa',1),
+(11,ARRAY[7.0,3.2,4.7,1.4],'Iris-versicolor',2),
+(12,ARRAY[6.4,3.2,4.5,1.5],'Iris-versicolor',2),
+(13,ARRAY[6.9,3.1,4.9,1.5],'Iris-versicolor',2),
+(14,ARRAY[5.5,2.3,4.0,1.3],'Iris-versicolor',2),
+(15,ARRAY[6.5,2.8,4.6,1.5],'Iris-versicolor',2),
+(16,ARRAY[5.7,2.8,4.5,1.3],'Iris-versicolor',2),
+(17,ARRAY[6.3,3.3,4.7,1.6],'Iris-versicolor',2),
+(18,ARRAY[4.9,2.4,3.3,1.0],'Iris-versicolor',2),
+(19,ARRAY[6.6,2.9,4.6,1.3],'Iris-versicolor',2),
+(20,ARRAY[5.2,2.7,3.9,1.4],'Iris-versicolor',2);
+</pre>
+-# Generate a multilayer perceptron with a single hidden layer of 5 units.
+Use the attributes column as the independent variables, and use the class
+column as the classification. Set the tolerance to 0 so that 5000
+iterations will be run. Use a hyperbolic tangent activation function.
+The model will be written to mlp_result.
+<pre class="example">
+SELECT madlib.mlp_classification(
+ 'iris_data_2', -- Source table
+ 'mlp_model', -- Destination table
+ 'attributes', -- Input features
+ 'class_text', -- Label
+ ARRAY[5], -- Number of units per layer
+ 'step_size=0.003,
+ n_iterations=5000,
+ tolerance=0', -- Optimizer params
+ 'tanh'); -- Activation function
+</pre>
+-# View the result for the model.
+<pre class="example">
+-- Set extended display on for easier reading of output
+\\x ON
+-- Neural net Initialization is non-deterministic, so your results may vary
+SELECT * FROM mlp_model;
+</pre>
+Result:
+<pre class="result">
+-[ RECORD 1 ]--+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+coeff | {1,1,1,1,1,0.136374930803,0.188739676875,0.662387810001,-1.03381622734,-0.469961067046,0.0614006983397,0.0811504589436,0.299008228258,-0.47391918521,-0.215098143699,0.10519213944,0.145844617525,0.511683525606,-0.800215552382,-0.36417142683,0.120751709056,0.167531106521,0.587074895969,-0.916946198095,-0.417055067449,0.0539541885146,0.0694359704131,0.262598585854,-0.419234805076,-0.189915344282,1,1,1,1,1,1,0.105645702152,1.46247470474,0.484457903226,0.965962824478,1.19361986431,0.419805760087,-0.105696503487,-1.46245956666,-0.484427811691,-0.965730981426,-1.19365280555,-0.419973628863}
+loss | 0.0184092375519
+num_iterations | 5000
+</pre>
+-# Next train a regression example. First create some test data. This dataset
+contains housing prices data.
+<pre class="example">
+CREATE TABLE lin_housing (id serial, x float8[], grp_by_col int, y float8);
+COPY lin_housing (x, grp_by_col, y) FROM STDIN NULL '?' DELIMITER '|';
+{1,0.00632,18.00,2.310,0,0.5380,6.5750,65.20,4.0900,1,296.0,15.30,396.90,4.98} | 1 | 24.00
+{1,0.02731,0.00,7.070,0,0.4690,6.4210,78.90,4.9671,2,242.0,17.80,396.90,9.14} | 1 | 21.60
+{1,0.02729,0.00,7.070,0,0.4690,7.1850,61.10,4.9671,2,242.0,17.80,392.83,4.03} | 1 | 34.70
+{1,0.03237,0.00,2.180,0,0.4580,6.9980,45.80,6.0622,3,222.0,18.70,394.63,2.94} | 1 | 33.40
+{1,0.06905,0.00,2.180,0,0.4580,7.1470,54.20,6.0622,3,222.0,18.70,396.90,5.33} | 1 | 36.20
+{1,0.02985,0.00,2.180,0,0.4580,6.4300,58.70,6.0622,3,222.0,18.70,394.12,5.21} | 1 | 28.70
+{1,0.08829,12.50,7.870,0,0.5240,6.0120,66.60,5.5605,5,311.0,15.20,395.60,12.43} | 1 | 22.90
+{1,0.14455,12.50,7.870,0,0.5240,6.1720,96.10,5.9505,5,311.0,15.20,396.90,19.15} | 1 | 27.10
+{1,0.21124,12.50,7.870,0,0.5240,5.6310,100.00,6.0821,5,311.0,15.20,386.63,29.93} | 1 | 16.50
+{1,0.17004,12.50,7.870,0,0.5240,6.0040,85.90,6.5921,5,311.0,15.20,386.71,17.10} | 1 | 18.90
+{1,0.22489,12.50,7.870,0,0.5240,6.3770,94.30,6.3467,5,311.0,15.20,392.52,20.45} | 1 | 15.00
+{1,0.11747,12.50,7.870,0,0.5240,6.0090,82.90,6.2267,5,311.0,15.20,396.90,13.27} | 1 | 18.90
+{1,0.09378,12.50,7.870,0,0.5240,5.8890,39.00,5.4509,5,311.0,15.20,390.50,15.71} | 1 | 21.70
+{1,0.62976,0.00,8.140,0,0.5380,5.9490,61.80,4.7075,4,307.0,21.00,396.90,8.26} | 1 | 20.40
+{1,0.63796,0.00,8.140,0,0.5380,6.0960,84.50,4.4619,4,307.0,21.00,380.02,10.26} | 1 | 18.20
+{1,0.62739,0.00,8.140,0,0.5380,5.8340,56.50,4.4986,4,307.0,21.00,395.62,8.47} | 1 | 19.90
+{1,1.05393,0.00,8.140,0,0.5380,5.9350,29.30,4.4986,4,307.0,21.00,386.85,6.58} | 1 | 23.10
+{1,0.78420,0.00,8.140,0,0.5380,5.9900,81.70,4.2579,4,307.0,21.00,386.75,14.67} | 1 | 17.50
+{1,0.80271,0.00,8.140,0,0.5380,5.4560,36.60,3.7965,4,307.0,21.00,288.99,11.69} | 1 | 20.20
+{1,0.72580,0.00,8.140,0,0.5380,5.7270,69.50,3.7965,4,307.0,21.00,390.95,11.28} | 1 | 18.20
+\.
+</pre>
+-# Now train a regression model using a multilayer perceptron a single hidden layer of two nodes.
+<pre class="example">
+SELECT mlp_regression(
+ 'lin_housing', -- Source table
+ 'mlp_regress', -- Desination table
+ 'x', -- Input features
+ 'y', -- Dependent variable
+ ARRAY[5,5], -- Number of units per layer
+ 'step_size=0.000007,
+ n_iterations=5000,
+ tolerance=0',
+ 'relu');
+</pre>
+-# Check the results of the model
+<pre class="example">
+-- Set extended display on for easier reading of output
+\\x ON
+-- Neural net Initialization is non-deterministic, so your results may vary
+SELECT * FROM mlp_regress;
+</pre>
+Result:
+<pre class="result">
+-[ RECORD 1 ]--+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------
+coeff | {1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2.79506311399e-05,3.56715008915e-05,-6.09333559685e-05,0.000251228318768,-0.000224772841379,-3.71863030857e-05,-3.5757865148e-06,5.27936784854e-05,-2.48474166186e-05,6.19731184294e-05,3.07638968743e-05,6.8964698578e-06,0.000106016701083,-1.71484730318e-05,1.18691881812e-05,-0.000163975464208,0.000170026304906,3.11688265279e-05,0.000177050148787,-1.58265976603e-05,2.70144422657e-05,0.000112667883422,3.77575139073e-05,8.12474658795e-05,-7.90458917626e-05,0.000107566386158,-2.63771171506e-06,2.47996880915e-05,-0.00012642310887,0.000203827391081,0.000139315565565,4.86147243454e-05,-0.000176126471913,-6.47820782916e-05,-8.51592776447e-06,-6.60601176758e-05,2.91421874156e-05,6.3556873752e-05,0.000197557443129,0.000220531367259,0.000135036310289,0.000143735913975,-4.75034117786e-05,-0.000179547345838,-1.6919846786e-05,0.000162784312994,0.000268595819851,-0.000460066553287,8.69756071591e-05,-0.00311762727057,0.000126024763103,0.00020598824
2921,0.003463432426,-0.00729789075286,0.00151625867549,-0.000890852767597,-0.00525016037249,0.0031043106659,0.00798041103839,-0.00552693050079,0.0232180415786,0.0230489850143,-0.0437890272341,0.0165765426407,-0.248554261758,-7.81336427846e-05,0.00558145591752,0.283465844585,-0.571699956182,0.133474351994,-0.0785181945605,-0.419269930709,0.249547772912,0.631761009875,-0.431305975666,1,1,1,1,1,1,0.0158747497572,-9.02809160806e-05,0.00015574347618,4.10805373863e-06,0.00121532434965,0.101790351335,0.0647558401493,-0.00013654998677,-9.92872075948e-06,-5.5319694394e-05,0.00519320756484,0.412736586036,0.0011998026977,-1.53688189815e-05,1.94817888201e-05,-4.63111489966e-05,7.24547899029e-05,0.00880394144485,5.45309822095e-05,-0.000140943219275,-7.96211486227e-05,-1.04337307472e-05,0.000161936762028,0.00136273797767,-4.54737243585e-05,-3.4083840736e-05,3.69286883662e-05,9.9047243188e-08,3.75014011824e-06,-9.45366086368e-08,1,1,1,1,1,1,6.67488547054,0.102754199001,0.41668912471,0.008868672964
79,0.00136206007228,-9.88642499013e-05}
+loss | 144.965776158
+num_iterations | 10000
+</pre>
+-# Now let's look at the prediction functions. In the following examples we will
+use the training data set for prediction as well, which is not usual but serves to
+show the syntax. First we will test the classification exampel.
+The prediction is in the the estimated_class_text column with the
+actual value in the class_text column.
+<pre class="example">
+SELECT mlp_predict(
--- End diff --
For consistency sake, should the reference to `mlp_predict` be `madlib.mlp_predict`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib issue #149: MLP: Multilayer Perceptron
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/incubator-madlib/pull/149
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/madlib-pr-build/109/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by edespino <gi...@git.apache.org>.
Github user edespino commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r127314140
--- Diff: src/ports/postgres/modules/convex/mlp.sql_in ---
@@ -0,0 +1,744 @@
+/* ----------------------------------------------------------------------- *//**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ *
+ * @file mlp.sql_in
+ *
+ * @brief SQL functions for multilayer perceptron
+ * @date June 2012
+ *
+ *
+ *//* ----------------------------------------------------------------------- */
+
+m4_include(`SQLCommon.m4')
+
+/**
+@addtogroup grp_mlp
+
+<div class="toc"><b>Contents</b><ul>
+<li class="level1"><a href="#mlp_classification">Classification</a></li>
+<li class="level1"><a href="#mlp_regression">Regression</a></li>
+<li class="level1"><a href="#optimization_params">Optimizer Parameters</a></li>
+<li class="level1"><a href="#predict">Prediction Functions/a></li>
+<li class="level1"><a href="#example">Examples</a></li>
+<li class="level1"><a href="#background">Technical Background</a></li>
+<li class="level1"><a href="#literature">Literature</a></li>
+<li class="level1"><a href="#related">Related Topics</a></li>
+</ul></div>
+
+Multilayer Perceptron (MLP) is a model for regression and
+classification
+
+Also called "vanilla neural networks", they consist of several
+fully connected hidden layers with non-linear activation
+functions. In the case of classification, the final layer of the
+neural net has as many nodes as classes, and the output of the
+neural net can be interpreted as the probability of a given input
+feature belonging to a specific class.
+
+
+@brief Solves classification and regression problems with several
+fully connected layers and nonlinear activation functions.
+
+@anchor mlp_classification
+@par Classification Training Function
+The mlp classification training function has the following format:
+<pre class="syntax">
+mlp_classification(
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+\b Arguments
+<DL class="arglist">
+ <DT>source_table</DT>
+ <DD>TEXT. Name of the table containing the training data.</DD>
+
+ <DT>model_table</DT>
+ <DD>TEXT. Name of the output table containing the model. Details of the output
+ tables are provided below.
+ </DD>
+
+ <DT>independent_varname</DT>
+ <DD>TEXT. Expression list to evaluate for the
+ independent variables. An intercept variable should not be included as part
+ of this expression. Please note that expression should be able to be cast
+ to DOUBLE PRECISION[].
+ </DD>
+
+ <DT>dependent_varname</DT>
+ <DD> TEXT. Name of the dependent variable column. For classification, supported types are:
+ text, varchar, character varying, char, character
+ integer, smallint, bigint, and boolean. </DD>
+
+ <DT>hidden_layer_sizes</DT>
+ <DD>INTEGER[], default: ARRAY[].
+ The number of neurons in each hidden layer. The length of this array will
+ determine the number of hidden layers. Empty for no hidden layers.
+ </DD>
+
+
+ <DT>optimizer_params (optional)</DT>
+ <DD>TEXT, default: NULL.
+ Parameters for optimization in a comma-separated string
+ of key-value pairs. See the description below for details.
+ </DD>
+
+ <DT>activation</DT>
+ <DD>TEXT, default: 'sigmoid'.
+ Activation function. Currently three functions are supported: 'sigmoid' (default),
+ 'relu', and 'tanh'. The text can be any subset of the three
+ strings; for e.g., activation='s' will use the sigmoid activation.
+ </DD>
+</DL>
+
+<b>Output tables</b>
+<br>
+ The model table produced by mlp contains the following columns:
+ <table class="output">
+ <tr>
+ <th>coeffs</th>
+ <td>FLOAT8[]. Flat array containing the weights of the neural net</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>INTEGER. Number of iterations completed by stochastic gradient descent
+ algorithm. The algorithm either converged in this number of iterations
+ or hit the maximum number specified in the optimization parameters. </td>
+ </tr>
+ <tr>
+ <th>loss</th>
+ <td>FLOAT8. The cross entropy over the training data.
+ See Technical Background section below for more details.</td>
+ </tr>
+ </table>
+
+
+A summary table named \<model_table\>_summary is also created, which has the following columns:
+ <table class="output">
+ <tr>
+ <th>source_table</th>
+ <td>The source table.</td>
+ </tr>
+ <tr>
+ <th>dependent_varname</th>
+ <td>The dependent variable.</td>
+ </tr>
+ <tr>
+ <th>independent_varname</th>
+ <td>The independent variables.</td>
+ </tr>
+ <tr>
+ <th>tolerance</th>
+ <td>The tolerance as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>step_size</th>
+ <td>The step size as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>The number of iterations run</td>
+ </tr>
+ <tr>
+ <th>n_tries</th>
+ <td>The number of tries as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>layer_sizes</th>
+ <td>The number of units in each layer including the input and output layer.</td>
+ </tr>
+ <tr>
+ <th>activation_function</th>
+ <td>The activation function.</td>
+ </tr>
+ <tr>
+ <th>is_classification</th>
+ <td>True if the model was trained for classification, False if it was trained
+ for regression</td>
+ </tr>
+ <tr>
+ <th>classes</th>
+ <td>The classes which were trained against (empty for regression)</td>
+ </tr>
+
+ </table>
+
+
+@anchor mlp_regression
+@par Regression Training Function
+The mlp regression training function has the following format:
+<pre class="syntax">
+mlp_regression(source_table,
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+
+\b Arguments
+
+Specifications for regression are largely the same as for classification. In the
+model table, the loss will refer to mean square error instead of cross entropy. In the
+summary table, there is classes column. The following
+arguments have specifications which differ from mlp_classification:
+<DL class="arglist">
+<DT>dependent_varname</DT>
+ <DD>TEXT. Name of the dependent variable column.
+ For regression supported types are any numeric type, or array
+ or numeric types (for multiple regression).
+ </DD>
+</DL>
+
+
+@anchor optimizer_params
+@par Optimizer Parameters
+Parameters in this section are supplied in the \e optimizer_params argument as a string
+containing a comma-delimited list of name-value pairs. All of these named
+parameters are optional, and their order does not matter. You must use the
+format "<param_name> = <value>" to specify the value of a parameter, otherwise
+the parameter is ignored.
+
+
+<pre class="syntax">
+ 'step_size = <value>,
+ n_iterations = <value>,
+ n_tries = <value>,
+ tolerance = <value>'
+</pre>
+\b Optimizer Parameters
+<DL class="arglist">
+
+<DT>step_size</dt>
+<DD>Default: [0.001].
+Also known as the learning rate. A small value is usually desirable to
+ensure convergence, while a large value provides more room for progress during
+training. Since the best value depends on the condition number of the data, in
+practice one often tunes this parameter.
+</DD>
+
+
+<DT>n_iterations</dt>
+<DD>Default: [100]. The maximum number of iterations allowed.
+</DD>
+<DT>n_tries</dt>
+<DD>Default: [1]. Number of times to retrain the network with randomly initialized
+weights
+</DD>
+
+<DT>tolerance</dt>
+<DD>Default: 0.001. The criterion to end iterations. The training stops whenever
+<the difference between the training models of two consecutive iterations is
+<smaller than \e tolerance or the iteration number is larger than \e max_iter.
+</DD>
+
+</DL>
+
+@anchor predict
+@par Prediction Function
+Used to generate predictions given a previously trained model on novel data.
+The same syntax is used for classification, and regression.
+<pre class="syntax">
+mlp_predict(model_table,
+ data_table,
+ id_col_name,
+ output_table,
+ pred_type)
+</pre>
+
+\b Arguments
+<DL class="arglist">
+ <DT>model_table</DT>
+ <DD>TEXT. Model table produced by the training function.</DD>
+
+ <DT>data_table</DT>
+ <DD>TEXT. Name of the table containing the data for prediction. This table is expected
+ to contain the same input features that were used during training. The table should
+ also contain id_col_name used for identifying each row.</DD>
+
+ <DT>id_col_name</DT>
+ <DD>TEXT. The name of the id column in the input table.</DD>
+
+ <DT>output_table</DT>
+ <DD>TEXT. Name of the table where output predictions are written. If this
+table name is already in use, then an error is returned. Table contains:</DD>
+ <table class="output">
+ <tr>
+ <th>id</th>
+ <td>Gives the 'id' for each prediction, corresponding to each row from the data_table.</td>
+ </tr>
+ <tr>
+ <th>estimated_<COL_NAME></th>
+ <td>
+ (For pred_type='response') The estimated class
+ for classification or value for regression, where
+ <COL_NAME> is the name of the column to be
+ predicted from training data
+ </td>
+ </tr>
+ <tr>
+ <th>prob_<CLASS></th>
+ <td>
+ (For pred_type='prob' for classification) The
+ probability of a given class <CLASS> as given by
+ softmax. There will be one column for each class
+ in the training data.
+ </td>
+ </tr>
+
+
+ <DT>pred_type</DT>
+ <DD>TEXT.
+
+the type of output requested:
+'response' gives the actual prediction,
+'prob' gives the probability of each class.
+for regression, only type='response' is defined.
+The name of the id column in the input table.</DD>
+</DL>
+</table>
+
+@anchor example
+@par Examples
+-# Create an input data set.
+<pre class="example">
+CREATE TABLE iris_data(
+ id integer,
+ attributes numeric[],
+ class_text varchar,
+ class integer
+);
+INSERT INTO iris_data VALUES
+(1,ARRAY[5.1,3.5,1.4,0.2],'Iris-setosa',1),
+(2,ARRAY[4.9,3.0,1.4,0.2],'Iris-setosa',1),
+(3,ARRAY[4.7,3.2,1.3,0.2],'Iris-setosa',1),
+(4,ARRAY[4.6,3.1,1.5,0.2],'Iris-setosa',1),
+(5,ARRAY[5.0,3.6,1.4,0.2],'Iris-setosa',1),
+(6,ARRAY[5.4,3.9,1.7,0.4],'Iris-setosa',1),
+(7,ARRAY[4.6,3.4,1.4,0.3],'Iris-setosa',1),
+(8,ARRAY[5.0,3.4,1.5,0.2],'Iris-setosa',1),
+(9,ARRAY[4.4,2.9,1.4,0.2],'Iris-setosa',1),
+(10,ARRAY[4.9,3.1,1.5,0.1],'Iris-setosa',1),
+(11,ARRAY[7.0,3.2,4.7,1.4],'Iris-versicolor',2),
+(12,ARRAY[6.4,3.2,4.5,1.5],'Iris-versicolor',2),
+(13,ARRAY[6.9,3.1,4.9,1.5],'Iris-versicolor',2),
+(14,ARRAY[5.5,2.3,4.0,1.3],'Iris-versicolor',2),
+(15,ARRAY[6.5,2.8,4.6,1.5],'Iris-versicolor',2),
+(16,ARRAY[5.7,2.8,4.5,1.3],'Iris-versicolor',2),
+(17,ARRAY[6.3,3.3,4.7,1.6],'Iris-versicolor',2),
+(18,ARRAY[4.9,2.4,3.3,1.0],'Iris-versicolor',2),
+(19,ARRAY[6.6,2.9,4.6,1.3],'Iris-versicolor',2),
+(20,ARRAY[5.2,2.7,3.9,1.4],'Iris-versicolor',2);
+</pre>
+-# Generate a multilayer perceptron with a single hidden layer of 5 units.
+Use the attributes column as the independent variables, and use the class
+column as the classification. Set the tolerance to 0 so that 5000
+iterations will be run. Use a hyperbolic tangent activation function.
+The model will be written to mlp_model.
+<pre class="example">
+SELECT madlib.mlp_classification(
+ 'iris_data', -- Source table
+ 'mlp_model', -- Destination table
+ 'attributes', -- Input features
+ 'class_text', -- Label
+ ARRAY[5], -- Number of units per layer
+ 'step_size=0.003,
+ n_iterations=5000,
+ tolerance=0', -- Optimizer params
+ 'tanh'); -- Activation function
+</pre>
+-# View the result for the model.
+<pre class="example">
+-- Set extended display on for easier reading of output
+\\x ON
+-- Neural net Initialization is non-deterministic, so your results may vary
+SELECT * FROM mlp_model;
+</pre>
+Result:
+<pre class="result">
+-[ RECORD 1 ]--+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+coeff | {1,1,1,1,1,0.136374930803,0.188739676875,0.662387810001,-1.03381622734,-0.469961067046,0.0614006983397,0.0811504589436,0.299008228258,-0.47391918521,-0.215098143699,0.10519213944,0.145844617525,0.511683525606,-0.800215552382,-0.36417142683,0.120751709056,0.167531106521,0.587074895969,-0.916946198095,-0.417055067449,0.0539541885146,0.0694359704131,0.262598585854,-0.419234805076,-0.189915344282,1,1,1,1,1,1,0.105645702152,1.46247470474,0.484457903226,0.965962824478,1.19361986431,0.419805760087,-0.105696503487,-1.46245956666,-0.484427811691,-0.965730981426,-1.19365280555,-0.419973628863}
+loss | 0.0184092375519
+num_iterations | 5000
+</pre>
+-# Next train a regression example. First create some test data. This dataset
+contains housing prices data.
+<pre class="example">
+CREATE TABLE lin_housing (id serial, x float8[], grp_by_col int, y float8);
+COPY lin_housing (x, grp_by_col, y) FROM STDIN NULL '?' DELIMITER '|';
+{1,0.00632,18.00,2.310,0,0.5380,6.5750,65.20,4.0900,1,296.0,15.30,396.90,4.98} | 1 | 24.00
+{1,0.02731,0.00,7.070,0,0.4690,6.4210,78.90,4.9671,2,242.0,17.80,396.90,9.14} | 1 | 21.60
+{1,0.02729,0.00,7.070,0,0.4690,7.1850,61.10,4.9671,2,242.0,17.80,392.83,4.03} | 1 | 34.70
+{1,0.03237,0.00,2.180,0,0.4580,6.9980,45.80,6.0622,3,222.0,18.70,394.63,2.94} | 1 | 33.40
+{1,0.06905,0.00,2.180,0,0.4580,7.1470,54.20,6.0622,3,222.0,18.70,396.90,5.33} | 1 | 36.20
+{1,0.02985,0.00,2.180,0,0.4580,6.4300,58.70,6.0622,3,222.0,18.70,394.12,5.21} | 1 | 28.70
+{1,0.08829,12.50,7.870,0,0.5240,6.0120,66.60,5.5605,5,311.0,15.20,395.60,12.43} | 1 | 22.90
+{1,0.14455,12.50,7.870,0,0.5240,6.1720,96.10,5.9505,5,311.0,15.20,396.90,19.15} | 1 | 27.10
+{1,0.21124,12.50,7.870,0,0.5240,5.6310,100.00,6.0821,5,311.0,15.20,386.63,29.93} | 1 | 16.50
+{1,0.17004,12.50,7.870,0,0.5240,6.0040,85.90,6.5921,5,311.0,15.20,386.71,17.10} | 1 | 18.90
+{1,0.22489,12.50,7.870,0,0.5240,6.3770,94.30,6.3467,5,311.0,15.20,392.52,20.45} | 1 | 15.00
+{1,0.11747,12.50,7.870,0,0.5240,6.0090,82.90,6.2267,5,311.0,15.20,396.90,13.27} | 1 | 18.90
+{1,0.09378,12.50,7.870,0,0.5240,5.8890,39.00,5.4509,5,311.0,15.20,390.50,15.71} | 1 | 21.70
+{1,0.62976,0.00,8.140,0,0.5380,5.9490,61.80,4.7075,4,307.0,21.00,396.90,8.26} | 1 | 20.40
+{1,0.63796,0.00,8.140,0,0.5380,6.0960,84.50,4.4619,4,307.0,21.00,380.02,10.26} | 1 | 18.20
+{1,0.62739,0.00,8.140,0,0.5380,5.8340,56.50,4.4986,4,307.0,21.00,395.62,8.47} | 1 | 19.90
+{1,1.05393,0.00,8.140,0,0.5380,5.9350,29.30,4.4986,4,307.0,21.00,386.85,6.58} | 1 | 23.10
+{1,0.78420,0.00,8.140,0,0.5380,5.9900,81.70,4.2579,4,307.0,21.00,386.75,14.67} | 1 | 17.50
+{1,0.80271,0.00,8.140,0,0.5380,5.4560,36.60,3.7965,4,307.0,21.00,288.99,11.69} | 1 | 20.20
+{1,0.72580,0.00,8.140,0,0.5380,5.7270,69.50,3.7965,4,307.0,21.00,390.95,11.28} | 1 | 18.20
+\.
--- End diff --
The leading slack (`\`) character is getting lost and this results in a copy command which cannot run properly. Please change this from `\.` to `\\.` to correct the issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by njayaram2 <gi...@git.apache.org>.
Github user njayaram2 commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r128065854
--- Diff: src/ports/postgres/modules/convex/mlp.sql_in ---
@@ -0,0 +1,752 @@
+/* ----------------------------------------------------------------------- *//**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ *
+ * @file mlp.sql_in
+ *
+ * @brief SQL functions for multilayer perceptron
+ * @date June 2012
+ *
+ *
+ *//* ----------------------------------------------------------------------- */
+
+m4_include(`SQLCommon.m4')
+
+/**
+@addtogroup grp_mlp
+
+<div class="toc"><b>Contents</b><ul>
+<li class="level1"><a href="#mlp_classification">Classification</a></li>
+<li class="level1"><a href="#mlp_regression">Regression</a></li>
+<li class="level1"><a href="#optimization_params">Optimizer Parameters</a></li>
+<li class="level1"><a href="#predict">Prediction Functions/a></li>
+<li class="level1"><a href="#example">Examples</a></li>
+<li class="level1"><a href="#background">Technical Background</a></li>
+<li class="level1"><a href="#literature">Literature</a></li>
+<li class="level1"><a href="#related">Related Topics</a></li>
+</ul></div>
+
+Multilayer Perceptron (MLP) is a model for regression and
+classification
+
+Also called "vanilla neural networks", they consist of several
+fully connected hidden layers with non-linear activation
+functions. In the case of classification, the final layer of the
+neural net has as many nodes as classes, and the output of the
+neural net can be interpreted as the probability of a given input
+feature belonging to a specific class.
+
+
+@brief Solves classification and regression problems with several
+fully connected layers and nonlinear activation functions.
+
+@anchor mlp_classification
+@par Classification Training Function
+The mlp classification training function has the following format:
+<pre class="syntax">
+mlp_classification(
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+\b Arguments
+<DL class="arglist">
+ <DT>source_table</DT>
+ <DD>TEXT. Name of the table containing the training data.</DD>
+
+ <DT>model_table</DT>
+ <DD>TEXT. Name of the output table containing the model. Details of the output
+ tables are provided below.
+ </DD>
+
+ <DT>independent_varname</DT>
+ <DD>TEXT. Expression list to evaluate for the
+ independent variables. An intercept variable should not be included as part
+ of this expression. Please note that expression should be able to be cast
+ to DOUBLE PRECISION[].
+ </DD>
+
+ <DT>dependent_varname</DT>
+ <DD> TEXT. Name of the dependent variable column. For classification, supported types are:
+ text, varchar, character varying, char, character
+ integer, smallint, bigint, and boolean. </DD>
+
+ <DT>hidden_layer_sizes</DT>
+ <DD>INTEGER[], default: ARRAY[].
+ The number of neurons in each hidden layer. The length of this array will
+ determine the number of hidden layers. Empty for no hidden layers.
+ </DD>
+
+
+ <DT>optimizer_params (optional)</DT>
+ <DD>TEXT, default: NULL.
+ Parameters for optimization in a comma-separated string
+ of key-value pairs. See the description below for details.
+ </DD>
+
+ <DT>activation</DT>
--- End diff --
Please add `(optional)`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by iyerr3 <gi...@git.apache.org>.
Github user iyerr3 commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r128583284
--- Diff: .gitignore ---
@@ -1,5 +1,6 @@
# Ignore build directory
/build*
+/build-docker*
--- End diff --
Does the `build*` not cover `build-docker*`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by edespino <gi...@git.apache.org>.
Github user edespino commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r127046071
--- Diff: src/ports/postgres/modules/convex/mlp.sql_in ---
@@ -0,0 +1,736 @@
+/* ----------------------------------------------------------------------- *//**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ *
+ * @file mlp.sql_in
+ *
+ * @brief SQL functions for multi-layer perceptron
+ * @date June 2012
+ *
+ *
+ *//* ----------------------------------------------------------------------- */
+
+m4_include(`SQLCommon.m4')
+
+/**
+@addtogroup grp_mlp
+
+<div class="toc"><b>Contents</b><ul>
+<li class="level1"><a href="#mlp_classification">Classification</a></li>
+<li class="level1"><a href="#mlp_regression">Regression</a></li>
+<li class="level1"><a href="#optimization_params">Optimization Parameters</a></li>
+<li class="level1"><a href="#predict">Prediction Functions/a></li>
--- End diff --
The closing `/a>` tag is missing it's leading `<` character. I noticed this with visual inspection of generated docs from this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by njayaram2 <gi...@git.apache.org>.
Github user njayaram2 commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r128116476
--- Diff: src/modules/convex/task/mlp.hpp ---
@@ -0,0 +1,342 @@
+/* ----------------------------------------------------------------------- *//**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ * @file mlp.hpp
+ *
+ * This file contains objective function related computation, which is called
+ * by classes in algo/, e.g., loss, gradient functions
+ *
+ *//* ----------------------------------------------------------------------- */
+
+#ifndef MADLIB_MODULES_CONVEX_TASK_MLP_HPP_
+#define MADLIB_MODULES_CONVEX_TASK_MLP_HPP_
+
+namespace madlib {
+
+namespace modules {
+
+namespace convex {
+
+// Use Eigen
+using namespace madlib::dbal::eigen_integration;
+
+template <class Model, class Tuple>
+class MLP {
+public:
+ typedef Model model_type;
+ typedef Tuple tuple_type;
+ typedef typename Tuple::independent_variables_type
+ independent_variables_type;
+ typedef typename Tuple::dependent_variable_type dependent_variable_type;
+
+ static void gradientInPlace(
+ model_type &model,
+ const independent_variables_type &y,
+ const dependent_variable_type &z,
+ const double &stepsize);
+
+ static double loss(
+ const model_type &model,
+ const independent_variables_type &y,
+ const dependent_variable_type &z);
+
+ static ColumnVector predict(
+ const model_type &model,
+ const independent_variables_type &y,
+ const int is_response);
+
+ const static int RELU = 0;
+ const static int SIGMOID = 1;
+ const static int TANH = 2;
+
+ static double sigmoid(const double &xi) {
+ return 1. / (1. + std::exp(-xi));
+ }
+
+ static double relu(const double &xi) {
+ return xi*(xi>0);
+ }
+
+ static double tanh(const double &xi) {
+ return std::tanh(xi);
+ }
+
+
+private:
+
+ static double sigmoidDerivative(const double &xi) {
+ double value = sigmoid(xi);
+ return value * (1. - value);
+ }
+
+ static double reluDerivative(const double &xi) {
+ return xi>0;
+ }
+
+ static double tanhDerivative(const double &xi) {
+ double value = tanh(xi);
+ return 1-value*value;
+ }
+
+ static void feedForward(
+ const model_type &model,
+ const independent_variables_type &y,
+ std::vector<ColumnVector> &net,
+ std::vector<ColumnVector> &x);
+
+ static void endLayerDeltaError(
+ const std::vector<ColumnVector> &net,
+ const std::vector<ColumnVector> &x,
+ const dependent_variable_type &z,
+ ColumnVector &delta_N);
+
+ static void errorBackPropagation(
+ const ColumnVector &delta_N,
+ const std::vector<ColumnVector> &net,
+ const model_type &model,
+ std::vector<ColumnVector> &delta);
+};
+
+template <class Model, class Tuple>
+void
+MLP<Model, Tuple>::gradientInPlace(
+ model_type &model,
+ const independent_variables_type &y,
+ const dependent_variable_type &z,
+ const double &stepsize) {
+ (void) model;
+ (void) z;
+ (void) y;
+ (void) stepsize;
+ std::vector<ColumnVector> net;
+ std::vector<ColumnVector> x;
+ std::vector<ColumnVector> delta;
+ ColumnVector delta_N;
+
+ feedForward(model, y, net, x);
+ endLayerDeltaError(net, x, z, delta_N);
+ errorBackPropagation(delta_N, net, model, delta);
+
+ uint16_t N = model.u.size(); // assuming nu. of layers >= 1
+ uint16_t k, s, j;
+ // #TODO fix lambda
+ float lambda = 0;
+
+ std::vector<uint16_t> n; n.clear(); //nu. of units in each layer
+
+ n.push_back(model.u[0].rows() - 1);
+ for (k = 1; k <= N; k ++) {
+ n.push_back(model.u[k-1].cols() - 1);
+ }
+
+ for (k=1; k <= N; k++){
+ for (s=0; s <= n[k-1]; s++){
+ for (j=1; j <= n[k]; j++){
+ model.u[k-1](s,j) -= stepsize * (delta[k](j) * x[k-1](s) + lambda * model.u[k-1](s,j));
+ }
+ }
+ }
+}
+
+template <class Model, class Tuple>
+double
+MLP<Model, Tuple>::loss(
+ const model_type &model,
+ const independent_variables_type &y,
+ const dependent_variable_type &z) {
+ // Here we compute the loss. In the case of regression we use sum of square errors
+ // In the case of classification the loss term is cross entropy.
+ std::vector<ColumnVector> net;
+ std::vector<ColumnVector> x;
+
+ feedForward(model, y, net, x);
+ double loss = 0.;
+ uint16_t j;
+
+ for (j = 1; j < z.rows() + 1; j ++) {
+ if(model.is_classification){
+ // Cross entropy: RHS term is negative
+ loss -= z(j-1)*std::log(x.back()(j)) + (1-z(j-1))*std::log(1-x.back()(j));
+ // Computed like https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/neural_network/_base.py#L222
+ //loss -= z(j-1)*std::log(x.back()(j));
+ }else{
+ double diff = x.back()(j) - z(j-1);
+ loss += diff * diff;
+ }
+ }
+ if(!model.is_classification){
+ loss /= 2.;
+ }else{
+ // Give the average
+ loss /= z.rows();
+ }
+ return loss;
+}
+
+template <class Model, class Tuple>
+ColumnVector
+MLP<Model, Tuple>::predict(
+ const model_type &model,
+ const independent_variables_type &y,
+ const int is_response
+ ) {
+ (void) model;
+ (void) y;
+ std::vector<ColumnVector> net;
+ std::vector<ColumnVector> x;
+
+ feedForward(model, y, net, x);
+ // Don't return the offset
+ ColumnVector output = x.back().tail(x.back().size()-1);
+ if(is_response){
+ int max_idx;
+ output.maxCoeff(&max_idx);
+ output.resize(1);
+ output[0] = (double)max_idx;
+ }
+ return output;
+}
+
+
+template <class Model, class Tuple>
+void
+// #TODO Change y to x and x to o
+MLP<Model, Tuple>::feedForward(
+ const model_type &model,
+ const independent_variables_type &y,
+ std::vector<ColumnVector> &net,
+ std::vector<ColumnVector> &x){
+ // meta data and x_k^0 = 1
+ uint16_t k, j, s;
+ uint16_t N = model.u.size(); // assuming >= 1
+ net.resize(N + 1);
+ x.resize(N + 1);
+
+ std::vector<uint16_t> n; n.clear();
+ n.push_back(model.u[0].rows() - 1);
+ x[0].resize(n[0] + 1);
+ x[0](0) = 1.;
+ for (k = 1; k <= N; k ++) {
+ n.push_back(model.u[k-1].cols() - 1);
+ net[k].resize(n[k] + 1);
+ x[k].resize(n[k] + 1);
+ // Bias
+ x[k](0) = 1.;
+ }
+
+ // y is a mapped parameter from DB, aligning with x here
+ for (j = 1; j <= n[0]; j ++) { x[0](j) = y(j-1); }
+
+ for (k = 1; k < N; k ++) {
+ for (j = 1; j <= n[k]; j ++) {
+ net[k](j) = 0.;
+ for (s = 0; s <= n[k-1]; s ++) {
+ net[k](j) += x[k-1](s) * model.u[k-1](s, j);
+ }
+ if(model.activation==RELU)
+ x[k](j) = relu(net[k](j));
+ else if(model.activation==SIGMOID)
+ x[k](j) = sigmoid(net[k](j));
+ else if(model.activation==TANH)
+ x[k](j) = tanh(net[k](j));
+ else
+ elog(WARNING,"Invalid activation");
--- End diff --
Ideally it won't come to this `else` because things are validated in python. If it indeed is invalid, then we should probably throw an error instead of a warning here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by edespino <gi...@git.apache.org>.
Github user edespino commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r127319839
--- Diff: src/ports/postgres/modules/convex/mlp.sql_in ---
@@ -0,0 +1,744 @@
+/* ----------------------------------------------------------------------- *//**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ *
+ * @file mlp.sql_in
+ *
+ * @brief SQL functions for multilayer perceptron
+ * @date June 2012
+ *
+ *
+ *//* ----------------------------------------------------------------------- */
+
+m4_include(`SQLCommon.m4')
+
+/**
+@addtogroup grp_mlp
+
+<div class="toc"><b>Contents</b><ul>
+<li class="level1"><a href="#mlp_classification">Classification</a></li>
+<li class="level1"><a href="#mlp_regression">Regression</a></li>
+<li class="level1"><a href="#optimization_params">Optimizer Parameters</a></li>
+<li class="level1"><a href="#predict">Prediction Functions/a></li>
+<li class="level1"><a href="#example">Examples</a></li>
+<li class="level1"><a href="#background">Technical Background</a></li>
+<li class="level1"><a href="#literature">Literature</a></li>
+<li class="level1"><a href="#related">Related Topics</a></li>
+</ul></div>
+
+Multilayer Perceptron (MLP) is a model for regression and
+classification
+
+Also called "vanilla neural networks", they consist of several
+fully connected hidden layers with non-linear activation
+functions. In the case of classification, the final layer of the
+neural net has as many nodes as classes, and the output of the
+neural net can be interpreted as the probability of a given input
+feature belonging to a specific class.
+
+
+@brief Solves classification and regression problems with several
+fully connected layers and nonlinear activation functions.
+
+@anchor mlp_classification
+@par Classification Training Function
+The mlp classification training function has the following format:
+<pre class="syntax">
+mlp_classification(
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+\b Arguments
+<DL class="arglist">
+ <DT>source_table</DT>
+ <DD>TEXT. Name of the table containing the training data.</DD>
+
+ <DT>model_table</DT>
+ <DD>TEXT. Name of the output table containing the model. Details of the output
+ tables are provided below.
+ </DD>
+
+ <DT>independent_varname</DT>
+ <DD>TEXT. Expression list to evaluate for the
+ independent variables. An intercept variable should not be included as part
+ of this expression. Please note that expression should be able to be cast
+ to DOUBLE PRECISION[].
+ </DD>
+
+ <DT>dependent_varname</DT>
+ <DD> TEXT. Name of the dependent variable column. For classification, supported types are:
+ text, varchar, character varying, char, character
+ integer, smallint, bigint, and boolean. </DD>
+
+ <DT>hidden_layer_sizes</DT>
+ <DD>INTEGER[], default: ARRAY[].
+ The number of neurons in each hidden layer. The length of this array will
+ determine the number of hidden layers. Empty for no hidden layers.
+ </DD>
+
+
+ <DT>optimizer_params (optional)</DT>
+ <DD>TEXT, default: NULL.
+ Parameters for optimization in a comma-separated string
+ of key-value pairs. See the description below for details.
+ </DD>
+
+ <DT>activation</DT>
+ <DD>TEXT, default: 'sigmoid'.
+ Activation function. Currently three functions are supported: 'sigmoid' (default),
+ 'relu', and 'tanh'. The text can be any subset of the three
+ strings; for e.g., activation='s' will use the sigmoid activation.
+ </DD>
+</DL>
+
+<b>Output tables</b>
+<br>
+ The model table produced by mlp contains the following columns:
+ <table class="output">
+ <tr>
+ <th>coeffs</th>
+ <td>FLOAT8[]. Flat array containing the weights of the neural net</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>INTEGER. Number of iterations completed by stochastic gradient descent
+ algorithm. The algorithm either converged in this number of iterations
+ or hit the maximum number specified in the optimization parameters. </td>
+ </tr>
+ <tr>
+ <th>loss</th>
+ <td>FLOAT8. The cross entropy over the training data.
+ See Technical Background section below for more details.</td>
+ </tr>
+ </table>
+
+
+A summary table named \<model_table\>_summary is also created, which has the following columns:
+ <table class="output">
+ <tr>
+ <th>source_table</th>
+ <td>The source table.</td>
+ </tr>
+ <tr>
+ <th>dependent_varname</th>
+ <td>The dependent variable.</td>
+ </tr>
+ <tr>
+ <th>independent_varname</th>
+ <td>The independent variables.</td>
+ </tr>
+ <tr>
+ <th>tolerance</th>
+ <td>The tolerance as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>step_size</th>
+ <td>The step size as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>The number of iterations run</td>
+ </tr>
+ <tr>
+ <th>n_tries</th>
+ <td>The number of tries as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>layer_sizes</th>
+ <td>The number of units in each layer including the input and output layer.</td>
+ </tr>
+ <tr>
+ <th>activation_function</th>
+ <td>The activation function.</td>
+ </tr>
+ <tr>
+ <th>is_classification</th>
+ <td>True if the model was trained for classification, False if it was trained
+ for regression</td>
+ </tr>
+ <tr>
+ <th>classes</th>
+ <td>The classes which were trained against (empty for regression)</td>
+ </tr>
+
+ </table>
+
+
+@anchor mlp_regression
+@par Regression Training Function
+The mlp regression training function has the following format:
+<pre class="syntax">
+mlp_regression(source_table,
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+
+\b Arguments
+
+Specifications for regression are largely the same as for classification. In the
+model table, the loss will refer to mean square error instead of cross entropy. In the
+summary table, there is classes column. The following
+arguments have specifications which differ from mlp_classification:
+<DL class="arglist">
+<DT>dependent_varname</DT>
+ <DD>TEXT. Name of the dependent variable column.
+ For regression supported types are any numeric type, or array
+ or numeric types (for multiple regression).
+ </DD>
+</DL>
+
+
+@anchor optimizer_params
+@par Optimizer Parameters
+Parameters in this section are supplied in the \e optimizer_params argument as a string
+containing a comma-delimited list of name-value pairs. All of these named
+parameters are optional, and their order does not matter. You must use the
+format "<param_name> = <value>" to specify the value of a parameter, otherwise
+the parameter is ignored.
+
+
+<pre class="syntax">
+ 'step_size = <value>,
+ n_iterations = <value>,
+ n_tries = <value>,
+ tolerance = <value>'
+</pre>
+\b Optimizer Parameters
+<DL class="arglist">
+
+<DT>step_size</dt>
+<DD>Default: [0.001].
+Also known as the learning rate. A small value is usually desirable to
+ensure convergence, while a large value provides more room for progress during
+training. Since the best value depends on the condition number of the data, in
+practice one often tunes this parameter.
+</DD>
+
+
+<DT>n_iterations</dt>
+<DD>Default: [100]. The maximum number of iterations allowed.
+</DD>
+<DT>n_tries</dt>
+<DD>Default: [1]. Number of times to retrain the network with randomly initialized
+weights
+</DD>
+
+<DT>tolerance</dt>
+<DD>Default: 0.001. The criterion to end iterations. The training stops whenever
+<the difference between the training models of two consecutive iterations is
+<smaller than \e tolerance or the iteration number is larger than \e max_iter.
+</DD>
+
+</DL>
+
+@anchor predict
+@par Prediction Function
+Used to generate predictions given a previously trained model on novel data.
+The same syntax is used for classification, and regression.
+<pre class="syntax">
+mlp_predict(model_table,
+ data_table,
+ id_col_name,
+ output_table,
+ pred_type)
+</pre>
+
+\b Arguments
+<DL class="arglist">
+ <DT>model_table</DT>
+ <DD>TEXT. Model table produced by the training function.</DD>
+
+ <DT>data_table</DT>
+ <DD>TEXT. Name of the table containing the data for prediction. This table is expected
+ to contain the same input features that were used during training. The table should
+ also contain id_col_name used for identifying each row.</DD>
+
+ <DT>id_col_name</DT>
+ <DD>TEXT. The name of the id column in the input table.</DD>
+
+ <DT>output_table</DT>
+ <DD>TEXT. Name of the table where output predictions are written. If this
+table name is already in use, then an error is returned. Table contains:</DD>
+ <table class="output">
+ <tr>
+ <th>id</th>
+ <td>Gives the 'id' for each prediction, corresponding to each row from the data_table.</td>
+ </tr>
+ <tr>
+ <th>estimated_<COL_NAME></th>
+ <td>
+ (For pred_type='response') The estimated class
+ for classification or value for regression, where
+ <COL_NAME> is the name of the column to be
+ predicted from training data
+ </td>
+ </tr>
+ <tr>
+ <th>prob_<CLASS></th>
+ <td>
+ (For pred_type='prob' for classification) The
+ probability of a given class <CLASS> as given by
+ softmax. There will be one column for each class
+ in the training data.
+ </td>
+ </tr>
+
+
+ <DT>pred_type</DT>
+ <DD>TEXT.
+
+the type of output requested:
+'response' gives the actual prediction,
+'prob' gives the probability of each class.
+for regression, only type='response' is defined.
+The name of the id column in the input table.</DD>
+</DL>
+</table>
+
+@anchor example
+@par Examples
+-# Create an input data set.
+<pre class="example">
+CREATE TABLE iris_data(
+ id integer,
+ attributes numeric[],
+ class_text varchar,
+ class integer
+);
+INSERT INTO iris_data VALUES
+(1,ARRAY[5.1,3.5,1.4,0.2],'Iris-setosa',1),
+(2,ARRAY[4.9,3.0,1.4,0.2],'Iris-setosa',1),
+(3,ARRAY[4.7,3.2,1.3,0.2],'Iris-setosa',1),
+(4,ARRAY[4.6,3.1,1.5,0.2],'Iris-setosa',1),
+(5,ARRAY[5.0,3.6,1.4,0.2],'Iris-setosa',1),
+(6,ARRAY[5.4,3.9,1.7,0.4],'Iris-setosa',1),
+(7,ARRAY[4.6,3.4,1.4,0.3],'Iris-setosa',1),
+(8,ARRAY[5.0,3.4,1.5,0.2],'Iris-setosa',1),
+(9,ARRAY[4.4,2.9,1.4,0.2],'Iris-setosa',1),
+(10,ARRAY[4.9,3.1,1.5,0.1],'Iris-setosa',1),
+(11,ARRAY[7.0,3.2,4.7,1.4],'Iris-versicolor',2),
+(12,ARRAY[6.4,3.2,4.5,1.5],'Iris-versicolor',2),
+(13,ARRAY[6.9,3.1,4.9,1.5],'Iris-versicolor',2),
+(14,ARRAY[5.5,2.3,4.0,1.3],'Iris-versicolor',2),
+(15,ARRAY[6.5,2.8,4.6,1.5],'Iris-versicolor',2),
+(16,ARRAY[5.7,2.8,4.5,1.3],'Iris-versicolor',2),
+(17,ARRAY[6.3,3.3,4.7,1.6],'Iris-versicolor',2),
+(18,ARRAY[4.9,2.4,3.3,1.0],'Iris-versicolor',2),
+(19,ARRAY[6.6,2.9,4.6,1.3],'Iris-versicolor',2),
+(20,ARRAY[5.2,2.7,3.9,1.4],'Iris-versicolor',2);
+</pre>
+-# Generate a multilayer perceptron with a single hidden layer of 5 units.
+Use the attributes column as the independent variables, and use the class
+column as the classification. Set the tolerance to 0 so that 5000
+iterations will be run. Use a hyperbolic tangent activation function.
+The model will be written to mlp_model.
+<pre class="example">
+SELECT madlib.mlp_classification(
+ 'iris_data', -- Source table
+ 'mlp_model', -- Destination table
+ 'attributes', -- Input features
+ 'class_text', -- Label
+ ARRAY[5], -- Number of units per layer
+ 'step_size=0.003,
+ n_iterations=5000,
+ tolerance=0', -- Optimizer params
+ 'tanh'); -- Activation function
+</pre>
+-# View the result for the model.
+<pre class="example">
+-- Set extended display on for easier reading of output
+\\x ON
+-- Neural net Initialization is non-deterministic, so your results may vary
+SELECT * FROM mlp_model;
+</pre>
+Result:
+<pre class="result">
+-[ RECORD 1 ]--+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+coeff | {1,1,1,1,1,0.136374930803,0.188739676875,0.662387810001,-1.03381622734,-0.469961067046,0.0614006983397,0.0811504589436,0.299008228258,-0.47391918521,-0.215098143699,0.10519213944,0.145844617525,0.511683525606,-0.800215552382,-0.36417142683,0.120751709056,0.167531106521,0.587074895969,-0.916946198095,-0.417055067449,0.0539541885146,0.0694359704131,0.262598585854,-0.419234805076,-0.189915344282,1,1,1,1,1,1,0.105645702152,1.46247470474,0.484457903226,0.965962824478,1.19361986431,0.419805760087,-0.105696503487,-1.46245956666,-0.484427811691,-0.965730981426,-1.19365280555,-0.419973628863}
+loss | 0.0184092375519
+num_iterations | 5000
+</pre>
+-# Next train a regression example. First create some test data. This dataset
+contains housing prices data.
+<pre class="example">
+CREATE TABLE lin_housing (id serial, x float8[], grp_by_col int, y float8);
+COPY lin_housing (x, grp_by_col, y) FROM STDIN NULL '?' DELIMITER '|';
+{1,0.00632,18.00,2.310,0,0.5380,6.5750,65.20,4.0900,1,296.0,15.30,396.90,4.98} | 1 | 24.00
+{1,0.02731,0.00,7.070,0,0.4690,6.4210,78.90,4.9671,2,242.0,17.80,396.90,9.14} | 1 | 21.60
+{1,0.02729,0.00,7.070,0,0.4690,7.1850,61.10,4.9671,2,242.0,17.80,392.83,4.03} | 1 | 34.70
+{1,0.03237,0.00,2.180,0,0.4580,6.9980,45.80,6.0622,3,222.0,18.70,394.63,2.94} | 1 | 33.40
+{1,0.06905,0.00,2.180,0,0.4580,7.1470,54.20,6.0622,3,222.0,18.70,396.90,5.33} | 1 | 36.20
+{1,0.02985,0.00,2.180,0,0.4580,6.4300,58.70,6.0622,3,222.0,18.70,394.12,5.21} | 1 | 28.70
+{1,0.08829,12.50,7.870,0,0.5240,6.0120,66.60,5.5605,5,311.0,15.20,395.60,12.43} | 1 | 22.90
+{1,0.14455,12.50,7.870,0,0.5240,6.1720,96.10,5.9505,5,311.0,15.20,396.90,19.15} | 1 | 27.10
+{1,0.21124,12.50,7.870,0,0.5240,5.6310,100.00,6.0821,5,311.0,15.20,386.63,29.93} | 1 | 16.50
+{1,0.17004,12.50,7.870,0,0.5240,6.0040,85.90,6.5921,5,311.0,15.20,386.71,17.10} | 1 | 18.90
+{1,0.22489,12.50,7.870,0,0.5240,6.3770,94.30,6.3467,5,311.0,15.20,392.52,20.45} | 1 | 15.00
+{1,0.11747,12.50,7.870,0,0.5240,6.0090,82.90,6.2267,5,311.0,15.20,396.90,13.27} | 1 | 18.90
+{1,0.09378,12.50,7.870,0,0.5240,5.8890,39.00,5.4509,5,311.0,15.20,390.50,15.71} | 1 | 21.70
+{1,0.62976,0.00,8.140,0,0.5380,5.9490,61.80,4.7075,4,307.0,21.00,396.90,8.26} | 1 | 20.40
+{1,0.63796,0.00,8.140,0,0.5380,6.0960,84.50,4.4619,4,307.0,21.00,380.02,10.26} | 1 | 18.20
+{1,0.62739,0.00,8.140,0,0.5380,5.8340,56.50,4.4986,4,307.0,21.00,395.62,8.47} | 1 | 19.90
+{1,1.05393,0.00,8.140,0,0.5380,5.9350,29.30,4.4986,4,307.0,21.00,386.85,6.58} | 1 | 23.10
+{1,0.78420,0.00,8.140,0,0.5380,5.9900,81.70,4.2579,4,307.0,21.00,386.75,14.67} | 1 | 17.50
+{1,0.80271,0.00,8.140,0,0.5380,5.4560,36.60,3.7965,4,307.0,21.00,288.99,11.69} | 1 | 20.20
+{1,0.72580,0.00,8.140,0,0.5380,5.7270,69.50,3.7965,4,307.0,21.00,390.95,11.28} | 1 | 18.20
+\.
+</pre>
+-# Now train a regression model using a multilayer perceptron a single hidden layer of two nodes.
+<pre class="example">
+SELECT madlib.mlp_regression(
+ 'lin_housing', -- Source table
+ 'mlp_regress', -- Desination table
+ 'x', -- Input features
+ 'y', -- Dependent variable
+ ARRAY[5,5], -- Number of units per layer
+ 'step_size=0.000007,
+ n_iterations=10000,
+ tolerance=0',
+ 'relu');
+</pre>
+-# Check the results of the model
+<pre class="example">
+-- Set extended display on for easier reading of output
+\\x ON
+-- Neural net Initialization is non-deterministic, so your results may vary
+SELECT * FROM mlp_regress;
+</pre>
+Result:
+<pre class="result">
+-[ RECORD 1 ]--+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------
+coeff | {1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2.79506311399e-05,3.56715008915e-05,-6.09333559685e-05,0.000251228318768,-0.000224772841379,-3.71863030857e-05,-3.5757865148e-06,5.27936784854e-05,-2.48474166186e-05,6.19731184294e-05,3.07638968743e-05,6.8964698578e-06,0.000106016701083,-1.71484730318e-05,1.18691881812e-05,-0.000163975464208,0.000170026304906,3.11688265279e-05,0.000177050148787,-1.58265976603e-05,2.70144422657e-05,0.000112667883422,3.77575139073e-05,8.12474658795e-05,-7.90458917626e-05,0.000107566386158,-2.63771171506e-06,2.47996880915e-05,-0.00012642310887,0.000203827391081,0.000139315565565,4.86147243454e-05,-0.000176126471913,-6.47820782916e-05,-8.51592776447e-06,-6.60601176758e-05,2.91421874156e-05,6.3556873752e-05,0.000197557443129,0.000220531367259,0.000135036310289,0.000143735913975,-4.75034117786e-05,-0.000179547345838,-1.6919846786e-05,0.000162784312994,0.000268595819851,-0.000460066553287,8.69756071591e-05,-0.00311762727057,0.000126024763103,0.00020598824
2921,0.003463432426,-0.00729789075286,0.00151625867549,-0.000890852767597,-0.00525016037249,0.0031043106659,0.00798041103839,-0.00552693050079,0.0232180415786,0.0230489850143,-0.0437890272341,0.0165765426407,-0.248554261758,-7.81336427846e-05,0.00558145591752,0.283465844585,-0.571699956182,0.133474351994,-0.0785181945605,-0.419269930709,0.249547772912,0.631761009875,-0.431305975666,1,1,1,1,1,1,0.0158747497572,-9.02809160806e-05,0.00015574347618,4.10805373863e-06,0.00121532434965,0.101790351335,0.0647558401493,-0.00013654998677,-9.92872075948e-06,-5.5319694394e-05,0.00519320756484,0.412736586036,0.0011998026977,-1.53688189815e-05,1.94817888201e-05,-4.63111489966e-05,7.24547899029e-05,0.00880394144485,5.45309822095e-05,-0.000140943219275,-7.96211486227e-05,-1.04337307472e-05,0.000161936762028,0.00136273797767,-4.54737243585e-05,-3.4083840736e-05,3.69286883662e-05,9.9047243188e-08,3.75014011824e-06,-9.45366086368e-08,1,1,1,1,1,1,6.67488547054,0.102754199001,0.41668912471,0.008868672964
79,0.00136206007228,-9.88642499013e-05}
+loss | 144.965776158
+num_iterations | 10000
+</pre>
+-# Now let's look at the prediction functions. In the following examples we will
+use the training data set for prediction as well, which is not usual but serves to
+show the syntax. First we will test the classification exampel.
+The prediction is in the the estimated_class_text column with the
+actual value in the class_text column.
+<pre class="example">
+SELECT madlib.mlp_predict(
+ 'mlp_model', -- Model table
+ 'iris_data', -- Test data table
+ 'id', -- Id column in test table
+ 'mlp_prediction', -- Output table for predictions
+ 'response' -- Output classes, not probabilities
+ );
+SELECT * FROM mlp_prediction JOIN iris_data USING (id);
--- End diff --
Move this `SELECT` with `JOIN` clause to its own `View results` section with the results.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by cooper-sloan <gi...@git.apache.org>.
Github user cooper-sloan commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r127504755
--- Diff: src/ports/postgres/modules/convex/mlp.sql_in ---
@@ -0,0 +1,744 @@
+/* ----------------------------------------------------------------------- *//**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ *
+ * @file mlp.sql_in
+ *
+ * @brief SQL functions for multilayer perceptron
+ * @date June 2012
+ *
+ *
+ *//* ----------------------------------------------------------------------- */
+
+m4_include(`SQLCommon.m4')
+
+/**
+@addtogroup grp_mlp
+
+<div class="toc"><b>Contents</b><ul>
+<li class="level1"><a href="#mlp_classification">Classification</a></li>
+<li class="level1"><a href="#mlp_regression">Regression</a></li>
+<li class="level1"><a href="#optimization_params">Optimizer Parameters</a></li>
+<li class="level1"><a href="#predict">Prediction Functions/a></li>
+<li class="level1"><a href="#example">Examples</a></li>
+<li class="level1"><a href="#background">Technical Background</a></li>
+<li class="level1"><a href="#literature">Literature</a></li>
+<li class="level1"><a href="#related">Related Topics</a></li>
+</ul></div>
+
+Multilayer Perceptron (MLP) is a model for regression and
+classification
+
+Also called "vanilla neural networks", they consist of several
+fully connected hidden layers with non-linear activation
+functions. In the case of classification, the final layer of the
+neural net has as many nodes as classes, and the output of the
+neural net can be interpreted as the probability of a given input
+feature belonging to a specific class.
+
+
+@brief Solves classification and regression problems with several
+fully connected layers and nonlinear activation functions.
+
+@anchor mlp_classification
+@par Classification Training Function
+The mlp classification training function has the following format:
+<pre class="syntax">
+mlp_classification(
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+\b Arguments
+<DL class="arglist">
+ <DT>source_table</DT>
+ <DD>TEXT. Name of the table containing the training data.</DD>
+
+ <DT>model_table</DT>
+ <DD>TEXT. Name of the output table containing the model. Details of the output
+ tables are provided below.
+ </DD>
+
+ <DT>independent_varname</DT>
+ <DD>TEXT. Expression list to evaluate for the
+ independent variables. An intercept variable should not be included as part
+ of this expression. Please note that expression should be able to be cast
+ to DOUBLE PRECISION[].
+ </DD>
+
+ <DT>dependent_varname</DT>
+ <DD> TEXT. Name of the dependent variable column. For classification, supported types are:
+ text, varchar, character varying, char, character
+ integer, smallint, bigint, and boolean. </DD>
+
+ <DT>hidden_layer_sizes</DT>
+ <DD>INTEGER[], default: ARRAY[].
+ The number of neurons in each hidden layer. The length of this array will
+ determine the number of hidden layers. Empty for no hidden layers.
+ </DD>
+
+
+ <DT>optimizer_params (optional)</DT>
+ <DD>TEXT, default: NULL.
+ Parameters for optimization in a comma-separated string
+ of key-value pairs. See the description below for details.
+ </DD>
+
+ <DT>activation</DT>
+ <DD>TEXT, default: 'sigmoid'.
+ Activation function. Currently three functions are supported: 'sigmoid' (default),
+ 'relu', and 'tanh'. The text can be any subset of the three
+ strings; for e.g., activation='s' will use the sigmoid activation.
+ </DD>
+</DL>
+
+<b>Output tables</b>
+<br>
+ The model table produced by mlp contains the following columns:
+ <table class="output">
+ <tr>
+ <th>coeffs</th>
+ <td>FLOAT8[]. Flat array containing the weights of the neural net</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>INTEGER. Number of iterations completed by stochastic gradient descent
+ algorithm. The algorithm either converged in this number of iterations
+ or hit the maximum number specified in the optimization parameters. </td>
+ </tr>
+ <tr>
+ <th>loss</th>
+ <td>FLOAT8. The cross entropy over the training data.
+ See Technical Background section below for more details.</td>
+ </tr>
+ </table>
+
+
+A summary table named \<model_table\>_summary is also created, which has the following columns:
+ <table class="output">
+ <tr>
+ <th>source_table</th>
+ <td>The source table.</td>
+ </tr>
+ <tr>
+ <th>dependent_varname</th>
+ <td>The dependent variable.</td>
+ </tr>
+ <tr>
+ <th>independent_varname</th>
+ <td>The independent variables.</td>
+ </tr>
+ <tr>
+ <th>tolerance</th>
+ <td>The tolerance as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>step_size</th>
+ <td>The step size as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>The number of iterations run</td>
+ </tr>
+ <tr>
+ <th>n_tries</th>
+ <td>The number of tries as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>layer_sizes</th>
+ <td>The number of units in each layer including the input and output layer.</td>
+ </tr>
+ <tr>
+ <th>activation_function</th>
+ <td>The activation function.</td>
+ </tr>
+ <tr>
+ <th>is_classification</th>
+ <td>True if the model was trained for classification, False if it was trained
+ for regression</td>
+ </tr>
+ <tr>
+ <th>classes</th>
+ <td>The classes which were trained against (empty for regression)</td>
+ </tr>
+
+ </table>
+
+
+@anchor mlp_regression
+@par Regression Training Function
+The mlp regression training function has the following format:
+<pre class="syntax">
+mlp_regression(source_table,
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+
+\b Arguments
+
+Specifications for regression are largely the same as for classification. In the
+model table, the loss will refer to mean square error instead of cross entropy. In the
+summary table, there is classes column. The following
+arguments have specifications which differ from mlp_classification:
+<DL class="arglist">
+<DT>dependent_varname</DT>
+ <DD>TEXT. Name of the dependent variable column.
+ For regression supported types are any numeric type, or array
+ or numeric types (for multiple regression).
+ </DD>
+</DL>
+
+
+@anchor optimizer_params
+@par Optimizer Parameters
+Parameters in this section are supplied in the \e optimizer_params argument as a string
+containing a comma-delimited list of name-value pairs. All of these named
+parameters are optional, and their order does not matter. You must use the
+format "<param_name> = <value>" to specify the value of a parameter, otherwise
+the parameter is ignored.
+
+
+<pre class="syntax">
+ 'step_size = <value>,
+ n_iterations = <value>,
+ n_tries = <value>,
+ tolerance = <value>'
+</pre>
+\b Optimizer Parameters
+<DL class="arglist">
+
+<DT>step_size</dt>
+<DD>Default: [0.001].
+Also known as the learning rate. A small value is usually desirable to
+ensure convergence, while a large value provides more room for progress during
+training. Since the best value depends on the condition number of the data, in
+practice one often tunes this parameter.
+</DD>
+
+
+<DT>n_iterations</dt>
+<DD>Default: [100]. The maximum number of iterations allowed.
+</DD>
+<DT>n_tries</dt>
+<DD>Default: [1]. Number of times to retrain the network with randomly initialized
+weights
+</DD>
+
+<DT>tolerance</dt>
+<DD>Default: 0.001. The criterion to end iterations. The training stops whenever
+<the difference between the training models of two consecutive iterations is
+<smaller than \e tolerance or the iteration number is larger than \e max_iter.
+</DD>
+
+</DL>
+
+@anchor predict
+@par Prediction Function
+Used to generate predictions given a previously trained model on novel data.
+The same syntax is used for classification, and regression.
+<pre class="syntax">
+mlp_predict(model_table,
+ data_table,
+ id_col_name,
+ output_table,
+ pred_type)
+</pre>
+
+\b Arguments
+<DL class="arglist">
+ <DT>model_table</DT>
+ <DD>TEXT. Model table produced by the training function.</DD>
+
+ <DT>data_table</DT>
+ <DD>TEXT. Name of the table containing the data for prediction. This table is expected
+ to contain the same input features that were used during training. The table should
+ also contain id_col_name used for identifying each row.</DD>
+
+ <DT>id_col_name</DT>
+ <DD>TEXT. The name of the id column in the input table.</DD>
+
+ <DT>output_table</DT>
+ <DD>TEXT. Name of the table where output predictions are written. If this
+table name is already in use, then an error is returned. Table contains:</DD>
+ <table class="output">
+ <tr>
+ <th>id</th>
+ <td>Gives the 'id' for each prediction, corresponding to each row from the data_table.</td>
+ </tr>
+ <tr>
+ <th>estimated_<COL_NAME></th>
+ <td>
+ (For pred_type='response') The estimated class
+ for classification or value for regression, where
+ <COL_NAME> is the name of the column to be
+ predicted from training data
+ </td>
+ </tr>
+ <tr>
+ <th>prob_<CLASS></th>
+ <td>
+ (For pred_type='prob' for classification) The
+ probability of a given class <CLASS> as given by
+ softmax. There will be one column for each class
+ in the training data.
+ </td>
+ </tr>
+
+
+ <DT>pred_type</DT>
+ <DD>TEXT.
+
+the type of output requested:
+'response' gives the actual prediction,
+'prob' gives the probability of each class.
+for regression, only type='response' is defined.
+The name of the id column in the input table.</DD>
+</DL>
+</table>
+
+@anchor example
+@par Examples
+-# Create an input data set.
+<pre class="example">
+CREATE TABLE iris_data(
+ id integer,
+ attributes numeric[],
+ class_text varchar,
+ class integer
+);
+INSERT INTO iris_data VALUES
+(1,ARRAY[5.1,3.5,1.4,0.2],'Iris-setosa',1),
+(2,ARRAY[4.9,3.0,1.4,0.2],'Iris-setosa',1),
+(3,ARRAY[4.7,3.2,1.3,0.2],'Iris-setosa',1),
+(4,ARRAY[4.6,3.1,1.5,0.2],'Iris-setosa',1),
+(5,ARRAY[5.0,3.6,1.4,0.2],'Iris-setosa',1),
+(6,ARRAY[5.4,3.9,1.7,0.4],'Iris-setosa',1),
+(7,ARRAY[4.6,3.4,1.4,0.3],'Iris-setosa',1),
+(8,ARRAY[5.0,3.4,1.5,0.2],'Iris-setosa',1),
+(9,ARRAY[4.4,2.9,1.4,0.2],'Iris-setosa',1),
+(10,ARRAY[4.9,3.1,1.5,0.1],'Iris-setosa',1),
+(11,ARRAY[7.0,3.2,4.7,1.4],'Iris-versicolor',2),
+(12,ARRAY[6.4,3.2,4.5,1.5],'Iris-versicolor',2),
+(13,ARRAY[6.9,3.1,4.9,1.5],'Iris-versicolor',2),
+(14,ARRAY[5.5,2.3,4.0,1.3],'Iris-versicolor',2),
+(15,ARRAY[6.5,2.8,4.6,1.5],'Iris-versicolor',2),
+(16,ARRAY[5.7,2.8,4.5,1.3],'Iris-versicolor',2),
+(17,ARRAY[6.3,3.3,4.7,1.6],'Iris-versicolor',2),
+(18,ARRAY[4.9,2.4,3.3,1.0],'Iris-versicolor',2),
+(19,ARRAY[6.6,2.9,4.6,1.3],'Iris-versicolor',2),
+(20,ARRAY[5.2,2.7,3.9,1.4],'Iris-versicolor',2);
+</pre>
+-# Generate a multilayer perceptron with a single hidden layer of 5 units.
+Use the attributes column as the independent variables, and use the class
+column as the classification. Set the tolerance to 0 so that 5000
+iterations will be run. Use a hyperbolic tangent activation function.
+The model will be written to mlp_model.
+<pre class="example">
+SELECT madlib.mlp_classification(
+ 'iris_data', -- Source table
+ 'mlp_model', -- Destination table
+ 'attributes', -- Input features
+ 'class_text', -- Label
+ ARRAY[5], -- Number of units per layer
+ 'step_size=0.003,
+ n_iterations=5000,
+ tolerance=0', -- Optimizer params
+ 'tanh'); -- Activation function
+</pre>
+-# View the result for the model.
+<pre class="example">
+-- Set extended display on for easier reading of output
+\\x ON
+-- Neural net Initialization is non-deterministic, so your results may vary
+SELECT * FROM mlp_model;
+</pre>
+Result:
+<pre class="result">
+-[ RECORD 1 ]--+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+coeff | {1,1,1,1,1,0.136374930803,0.188739676875,0.662387810001,-1.03381622734,-0.469961067046,0.0614006983397,0.0811504589436,0.299008228258,-0.47391918521,-0.215098143699,0.10519213944,0.145844617525,0.511683525606,-0.800215552382,-0.36417142683,0.120751709056,0.167531106521,0.587074895969,-0.916946198095,-0.417055067449,0.0539541885146,0.0694359704131,0.262598585854,-0.419234805076,-0.189915344282,1,1,1,1,1,1,0.105645702152,1.46247470474,0.484457903226,0.965962824478,1.19361986431,0.419805760087,-0.105696503487,-1.46245956666,-0.484427811691,-0.965730981426,-1.19365280555,-0.419973628863}
+loss | 0.0184092375519
+num_iterations | 5000
+</pre>
+-# Next train a regression example. First create some test data. This dataset
+contains housing prices data.
+<pre class="example">
+CREATE TABLE lin_housing (id serial, x float8[], grp_by_col int, y float8);
+COPY lin_housing (x, grp_by_col, y) FROM STDIN NULL '?' DELIMITER '|';
+{1,0.00632,18.00,2.310,0,0.5380,6.5750,65.20,4.0900,1,296.0,15.30,396.90,4.98} | 1 | 24.00
+{1,0.02731,0.00,7.070,0,0.4690,6.4210,78.90,4.9671,2,242.0,17.80,396.90,9.14} | 1 | 21.60
+{1,0.02729,0.00,7.070,0,0.4690,7.1850,61.10,4.9671,2,242.0,17.80,392.83,4.03} | 1 | 34.70
+{1,0.03237,0.00,2.180,0,0.4580,6.9980,45.80,6.0622,3,222.0,18.70,394.63,2.94} | 1 | 33.40
+{1,0.06905,0.00,2.180,0,0.4580,7.1470,54.20,6.0622,3,222.0,18.70,396.90,5.33} | 1 | 36.20
+{1,0.02985,0.00,2.180,0,0.4580,6.4300,58.70,6.0622,3,222.0,18.70,394.12,5.21} | 1 | 28.70
+{1,0.08829,12.50,7.870,0,0.5240,6.0120,66.60,5.5605,5,311.0,15.20,395.60,12.43} | 1 | 22.90
+{1,0.14455,12.50,7.870,0,0.5240,6.1720,96.10,5.9505,5,311.0,15.20,396.90,19.15} | 1 | 27.10
+{1,0.21124,12.50,7.870,0,0.5240,5.6310,100.00,6.0821,5,311.0,15.20,386.63,29.93} | 1 | 16.50
+{1,0.17004,12.50,7.870,0,0.5240,6.0040,85.90,6.5921,5,311.0,15.20,386.71,17.10} | 1 | 18.90
+{1,0.22489,12.50,7.870,0,0.5240,6.3770,94.30,6.3467,5,311.0,15.20,392.52,20.45} | 1 | 15.00
+{1,0.11747,12.50,7.870,0,0.5240,6.0090,82.90,6.2267,5,311.0,15.20,396.90,13.27} | 1 | 18.90
+{1,0.09378,12.50,7.870,0,0.5240,5.8890,39.00,5.4509,5,311.0,15.20,390.50,15.71} | 1 | 21.70
+{1,0.62976,0.00,8.140,0,0.5380,5.9490,61.80,4.7075,4,307.0,21.00,396.90,8.26} | 1 | 20.40
+{1,0.63796,0.00,8.140,0,0.5380,6.0960,84.50,4.4619,4,307.0,21.00,380.02,10.26} | 1 | 18.20
+{1,0.62739,0.00,8.140,0,0.5380,5.8340,56.50,4.4986,4,307.0,21.00,395.62,8.47} | 1 | 19.90
+{1,1.05393,0.00,8.140,0,0.5380,5.9350,29.30,4.4986,4,307.0,21.00,386.85,6.58} | 1 | 23.10
+{1,0.78420,0.00,8.140,0,0.5380,5.9900,81.70,4.2579,4,307.0,21.00,386.75,14.67} | 1 | 17.50
+{1,0.80271,0.00,8.140,0,0.5380,5.4560,36.60,3.7965,4,307.0,21.00,288.99,11.69} | 1 | 20.20
+{1,0.72580,0.00,8.140,0,0.5380,5.7270,69.50,3.7965,4,307.0,21.00,390.95,11.28} | 1 | 18.20
+\.
+</pre>
+-# Now train a regression model using a multilayer perceptron a single hidden layer of two nodes.
+<pre class="example">
+SELECT madlib.mlp_regression(
+ 'lin_housing', -- Source table
+ 'mlp_regress', -- Desination table
+ 'x', -- Input features
+ 'y', -- Dependent variable
+ ARRAY[5,5], -- Number of units per layer
+ 'step_size=0.000007,
+ n_iterations=10000,
+ tolerance=0',
+ 'relu');
+</pre>
+-# Check the results of the model
+<pre class="example">
+-- Set extended display on for easier reading of output
+\\x ON
+-- Neural net Initialization is non-deterministic, so your results may vary
+SELECT * FROM mlp_regress;
+</pre>
+Result:
+<pre class="result">
+-[ RECORD 1 ]--+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------
+coeff | {1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2.79506311399e-05,3.56715008915e-05,-6.09333559685e-05,0.000251228318768,-0.000224772841379,-3.71863030857e-05,-3.5757865148e-06,5.27936784854e-05,-2.48474166186e-05,6.19731184294e-05,3.07638968743e-05,6.8964698578e-06,0.000106016701083,-1.71484730318e-05,1.18691881812e-05,-0.000163975464208,0.000170026304906,3.11688265279e-05,0.000177050148787,-1.58265976603e-05,2.70144422657e-05,0.000112667883422,3.77575139073e-05,8.12474658795e-05,-7.90458917626e-05,0.000107566386158,-2.63771171506e-06,2.47996880915e-05,-0.00012642310887,0.000203827391081,0.000139315565565,4.86147243454e-05,-0.000176126471913,-6.47820782916e-05,-8.51592776447e-06,-6.60601176758e-05,2.91421874156e-05,6.3556873752e-05,0.000197557443129,0.000220531367259,0.000135036310289,0.000143735913975,-4.75034117786e-05,-0.000179547345838,-1.6919846786e-05,0.000162784312994,0.000268595819851,-0.000460066553287,8.69756071591e-05,-0.00311762727057,0.000126024763103,0.00020598824
2921,0.003463432426,-0.00729789075286,0.00151625867549,-0.000890852767597,-0.00525016037249,0.0031043106659,0.00798041103839,-0.00552693050079,0.0232180415786,0.0230489850143,-0.0437890272341,0.0165765426407,-0.248554261758,-7.81336427846e-05,0.00558145591752,0.283465844585,-0.571699956182,0.133474351994,-0.0785181945605,-0.419269930709,0.249547772912,0.631761009875,-0.431305975666,1,1,1,1,1,1,0.0158747497572,-9.02809160806e-05,0.00015574347618,4.10805373863e-06,0.00121532434965,0.101790351335,0.0647558401493,-0.00013654998677,-9.92872075948e-06,-5.5319694394e-05,0.00519320756484,0.412736586036,0.0011998026977,-1.53688189815e-05,1.94817888201e-05,-4.63111489966e-05,7.24547899029e-05,0.00880394144485,5.45309822095e-05,-0.000140943219275,-7.96211486227e-05,-1.04337307472e-05,0.000161936762028,0.00136273797767,-4.54737243585e-05,-3.4083840736e-05,3.69286883662e-05,9.9047243188e-08,3.75014011824e-06,-9.45366086368e-08,1,1,1,1,1,1,6.67488547054,0.102754199001,0.41668912471,0.008868672964
79,0.00136206007228,-9.88642499013e-05}
+loss | 144.965776158
+num_iterations | 10000
+</pre>
+-# Now let's look at the prediction functions. In the following examples we will
+use the training data set for prediction as well, which is not usual but serves to
+show the syntax. First we will test the classification exampel.
+The prediction is in the the estimated_class_text column with the
+actual value in the class_text column.
+<pre class="example">
+SELECT madlib.mlp_predict(
+ 'mlp_model', -- Model table
+ 'iris_data', -- Test data table
+ 'id', -- Id column in test table
+ 'mlp_prediction', -- Output table for predictions
+ 'response' -- Output classes, not probabilities
+ );
+SELECT * FROM mlp_prediction JOIN iris_data USING (id);
+</pre>
+Result:
+<pre class="result">
+ id | estimated_class_text | attributes | class_text | class
+----+----------------------+-------------------+-----------------+-------
+ 1 | Iris-setosa | {5.1,3.5,1.4,0.2} | Iris-setosa | 1
+ 2 | Iris-setosa | {4.9,3.0,1.4,0.2} | Iris-setosa | 1
+ 3 | Iris-setosa | {4.7,3.2,1.3,0.2} | Iris-setosa | 1
+ 4 | Iris-setosa | {4.6,3.1,1.5,0.2} | Iris-setosa | 1
+ 5 | Iris-setosa | {5.0,3.6,1.4,0.2} | Iris-setosa | 1
+ 6 | Iris-setosa | {5.4,3.9,1.7,0.4} | Iris-setosa | 1
+ 7 | Iris-setosa | {4.6,3.4,1.4,0.3} | Iris-setosa | 1
+ 8 | Iris-setosa | {5.0,3.4,1.5,0.2} | Iris-setosa | 1
+ 9 | Iris-setosa | {4.4,2.9,1.4,0.2} | Iris-setosa | 1
+ 10 | Iris-setosa | {4.9,3.1,1.5,0.1} | Iris-setosa | 1
+ 11 | Iris-versicolor | {7.0,3.2,4.7,1.4} | Iris-versicolor | 2
+ 12 | Iris-versicolor | {6.4,3.2,4.5,1.5} | Iris-versicolor | 2
+ 13 | Iris-versicolor | {6.9,3.1,4.9,1.5} | Iris-versicolor | 2
+ 14 | Iris-versicolor | {5.5,2.3,4.0,1.3} | Iris-versicolor | 2
+ 15 | Iris-versicolor | {6.5,2.8,4.6,1.5} | Iris-versicolor | 2
+ 16 | Iris-versicolor | {5.7,2.8,4.5,1.3} | Iris-versicolor | 2
+ 17 | Iris-versicolor | {6.3,3.3,4.7,1.6} | Iris-versicolor | 2
+ 18 | Iris-versicolor | {4.9,2.4,3.3,1.0} | Iris-versicolor | 2
+ 19 | Iris-versicolor | {6.6,2.9,4.6,1.3} | Iris-versicolor | 2
+ 20 | Iris-versicolor | {5.2,2.7,3.9,1.4} | Iris-versicolor | 2
+</pre>
+Prediction using the regression model:
+<pre class="example">
+SELECT madlib.mlp_predict(
+ 'mlp_regress', -- Model table
+ 'lin_housing', -- Test data table
+ 'id', -- Id column in test table
+ 'mlp_regress_prediction', -- Output table for predictions
+ 'response' -- Output values, not probabilities
+ );
+</pre>
+-# View results
+<pre class="example">
+SELECT * FROM lin_housing JOIN mlp_regress_prediction USING (id);
+</pre>
+Result for the regression model:
+<pre class="result">
+ id | x | grp_by_col | y | estimated_y
+----+-------------------------------------------------------------------------+------------+------+--------------------
+ 1 | {1,0.00632,18,2.31,0,0.538,6.575,65.2,4.09,1,296,15.3,396.9,4.98} | 1 | 24 | {23.2627062018087}
+ 2 | {1,0.02731,0,7.07,0,0.469,6.421,78.9,4.9671,2,242,17.8,396.9,9.14} | 1 | 21.6 | {25.7088419115781}
+ 3 | {1,0.02729,0,7.07,0,0.469,7.185,61.1,4.9671,2,242,17.8,392.83,4.03} | 1 | 34.7 | {27.5587003901404}
+ 4 | {1,0.03237,0,2.18,0,0.458,6.998,45.8,6.0622,3,222,18.7,394.63,2.94} | 1 | 33.4 | {31.1812237427816}
+ 5 | {1,0.06905,0,2.18,0,0.458,7.147,54.2,6.0622,3,222,18.7,396.9,5.33} | 1 | 36.2 | {30.3696873085477}
+ 6 | {1,0.02985,0,2.18,0,0.458,6.43,58.7,6.0622,3,222,18.7,394.12,5.21} | 1 | 28.7 | {29.5290259241882}
+ 7 | {1,0.08829,12.5,7.87,0,0.524,6.012,66.6,5.5605,5,311,15.2,395.6,12.43} | 1 | 22.9 | {21.1576051716888}
+ 8 | {1,0.14455,12.5,7.87,0,0.524,6.172,96.1,5.9505,5,311,15.2,396.9,19.15} | 1 | 27.1 | {17.6194200563055}
+ 9 | {1,0.21124,12.5,7.87,0,0.524,5.631,100,6.0821,5,311,15.2,386.63,29.93} | 1 | 16.5 | {15.1366297774139}
+10 | {1,0.17004,12.5,7.87,0,0.524,6.004,85.9,6.5921,5,311,15.2,386.71,17.1} | 1 | 18.9 | {17.6528662199369}
+11 | {1,0.22489,12.5,7.87,0,0.524,6.377,94.3,6.3467,5,311,15.2,392.52,20.45} | 1 | 15 | {17.2017487668181}
+12 | {1,0.11747,12.5,7.87,0,0.524,6.009,82.9,6.2267,5,311,15.2,396.9,13.27} | 1 | 18.9 | {19.4893860319992}
+13 | {1,0.09378,12.5,7.87,0,0.524,5.889,39,5.4509,5,311,15.2,390.5,15.71} | 1 | 21.7 | {23.2917226708039}
+14 | {1,0.62976,0,8.14,0,0.538,5.949,61.8,4.7075,4,307,21,396.9,8.26} | 1 | 20.4 | {22.8904812605193}
+15 | {1,0.63796,0,8.14,0,0.538,6.096,84.5,4.4619,4,307,21,380.02,10.26} | 1 | 18.2 | {18.2386754423677}
+16 | {1,0.62739,0,8.14,0,0.538,5.834,56.5,4.4986,4,307,21,395.62,8.47} | 1 | 19.9 | {23.28949550874}
+17 | {1,1.05393,0,8.14,0,0.538,5.935,29.3,4.4986,4,307,21,386.85,6.58} | 1 | 23.1 | {25.3288762085473}
+18 | {1,0.7842,0,8.14,0,0.538,5.99,81.7,4.2579,4,307,21,386.75,14.67} | 1 | 17.5 | {19.0203738118451}
+19 | {1,0.80271,0,8.14,0,0.538,5.456,36.6,3.7965,4,307,21,288.99,11.69} | 1 | 20.2 | {12.3162005347545}
+20 | {1,0.7258,0,8.14,0,0.538,5.727,69.5,3.7965,4,307,21,390.95,11.28} | 1 | 18.2 | {21.0902211848747}
+</pre>
+Note that the results you get for all examples may vary with the platform you are using.
+
+@anchor background
+@par Technical Background
--- End diff --
There is a design document (also available to users) which goes into much more detail on the specific implementation. I'm not exactly sure how much detail to go into for the user doc, so I will get input the team.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by njayaram2 <gi...@git.apache.org>.
Github user njayaram2 commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r128067814
--- Diff: src/ports/postgres/modules/convex/mlp.sql_in ---
@@ -0,0 +1,752 @@
+/* ----------------------------------------------------------------------- *//**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ *
+ * @file mlp.sql_in
+ *
+ * @brief SQL functions for multilayer perceptron
+ * @date June 2012
+ *
+ *
+ *//* ----------------------------------------------------------------------- */
+
+m4_include(`SQLCommon.m4')
+
+/**
+@addtogroup grp_mlp
+
+<div class="toc"><b>Contents</b><ul>
+<li class="level1"><a href="#mlp_classification">Classification</a></li>
+<li class="level1"><a href="#mlp_regression">Regression</a></li>
+<li class="level1"><a href="#optimization_params">Optimizer Parameters</a></li>
+<li class="level1"><a href="#predict">Prediction Functions/a></li>
+<li class="level1"><a href="#example">Examples</a></li>
+<li class="level1"><a href="#background">Technical Background</a></li>
+<li class="level1"><a href="#literature">Literature</a></li>
+<li class="level1"><a href="#related">Related Topics</a></li>
+</ul></div>
+
+Multilayer Perceptron (MLP) is a model for regression and
+classification
+
+Also called "vanilla neural networks", they consist of several
+fully connected hidden layers with non-linear activation
+functions. In the case of classification, the final layer of the
+neural net has as many nodes as classes, and the output of the
+neural net can be interpreted as the probability of a given input
+feature belonging to a specific class.
+
+
+@brief Solves classification and regression problems with several
+fully connected layers and nonlinear activation functions.
+
+@anchor mlp_classification
+@par Classification Training Function
+The mlp classification training function has the following format:
+<pre class="syntax">
+mlp_classification(
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+\b Arguments
+<DL class="arglist">
+ <DT>source_table</DT>
+ <DD>TEXT. Name of the table containing the training data.</DD>
+
+ <DT>model_table</DT>
+ <DD>TEXT. Name of the output table containing the model. Details of the output
+ tables are provided below.
+ </DD>
+
+ <DT>independent_varname</DT>
+ <DD>TEXT. Expression list to evaluate for the
+ independent variables. An intercept variable should not be included as part
+ of this expression. Please note that expression should be able to be cast
+ to DOUBLE PRECISION[].
+ </DD>
+
+ <DT>dependent_varname</DT>
+ <DD> TEXT. Name of the dependent variable column. For classification, supported types are:
+ text, varchar, character varying, char, character
+ integer, smallint, bigint, and boolean. </DD>
+
+ <DT>hidden_layer_sizes</DT>
+ <DD>INTEGER[], default: ARRAY[].
+ The number of neurons in each hidden layer. The length of this array will
+ determine the number of hidden layers. Empty for no hidden layers.
+ </DD>
+
+
+ <DT>optimizer_params (optional)</DT>
+ <DD>TEXT, default: NULL.
+ Parameters for optimization in a comma-separated string
+ of key-value pairs. See the description below for details.
+ </DD>
+
+ <DT>activation</DT>
+ <DD>TEXT, default: 'sigmoid'.
+ Activation function. Currently three functions are supported: 'sigmoid' (default),
+ 'relu', and 'tanh'. The text can be any prefix of the three
+ strings; for e.g., activation='s' will use the sigmoid activation.
+ </DD>
+</DL>
+
+<b>Output tables</b>
+<br>
+ The model table produced by mlp contains the following columns:
+ <table class="output">
+ <tr>
+ <th>coeffs</th>
+ <td>FLOAT8[]. Flat array containing the weights of the neural net</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>INTEGER. Number of iterations completed by stochastic gradient descent
+ algorithm. The algorithm either converged in this number of iterations
+ or hit the maximum number specified in the optimization parameters. </td>
+ </tr>
+ <tr>
+ <th>loss</th>
+ <td>FLOAT8. The cross entropy over the training data.
+ See Technical Background section below for more details.</td>
+ </tr>
+ </table>
+
+
+A summary table named \<model_table\>_summary is also created, which has the following columns:
+ <table class="output">
+ <tr>
+ <th>source_table</th>
+ <td>The source table.</td>
+ </tr>
+ <tr>
+ <th>dependent_varname</th>
+ <td>The dependent variable.</td>
+ </tr>
+ <tr>
+ <th>independent_varname</th>
+ <td>The independent variables.</td>
+ </tr>
+ <tr>
+ <th>tolerance</th>
+ <td>The tolerance as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>step_size</th>
+ <td>The step size as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>The number of iterations run</td>
+ </tr>
+ <tr>
+ <th>n_tries</th>
+ <td>The number of tries as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>layer_sizes</th>
+ <td>The number of units in each layer including the input and output layer.</td>
+ </tr>
+ <tr>
+ <th>activation_function</th>
+ <td>The activation function.</td>
+ </tr>
+ <tr>
+ <th>is_classification</th>
+ <td>True if the model was trained for classification, False if it was trained
+ for regression</td>
+ </tr>
+ <tr>
+ <th>classes</th>
+ <td>The classes which were trained against (empty for regression)</td>
+ </tr>
+
+ </table>
+
+
+@anchor mlp_regression
+@par Regression Training Function
+The mlp regression training function has the following format:
+<pre class="syntax">
+mlp_regression(source_table,
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+
+\b Arguments
+
+Specifications for regression are largely the same as for classification. In the
+model table, the loss will refer to mean square error instead of cross entropy. In the
+summary table, there is classes column. The following
+arguments have specifications which differ from mlp_classification:
+<DL class="arglist">
+<DT>dependent_varname</DT>
+ <DD>TEXT. Name of the dependent variable column.
+ For regression supported types are any numeric type, or array
+ or numeric types (for multiple regression).
+ </DD>
+</DL>
+
+
+@anchor optimizer_params
+@par Optimizer Parameters
+Parameters in this section are supplied in the \e optimizer_params argument as a string
+containing a comma-delimited list of name-value pairs. All of these named
+parameters are optional, and their order does not matter. You must use the
+format "<param_name> = <value>" to specify the value of a parameter, otherwise
+the parameter is ignored.
+
+
+<pre class="syntax">
+ 'step_size = <value>,
+ n_iterations = <value>,
+ n_tries = <value>,
+ tolerance = <value>'
+</pre>
+\b Optimizer Parameters
+<DL class="arglist">
+
+<DT>step_size</dt>
+<DD>Default: [0.001].
+Also known as the learning rate. A small value is usually desirable to
+ensure convergence, while a large value provides more room for progress during
+training. Since the best value depends on the condition number of the data, in
+practice one often tunes this parameter.
+</DD>
+
+
+<DT>n_iterations</dt>
+<DD>Default: [100]. The maximum number of iterations allowed.
+</DD>
+<DT>n_tries</dt>
+<DD>Default: [1]. Number of times to retrain the network with randomly initialized
+weights
+</DD>
+
+<DT>tolerance</dt>
+<DD>Default: 0.001. The criterion to end iterations. The training stops whenever
+<the difference between the training models of two consecutive iterations is
+<smaller than \e tolerance or the iteration number is larger than \e max_iter.
+</DD>
+
+</DL>
+
+@anchor predict
+@par Prediction Function
+Used to generate predictions given a previously trained model on novel data.
+The same syntax is used for classification, and regression.
+<pre class="syntax">
+mlp_predict(model_table,
+ data_table,
+ id_col_name,
+ output_table,
+ pred_type)
+</pre>
+
+\b Arguments
+<DL class="arglist">
+ <DT>model_table</DT>
+ <DD>TEXT. Model table produced by the training function.</DD>
+
+ <DT>data_table</DT>
+ <DD>TEXT. Name of the table containing the data for prediction. This table is expected
+ to contain the same input features that were used during training. The table should
+ also contain id_col_name used for identifying each row.</DD>
+
+ <DT>id_col_name</DT>
+ <DD>TEXT. The name of the id column in the input table.</DD>
+
+ <DT>output_table</DT>
+ <DD>TEXT. Name of the table where output predictions are written. If this
+table name is already in use, then an error is returned. Table contains:</DD>
+ <table class="output">
+ <tr>
+ <th>id</th>
+ <td>Gives the 'id' for each prediction, corresponding to each row from the data_table.</td>
+ </tr>
+ <tr>
+ <th>estimated_<COL_NAME></th>
+ <td>
+ (For pred_type='response') The estimated class
+ for classification or value for regression, where
+ <COL_NAME> is the name of the column to be
+ predicted from training data
+ </td>
+ </tr>
+ <tr>
+ <th>prob_<CLASS></th>
+ <td>
+ (For pred_type='prob' for classification) The
+ probability of a given class <CLASS> as given by
+ softmax. There will be one column for each class
+ in the training data.
+ </td>
+ </tr>
+
+
+ <DT>pred_type</DT>
+ <DD>TEXT.
+
+the type of output requested:
+'response' gives the actual prediction,
+'prob' gives the probability of each class.
+for regression, only type='response' is defined.
+The name of the id column in the input table.</DD>
+</DL>
+</table>
+
+@anchor example
+@par Examples
+-# Create an input data set.
+<pre class="example">
+CREATE TABLE iris_data(
+ id integer,
+ attributes numeric[],
+ class_text varchar,
+ class integer
+);
+INSERT INTO iris_data VALUES
+(1,ARRAY[5.1,3.5,1.4,0.2],'Iris-setosa',1),
+(2,ARRAY[4.9,3.0,1.4,0.2],'Iris-setosa',1),
+(3,ARRAY[4.7,3.2,1.3,0.2],'Iris-setosa',1),
+(4,ARRAY[4.6,3.1,1.5,0.2],'Iris-setosa',1),
+(5,ARRAY[5.0,3.6,1.4,0.2],'Iris-setosa',1),
+(6,ARRAY[5.4,3.9,1.7,0.4],'Iris-setosa',1),
+(7,ARRAY[4.6,3.4,1.4,0.3],'Iris-setosa',1),
+(8,ARRAY[5.0,3.4,1.5,0.2],'Iris-setosa',1),
+(9,ARRAY[4.4,2.9,1.4,0.2],'Iris-setosa',1),
+(10,ARRAY[4.9,3.1,1.5,0.1],'Iris-setosa',1),
+(11,ARRAY[7.0,3.2,4.7,1.4],'Iris-versicolor',2),
+(12,ARRAY[6.4,3.2,4.5,1.5],'Iris-versicolor',2),
+(13,ARRAY[6.9,3.1,4.9,1.5],'Iris-versicolor',2),
+(14,ARRAY[5.5,2.3,4.0,1.3],'Iris-versicolor',2),
+(15,ARRAY[6.5,2.8,4.6,1.5],'Iris-versicolor',2),
+(16,ARRAY[5.7,2.8,4.5,1.3],'Iris-versicolor',2),
+(17,ARRAY[6.3,3.3,4.7,1.6],'Iris-versicolor',2),
+(18,ARRAY[4.9,2.4,3.3,1.0],'Iris-versicolor',2),
+(19,ARRAY[6.6,2.9,4.6,1.3],'Iris-versicolor',2),
+(20,ARRAY[5.2,2.7,3.9,1.4],'Iris-versicolor',2);
+</pre>
+-# Generate a multilayer perceptron with a single hidden layer of 5 units.
+Use the attributes column as the independent variables, and use the class
+column as the classification. Set the tolerance to 0 so that 5000
+iterations will be run. Use a hyperbolic tangent activation function.
+The model will be written to mlp_model.
+<pre class="example">
+DROP TABLE IF EXISTS mlp_model;
+DROP TABLE IF EXISTS mlp_model_summary;
+SELECT madlib.mlp_classification(
+ 'iris_data', -- Source table
+ 'mlp_model', -- Destination table
+ 'attributes', -- Input features
+ 'class_text', -- Label
+ ARRAY[5], -- Number of units per layer
+ 'step_size=0.003,
+ n_iterations=5000,
+ tolerance=0', -- Optimizer params
+ 'tanh'); -- Activation function
+</pre>
+-# View the result for the model.
+<pre class="example">
+-- Set extended display on for easier reading of output
+\\x ON
+-- Neural net Initialization is non-deterministic, so your results may vary
+SELECT * FROM mlp_model;
+</pre>
+Result:
+<pre class="result">
+-[ RECORD 1 ]--+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+coeff | {1,1,1,1,1,0.136374930803,0.188739676875,0.662387810001,-1.03381622734,-0.469961067046,0.0614006983397,0.0811504589436,0.299008228258,-0.47391918521,-0.215098143699,0.10519213944,0.145844617525,0.511683525606,-0.800215552382,-0.36417142683,0.120751709056,0.167531106521,0.587074895969,-0.916946198095,-0.417055067449,0.0539541885146,0.0694359704131,0.262598585854,-0.419234805076,-0.189915344282,1,1,1,1,1,1,0.105645702152,1.46247470474,0.484457903226,0.965962824478,1.19361986431,0.419805760087,-0.105696503487,-1.46245956666,-0.484427811691,-0.965730981426,-1.19365280555,-0.419973628863}
+loss | 0.0184092375519
+num_iterations | 5000
+</pre>
+-# Next train a regression example. First create some test data. This dataset
+contains housing prices data.
+<pre class="example">
+CREATE TABLE lin_housing (id serial, x float8[], grp_by_col int, y float8);
+COPY lin_housing (x, grp_by_col, y) FROM STDIN NULL '?' DELIMITER '|';
+{1,0.00632,18.00,2.310,0,0.5380,6.5750,65.20,4.0900,1,296.0,15.30,396.90,4.98} | 1 | 24.00
+{1,0.02731,0.00,7.070,0,0.4690,6.4210,78.90,4.9671,2,242.0,17.80,396.90,9.14} | 1 | 21.60
+{1,0.02729,0.00,7.070,0,0.4690,7.1850,61.10,4.9671,2,242.0,17.80,392.83,4.03} | 1 | 34.70
+{1,0.03237,0.00,2.180,0,0.4580,6.9980,45.80,6.0622,3,222.0,18.70,394.63,2.94} | 1 | 33.40
+{1,0.06905,0.00,2.180,0,0.4580,7.1470,54.20,6.0622,3,222.0,18.70,396.90,5.33} | 1 | 36.20
+{1,0.02985,0.00,2.180,0,0.4580,6.4300,58.70,6.0622,3,222.0,18.70,394.12,5.21} | 1 | 28.70
+{1,0.08829,12.50,7.870,0,0.5240,6.0120,66.60,5.5605,5,311.0,15.20,395.60,12.43} | 1 | 22.90
+{1,0.14455,12.50,7.870,0,0.5240,6.1720,96.10,5.9505,5,311.0,15.20,396.90,19.15} | 1 | 27.10
+{1,0.21124,12.50,7.870,0,0.5240,5.6310,100.00,6.0821,5,311.0,15.20,386.63,29.93} | 1 | 16.50
+{1,0.17004,12.50,7.870,0,0.5240,6.0040,85.90,6.5921,5,311.0,15.20,386.71,17.10} | 1 | 18.90
+{1,0.22489,12.50,7.870,0,0.5240,6.3770,94.30,6.3467,5,311.0,15.20,392.52,20.45} | 1 | 15.00
+{1,0.11747,12.50,7.870,0,0.5240,6.0090,82.90,6.2267,5,311.0,15.20,396.90,13.27} | 1 | 18.90
+{1,0.09378,12.50,7.870,0,0.5240,5.8890,39.00,5.4509,5,311.0,15.20,390.50,15.71} | 1 | 21.70
+{1,0.62976,0.00,8.140,0,0.5380,5.9490,61.80,4.7075,4,307.0,21.00,396.90,8.26} | 1 | 20.40
+{1,0.63796,0.00,8.140,0,0.5380,6.0960,84.50,4.4619,4,307.0,21.00,380.02,10.26} | 1 | 18.20
+{1,0.62739,0.00,8.140,0,0.5380,5.8340,56.50,4.4986,4,307.0,21.00,395.62,8.47} | 1 | 19.90
+{1,1.05393,0.00,8.140,0,0.5380,5.9350,29.30,4.4986,4,307.0,21.00,386.85,6.58} | 1 | 23.10
+{1,0.78420,0.00,8.140,0,0.5380,5.9900,81.70,4.2579,4,307.0,21.00,386.75,14.67} | 1 | 17.50
+{1,0.80271,0.00,8.140,0,0.5380,5.4560,36.60,3.7965,4,307.0,21.00,288.99,11.69} | 1 | 20.20
+{1,0.72580,0.00,8.140,0,0.5380,5.7270,69.50,3.7965,4,307.0,21.00,390.95,11.28} | 1 | 18.20
+\\.
--- End diff --
Copying the above lines as is gave me an error while executing it in psql. It's probably due to the space/tab between the `|` inside each copy. Please remove the spaces and have something like this instead:
`{1,0.00632,18.00,2.310,0,0.5380,6.5750,65.20,4.0900,1,296.0,15.30,396.90,4.98}|1|24.00`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by njayaram2 <gi...@git.apache.org>.
Github user njayaram2 commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r128116084
--- Diff: src/modules/convex/task/mlp.hpp ---
@@ -0,0 +1,342 @@
+/* ----------------------------------------------------------------------- *//**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ * @file mlp.hpp
+ *
+ * This file contains objective function related computation, which is called
+ * by classes in algo/, e.g., loss, gradient functions
+ *
+ *//* ----------------------------------------------------------------------- */
+
+#ifndef MADLIB_MODULES_CONVEX_TASK_MLP_HPP_
+#define MADLIB_MODULES_CONVEX_TASK_MLP_HPP_
+
+namespace madlib {
+
+namespace modules {
+
+namespace convex {
+
+// Use Eigen
+using namespace madlib::dbal::eigen_integration;
+
+template <class Model, class Tuple>
+class MLP {
+public:
+ typedef Model model_type;
+ typedef Tuple tuple_type;
+ typedef typename Tuple::independent_variables_type
+ independent_variables_type;
+ typedef typename Tuple::dependent_variable_type dependent_variable_type;
+
+ static void gradientInPlace(
+ model_type &model,
+ const independent_variables_type &y,
+ const dependent_variable_type &z,
+ const double &stepsize);
+
+ static double loss(
+ const model_type &model,
+ const independent_variables_type &y,
+ const dependent_variable_type &z);
+
+ static ColumnVector predict(
+ const model_type &model,
+ const independent_variables_type &y,
+ const int is_response);
+
+ const static int RELU = 0;
+ const static int SIGMOID = 1;
+ const static int TANH = 2;
+
+ static double sigmoid(const double &xi) {
+ return 1. / (1. + std::exp(-xi));
+ }
+
+ static double relu(const double &xi) {
+ return xi*(xi>0);
+ }
+
+ static double tanh(const double &xi) {
+ return std::tanh(xi);
+ }
+
+
+private:
+
+ static double sigmoidDerivative(const double &xi) {
+ double value = sigmoid(xi);
+ return value * (1. - value);
+ }
+
+ static double reluDerivative(const double &xi) {
+ return xi>0;
+ }
+
+ static double tanhDerivative(const double &xi) {
+ double value = tanh(xi);
+ return 1-value*value;
+ }
+
+ static void feedForward(
+ const model_type &model,
+ const independent_variables_type &y,
+ std::vector<ColumnVector> &net,
+ std::vector<ColumnVector> &x);
+
+ static void endLayerDeltaError(
+ const std::vector<ColumnVector> &net,
+ const std::vector<ColumnVector> &x,
+ const dependent_variable_type &z,
+ ColumnVector &delta_N);
+
+ static void errorBackPropagation(
+ const ColumnVector &delta_N,
+ const std::vector<ColumnVector> &net,
+ const model_type &model,
+ std::vector<ColumnVector> &delta);
+};
+
+template <class Model, class Tuple>
+void
+MLP<Model, Tuple>::gradientInPlace(
+ model_type &model,
+ const independent_variables_type &y,
+ const dependent_variable_type &z,
+ const double &stepsize) {
+ (void) model;
+ (void) z;
+ (void) y;
+ (void) stepsize;
+ std::vector<ColumnVector> net;
+ std::vector<ColumnVector> x;
+ std::vector<ColumnVector> delta;
+ ColumnVector delta_N;
+
+ feedForward(model, y, net, x);
+ endLayerDeltaError(net, x, z, delta_N);
+ errorBackPropagation(delta_N, net, model, delta);
+
+ uint16_t N = model.u.size(); // assuming nu. of layers >= 1
+ uint16_t k, s, j;
+ // #TODO fix lambda
+ float lambda = 0;
+
+ std::vector<uint16_t> n; n.clear(); //nu. of units in each layer
+
+ n.push_back(model.u[0].rows() - 1);
+ for (k = 1; k <= N; k ++) {
+ n.push_back(model.u[k-1].cols() - 1);
+ }
+
+ for (k=1; k <= N; k++){
+ for (s=0; s <= n[k-1]; s++){
+ for (j=1; j <= n[k]; j++){
+ model.u[k-1](s,j) -= stepsize * (delta[k](j) * x[k-1](s) + lambda * model.u[k-1](s,j));
+ }
+ }
+ }
+}
+
+template <class Model, class Tuple>
+double
+MLP<Model, Tuple>::loss(
+ const model_type &model,
+ const independent_variables_type &y,
+ const dependent_variable_type &z) {
+ // Here we compute the loss. In the case of regression we use sum of square errors
+ // In the case of classification the loss term is cross entropy.
+ std::vector<ColumnVector> net;
+ std::vector<ColumnVector> x;
+
+ feedForward(model, y, net, x);
+ double loss = 0.;
+ uint16_t j;
+
+ for (j = 1; j < z.rows() + 1; j ++) {
+ if(model.is_classification){
+ // Cross entropy: RHS term is negative
+ loss -= z(j-1)*std::log(x.back()(j)) + (1-z(j-1))*std::log(1-x.back()(j));
+ // Computed like https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/neural_network/_base.py#L222
+ //loss -= z(j-1)*std::log(x.back()(j));
+ }else{
+ double diff = x.back()(j) - z(j-1);
+ loss += diff * diff;
+ }
+ }
+ if(!model.is_classification){
+ loss /= 2.;
+ }else{
+ // Give the average
+ loss /= z.rows();
+ }
+ return loss;
+}
+
+template <class Model, class Tuple>
+ColumnVector
+MLP<Model, Tuple>::predict(
+ const model_type &model,
+ const independent_variables_type &y,
+ const int is_response
+ ) {
+ (void) model;
+ (void) y;
+ std::vector<ColumnVector> net;
+ std::vector<ColumnVector> x;
+
+ feedForward(model, y, net, x);
+ // Don't return the offset
+ ColumnVector output = x.back().tail(x.back().size()-1);
+ if(is_response){
+ int max_idx;
+ output.maxCoeff(&max_idx);
+ output.resize(1);
+ output[0] = (double)max_idx;
+ }
+ return output;
+}
+
+
+template <class Model, class Tuple>
+void
+// #TODO Change y to x and x to o
+MLP<Model, Tuple>::feedForward(
+ const model_type &model,
+ const independent_variables_type &y,
+ std::vector<ColumnVector> &net,
+ std::vector<ColumnVector> &x){
+ // meta data and x_k^0 = 1
+ uint16_t k, j, s;
+ uint16_t N = model.u.size(); // assuming >= 1
+ net.resize(N + 1);
+ x.resize(N + 1);
+
+ std::vector<uint16_t> n; n.clear();
+ n.push_back(model.u[0].rows() - 1);
+ x[0].resize(n[0] + 1);
+ x[0](0) = 1.;
+ for (k = 1; k <= N; k ++) {
+ n.push_back(model.u[k-1].cols() - 1);
+ net[k].resize(n[k] + 1);
+ x[k].resize(n[k] + 1);
+ // Bias
+ x[k](0) = 1.;
+ }
+
+ // y is a mapped parameter from DB, aligning with x here
+ for (j = 1; j <= n[0]; j ++) { x[0](j) = y(j-1); }
+
+ for (k = 1; k < N; k ++) {
+ for (j = 1; j <= n[k]; j ++) {
+ net[k](j) = 0.;
+ for (s = 0; s <= n[k-1]; s ++) {
+ net[k](j) += x[k-1](s) * model.u[k-1](s, j);
+ }
+ if(model.activation==RELU)
+ x[k](j) = relu(net[k](j));
+ else if(model.activation==SIGMOID)
+ x[k](j) = sigmoid(net[k](j));
+ else if(model.activation==TANH)
+ x[k](j) = tanh(net[k](j));
+ else
+ elog(WARNING,"Invalid activation");
+ }
+ }
+
+ // output layer computation
+ for (j = 1; j <= n[N]; j ++) {
+ x[N](j) = 0.;
+ for (s = 0; s <= n[N-1]; s ++) {
+ x[N](j) += x[N-1](s) * model.u[N-1](s, j);
+ }
+ }
+ // Numerically stable calculation of softmax
+ ColumnVector last_x = x[N].tail(n[N]);
+ if(model.is_classification){
+ double max_x = last_x.maxCoeff();
+ last_x = (last_x.array() - max_x).exp();
+ last_x /= last_x.sum();
+ }
+ x[N].tail(n[N]) = last_x;
+}
+
+template <class Model, class Tuple>
+void
+MLP<Model, Tuple>::endLayerDeltaError(
+ const std::vector<ColumnVector> &net,
+ const std::vector<ColumnVector> &x,
+ const dependent_variable_type &z,
+ ColumnVector &delta_N) {
+ //meta data
+ uint16_t t;
+ uint16_t N = x.size() - 1; // assuming >= 1
+ uint16_t n_N = x[N].rows() - 1;
+ delta_N.resize(n_N + 1);
+
+ for (t = 1; t <= n_N; t ++) {
+ delta_N(t) = (x[N](t) - z(t-1));
+ }
+}
+
+template <class Model, class Tuple>
+void
+MLP<Model, Tuple>::errorBackPropagation(
+ const ColumnVector &delta_N,
+ const std::vector<ColumnVector> &net,
+ const model_type &model,
+ std::vector<ColumnVector> &delta) {
+ // meta data
+ uint16_t k, j, t;
+ uint16_t N = model.u.size(); // assuming >= 1
+ delta.resize(N + 1);
+
+ std::vector<uint16_t> n; n.clear();
+ n.push_back(model.u[0].rows() - 1);
+ for (k = 1; k <= N; k ++) {
+ n.push_back(model.u[k-1].cols() - 1);
+ delta[k].resize(n[k]+1);
+ }
+ delta[N] = delta_N;
+
+ for (k = N - 1; k >= 1; k --) {
+ for (j = 0; j <= n[k]; j ++) {
+ delta[k](j) = 0.;
+ for (t = 1; t <= n[k+1]; t ++) {
+ delta[k](j) += delta[k+1](t) * model.u[k](j, t);
+ }
+ if(model.activation==RELU)
+ delta[k](j) = delta[k](j) * reluDerivative(net[k](j));
+ else if(model.activation==SIGMOID)
+ delta[k](j) = delta[k](j) * sigmoidDerivative(net[k](j));
+ else if(model.activation==TANH)
+ delta[k](j) = delta[k](j) * tanhDerivative(net[k](j));
+ else
+ elog(WARNING,"Invalid activation");
--- End diff --
Ideally it won't come to this `else` because things are validated in python. If it indeed is invalid, then we should probably throw an error instead of a warning here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by cooper-sloan <gi...@git.apache.org>.
Github user cooper-sloan commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r128626337
--- Diff: src/modules/convex/task/mlp.hpp ---
@@ -0,0 +1,334 @@
+/* ----------------------------------------------------------------------- *//**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ * @file mlp.hpp
+ *
+ * This file contains objective function related computation, which is called
+ * by classes in algo/, e.g., loss, gradient functions
+ *
+ *//* ----------------------------------------------------------------------- */
+
+#ifndef MADLIB_MODULES_CONVEX_TASK_MLP_HPP_
+#define MADLIB_MODULES_CONVEX_TASK_MLP_HPP_
+
+namespace madlib {
+
+namespace modules {
+
+namespace convex {
+
+// Use Eigen
+using namespace madlib::dbal::eigen_integration;
+
+template <class Model, class Tuple>
+class MLP {
+public:
+ typedef Model model_type;
+ typedef Tuple tuple_type;
+ typedef typename Tuple::independent_variables_type
+ independent_variables_type;
+ typedef typename Tuple::dependent_variable_type dependent_variable_type;
+
+ static void gradientInPlace(
+ model_type &model,
+ const independent_variables_type &y,
+ const dependent_variable_type &z,
+ const double &stepsize);
+
+ static double loss(
+ const model_type &model,
+ const independent_variables_type &y,
+ const dependent_variable_type &z);
+
+ static ColumnVector predict(
+ const model_type &model,
+ const independent_variables_type &y,
+ const bool get_class);
+
+ const static int RELU = 0;
+ const static int SIGMOID = 1;
+ const static int TANH = 2;
+
+ static double sigmoid(const double &xi) {
+ return 1. / (1. + std::exp(-xi));
+ }
+
+ static double relu(const double &xi) {
+ return xi*(xi>0);
+ }
+
+ static double tanh(const double &xi) {
+ return std::tanh(xi);
+ }
+
+
+private:
+
+ static double sigmoidDerivative(const double &xi) {
+ double value = sigmoid(xi);
+ return value * (1. - value);
+ }
+
+ static double reluDerivative(const double &xi) {
+ return xi>0;
+ }
+
+ static double tanhDerivative(const double &xi) {
+ double value = tanh(xi);
+ return 1-value*value;
+ }
+
+ static void feedForward(
+ const model_type &model,
+ const independent_variables_type &y,
+ std::vector<ColumnVector> &net,
+ std::vector<ColumnVector> &x);
+
+ static void endLayerDeltaError(
+ const std::vector<ColumnVector> &net,
+ const std::vector<ColumnVector> &x,
+ const dependent_variable_type &z,
+ ColumnVector &delta_N);
+
+ static void errorBackPropagation(
+ const ColumnVector &delta_N,
+ const std::vector<ColumnVector> &net,
+ const model_type &model,
+ std::vector<ColumnVector> &delta);
+};
+
+template <class Model, class Tuple>
+void
+MLP<Model, Tuple>::gradientInPlace(
+ model_type &model,
+ const independent_variables_type &y,
+ const dependent_variable_type &z,
+ const double &stepsize) {
+ (void) model;
+ (void) z;
+ (void) y;
+ (void) stepsize;
+ std::vector<ColumnVector> net;
+ std::vector<ColumnVector> x;
+ std::vector<ColumnVector> delta;
+ ColumnVector delta_N;
+
+ feedForward(model, y, net, x);
+ endLayerDeltaError(net, x, z, delta_N);
+ errorBackPropagation(delta_N, net, model, delta);
+
+ uint16_t N = model.u.size(); // assuming nu. of layers >= 1
+ uint16_t k, s, j;
+ // #TODO fix lambda
--- End diff --
We will use a different approach, making use of the convex architecture. So I will remove this altogether.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by edespino <gi...@git.apache.org>.
Github user edespino commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r127056665
--- Diff: src/ports/postgres/modules/convex/mlp.sql_in ---
@@ -0,0 +1,736 @@
+/* ----------------------------------------------------------------------- *//**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ *
+ * @file mlp.sql_in
+ *
+ * @brief SQL functions for multi-layer perceptron
+ * @date June 2012
+ *
+ *
+ *//* ----------------------------------------------------------------------- */
+
+m4_include(`SQLCommon.m4')
+
+/**
+@addtogroup grp_mlp
+
+<div class="toc"><b>Contents</b><ul>
+<li class="level1"><a href="#mlp_classification">Classification</a></li>
+<li class="level1"><a href="#mlp_regression">Regression</a></li>
+<li class="level1"><a href="#optimization_params">Optimization Parameters</a></li>
+<li class="level1"><a href="#predict">Prediction Functions/a></li>
+<li class="level1"><a href="#example">Examples</a></li>
+<li class="level1"><a href="#background">Technical Background</a></li>
+<li class="level1"><a href="#literature">Literature</a></li>
+<li class="level1"><a href="#related">Related Topics</a></li>
+</ul></div>
+
+Multi Layer Perceptron (MLP) is a model for regression and
+classification
+
+Also called "vanilla neural networks", they consist of several
+fully connected hidden layers with non-linear activation
+functions. In the case of classification, the final layer of the
+neural net has as many nodes as classes, and the output of the
+neural net can be interpreted as the probability of a given input
+feature belonging to a specific class.
+
+
+@brief Solves classification and regression problems with several
+fully connected layers and nonlinear activation functions.
+
+@anchor mlp_classification
+@par Classification Training Function
+The mlp classification training function has the following format:
+<pre class="syntax">
+mlp_classification(
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+\b Arguments
+<DL class="arglist">
+ <DT>source_table</DT>
+ <DD>TEXT. Name of the table containing the training data.</DD>
+
+ <DT>model_table</DT>
+ <DD>TEXT. Name of the output table containing the model. Details of the output
+ tables are provided below.
+ </DD>
+
+ <DT>independent_varname</DT>
+ <DD>TEXT. Expression list to evaluate for the
+ independent variables. An intercept variable should not be included as part
+ of this expression. Please note that expression should be able to be cast
+ to DOUBLE PRECISION[].
+ </DD>
+
+ <DT>dependent_varname</DT>
+ <DD> TEXT. Name of the dependent variable column. For classification, supported types are:
+ text, varchar, character varying, char, character
+ integer, smallint, bigint, and boolean. </DD>
+
+ <DT>hidden_layer_sizes</DT>
+ <DD>INTEGER[], default: ARRAY[].
+ The number of neurons in each hidden layer. The length of this array will
+ determine the number of hidden layers. Empty for no hidden layers.
+ </DD>
+
+
+ <DT>optimizer_params (optional)</DT>
+ <DD>TEXT, default: NULL.
+ Parameters for optimization in a comma-separated string
+ of key-value pairs. See the description below for details.
+ </DD>
+
+ <DT>activation</DT>
+ <DD>TEXT, default: 'sigmoid'.
+ Activation function. Currently three functions are supported: 'sigmoid' (default),
+ 'relu', and 'tanh'. The text can be any subset of the three
+ strings; for e.g., activation='s' will use the sigmoid activation.
+ </DD>
+</DL>
+
+<b>Output tables</b>
+<br>
+ The model table produced by mlp contains the following columns:
+ <table class="output">
+ <tr>
+ <th>coeffs</th>
+ <td>FLOAT8[]. Flat array containing the weights of the neural net</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>INTEGER. Number of iterations completed by stochastic gradient descent
+ algorithm. The algorithm either converged in this number of iterations
+ or hit the maximum number specified in the optimization parameters. </td>
+ </tr>
+ <tr>
+ <th>loss</th>
+ <td>FLOAT8. The cross entropy over the training data.
+ See Technical Background section below for more details.</td>
+ </tr>
+ </table>
+
+
+A summary table named \<model_table\>_summary is also created, which has the following columns:
+ <table class="output">
+ <tr>
+ <th>source_table</th>
+ <td>The source table.</td>
+ </tr>
+ <tr>
+ <th>dependent_varname</th>
+ <td>The dependent variable.</td>
+ </tr>
+ <tr>
+ <th>independent_varname</th>
+ <td>The independent variables.</td>
+ </tr>
+ <tr>
+ <th>tolerance</th>
+ <td>The tolerance as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>step_size</th>
+ <td>The step size as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>The number of iterations run</td>
+ </tr>
+ <tr>
+ <th>n_tries</th>
+ <td>The number of tries as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>layer_sizes</th>
+ <td>The number of units in each layer including the input and output layer.</td>
+ </tr>
+ <tr>
+ <th>activation_function</th>
+ <td>The activation function.</td>
+ </tr>
+ <tr>
+ <th>is_classification</th>
+ <td>True if the model was trained for classification, False if it was trained
+ for regression</td>
+ </tr>
+ <tr>
+ <th>classes</th>
+ <td>The classes which were trained against (empty for regression)</td>
+ </tr>
+
+ </table>
+
+
+@anchor mlp_regression
+@par Regression Training Function
+The mlp regression training function has the following format:
+<pre class="syntax">
+mlp_regression(source_table,
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+
+\b Arguments
+
+Specifications for regression are largely the same as for classification. In the
+model table, the loss will refer to mean square error instead of cross entropy. In the
+summary table, there is classes column. The following
+arguments have specifications which differ from mlp_classification:
+<DL class="arglist">
+<DT>dependent_varname</DT>
+ <DD>TEXT. Name of the dependent variable column.
+ For regression supported types are any numeric type, or array
+ or numeric types (for multiple regression).
+ </DD>
+</DL>
+
+
+@anchor optimizer_params
+@par Optimizer Parameters
+Parameters in this section are supplied in the \e optimizer_params argument as a string
+containing a comma-delimited list of name-value pairs. All of these named
+parameters are optional, and their order does not matter. You must use the
+format "<param_name> = <value>" to specify the value of a parameter, otherwise
+the parameter is ignored.
+
+
+<pre class="syntax">
+ 'step_size = <value>,
+ n_iterations = <value>,
+ n_tries = <value>,
+ tolerance = <value>'
+</pre>
+\b Optimizer Parameters
+<DL class="arglist">
+
+<DT>step_size</dt>
+<DD>Default: [0.001].
+Also known as the learning rate. A small value is usually desirable to
+ensure convergence, while a large value provides more room for progress during
+training. Since the best value depends on the condition number of the data, in
+practice one often tunes this parameter.
+</DD>
+
+
+<DT>n_iterations</dt>
+<DD>Default: [100]. The maximum number of iterations allowed.
+</DD>
+<DT>n_tries</dt>
+<DD>Default: [1]. Number of times to retrain the network with randomly initialized
+weights
+</DD>
+
+<DT>tolerance</dt>
+<DD>Default: 0.001. The criterion to end iterations. The training stops whenever
+<the difference between the training models of two consecutive iterations is
+<smaller than \e tolerance or the iteration number is larger than \e max_iter.
+</DD>
+
+</DL>
+
+@anchor predict
+@par Prediction Function
+Used to generate predictions given a previously trained model on novel data.
+The same syntax is used for classification, and regression.
+<pre class="syntax">
+mlp_predict(model_table,
+ data_table,
+ id_col_name,
+ output_table,
+ pred_type)
+</pre>
+
+\b Arguments
+<DL class="arglist">
+ <DT>model_table</DT>
+ <DD>TEXT. Model table produced by the training function.</DD>
+
+ <DT>data_table</DT>
+ <DD>TEXT. Name of the table containing the data for prediction. This table is expected
+ to contain the same input features that were used during training. The table should
+ also contain id_col_name used for identifying each row.</DD>
+
+ <DT>id_col_name</DT>
+ <DD>TEXT. The name of the id column in the input table.</DD>
+
+ <DT>output_table</DT>
+ <DD>TEXT. Name of the table where output predictions are written. If this
+table name is already in use, then an error is returned. Table contains:</DD>
+ <table class="output">
+ <tr>
+ <th>id</th>
+ <td>Gives the 'id' for each prediction, corresponding to each row from the data_table.</td>
+ </tr>
+ <tr>
+ <th>estimated_<COL_NAME></th>
+ <td>
+ (For pred_type='response') The estimated class
+ for classification or value for regression, where
+ <COL_NAME> is the name of the column to be
+ predicted from training data
+ </td>
+ </tr>
+ <tr>
+ <th>prob_<CLASS></th>
+ <td>
+ (For pred_type='prob' for classification) The
+ probability of a given class <CLASS> as given by
+ softmax. There will be one column for each class
+ in the training data.
+ </td>
+ </tr>
+
+
+ <DT>pred_type</DT>
+ <DD>TEXT.
+
+the type of output requested:
+'response' gives the actual prediction,
+'prob' gives the probability of each class.
+for regression, only type='response' is defined.
+The name of the id column in the input table.</DD>
+</DL>
+</table>
+
+@anchor example
+@par Examples
+-# Create an input data set.
+<pre class="example">
+CREATE TABLE iris_data(
+ id integer,
+ attributes numeric[],
+ class_text varchar,
+ class integer
+);
+INSERT INTO iris_data VALUES
+(1,ARRAY[5.1,3.5,1.4,0.2],'Iris-setosa',1),
+(2,ARRAY[4.9,3.0,1.4,0.2],'Iris-setosa',1),
+(3,ARRAY[4.7,3.2,1.3,0.2],'Iris-setosa',1),
+(4,ARRAY[4.6,3.1,1.5,0.2],'Iris-setosa',1),
+(5,ARRAY[5.0,3.6,1.4,0.2],'Iris-setosa',1),
+(6,ARRAY[5.4,3.9,1.7,0.4],'Iris-setosa',1),
+(7,ARRAY[4.6,3.4,1.4,0.3],'Iris-setosa',1),
+(8,ARRAY[5.0,3.4,1.5,0.2],'Iris-setosa',1),
+(9,ARRAY[4.4,2.9,1.4,0.2],'Iris-setosa',1),
+(10,ARRAY[4.9,3.1,1.5,0.1],'Iris-setosa',1),
+(11,ARRAY[7.0,3.2,4.7,1.4],'Iris-versicolor',2),
+(12,ARRAY[6.4,3.2,4.5,1.5],'Iris-versicolor',2),
+(13,ARRAY[6.9,3.1,4.9,1.5],'Iris-versicolor',2),
+(14,ARRAY[5.5,2.3,4.0,1.3],'Iris-versicolor',2),
+(15,ARRAY[6.5,2.8,4.6,1.5],'Iris-versicolor',2),
+(16,ARRAY[5.7,2.8,4.5,1.3],'Iris-versicolor',2),
+(17,ARRAY[6.3,3.3,4.7,1.6],'Iris-versicolor',2),
+(18,ARRAY[4.9,2.4,3.3,1.0],'Iris-versicolor',2),
+(19,ARRAY[6.6,2.9,4.6,1.3],'Iris-versicolor',2),
+(20,ARRAY[5.2,2.7,3.9,1.4],'Iris-versicolor',2);
+</pre>
+-# Generate a multilayer perceptron with a single hidden layer of 5 units.
+Use the attributes column as the independent variables, and use the class
+column as the classification. Set the tolerance to 0 so that 5000
+iterations will be run. Use a hyperbolic tangent activation function.
+The model will be written to mlp_result.
--- End diff --
In the example, should the model destination table be written to `mlp_model` instead of `mlp_result`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by edespino <gi...@git.apache.org>.
Github user edespino commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r127094613
--- Diff: src/ports/postgres/modules/convex/mlp.sql_in ---
@@ -0,0 +1,736 @@
+/* ----------------------------------------------------------------------- *//**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ *
+ * @file mlp.sql_in
+ *
+ * @brief SQL functions for multi-layer perceptron
+ * @date June 2012
+ *
+ *
+ *//* ----------------------------------------------------------------------- */
+
+m4_include(`SQLCommon.m4')
+
+/**
+@addtogroup grp_mlp
+
+<div class="toc"><b>Contents</b><ul>
+<li class="level1"><a href="#mlp_classification">Classification</a></li>
+<li class="level1"><a href="#mlp_regression">Regression</a></li>
+<li class="level1"><a href="#optimization_params">Optimization Parameters</a></li>
+<li class="level1"><a href="#predict">Prediction Functions/a></li>
+<li class="level1"><a href="#example">Examples</a></li>
+<li class="level1"><a href="#background">Technical Background</a></li>
+<li class="level1"><a href="#literature">Literature</a></li>
+<li class="level1"><a href="#related">Related Topics</a></li>
+</ul></div>
+
+Multi Layer Perceptron (MLP) is a model for regression and
+classification
+
+Also called "vanilla neural networks", they consist of several
+fully connected hidden layers with non-linear activation
+functions. In the case of classification, the final layer of the
+neural net has as many nodes as classes, and the output of the
+neural net can be interpreted as the probability of a given input
+feature belonging to a specific class.
+
+
+@brief Solves classification and regression problems with several
+fully connected layers and nonlinear activation functions.
+
+@anchor mlp_classification
+@par Classification Training Function
+The mlp classification training function has the following format:
+<pre class="syntax">
+mlp_classification(
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+\b Arguments
+<DL class="arglist">
+ <DT>source_table</DT>
+ <DD>TEXT. Name of the table containing the training data.</DD>
+
+ <DT>model_table</DT>
+ <DD>TEXT. Name of the output table containing the model. Details of the output
+ tables are provided below.
+ </DD>
+
+ <DT>independent_varname</DT>
+ <DD>TEXT. Expression list to evaluate for the
+ independent variables. An intercept variable should not be included as part
+ of this expression. Please note that expression should be able to be cast
+ to DOUBLE PRECISION[].
+ </DD>
+
+ <DT>dependent_varname</DT>
+ <DD> TEXT. Name of the dependent variable column. For classification, supported types are:
+ text, varchar, character varying, char, character
+ integer, smallint, bigint, and boolean. </DD>
+
+ <DT>hidden_layer_sizes</DT>
+ <DD>INTEGER[], default: ARRAY[].
+ The number of neurons in each hidden layer. The length of this array will
+ determine the number of hidden layers. Empty for no hidden layers.
+ </DD>
+
+
+ <DT>optimizer_params (optional)</DT>
+ <DD>TEXT, default: NULL.
+ Parameters for optimization in a comma-separated string
+ of key-value pairs. See the description below for details.
+ </DD>
+
+ <DT>activation</DT>
+ <DD>TEXT, default: 'sigmoid'.
+ Activation function. Currently three functions are supported: 'sigmoid' (default),
+ 'relu', and 'tanh'. The text can be any subset of the three
+ strings; for e.g., activation='s' will use the sigmoid activation.
+ </DD>
+</DL>
+
+<b>Output tables</b>
+<br>
+ The model table produced by mlp contains the following columns:
+ <table class="output">
+ <tr>
+ <th>coeffs</th>
+ <td>FLOAT8[]. Flat array containing the weights of the neural net</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>INTEGER. Number of iterations completed by stochastic gradient descent
+ algorithm. The algorithm either converged in this number of iterations
+ or hit the maximum number specified in the optimization parameters. </td>
+ </tr>
+ <tr>
+ <th>loss</th>
+ <td>FLOAT8. The cross entropy over the training data.
+ See Technical Background section below for more details.</td>
+ </tr>
+ </table>
+
+
+A summary table named \<model_table\>_summary is also created, which has the following columns:
+ <table class="output">
+ <tr>
+ <th>source_table</th>
+ <td>The source table.</td>
+ </tr>
+ <tr>
+ <th>dependent_varname</th>
+ <td>The dependent variable.</td>
+ </tr>
+ <tr>
+ <th>independent_varname</th>
+ <td>The independent variables.</td>
+ </tr>
+ <tr>
+ <th>tolerance</th>
+ <td>The tolerance as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>step_size</th>
+ <td>The step size as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>The number of iterations run</td>
+ </tr>
+ <tr>
+ <th>n_tries</th>
+ <td>The number of tries as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>layer_sizes</th>
+ <td>The number of units in each layer including the input and output layer.</td>
+ </tr>
+ <tr>
+ <th>activation_function</th>
+ <td>The activation function.</td>
+ </tr>
+ <tr>
+ <th>is_classification</th>
+ <td>True if the model was trained for classification, False if it was trained
+ for regression</td>
+ </tr>
+ <tr>
+ <th>classes</th>
+ <td>The classes which were trained against (empty for regression)</td>
+ </tr>
+
+ </table>
+
+
+@anchor mlp_regression
+@par Regression Training Function
+The mlp regression training function has the following format:
+<pre class="syntax">
+mlp_regression(source_table,
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+
+\b Arguments
+
+Specifications for regression are largely the same as for classification. In the
+model table, the loss will refer to mean square error instead of cross entropy. In the
+summary table, there is classes column. The following
+arguments have specifications which differ from mlp_classification:
+<DL class="arglist">
+<DT>dependent_varname</DT>
+ <DD>TEXT. Name of the dependent variable column.
+ For regression supported types are any numeric type, or array
+ or numeric types (for multiple regression).
+ </DD>
+</DL>
+
+
+@anchor optimizer_params
+@par Optimizer Parameters
+Parameters in this section are supplied in the \e optimizer_params argument as a string
+containing a comma-delimited list of name-value pairs. All of these named
+parameters are optional, and their order does not matter. You must use the
+format "<param_name> = <value>" to specify the value of a parameter, otherwise
+the parameter is ignored.
+
+
+<pre class="syntax">
+ 'step_size = <value>,
+ n_iterations = <value>,
+ n_tries = <value>,
+ tolerance = <value>'
+</pre>
+\b Optimizer Parameters
+<DL class="arglist">
+
+<DT>step_size</dt>
+<DD>Default: [0.001].
+Also known as the learning rate. A small value is usually desirable to
+ensure convergence, while a large value provides more room for progress during
+training. Since the best value depends on the condition number of the data, in
+practice one often tunes this parameter.
+</DD>
+
+
+<DT>n_iterations</dt>
+<DD>Default: [100]. The maximum number of iterations allowed.
+</DD>
+<DT>n_tries</dt>
+<DD>Default: [1]. Number of times to retrain the network with randomly initialized
+weights
+</DD>
+
+<DT>tolerance</dt>
+<DD>Default: 0.001. The criterion to end iterations. The training stops whenever
+<the difference between the training models of two consecutive iterations is
+<smaller than \e tolerance or the iteration number is larger than \e max_iter.
+</DD>
+
+</DL>
+
+@anchor predict
+@par Prediction Function
+Used to generate predictions given a previously trained model on novel data.
+The same syntax is used for classification, and regression.
+<pre class="syntax">
+mlp_predict(model_table,
+ data_table,
+ id_col_name,
+ output_table,
+ pred_type)
+</pre>
+
+\b Arguments
+<DL class="arglist">
+ <DT>model_table</DT>
+ <DD>TEXT. Model table produced by the training function.</DD>
+
+ <DT>data_table</DT>
+ <DD>TEXT. Name of the table containing the data for prediction. This table is expected
+ to contain the same input features that were used during training. The table should
+ also contain id_col_name used for identifying each row.</DD>
+
+ <DT>id_col_name</DT>
+ <DD>TEXT. The name of the id column in the input table.</DD>
+
+ <DT>output_table</DT>
+ <DD>TEXT. Name of the table where output predictions are written. If this
+table name is already in use, then an error is returned. Table contains:</DD>
+ <table class="output">
+ <tr>
+ <th>id</th>
+ <td>Gives the 'id' for each prediction, corresponding to each row from the data_table.</td>
+ </tr>
+ <tr>
+ <th>estimated_<COL_NAME></th>
+ <td>
+ (For pred_type='response') The estimated class
+ for classification or value for regression, where
+ <COL_NAME> is the name of the column to be
+ predicted from training data
+ </td>
+ </tr>
+ <tr>
+ <th>prob_<CLASS></th>
+ <td>
+ (For pred_type='prob' for classification) The
+ probability of a given class <CLASS> as given by
+ softmax. There will be one column for each class
+ in the training data.
+ </td>
+ </tr>
+
+
+ <DT>pred_type</DT>
+ <DD>TEXT.
+
+the type of output requested:
+'response' gives the actual prediction,
+'prob' gives the probability of each class.
+for regression, only type='response' is defined.
+The name of the id column in the input table.</DD>
+</DL>
+</table>
+
+@anchor example
+@par Examples
+-# Create an input data set.
+<pre class="example">
+CREATE TABLE iris_data(
+ id integer,
+ attributes numeric[],
+ class_text varchar,
+ class integer
+);
+INSERT INTO iris_data VALUES
+(1,ARRAY[5.1,3.5,1.4,0.2],'Iris-setosa',1),
+(2,ARRAY[4.9,3.0,1.4,0.2],'Iris-setosa',1),
+(3,ARRAY[4.7,3.2,1.3,0.2],'Iris-setosa',1),
+(4,ARRAY[4.6,3.1,1.5,0.2],'Iris-setosa',1),
+(5,ARRAY[5.0,3.6,1.4,0.2],'Iris-setosa',1),
+(6,ARRAY[5.4,3.9,1.7,0.4],'Iris-setosa',1),
+(7,ARRAY[4.6,3.4,1.4,0.3],'Iris-setosa',1),
+(8,ARRAY[5.0,3.4,1.5,0.2],'Iris-setosa',1),
+(9,ARRAY[4.4,2.9,1.4,0.2],'Iris-setosa',1),
+(10,ARRAY[4.9,3.1,1.5,0.1],'Iris-setosa',1),
+(11,ARRAY[7.0,3.2,4.7,1.4],'Iris-versicolor',2),
+(12,ARRAY[6.4,3.2,4.5,1.5],'Iris-versicolor',2),
+(13,ARRAY[6.9,3.1,4.9,1.5],'Iris-versicolor',2),
+(14,ARRAY[5.5,2.3,4.0,1.3],'Iris-versicolor',2),
+(15,ARRAY[6.5,2.8,4.6,1.5],'Iris-versicolor',2),
+(16,ARRAY[5.7,2.8,4.5,1.3],'Iris-versicolor',2),
+(17,ARRAY[6.3,3.3,4.7,1.6],'Iris-versicolor',2),
+(18,ARRAY[4.9,2.4,3.3,1.0],'Iris-versicolor',2),
+(19,ARRAY[6.6,2.9,4.6,1.3],'Iris-versicolor',2),
+(20,ARRAY[5.2,2.7,3.9,1.4],'Iris-versicolor',2);
+</pre>
+-# Generate a multilayer perceptron with a single hidden layer of 5 units.
+Use the attributes column as the independent variables, and use the class
+column as the classification. Set the tolerance to 0 so that 5000
+iterations will be run. Use a hyperbolic tangent activation function.
+The model will be written to mlp_result.
--- End diff --
There is a potential similar issue in `src/ports/postgres/modules/convex/mlp_igd.py_in`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by edespino <gi...@git.apache.org>.
Github user edespino commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r127066684
--- Diff: src/modules/convex/mlp_igd.cpp ---
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ * @file mlp_igd.cpp
+ *
+ * @brief Low-rank Matrix Factorization functions
+ *
+ *//* ----------------------------------------------------------------------- */
+#include <boost/lexical_cast.hpp>
+
+#include <dbconnector/dbconnector.hpp>
+#include <modules/shared/HandleTraits.hpp>
+
+#include "mlp_igd.hpp"
+
+#include "task/mlp.hpp"
+#include "algo/igd.hpp"
+#include "algo/loss.hpp"
+
+#include "type/tuple.hpp"
+#include "type/model.hpp"
+#include "type/state.hpp"
+
+namespace madlib {
+
+namespace modules {
+
+namespace convex {
+
+// These 2 classes contain public static methods that can be called
+typedef IGD<MLPIGDState<MutableArrayHandle<double> >, MLPIGDState<ArrayHandle<double> >,
+ MLP<MLPModel<MutableArrayHandle<double> >, MLPTuple > > MLPIGDAlgorithm;
+
+typedef Loss<MLPIGDState<MutableArrayHandle<double> >, MLPIGDState<ArrayHandle<double> >,
+ MLP<MLPModel<MutableArrayHandle<double> >, MLPTuple > > MLPLossAlgorithm;
+
+typedef MLP<MLPModel<MutableArrayHandle<double> >,MLPTuple> MLPTask;
+
+/**
+ * @brief Perform the multi-layer perceptron transition step
+ *
+ * Called for each tuple.
+ */
+AnyType
+mlp_igd_transition::run(AnyType &args) {
+ // For the first tuple: args[0] is nothing more than a marker that
+ // indicates that we should do some initial operations.
+ // For other tuples: args[0] holds the computation state until last tuple
+ MLPIGDState<MutableArrayHandle<double> > state = args[0];
+
+ // initilize the state if first tuple
+ if (state.algo.numRows == 0) {
+ if (!args[3].isNull()) {
+ MLPIGDState<ArrayHandle<double> > previousState = args[3];
+
+ state.allocate(*this, previousState.task.numberOfStages,
+ previousState.task.numbersOfUnits);
+ state = previousState;
+ } else {
+ // configuration parameters
+ ArrayHandle<double> numbersOfUnits = args[4].getAs<ArrayHandle<double> >();
+ if (numbersOfUnits.size() <= 1) {
+ throw std::runtime_error("Invalid parameter: numbers_of_units "
+ "has too few entries");
+ } else if (numbersOfUnits.size() >=
+ std::numeric_limits<uint16_t>::max()) {
+ throw std::runtime_error("Invalid parameter: numbers_of_units "
+ "has too many entries");
+ }
+ size_t k;
+ for (k = 0; k < numbersOfUnits.size(); k ++) {
+ if (numbersOfUnits[k] == 0) {
+ throw std::runtime_error("Invalid parameter: numbers_of_units "
+ "has zero entry");
+ }
+ }
+
+ double stepsize = args[5].getAs<double>();
+ if (stepsize <= 0.) {
+ throw std::runtime_error("Invalid parameter: stepsize <= 0.0");
--- End diff --
Suggest changing `stepsize` string in error message to match the actual parameter name (`step_size`).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by cooper-sloan <gi...@git.apache.org>.
Github user cooper-sloan commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r127092987
--- Diff: src/ports/postgres/modules/convex/mlp_igd.py_in ---
@@ -0,0 +1,754 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+@file mlp_igd.py_in
+
+@brief Multilayer perceptron using IGD: Driver functions
+
+@namespace mlp_igd
+"""
+import plpy
+
+from utilities.control import MinWarning
+from utilities.utilities import add_postfix
+from utilities.utilities import py_list_to_sql_string
+from utilities.utilities import extract_keyvalue_params
+from utilities.utilities import _assert
+from utilities.utilities import unique_string
+from utilities.utilities import strip_end_quotes
+
+from utilities.validate_args import cols_in_tbl_valid
+from utilities.validate_args import input_tbl_valid
+from utilities.validate_args import is_var_valid
+from utilities.validate_args import output_tbl_valid
+from utilities.validate_args import get_expr_type
+from utilities.validate_args import array_col_has_same_dimension
+from utilities.validate_args import array_col_dimension
+
+
+def mlp(schema_madlib, source_table, output_table, independent_varname,
+ dependent_varname, hidden_layer_sizes,
+ optimizer_param_str, activation, is_classification, **kwargs):
+ """
+ Args:
+ @param schema_madlib
+ @param source_table
+ @param output_table
+ @param independent_varname
+ @param dependent_varname
+ @param hidden_layer_sizes
+ @param optimizer_param_str
+
+ Returns:
+ None
+ """
+ with MinWarning('info'):
+ optimizer_params = _get_optimizer_params(optimizer_param_str or "")
+ _validate_args(source_table, output_table, independent_varname,
+ dependent_varname, hidden_layer_sizes,
+ optimizer_params, is_classification)
+
+ current_iteration = 1
+ prev_state = None
+ tolerance = optimizer_params["tolerance"]
+ n_iterations = optimizer_params["n_iterations"]
+ step_size = optimizer_params["step_size"]
+ n_tries = optimizer_params["n_tries"]
+ activation_name = _get_activation_function_name(activation)
+ activation_index = _get_activation_index(activation_name)
+ num_input_nodes = array_col_dimension(source_table, independent_varname)
+ num_output_nodes = 0
+ classes = []
+ dependent_type = get_expr_type(dependent_varname, source_table)
+ original_dependent_varname = dependent_varname
+
+ if(is_classification):
+ dependent_variable_sql = """
+ SELECT DISTINCT {dependent_varname}
+ FROM {source_table}
+ """.format(dependent_varname=dependent_varname,
+ source_table=source_table)
+ labels = plpy.execute(dependent_variable_sql)
+ one_hot_dependent_varname = 'ARRAY['
+ num_output_nodes = len(labels)
+ for x in labels:
+ label = _format_label(x[dependent_varname])
+ classes.append(label)
+ one_hot_dependent_varname += dependent_varname + "=" + str(label) + ","
+ # Remove the last comma
+ one_hot_dependent_varname = one_hot_dependent_varname[:-1]
+ one_hot_dependent_varname += ']::integer[]'
+ dependent_varname = one_hot_dependent_varname
+ if(not is_classification):
+ if "[]" not in dependent_type:
+ dependent_varname = "ARRAY[" + dependent_varname + "]"
+ num_output_nodes = array_col_dimension(source_table, dependent_varname)
+ layer_sizes = [num_input_nodes] + hidden_layer_sizes + [num_output_nodes]
+
+ while True:
+ if prev_state:
+ prev_state_str = py_list_to_sql_string(prev_state, array_type="double precision")
+ else:
+ prev_state_str = "(NULL)::DOUBLE PRECISION[]"
+ train_sql = """
+ SELECT
+ {schema_madlib}.mlp_igd_step(
+ ({independent_varname})::DOUBLE PRECISION[],
+ ({dependent_varname})::DOUBLE PRECISION[],
+ {prev_state},
+ {layer_sizes},
+ ({step_size})::FLOAT8,
+ {activation},
+ {is_classification}) as curr_state
+ FROM {source_table} AS _src
+ """.format(schema_madlib=schema_madlib,
+ independent_varname=independent_varname,
+ dependent_varname=dependent_varname,
+ prev_state=prev_state_str,
+ # C++ uses double internally
+ layer_sizes=py_list_to_sql_string(layer_sizes,
+ array_type="double precision"),
+ step_size=step_size,
+ source_table=source_table,
+ activation=activation_index,
+ is_classification=int(is_classification))
+ curr_state = plpy.execute(train_sql)[0]["curr_state"]
+ dist_sql = """
+ SELECT {schema_madlib}.internal_mlp_igd_distance(
+ {prev_state},
+ {curr_state}) as state_dist
+ """.format(schema_madlib=schema_madlib,
+ prev_state=prev_state_str,
+ curr_state=py_list_to_sql_string(curr_state, "double precision"),
+ tolerance=tolerance)
+ state_dist = plpy.execute(dist_sql)[0]["state_dist"]
+ if ((state_dist and state_dist < tolerance) or
+ current_iteration > n_iterations):
+ break
+ prev_state = curr_state
+ # plpy.info("Iteration: "+str(current_iteration)+"/"+str(n_iterations))
+ current_iteration += 1
+ _build_model_table(schema_madlib, output_table, curr_state, n_iterations)
+ layer_sizes_str=py_list_to_sql_string(layer_sizes, array_type="integer")
+ classes_str=py_list_to_sql_string([strip_end_quotes(cl, "'") for cl in classes],
+ array_type=dependent_type)
+ # TODO validation for summary
+ summary_table_creation_query = """
+ CREATE TABLE {output_table}_summary(
+ source_table TEXT,
+ independent_varname TEXT,
+ dependent_varname TEXT,
+ tolerance FLOAT,
+ step_size FLOAT,
+ n_iterations INTEGER,
+ n_tries INTEGER,
+ layer_sizes INTEGER[],
+ activation_function TEXT,
+ is_classification BOOLEAN,
+ classes {dependent_type}[]
+ )""".format(output_table=output_table,
+ dependent_type=dependent_type)
+
+ summary_table_update_query = """
+ INSERT INTO {output_table}_summary VALUES(
+ '{source_table}',
+ '{independent_varname}',
+ '{original_dependent_varname}',
+ {tolerance},
+ {step_size},
+ {n_iterations},
+ {n_tries},
+ {layer_sizes_str},
+ '{activation_name}',
+ {is_classification},
+ {classes_str}
+ )
+ """.format(**locals())
+ plpy.execute(summary_table_creation_query)
+ plpy.execute(summary_table_update_query)
+# ----------------------------------------------------------------------
+
+
+def _build_model_table(schema_madlib, output_table, final_state, n_iterations):
+ final_state_str = py_list_to_sql_string(final_state, array_type="double precision")
+
+ model_table_query = """
+ CREATE TABLE {output_table} AS
+ SELECT
+ (result).coeff AS coeff,
+ (result).loss AS loss,
+ {n_iterations} AS num_iterations
+ -- (result).num_rows_processed AS num_rows_processed,
+ -- n_tuples_including_nulls - (result).num_rows_processed
+ FROM (
+ SELECT
+ {schema_madlib}.internal_mlp_igd_result(
+ {final_state_str}
+ ) AS result
+ ) rel_state_subq
+ """.format(**locals())
+ plpy.execute(model_table_query)
+# ----------------------------------------------------------------------
+
+
+def _get_optimizer_params(param_str):
+ params_defaults = {
+ "step_size": (0.001, float),
+ "n_iterations": (100, int),
+ "n_tries": (1, int),
+ "tolerance": (0.001, float),
+ }
+ param_defaults = dict([(k, v[0]) for k, v in params_defaults.items()])
+ param_types = dict([(k, v[1]) for k, v in params_defaults.items()])
+
+ if not param_str:
+ return param_defaults
+
+ name_value = extract_keyvalue_params(param_str, param_types, param_defaults,
+ ignore_invalid=True)
--- End diff --
In regards to 'nan' issue. We have run across this in the past. It has to do with regularization, which is scheduled for phase 2. Once we implement regularization this issue should go away. However I will try recreate this bug and verify that is the true issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib pull request #149: MLP: Multilayer Perceptron
Posted by edespino <gi...@git.apache.org>.
Github user edespino commented on a diff in the pull request:
https://github.com/apache/incubator-madlib/pull/149#discussion_r127317719
--- Diff: src/ports/postgres/modules/convex/mlp.sql_in ---
@@ -0,0 +1,744 @@
+/* ----------------------------------------------------------------------- *//**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ *
+ * @file mlp.sql_in
+ *
+ * @brief SQL functions for multilayer perceptron
+ * @date June 2012
+ *
+ *
+ *//* ----------------------------------------------------------------------- */
+
+m4_include(`SQLCommon.m4')
+
+/**
+@addtogroup grp_mlp
+
+<div class="toc"><b>Contents</b><ul>
+<li class="level1"><a href="#mlp_classification">Classification</a></li>
+<li class="level1"><a href="#mlp_regression">Regression</a></li>
+<li class="level1"><a href="#optimization_params">Optimizer Parameters</a></li>
+<li class="level1"><a href="#predict">Prediction Functions/a></li>
+<li class="level1"><a href="#example">Examples</a></li>
+<li class="level1"><a href="#background">Technical Background</a></li>
+<li class="level1"><a href="#literature">Literature</a></li>
+<li class="level1"><a href="#related">Related Topics</a></li>
+</ul></div>
+
+Multilayer Perceptron (MLP) is a model for regression and
+classification
+
+Also called "vanilla neural networks", they consist of several
+fully connected hidden layers with non-linear activation
+functions. In the case of classification, the final layer of the
+neural net has as many nodes as classes, and the output of the
+neural net can be interpreted as the probability of a given input
+feature belonging to a specific class.
+
+
+@brief Solves classification and regression problems with several
+fully connected layers and nonlinear activation functions.
+
+@anchor mlp_classification
+@par Classification Training Function
+The mlp classification training function has the following format:
+<pre class="syntax">
+mlp_classification(
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+\b Arguments
+<DL class="arglist">
+ <DT>source_table</DT>
+ <DD>TEXT. Name of the table containing the training data.</DD>
+
+ <DT>model_table</DT>
+ <DD>TEXT. Name of the output table containing the model. Details of the output
+ tables are provided below.
+ </DD>
+
+ <DT>independent_varname</DT>
+ <DD>TEXT. Expression list to evaluate for the
+ independent variables. An intercept variable should not be included as part
+ of this expression. Please note that expression should be able to be cast
+ to DOUBLE PRECISION[].
+ </DD>
+
+ <DT>dependent_varname</DT>
+ <DD> TEXT. Name of the dependent variable column. For classification, supported types are:
+ text, varchar, character varying, char, character
+ integer, smallint, bigint, and boolean. </DD>
+
+ <DT>hidden_layer_sizes</DT>
+ <DD>INTEGER[], default: ARRAY[].
+ The number of neurons in each hidden layer. The length of this array will
+ determine the number of hidden layers. Empty for no hidden layers.
+ </DD>
+
+
+ <DT>optimizer_params (optional)</DT>
+ <DD>TEXT, default: NULL.
+ Parameters for optimization in a comma-separated string
+ of key-value pairs. See the description below for details.
+ </DD>
+
+ <DT>activation</DT>
+ <DD>TEXT, default: 'sigmoid'.
+ Activation function. Currently three functions are supported: 'sigmoid' (default),
+ 'relu', and 'tanh'. The text can be any subset of the three
+ strings; for e.g., activation='s' will use the sigmoid activation.
+ </DD>
+</DL>
+
+<b>Output tables</b>
+<br>
+ The model table produced by mlp contains the following columns:
+ <table class="output">
+ <tr>
+ <th>coeffs</th>
+ <td>FLOAT8[]. Flat array containing the weights of the neural net</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>INTEGER. Number of iterations completed by stochastic gradient descent
+ algorithm. The algorithm either converged in this number of iterations
+ or hit the maximum number specified in the optimization parameters. </td>
+ </tr>
+ <tr>
+ <th>loss</th>
+ <td>FLOAT8. The cross entropy over the training data.
+ See Technical Background section below for more details.</td>
+ </tr>
+ </table>
+
+
+A summary table named \<model_table\>_summary is also created, which has the following columns:
+ <table class="output">
+ <tr>
+ <th>source_table</th>
+ <td>The source table.</td>
+ </tr>
+ <tr>
+ <th>dependent_varname</th>
+ <td>The dependent variable.</td>
+ </tr>
+ <tr>
+ <th>independent_varname</th>
+ <td>The independent variables.</td>
+ </tr>
+ <tr>
+ <th>tolerance</th>
+ <td>The tolerance as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>step_size</th>
+ <td>The step size as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>n_iterations</th>
+ <td>The number of iterations run</td>
+ </tr>
+ <tr>
+ <th>n_tries</th>
+ <td>The number of tries as given in optimizer_params.</td>
+ </tr>
+ <tr>
+ <th>layer_sizes</th>
+ <td>The number of units in each layer including the input and output layer.</td>
+ </tr>
+ <tr>
+ <th>activation_function</th>
+ <td>The activation function.</td>
+ </tr>
+ <tr>
+ <th>is_classification</th>
+ <td>True if the model was trained for classification, False if it was trained
+ for regression</td>
+ </tr>
+ <tr>
+ <th>classes</th>
+ <td>The classes which were trained against (empty for regression)</td>
+ </tr>
+
+ </table>
+
+
+@anchor mlp_regression
+@par Regression Training Function
+The mlp regression training function has the following format:
+<pre class="syntax">
+mlp_regression(source_table,
+ source_table,
+ output_table,
+ independent_varname,
+ dependent_varname,
+ hidden_layer_sizes,
+ optimizer_params,
+ activation
+ )
+</pre>
+
+\b Arguments
+
+Specifications for regression are largely the same as for classification. In the
+model table, the loss will refer to mean square error instead of cross entropy. In the
+summary table, there is classes column. The following
+arguments have specifications which differ from mlp_classification:
+<DL class="arglist">
+<DT>dependent_varname</DT>
+ <DD>TEXT. Name of the dependent variable column.
+ For regression supported types are any numeric type, or array
+ or numeric types (for multiple regression).
+ </DD>
+</DL>
+
+
+@anchor optimizer_params
+@par Optimizer Parameters
+Parameters in this section are supplied in the \e optimizer_params argument as a string
+containing a comma-delimited list of name-value pairs. All of these named
+parameters are optional, and their order does not matter. You must use the
+format "<param_name> = <value>" to specify the value of a parameter, otherwise
+the parameter is ignored.
+
+
+<pre class="syntax">
+ 'step_size = <value>,
+ n_iterations = <value>,
+ n_tries = <value>,
+ tolerance = <value>'
+</pre>
+\b Optimizer Parameters
+<DL class="arglist">
+
+<DT>step_size</dt>
+<DD>Default: [0.001].
+Also known as the learning rate. A small value is usually desirable to
+ensure convergence, while a large value provides more room for progress during
+training. Since the best value depends on the condition number of the data, in
+practice one often tunes this parameter.
+</DD>
+
+
+<DT>n_iterations</dt>
+<DD>Default: [100]. The maximum number of iterations allowed.
+</DD>
+<DT>n_tries</dt>
+<DD>Default: [1]. Number of times to retrain the network with randomly initialized
+weights
+</DD>
+
+<DT>tolerance</dt>
+<DD>Default: 0.001. The criterion to end iterations. The training stops whenever
+<the difference between the training models of two consecutive iterations is
+<smaller than \e tolerance or the iteration number is larger than \e max_iter.
+</DD>
+
+</DL>
+
+@anchor predict
+@par Prediction Function
+Used to generate predictions given a previously trained model on novel data.
+The same syntax is used for classification, and regression.
+<pre class="syntax">
+mlp_predict(model_table,
+ data_table,
+ id_col_name,
+ output_table,
+ pred_type)
+</pre>
+
+\b Arguments
+<DL class="arglist">
+ <DT>model_table</DT>
+ <DD>TEXT. Model table produced by the training function.</DD>
+
+ <DT>data_table</DT>
+ <DD>TEXT. Name of the table containing the data for prediction. This table is expected
+ to contain the same input features that were used during training. The table should
+ also contain id_col_name used for identifying each row.</DD>
+
+ <DT>id_col_name</DT>
+ <DD>TEXT. The name of the id column in the input table.</DD>
+
+ <DT>output_table</DT>
+ <DD>TEXT. Name of the table where output predictions are written. If this
+table name is already in use, then an error is returned. Table contains:</DD>
+ <table class="output">
+ <tr>
+ <th>id</th>
+ <td>Gives the 'id' for each prediction, corresponding to each row from the data_table.</td>
+ </tr>
+ <tr>
+ <th>estimated_<COL_NAME></th>
+ <td>
+ (For pred_type='response') The estimated class
+ for classification or value for regression, where
+ <COL_NAME> is the name of the column to be
+ predicted from training data
+ </td>
+ </tr>
+ <tr>
+ <th>prob_<CLASS></th>
+ <td>
+ (For pred_type='prob' for classification) The
+ probability of a given class <CLASS> as given by
+ softmax. There will be one column for each class
+ in the training data.
+ </td>
+ </tr>
+
+
+ <DT>pred_type</DT>
+ <DD>TEXT.
+
+the type of output requested:
+'response' gives the actual prediction,
+'prob' gives the probability of each class.
+for regression, only type='response' is defined.
+The name of the id column in the input table.</DD>
+</DL>
+</table>
+
+@anchor example
+@par Examples
+-# Create an input data set.
+<pre class="example">
+CREATE TABLE iris_data(
+ id integer,
+ attributes numeric[],
+ class_text varchar,
+ class integer
+);
+INSERT INTO iris_data VALUES
+(1,ARRAY[5.1,3.5,1.4,0.2],'Iris-setosa',1),
+(2,ARRAY[4.9,3.0,1.4,0.2],'Iris-setosa',1),
+(3,ARRAY[4.7,3.2,1.3,0.2],'Iris-setosa',1),
+(4,ARRAY[4.6,3.1,1.5,0.2],'Iris-setosa',1),
+(5,ARRAY[5.0,3.6,1.4,0.2],'Iris-setosa',1),
+(6,ARRAY[5.4,3.9,1.7,0.4],'Iris-setosa',1),
+(7,ARRAY[4.6,3.4,1.4,0.3],'Iris-setosa',1),
+(8,ARRAY[5.0,3.4,1.5,0.2],'Iris-setosa',1),
+(9,ARRAY[4.4,2.9,1.4,0.2],'Iris-setosa',1),
+(10,ARRAY[4.9,3.1,1.5,0.1],'Iris-setosa',1),
+(11,ARRAY[7.0,3.2,4.7,1.4],'Iris-versicolor',2),
+(12,ARRAY[6.4,3.2,4.5,1.5],'Iris-versicolor',2),
+(13,ARRAY[6.9,3.1,4.9,1.5],'Iris-versicolor',2),
+(14,ARRAY[5.5,2.3,4.0,1.3],'Iris-versicolor',2),
+(15,ARRAY[6.5,2.8,4.6,1.5],'Iris-versicolor',2),
+(16,ARRAY[5.7,2.8,4.5,1.3],'Iris-versicolor',2),
+(17,ARRAY[6.3,3.3,4.7,1.6],'Iris-versicolor',2),
+(18,ARRAY[4.9,2.4,3.3,1.0],'Iris-versicolor',2),
+(19,ARRAY[6.6,2.9,4.6,1.3],'Iris-versicolor',2),
+(20,ARRAY[5.2,2.7,3.9,1.4],'Iris-versicolor',2);
+</pre>
+-# Generate a multilayer perceptron with a single hidden layer of 5 units.
+Use the attributes column as the independent variables, and use the class
+column as the classification. Set the tolerance to 0 so that 5000
+iterations will be run. Use a hyperbolic tangent activation function.
+The model will be written to mlp_model.
+<pre class="example">
+SELECT madlib.mlp_classification(
+ 'iris_data', -- Source table
+ 'mlp_model', -- Destination table
+ 'attributes', -- Input features
+ 'class_text', -- Label
+ ARRAY[5], -- Number of units per layer
+ 'step_size=0.003,
+ n_iterations=5000,
+ tolerance=0', -- Optimizer params
+ 'tanh'); -- Activation function
+</pre>
+-# View the result for the model.
+<pre class="example">
+-- Set extended display on for easier reading of output
+\\x ON
+-- Neural net Initialization is non-deterministic, so your results may vary
+SELECT * FROM mlp_model;
+</pre>
+Result:
+<pre class="result">
+-[ RECORD 1 ]--+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+coeff | {1,1,1,1,1,0.136374930803,0.188739676875,0.662387810001,-1.03381622734,-0.469961067046,0.0614006983397,0.0811504589436,0.299008228258,-0.47391918521,-0.215098143699,0.10519213944,0.145844617525,0.511683525606,-0.800215552382,-0.36417142683,0.120751709056,0.167531106521,0.587074895969,-0.916946198095,-0.417055067449,0.0539541885146,0.0694359704131,0.262598585854,-0.419234805076,-0.189915344282,1,1,1,1,1,1,0.105645702152,1.46247470474,0.484457903226,0.965962824478,1.19361986431,0.419805760087,-0.105696503487,-1.46245956666,-0.484427811691,-0.965730981426,-1.19365280555,-0.419973628863}
+loss | 0.0184092375519
+num_iterations | 5000
+</pre>
+-# Next train a regression example. First create some test data. This dataset
+contains housing prices data.
+<pre class="example">
+CREATE TABLE lin_housing (id serial, x float8[], grp_by_col int, y float8);
+COPY lin_housing (x, grp_by_col, y) FROM STDIN NULL '?' DELIMITER '|';
+{1,0.00632,18.00,2.310,0,0.5380,6.5750,65.20,4.0900,1,296.0,15.30,396.90,4.98} | 1 | 24.00
+{1,0.02731,0.00,7.070,0,0.4690,6.4210,78.90,4.9671,2,242.0,17.80,396.90,9.14} | 1 | 21.60
+{1,0.02729,0.00,7.070,0,0.4690,7.1850,61.10,4.9671,2,242.0,17.80,392.83,4.03} | 1 | 34.70
+{1,0.03237,0.00,2.180,0,0.4580,6.9980,45.80,6.0622,3,222.0,18.70,394.63,2.94} | 1 | 33.40
+{1,0.06905,0.00,2.180,0,0.4580,7.1470,54.20,6.0622,3,222.0,18.70,396.90,5.33} | 1 | 36.20
+{1,0.02985,0.00,2.180,0,0.4580,6.4300,58.70,6.0622,3,222.0,18.70,394.12,5.21} | 1 | 28.70
+{1,0.08829,12.50,7.870,0,0.5240,6.0120,66.60,5.5605,5,311.0,15.20,395.60,12.43} | 1 | 22.90
+{1,0.14455,12.50,7.870,0,0.5240,6.1720,96.10,5.9505,5,311.0,15.20,396.90,19.15} | 1 | 27.10
+{1,0.21124,12.50,7.870,0,0.5240,5.6310,100.00,6.0821,5,311.0,15.20,386.63,29.93} | 1 | 16.50
+{1,0.17004,12.50,7.870,0,0.5240,6.0040,85.90,6.5921,5,311.0,15.20,386.71,17.10} | 1 | 18.90
+{1,0.22489,12.50,7.870,0,0.5240,6.3770,94.30,6.3467,5,311.0,15.20,392.52,20.45} | 1 | 15.00
+{1,0.11747,12.50,7.870,0,0.5240,6.0090,82.90,6.2267,5,311.0,15.20,396.90,13.27} | 1 | 18.90
+{1,0.09378,12.50,7.870,0,0.5240,5.8890,39.00,5.4509,5,311.0,15.20,390.50,15.71} | 1 | 21.70
+{1,0.62976,0.00,8.140,0,0.5380,5.9490,61.80,4.7075,4,307.0,21.00,396.90,8.26} | 1 | 20.40
+{1,0.63796,0.00,8.140,0,0.5380,6.0960,84.50,4.4619,4,307.0,21.00,380.02,10.26} | 1 | 18.20
+{1,0.62739,0.00,8.140,0,0.5380,5.8340,56.50,4.4986,4,307.0,21.00,395.62,8.47} | 1 | 19.90
+{1,1.05393,0.00,8.140,0,0.5380,5.9350,29.30,4.4986,4,307.0,21.00,386.85,6.58} | 1 | 23.10
+{1,0.78420,0.00,8.140,0,0.5380,5.9900,81.70,4.2579,4,307.0,21.00,386.75,14.67} | 1 | 17.50
+{1,0.80271,0.00,8.140,0,0.5380,5.4560,36.60,3.7965,4,307.0,21.00,288.99,11.69} | 1 | 20.20
+{1,0.72580,0.00,8.140,0,0.5380,5.7270,69.50,3.7965,4,307.0,21.00,390.95,11.28} | 1 | 18.20
+\.
+</pre>
+-# Now train a regression model using a multilayer perceptron a single hidden layer of two nodes.
+<pre class="example">
+SELECT madlib.mlp_regression(
+ 'lin_housing', -- Source table
+ 'mlp_regress', -- Desination table
+ 'x', -- Input features
+ 'y', -- Dependent variable
+ ARRAY[5,5], -- Number of units per layer
+ 'step_size=0.000007,
+ n_iterations=10000,
+ tolerance=0',
+ 'relu');
+</pre>
+-# Check the results of the model
+<pre class="example">
+-- Set extended display on for easier reading of output
+\\x ON
+-- Neural net Initialization is non-deterministic, so your results may vary
+SELECT * FROM mlp_regress;
+</pre>
+Result:
+<pre class="result">
+-[ RECORD 1 ]--+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------
+coeff | {1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2.79506311399e-05,3.56715008915e-05,-6.09333559685e-05,0.000251228318768,-0.000224772841379,-3.71863030857e-05,-3.5757865148e-06,5.27936784854e-05,-2.48474166186e-05,6.19731184294e-05,3.07638968743e-05,6.8964698578e-06,0.000106016701083,-1.71484730318e-05,1.18691881812e-05,-0.000163975464208,0.000170026304906,3.11688265279e-05,0.000177050148787,-1.58265976603e-05,2.70144422657e-05,0.000112667883422,3.77575139073e-05,8.12474658795e-05,-7.90458917626e-05,0.000107566386158,-2.63771171506e-06,2.47996880915e-05,-0.00012642310887,0.000203827391081,0.000139315565565,4.86147243454e-05,-0.000176126471913,-6.47820782916e-05,-8.51592776447e-06,-6.60601176758e-05,2.91421874156e-05,6.3556873752e-05,0.000197557443129,0.000220531367259,0.000135036310289,0.000143735913975,-4.75034117786e-05,-0.000179547345838,-1.6919846786e-05,0.000162784312994,0.000268595819851,-0.000460066553287,8.69756071591e-05,-0.00311762727057,0.000126024763103,0.00020598824
2921,0.003463432426,-0.00729789075286,0.00151625867549,-0.000890852767597,-0.00525016037249,0.0031043106659,0.00798041103839,-0.00552693050079,0.0232180415786,0.0230489850143,-0.0437890272341,0.0165765426407,-0.248554261758,-7.81336427846e-05,0.00558145591752,0.283465844585,-0.571699956182,0.133474351994,-0.0785181945605,-0.419269930709,0.249547772912,0.631761009875,-0.431305975666,1,1,1,1,1,1,0.0158747497572,-9.02809160806e-05,0.00015574347618,4.10805373863e-06,0.00121532434965,0.101790351335,0.0647558401493,-0.00013654998677,-9.92872075948e-06,-5.5319694394e-05,0.00519320756484,0.412736586036,0.0011998026977,-1.53688189815e-05,1.94817888201e-05,-4.63111489966e-05,7.24547899029e-05,0.00880394144485,5.45309822095e-05,-0.000140943219275,-7.96211486227e-05,-1.04337307472e-05,0.000161936762028,0.00136273797767,-4.54737243585e-05,-3.4083840736e-05,3.69286883662e-05,9.9047243188e-08,3.75014011824e-06,-9.45366086368e-08,1,1,1,1,1,1,6.67488547054,0.102754199001,0.41668912471,0.008868672964
79,0.00136206007228,-9.88642499013e-05}
+loss | 144.965776158
+num_iterations | 10000
+</pre>
+-# Now let's look at the prediction functions. In the following examples we will
+use the training data set for prediction as well, which is not usual but serves to
+show the syntax. First we will test the classification exampel.
+The prediction is in the the estimated_class_text column with the
+actual value in the class_text column.
+<pre class="example">
+SELECT madlib.mlp_predict(
+ 'mlp_model', -- Model table
+ 'iris_data', -- Test data table
+ 'id', -- Id column in test table
+ 'mlp_prediction', -- Output table for predictions
+ 'response' -- Output classes, not probabilities
+ );
+SELECT * FROM mlp_prediction JOIN iris_data USING (id);
+</pre>
+Result:
+<pre class="result">
+ id | estimated_class_text | attributes | class_text | class
+----+----------------------+-------------------+-----------------+-------
+ 1 | Iris-setosa | {5.1,3.5,1.4,0.2} | Iris-setosa | 1
+ 2 | Iris-setosa | {4.9,3.0,1.4,0.2} | Iris-setosa | 1
+ 3 | Iris-setosa | {4.7,3.2,1.3,0.2} | Iris-setosa | 1
+ 4 | Iris-setosa | {4.6,3.1,1.5,0.2} | Iris-setosa | 1
+ 5 | Iris-setosa | {5.0,3.6,1.4,0.2} | Iris-setosa | 1
+ 6 | Iris-setosa | {5.4,3.9,1.7,0.4} | Iris-setosa | 1
+ 7 | Iris-setosa | {4.6,3.4,1.4,0.3} | Iris-setosa | 1
+ 8 | Iris-setosa | {5.0,3.4,1.5,0.2} | Iris-setosa | 1
+ 9 | Iris-setosa | {4.4,2.9,1.4,0.2} | Iris-setosa | 1
+ 10 | Iris-setosa | {4.9,3.1,1.5,0.1} | Iris-setosa | 1
+ 11 | Iris-versicolor | {7.0,3.2,4.7,1.4} | Iris-versicolor | 2
+ 12 | Iris-versicolor | {6.4,3.2,4.5,1.5} | Iris-versicolor | 2
+ 13 | Iris-versicolor | {6.9,3.1,4.9,1.5} | Iris-versicolor | 2
+ 14 | Iris-versicolor | {5.5,2.3,4.0,1.3} | Iris-versicolor | 2
+ 15 | Iris-versicolor | {6.5,2.8,4.6,1.5} | Iris-versicolor | 2
+ 16 | Iris-versicolor | {5.7,2.8,4.5,1.3} | Iris-versicolor | 2
+ 17 | Iris-versicolor | {6.3,3.3,4.7,1.6} | Iris-versicolor | 2
+ 18 | Iris-versicolor | {4.9,2.4,3.3,1.0} | Iris-versicolor | 2
+ 19 | Iris-versicolor | {6.6,2.9,4.6,1.3} | Iris-versicolor | 2
+ 20 | Iris-versicolor | {5.2,2.7,3.9,1.4} | Iris-versicolor | 2
+</pre>
+Prediction using the regression model:
+<pre class="example">
+SELECT madlib.mlp_predict(
--- End diff --
In case user attempts to run multiple times, precede `SELECT` statement with:
```
DROP TABLE IF EXISTS mlp_regress_prediction;
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-madlib issue #149: MLP: Multilayer Perceptron
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/incubator-madlib/pull/149
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---